EVP: Fix EVP_Digest{Sign,Verify}Init() to handle no default digest

EVP_DigestSignInit() and EVP_DigestVerifyInit() would detect if there
is no default digest when using legacy (EVP_PKEY_ASN1_METHOD)
implementations.  However, it doesn't do that when provider side keys
are used.

Furthermore, because EVP_PKEY_get_default_digest_name() was used in
the portion of the code that uses the provider implementation, the
EVP_PKEY_ASN1_METHOD would be used if the key has one attached.  This
is now changed to use evp_keymgmt_util_get_deflt_digest_name()
instead.

Finally, we make sure to detect if the provider implementation
supports the digest name parameters (default or mandatory), and
returns with error if not.  This is what the legacy portion of the
code does.

Fixes #11571

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11576)

diff --git a/crypto/evp/m_sigver.c b/crypto/evp/m_sigver.c
index 8dd4207d1783e5421145ddacccfd1dcf53126eec..148d29fd9017f9675bd721ca85c81d6ea5dd70d4 100644 (file)
--- a/crypto/evp/m_sigver.c
+++ b/crypto/evp/m_sigver.c
@@ -152,10 +152,16 @@ static int do_sigver_init(EVP_MD_CTX *ctx, EVP_PKEY_CTX **pctx,
         if (mdname == NULL)
             mdname = canon_mdname(EVP_MD_name(type));
     } else {
-        if (mdname == NULL
-            && EVP_PKEY_get_default_digest_name(locpctx->pkey, locmdname,
-                                                sizeof(locmdname)))
-            mdname = canon_mdname(locmdname);
+        if (mdname == NULL) {
+            if (evp_keymgmt_util_get_deflt_digest_name(tmp_keymgmt, provkey,
+                                                       locmdname,
+                                                       sizeof(locmdname)) > 0) {
+                mdname = canon_mdname(locmdname);
+            } else {
+                EVPerr(EVP_F_DO_SIGVER_INIT, EVP_R_NO_DEFAULT_DIGEST);
+                return 0;
+            }
+        }
 
         if (mdname != NULL) {
             /*