Ensure EVP_PKEY_set1_DH detects X9.42 keys

OpenSSL supports both PKCS#3 and X9.42 DH keys. By default we use PKCS#3
keys. The function `EVP_PKEY_set1_DH` was assuming that the supplied DH
key was a PKCS#3 key. It should detect what type of key it is and assign
the correct type as appropriate.

Fixes #10592

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10593)

(cherry picked from commit 32c869ffaba67822602ea9fec611272ff8e8db58)

diff --git a/crypto/evp/p_lib.c b/crypto/evp/p_lib.c
index 451bc95eaec21914cee4fc1e3b44439c6fd9e7b4..9f1a485a5b832b212e36bcb82399bbe02ffd5bf4 100644 (file)
--- a/crypto/evp/p_lib.c
+++ b/crypto/evp/p_lib.c
@@ -540,7 +540,9 @@ EC_KEY *EVP_PKEY_get1_EC_KEY(EVP_PKEY *pkey)
 
 int EVP_PKEY_set1_DH(EVP_PKEY *pkey, DH *key)
 {
-    int ret = EVP_PKEY_assign_DH(pkey, key);
+    int type = DH_get0_q(key) == NULL ? EVP_PKEY_DH : EVP_PKEY_DHX;
+    int ret = EVP_PKEY_assign(pkey, type, key);
+
     if (ret)
         DH_up_ref(key);
     return ret;