Fix incorrect return code on ECDSA key verification

ECDSA_do_verify() is a function that verifies a ECDSA signature given a hash and a public EC key. The function is supposed to return 1 on valid signature, 0 on invalid signature and -1 on error. Previously, we returned 0 if the key did not have a verify_sig method. This is actually an error case and not an invalid signature. Consequently, this patch updates the return code to -1.

Fixes #8766

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/10693)

diff --git a/crypto/ec/ecdsa_vrf.c b/crypto/ec/ecdsa_vrf.c
index 75dfe5b92b622a5db11ecb4eca1c0aea71333c97..60f4af04263e3d007a0d6419d7f43acf6bdd86dc 100644 (file)
--- a/crypto/ec/ecdsa_vrf.c
+++ b/crypto/ec/ecdsa_vrf.c
@@ -23,7 +23,7 @@ int ECDSA_do_verify(const unsigned char *dgst, int dgst_len,
     if (eckey->meth->verify_sig != NULL)
         return eckey->meth->verify_sig(dgst, dgst_len, sig, eckey);
     ECerr(EC_F_ECDSA_DO_VERIFY, EC_R_OPERATION_NOT_SUPPORTED);
-    return 0;
+    return -1;
 }
 
 /*-
@@ -39,5 +39,5 @@ int ECDSA_verify(int type, const unsigned char *dgst, int dgst_len,
         return eckey->meth->verify(type, dgst, dgst_len, sigbuf, sig_len,
                                    eckey);
     ECerr(EC_F_ECDSA_VERIFY, EC_R_OPERATION_NOT_SUPPORTED);
-    return 0;
+    return -1;
 }