Change returned -2 to 0 in EVP_Digest{Sign,Verify}Init()

The returned -2 was to mark when these operations are unsupported.
However, that breaks away from the previous API and expectations, and
there's not enough justification for that not being zero.

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/10815)

diff --git a/CHANGES b/CHANGES
index 215fd13b6080609de858e7f6eeb9f17eab7ed9ee..1f0fe2d06b2ad9ef866f6868a1e8753bd3df51be 100644 (file)
--- a/CHANGES
+++ b/CHANGES
@@ -9,6 +9,15 @@
 
  Changes between 1.1.1 and 3.0.0 [xx XXX xxxx]
 
+  *) Corrected the documentation of the return values from the EVP_DigestSign*
+     set of functions.  The documentation mentioned negative values for some
+     errors, but this was never the case, so the mention of negative values
+     was removed.
+
+     Code that followed the documentation and thereby check with something
+     like 'EVP_DigestSignInit(...) <= 0' will continue to work undisturbed.
+     [Richard Levitte]
+
   *) All of the low level Blowfish functions have been deprecated including:
      BF_set_key, BF_encrypt, BF_decrypt, BF_ecb_encrypt, BF_cbc_encrypt,
      BF_cfb64_encrypt, BF_ofb64_encrypt, and BF_options.

diff --git a/crypto/evp/m_sigver.c b/crypto/evp/m_sigver.c
index faee5073426f0067bae80bb96b66f21c3faa0bb8..dbfa01b3ed20138ef0e2b83eb22bf19e390a26c2 100644 (file)
--- a/crypto/evp/m_sigver.c
+++ b/crypto/evp/m_sigver.c
@@ -167,7 +167,7 @@ static int do_sigver_init(EVP_MD_CTX *ctx, EVP_PKEY_CTX **pctx,
  legacy:
     if (ctx->pctx->pmeth == NULL) {
         EVPerr(0, EVP_R_OPERATION_NOT_SUPPORTED_FOR_THIS_KEYTYPE);
-        return -2;
+        return 0;
     }
 
     if (!(ctx->pctx->pmeth->flags & EVP_PKEY_FLAG_SIGCTX_CUSTOM)) {

diff --git a/doc/man3/EVP_DigestSignInit.pod b/doc/man3/EVP_DigestSignInit.pod
index 602c8eda96793e76ba781df122a85bc4ec3bbd49..7e105b29bc44c2106a066b5b63a9cbd32f1fdee5 100644 (file)
--- a/doc/man3/EVP_DigestSignInit.pod
+++ b/doc/man3/EVP_DigestSignInit.pod
@@ -129,9 +129,7 @@ EVP_DigestSignFinal().
 =head1 RETURN VALUES
 
 EVP_DigestSignInit(), EVP_DigestSignUpdate(), EVP_DigestSignaFinal() and
-EVP_DigestSign() return 1 for success and 0 or a negative value for failure. In
-particular, a return value of -2 indicates the operation is not supported by the
-public key algorithm.
+EVP_DigestSign() return 1 for success and 0 for failure.
 
 The error codes can be obtained from L<ERR_get_error(3)>.