Fix BN error reporting

Commit ed57f7f935 implemented the macro ERR_raise and updated err.h to use
it. A typo in err.h means that errors in the BN library are mistakenly
attributed to the RSA library.

This was found due to the following error appearing in a travis log:

00:07:CB:13:05:7F:00:00:error:0400006C:rsa routines::data greater than mod
len:crypto/bn/bn_gcd.c:613:
00:07:CB:13:05:7F:00:00:error:04000003:rsa routines::BN
lib:crypto/rsa/rsa_gen.c:393:
/home/travis/build/openssl/openssl/util/shlib_wrap.sh
/home/travis/build/openssl/openssl/apps/openssl genrsa -out rsamptest.pem
-primes 5 8192 => 1
not ok 12 - genrsa 8192p5

The line in question (crypto/bn/bn_gcd.c:613) actually looks like this:

        BNerr(BN_F_BN_MOD_INVERSE_NO_BRANCH, BN_R_NO_INVERSE);

The test was checking for that error being raised, but was instead seeing
a different error and thus failing.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Yang <kaishen.yy@antfin.com>
(Merged from https://github.com/openssl/openssl/pull/9539)

diff --git a/include/openssl/err.h b/include/openssl/err.h
index ad5ed5a29206af25d7ba7bd2abf266983421ed04..e5a1a0dc1eeaf4493ddb057f41c6734e24bdbcb1 100644 (file)
--- a/include/openssl/err.h
+++ b/include/openssl/err.h
@@ -110,7 +110,7 @@ typedef struct err_state_st {
 # if ! OPENSSL_API_3
 #  define SYSerr(f,r)  ERR_raise(ERR_LIB_SYS,(r))
 # endif
-# define BNerr(f,r)   ERR_raise(ERR_LIB_RSA,(r))
+# define BNerr(f,r)   ERR_raise(ERR_LIB_BN,(r))
 # define RSAerr(f,r)  ERR_raise(ERR_LIB_RSA,(r))
 # define DHerr(f,r)   ERR_raise(ERR_LIB_DH,(r))
 # define EVPerr(f,r)  ERR_raise(ERR_LIB_EVP,(r))