Return an error from BN_mod_inverse if n is 1 (or -1)
authorMatt Caswell <matt@openssl.org>
Fri, 27 Apr 2018 16:36:11 +0000 (17:36 +0100)
committerMatt Caswell <matt@openssl.org>
Thu, 3 May 2018 09:35:49 +0000 (10:35 +0100)
commitf4be878edae21755fe83ef26f3f3821a2fe58a97
treeec0fc82b571fcd0d72a898a62c1b2fb5e57f2ef8
parent70ec1c96a7cdf8ee6a500535a71cf46e930fb2f3
Return an error from BN_mod_inverse if n is 1 (or -1)

Calculating BN_mod_inverse where n is 1 (or -1) doesn't make sense. We
should return an error in that case. Instead we were returning a valid
result with value 0.

Fixes #6004

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6119)

(cherry picked from commit b1860d6c71733314417d053a72af66ae72e8268e)
crypto/bn/bn_gcd.c
crypto/bn/bn_mont.c