Increase rounds of Miller-Rabin testing DH_check

DH_check is used to test the validity of Diffie-Hellman parameter sets (p, q, g). Among the tests performed are primality tests on p and q, for this BN_is_prime_ex is called with the rounds of Miller-Rabin set as default. This will therefore use the average case error estimates derived from the function BN_prime_checks_for_size based on the bit size of the number tested.

However, these bounds are only accurate on testing random input. Within this testing scenario, where we are checking the validity of a DH parameter set, we can not assert that these parameters are randomly generated. Thus we must treat them as if they are adversarial in nature and increase the rounds of Miller-Rabin performed.

Generally, each round of Miller-Rabin can declare a composite number prime with probability at most (1/4), thus 64 rounds is sufficient in thwarting known generation techniques (even in safe prime settings - see https://eprint.iacr.org/2019/032 for full analysis). The choice of 64 rounds is also consistent with SRP_NUMBER_ITERATIONS_FOR_PRIME 64 as used in srp_Verify_N_and_g in openssl/apps/s_client.c.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8593)

diff --git a/crypto/dh/dh_check.c b/crypto/dh/dh_check.c
index cd7f70b67278aae413cbce44d31915f3a70be1ec..8be2b91a9e3d6d6445cb6c483ef11cab21caa2d2 100644 (file)
--- a/crypto/dh/dh_check.c
+++ b/crypto/dh/dh_check.c
@@ -12,6 +12,8 @@
 #include <openssl/bn.h>
 #include "dh_locl.h"
 
+# define DH_NUMBER_ITERATIONS_FOR_PRIME 64
+
 /*-
  * Check that p and g are suitable enough
  *
@@ -125,7 +127,7 @@ int DH_check(const DH *dh, int *ret)
             if (!BN_is_one(t1))
                 *ret |= DH_NOT_SUITABLE_GENERATOR;
         }
-        r = BN_is_prime_ex(dh->q, BN_prime_checks, ctx, NULL);
+        r = BN_is_prime_ex(dh->q, DH_NUMBER_ITERATIONS_FOR_PRIME, ctx, NULL);
         if (r < 0)
             goto err;
         if (!r)
@@ -153,7 +155,7 @@ int DH_check(const DH *dh, int *ret)
     } else
         *ret |= DH_UNABLE_TO_CHECK_GENERATOR;
 
-    r = BN_is_prime_ex(dh->p, BN_prime_checks, ctx, NULL);
+    r = BN_is_prime_ex(dh->p, DH_NUMBER_ITERATIONS_FOR_PRIME, ctx, NULL);
     if (r < 0)
         goto err;
     if (!r)
@@ -161,7 +163,7 @@ int DH_check(const DH *dh, int *ret)
     else if (!dh->q) {
         if (!BN_rshift1(t1, dh->p))
             goto err;
-        r = BN_is_prime_ex(t1, BN_prime_checks, ctx, NULL);
+        r = BN_is_prime_ex(t1, DH_NUMBER_ITERATIONS_FOR_PRIME, ctx, NULL);
         if (r < 0)
             goto err;
         if (!r)