Fix infinite loops in secure memory allocation.
Remove assertion when mmap() fails.
Only give the 1<<31 limit test as an example.
Fix the small arena test to just check for the symptom of the infinite
loop (i.e. initialized set on failure), rather than the actual infinite
loop. This avoids some valgrind errors.
Backport of:
PR #3512 commit
fee423bb68869de02fceaceefbc847e98213574b
PR #3510 commit
a486561b691d6293a901b412172ca0c6d1ffc0dc
PR #3455 commit
c8e89d58a5d44b9dd657d6d13a5a10d1d4d30733
PR #3449 commit
7031ddac94d0ae616d1b0670263a9265ce672cd2
Issue 1:
sh.bittable_size is a size_t but i is and int, which can result in
freelist == -1 if sh.bittable_size exceeds an int.
This seems to result in an OPENSSL_assert due to invalid allocation
size, so maybe that is "ok."
Worse, if sh.bittable_size is exactly 1<<31, then this becomes an
infinite loop (because 1<<31 is a negative int, so it can be shifted
right forever and sticks at -1).
Issue 2:
CRYPTO_secure_malloc_init() sets secure_mem_initialized=1 even when
sh_init() returns 0.
If sh_init() fails, we end up with secure_mem_initialized=1 but
sh.minsize=0. If you then call secure_malloc(), which then calls,
sh_malloc(), this then enters an infite loop since 0 << anything will
never be larger than size.
Issue 3:
That same sh_malloc loop will loop forever for a size greater
than size_t/2 because i will proceed (assuming sh.minsize=16):
i=16, 32, 64, ..., size_t/8, size_t/4, size_t/2, 0, 0, 0, 0, ....
This sequence will never be larger than "size".
Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3453)