Fix typo in HKDF example documentation

Out-of-bounds array access in the example documentation of
EVP_PKEY_CTX_set_hkdf_md fixed.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Stephen Henson <steve@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4081)

Add entropy sanity check

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/4092)

Set randomness buffer pointer in get_entropy calls.

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/4092)

Consolidate to a single asn1_time_from_tm() function

Add missing ASN1_TIME functions

Do some cleanup of the ASN1_TIME code.
Add ASN1_TIME_normalize() to normalize ASN1_TIME structures.
Add ASN1_TIME_compare() to compare two ASN1_TIME structures.
Add ASN1_TIME_cmp_time_t() to compare an ASN1_TIME to time_t
(generic version of ASN1_UTCTIME_cmp_time_t()).

Replace '0' .. '9' compares with isdigit()

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2753)

Test fixtures changed to pointers.

Change the fixture types to pointers to structures that are heap allocated in the tests that use SETUP_TEST_FIXTURE.  This will permit error returns from the setup function and allow for future running tests in parallel.

Also removed a call of `exit(2)` which allows the remaining tests to run if one fails to initialise.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4071)

recipes/80-test_ca.t: make it work with spaces in pathnames.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Add RAND_priv_bytes() for private keys

Add a new global DRBG for private keys used by RAND_priv_bytes.

Add BN_priv_rand() and BN_priv_rand_range() which use RAND_priv_bytes().
Change callers to use the appropriate BN_priv... function.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/4076)

Add a DRBG to each SSL object

Give each SSL object it's own DRBG, chained to the parent global
DRBG which is used only as a source of randomness into the per-SSL
DRBG.  This is used for all session, ticket, and pre-master secret keys.
It is NOT used for ECDH key generation which use only the global
DRBG. (Doing that without changing the API is tricky, if not impossible.)

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/4050)

Switch from ossl_rand to DRBG rand

If RAND_add wraps around, XOR with existing. Add test to drbgtest that
does the wrap-around.

Re-order seeding and stop after first success.

Add RAND_poll_ex()

Use the DF and therefore lower RANDOMNESS_NEEDED.  Also, for child DRBG's,
mix in the address as the personalization bits.

Centralize the entropy callbacks, from drbg_lib to rand_lib.
(Conceptually, entropy is part of the enclosing application.)
Thanks to Dr. Matthias St Pierre for the suggestion.

Various code cleanups:
    -Make state an enum; inline RANDerr calls.
    -Add RAND_POLL_RETRIES (thanks Pauli for the idea)
    -Remove most RAND_seed calls from rest of library
    -Rename DRBG_CTX to RAND_DRBG, etc.
    -Move some code from drbg_lib to drbg_rand; drbg_lib is now only the
     implementation of NIST DRBG.
    -Remove blocklength

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/4019)

Move ossl_assert

Move the definition of ossl_assert() out of e_os.h which is intended for OS
specific things. Instead it is moved into internal/cryptlib.h.

This also changes the definition to remove the (int) cast.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4073)

remove horrible pragma macro and remove __owur from SSL_CTX_add_session() declaration

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4014)

Remove the obsolete misleading comment and code related to it.

The comment "The following should not return 1, otherwise, things
are very strange" is from the very first commit of OpenSSL. The
really meaning of the comment is if the identical session can be
found from internal cache after calling get_session_cb but not
found before calling get_session_cb, it is just strange.

The value 1 was originated from the old doc of SSLeay, reversed
from the actual return value of SSL_CTX_add_session().

Anyway either return value of SSL_CTX_add_session() should not
interrupt the session resumption process. So the checking of
return value of SSL_CTX_add_session() is not necessary.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4014)

Allow use of long name for KDFs

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/4079)

Fix indentation

Conform to coding guidelines.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Stephen Henson <steve@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3989)

Added differentiation between missing secret and missing seed

This was previously mistakenly handled as a single error code.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Stephen Henson <steve@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3989)

Changed use of EVP_PKEY_CTX_md() and more specific error codes

Changed HKDF to use EVP_PKEY_CTX_md() (review comment of @snhenson) and
introduced more specific error codes (not only indicating *that* some
parameter is missing, but actually *which* one it is).

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Stephen Henson <steve@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3989)

More error handling to HKDF and one more case in TLS1-PRF

HKDF now handles an invalid digest like TLS1-PRF does (i.e., returns
KDF_R_INVALID_DIGEST if the passed digest is not known). Both KDFs now
set the error code KDF_R_UNKNOWN_PARAMETER_TYPE if a type was passed
that is not recognized. This will have the effect of improving debugging
output in case a user uses "openssl pkeyutl -kdf ..." in a wrong way and
result in an actual error code (instead of just "failure" and an empty
error stack).

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Stephen Henson <steve@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3989)

Set error when HKDF used without parameters

Introduce KDF_F_PKEY_HKDF_DERIVE and return the KDF_R_MISSING_PARAMETER
error code when required parameters have not been set. This will make
"openssl pkeyutl -kdf HKDF" return a meaningful error message instead of
simply "Public Key operation error".

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Stephen Henson <steve@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3989)

sha/asm/keccak1600-armv4.pl: improve non-NEON performance by ~10%.

This is achieved mostly by ~10% reduction of amount of instructions
per round thanks to a) switch to KECCAK_2X variant; b) merge of
almost 1/2 rotations with logical instructions. Performance is
improved on all observed processors except on Cortex-A15. This is
because it's capable of exploiting more parallelism and can execute
original code for same amount of time.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/4057)

Simplify some of the sslapitest code.

Removing the use of SETUP_TEST_FIXTURE reduces complxity in those tests that
used it.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4066)

Adding NID_hmac_sha1 and _md5 to builtin_pbe[]

The OID for {1 3 6 1 5 5 8 1 2} HMAC-SHA1 (NID_hmac_sha1) is explicitly
referenced by RFC 2510, RFC 3370, and RFC 4210. This is essential for the
common implementations of CMP (Certificate Managing Protocol, RFC4210).

HMAC-MD5's OID {1 3 6 1 5 5 8 1 1} (NID_hmac_md5) is in the same branch and
it seems to generally exist (-> Internet search), but it is unclear where it is
actually defined as it appears not to be referenced by RFCs and practically
rather unused.

Those OIDs are both duplicates to OIDs from an RSA OID branch, which are already
included in builtin_pbe[]:

HMAC-SHA1 also has another OID defined in PKCS#5/RFC2898 (NID_hmacWithSHA1).

It is also unclear where the other OID for HMAC-MD5 (NID_hmacWithMD5) from the
RSA branch is officially specified, as only HMAC-SHA1 from PKCS#5 was found to be
defined. Anyway, HMAC-MD5 likely only plays a neglectable role in the future.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/3811)

Remove EXECUTE_TEST_NO_TEARDOWN.

Simplify the only test that uses this macro so it doesn't need it anymore.
Clean up the formatting a little.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4034)

sha/keccak1600.c: choose more sensible default parameters.

"More" refers to the fact that we make active BIT_INTERLEAVE choice
in some specific cases. Update commentary correspondingly.

Reviewed-by: Rich Salz <rsalz@openssl.org>

asn1/a_time.c: make handling of 'fractional point' formally correct.

Even though tm->length >= 15 && v[14] == '.' works in practice,
[because "YYYYMMDDHHMMSS." would be rejected as invalid by
asn1_time_to_tm,] formal correctness with respect to buffer
overstep in few lines vicinity improves readability.

[Also fold one if condition and improve expression readability.]

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4058)

Add test case for ASN1_TIME_print

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4061)

Add EC key generation paragraph in doc/HOWTO/keys.txt

Seems this documentation is not dead, so add this missing part

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4037)

RSA_get0_ functions permit NULL parameters

Document that the RSA_get0_ functions permit a NULL BIGNUM **. Those output parameters are ignored.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4064)

Add a test to check we get a new session even if s->hit is true in TLSv1.3

In TLSv1.3 we can resume, but still get a new session. This adds a test to
make sure that is happening.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/4068)

Fix new_session_cb calls in TLSv1.3

If a new_session_cb is set then it was only ever getting invoked if !s->hit
is true. This is sensible for <=TLSv1.2 but does not work for TLSv1.3.

Fixes #4045

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/4068)

Add '-ext' option to display extensions in 'x509'

This is to address issue #3932. Support comma-separated string
to specify what extensions to be displayed.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4016)

Remove redundant variable

[to be squashed]

Fix typo in documents

I scanned all files in the doc folder with a spell checker (https://github.com/EWSoftware/VSSpellChecker).
This patch (hopefully) corrected all spell errors that it found.

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/4065)

Fix an information leak in the RSA padding check code.
The memory blocks contain secret data and must be
cleared before returning to the system heap.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4062)

Fix SSL_set_tlsext_debug_callback/-tlsextdebug

Some extensions were being displayed twice, before they were parsed, and
again after they were parsed.
The supported_versions extension was not being fully displayed, as it
was processed differently than other extensions.

Move the debug callback to where the extensions are first collected, to
catch all the extensions as they come in, so they are ordered correctly.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3911)

Fix the names of older ciphers.

The names of these ciphers have an "SSL_" prefix, but the RFC names use
"TLS_":
https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-4

This dates back to these ciphers being originally defined in SSLv3. As
SSLv3 is on its way out anyway and this is a new set of APIs,
consistently use the TLS names.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4007)

Fix errors in SSL_state_string_long

TLS_ST_SR_NEXT_PROTO means "SSLv3/TLS read next proto"
Fix typo in the message for TLS_ST_SW_CERT_STATUS

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4054)

app_isdir() cleanup

I think it's better to use `GetFileAttributes` to obtain the attributes
of a file than `FindFirstFile`. If the input name contains `*`, this
function should return failure rather than check whether the first match
happens to be a file or a directory.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/3991)

bn/bn_lcl.h: restore formatting.

Reviewed-by: Rich Salz <rsalz@openssl.org>

bn/bn_lcl.h: use __int128 whenever possible, not only on MIPS.

Reviewed-by: Rich Salz <rsalz@openssl.org>

bn/bn_lcl.h: improve inline assembly coverage on PPC64.

[And move misplaced macros.]

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix typo in sha1-thumb.pl

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4056)

Refactor ASN1_TIME_print functions

Check time string format before parsing
Reduce more duplicated code
By involving asn1_time_to_tm, we can now get information we mostly need
to print a time string.

This follows what was discussed at
https://github.com/openssl/openssl/pull/4001#discussion_r129092251

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4039)

Fix typo in ASN1_TIME_set.pod

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4053)

Fix typos in files in ssl directory

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4052)

Update ASN1_TIME_to_tm's documentation

To state the fractional seconds part will be lost in the conversion.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4041)

Update copyright header

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4049)

Remove redundant declarations in ssl_locl.h

Remove the function prototypes for ssl_cert_get0_next_certificate, ssl_set_default_md, tls1_shared_list,
dtls1_send_newsession_ticket, tls1_ctrl, and tls1_callback_ctrl, all of which are not defined.

It also changed the signature of the function pqueue_next to `pitem *pqueue_next(piterator *item)` in
pqueue.c, making it match the prototype in ssl_locl.h. (`piterator *` is equivalent to `pitem **`.)

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4049)

Remove redundant declarations in record_locl.h

This patch removes the prototype of function RECORD_LAYER_set_write_sequence from record_locl.h, since this function is not defined.

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4051)

make update

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4015)

Add list -public-key-methods

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4015)

Add public key method enumeration function.

Add functions to enumerate public key methods. Add test to ensure table
is in the correct order.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4015)

Add some test coverage for OPENSSL_secure_clear_free

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4044)

Use OPENSSL_secure_clear_free in STORE file_load

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4044)

Use OPENSSL_secure_clear_free in PEM_read_bio_PrivateKey and PEM_read_bio_ex

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4044)

Document OPENSSL_secure_clear_free

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4044)

Use OPENSSL_secure_clear_free for secure mem BIOs and X25519 private keys

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4044)

Implement the CRYPTO_secure_clear_free function

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4044)

Add some test coverage for PEM_read_bio_PrivateKey

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4047)

Clean password buffer on stack for PEM_read_bio_PrivateKey
and d2i_PKCS8PrivateKey_bio before it goes out of scope.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4047)

Fix rsa -check option

original problem: if a private key is invaild, nothing outputted.

the error filter in apps/rsa.c is not working any more.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4043)

Make SSL_set_tlsext_host_name no effect from server side

Documentation and test cases are also updated

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3783)

Fix coding style nits in crypto/asn1/a_strnid.c

Since this file is modified in PR #3934, so should be cleaned up
incidentially.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4035)

Fix a reference nit in doc

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4036)

Perl: Use File::Glob::bsd_glob rather than File::Glob::glob

File::Glob::glob is deprecated, it's use generates this kind of
message:

    File::Glob::glob() will disappear in perl 5.30. Use File::Glob::bsd_glob() instead. at ../master/Configure line 277.

So instead, use a construction that makes the caller glob() use
File::Glob::bsd_glob().

Note that we're still excluding VMS, as it's directory specs use '['
and ']', which have a different meaning with bsd_glob and would need
some extra quoting.  This might change, but later.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4040)

Formatting & declaration cleanse.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4032)

OPENSSL_NO_ENGINE defined causes build failures

TEST_note as an undefined link symbol is the symptom.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4032)

Remove trailing whitespace from store-related man pages

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3860)

Improve style

Spaces around operators.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3860)

Catch up to the removal of OSSL_STORE_open_file()

Remove references to it in documentation.

Unfortunately, it is too late to renumber symbols in libcrypto.num
and avoid the NOEXIST entry there.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3860)

Fixups for STORE commit

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3860)

Define a value for SYS_F_FCNTL

This symbol was added in commit d33b215b331116e50947ca7e75d210e1db39b78d
but was only used in certain (presumed uncommon) preprocessor conditionals,
as no build failures have been reported yet.

Reported by Balaji Marisetti.

Closes: #4029

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4030)

Fix trivial coding style nits in a_time/a_tm files
Clean up some true/false returns

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/4001)

Fix comment typo.

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4023)

Various doc fixes.

Fix typo in NOTES.WIN: this -> these

Fix wrong capital letter in certificates.txt

Make number of characters in each line more even
Remove redundant empty line

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3986)

Install custom RAND_METHOD for fuzzing

Instead of setting a "magic" global variable to force RAND to keep
consistent state and always generate the same bytestream, have
the fuzzing code install its own RAND_METHOD that does this.  For
BN_RAND_DEBUG, we just don't do it; that debugging was about mucking
with BN's internal representation, not requiring predictable rand
bytes.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/4025)

Fix the two new tests since approval.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3953)

Update the test framework so that the need for test_main is removed.  Everything
that needed test_main now works using the same infrastructure as tests that used
register_tests.

This meant:
* renaming register_tests to setup_tests and giving it a success/failure return.
* renaming the init_test function to setup_test_framework.
* renaming the finish_test function to pulldown_test_framework.
* adding a user provided global_init function that runs before the test frame
    work is initialised.  It returns a failure indication that stops the stest.
* adding helper functions that permit tests to access their command line args.
* spliting the BIO initialisation and finalisation out from the test setup and
    teardown.
* hiding some of the now test internal functions.
* fix the comments in testutil.h

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3953)

Simplify the handling of shared library version numbers

$(SHLIB_MAJOR).$(SHLIB_MINOR) is really a synonym for
$(SHLIB_VERSION_NUMBER), and is therefore an added complexity,
so better to use $(SHLIB_VERSION_NUMBER) directly.  SHLIB_MAJOR and
SHLIB_MINOR are now unused, but are kept around purely as information
in case someone relies on their existence.

At the same time, add support for custom shared library extensions
with the three new Makefile variables SHLIB_EXT, SHLIB_EXT_SIMPLE and
SHLIB_EXT_IMPORT.  By default, they hold the variants of shared
library extensions we support.  On mingw and cygwin, SHLIB_EXT_IMPORT
is defined; on all other Unix platforms, it's empty.

An example to get shared libraries with a slightly different SOVER name:

    $ make SHLIB_EXT='.$(SHLIB_VERSION_NUMBER).so'

Fixes #3902

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3964)

add basic references to the new methods in documentation

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

handle scrypt PBKDF in PKCS#12 files info

Print the parameters for scrypt PBKDF when used in PKCS#12 files.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

nicer formatting for MAC info

The info printing for PBES2 already uses space after type and commas to
separate items in the line so use the same format for MAC info too.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

make scrypt ASN.1 parameter functions public

Since scrypt PBKDF can be used both in PKCS#5 and PKCS#12 files,
do share the code between them.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

more info about PKCS#12 structure MAC

report additional information about the MAC used over the
PKCS#12 structure: size of mac and salt as well as the
hash algorithm used for creating it

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

pkcs12.c better formatting for unsupported params

since when we get to this point, other information was already
printed, we should insert some whitespace between already
printed data and this "unsupported parameters" error message

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

fix OSSL_STORE man pages

the man pages have lines that contain nothing but whitespace,
clean it up by removing that whitespace

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

Add test cases and docs for ASN1_STRING_TABLE_* functions

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3998)

Fix async engine pause dead lock in error case.

In 'crypto/rand/ossl_rand.c', a call to
'ASYNC_unblock_pause()' is missing in an error case.

CLA: trivial

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/4020)

Add support to free/allocate SSL buffers

OpenSSL already has the feature of SSL_MODE_RELEASE_BUFFERS that can
be set to release the read or write buffers when data has finished
reading or writing. OpenSSL will automatically re-allocate the buffers
as needed. This can be quite aggressive in terms of memory allocation.

This provides a manual mechanism. SSL_free_buffers() will free
the data buffers if there's no pending data. SSL_alloc_buffers()
will realloc them; but this function is not strictly necessary, as it's
still done automatically in the state machine.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2240)

Fix potential use-after-free and memory leak

In function wait_for_async(), allocated async fds is freed if
`SSL_get_all_async_fds` fails, but later `fds` is used. Interestingly,
it is not freed when everything succeeds.

Rewrite the FD set loop to make it more readable and to not modify the allocated
pointer so it can be freed.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/3992)

Wire SHA3 EVPs and add tests.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3943)

Add evp/m_sha3.c.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3943)

sha/keccak1600.c: build and make it work with strict warnings.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3943)

crypto/x86_64cpuid.pl: fix typo in Knights Landing detection.

Thanks to David Benjamin for spotting this!

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4009)

aes/asm/aesni-sha*-x86_64.pl: add SHAEXT performance results.

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/3898)

evp/e_aes_cbc_hmac_sha256.c: give SHAEXT right priority.

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/3898)

sha/asm/keccak1600-avx512.pl: improve performance by 17%.

Improvement is result of combination of data layout ideas from
Keccak Code Package and initial version of this module.

Hardware used for benchmarking courtesy of Atos, experiments run by
Romain Dolbeau <romain.dolbeau@atos.net>. Kudos!

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Rich Salz <rsalz@openssl.org>

schlock global variable needs to be volatile

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4000)

Fix nid assignment in ASN1_STRING_TABLE_add

CLA: trivial

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3934)

Makefile.shared: Make link_shlib.linux-shared less verbose again

A previous change inavertently removed a silencing '@'

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4003)

test/recipes/80-test_tsa.t: Don't trust 'OPENSSL_CONF'

There's a case when the environment variable OPENSSL_CONF is
useless...  when cross compiling for mingw and your wine environment
has an environment variable OPENSSL_CONF.  The latter will override
anything that's given when starting wine and there make the use of
that environment variable useless in our tests.

Therefore, we should not trust it, and use explicit '-config' options
instead.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3994)

Simplify Makefile.shared

Makefile.shared was designed to figure out static library names,
shared library names, library version compatibility, import library
names and the like on its own.  This was a design for pre-1.1.0
OpenSSL because the main Makefile didn't have all that knowledge.

With 1.1.0, the situation isn't the same, a lot more knowledge is
included in the main Makefile, and while Makefile.shared did things
right most of the time (there are some corner cases, such as the
choice of .sl or .so as DSO extension on some HPUX versions), there's
still an inherent fragility when one has to keep an eye on
Makefile.shared to make sure it produces what the main Makefile
produces.

This change simplifies Makefile.shared by removing all its
"intelligence" and have it depend entirely on the input from the main
Makefile instead.  That way, all the naming is driven from
configuration data.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3983)