Remove old KDF initialisation

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9662)

Fix TLS/SSL PRF usages.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9662)

Move KDFs to the provider.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9662)

Teach TLSProxy how to parse CertificateRequest messages

We also use this in test_tls13messages to check that the extensions we
expect to see in a CertificateRequest are there.

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9780)

Don't send a status_request extension in a CertificateRequest message

If a TLSv1.3 server configured to respond to the status_request extension
also attempted to send a CertificateRequest then it was incorrectly
inserting a non zero length status_request extension into that message.

The TLSv1.3 RFC does allow that extension in that message but it must
always be zero length.

In fact we should not be sending the extension at all in that message
because we don't support it.

Fixes #9767

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9780)

Undeprecate OpenSSL_version_num and OPENSSL_VERSION_NUMBER

The OpenSSL_version_num() function returns at runtime the
OPENSSL_VERSION_NUMBER of the compiled OpenSSL library.  This is a
used and useful interface, and should not (at least yet) be
deprecated, we just introduced the new versioning schema, it seems
too early to deprecate the old.

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7853)

Reverting check to correct

Fixes #9773.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9776)

Disallow change EVP_CIPHER properties once set

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9764)

Disallow change EVP_MD properties once set

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9764)

[test] computing ECC cofactors: regression test

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9827)

[crypto/ec] for ECC parameters with NULL or zero cofactor, compute it

The cofactor argument to EC_GROUP_set_generator is optional, and SCA mitigations for ECC currently use it. So the library currently falls back to very old SCA-vulnerable code if the cofactor is not present.

This PR allows EC_GROUP_set_generator to compute the cofactor for all curves of cryptographic interest. Steering scalar multiplication to more SCA-robust code.

This issue affects persisted private keys in explicit parameter form, where the (optional) cofactor field is zero or absent.

It also affects curves not built-in to the library, but constructed programatically with explicit parameters, then calling EC_GROUP_set_generator with a nonsensical value (NULL, zero).

The very old scalar multiplication code is known to be vulnerable to local uarch attacks, outside of the OpenSSL threat model. New results suggest the code path is also vulnerable to traditional wall clock timing attacks.

CVE-2019-1547

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9827)

Make failed messages easier to find

Now that we use travis_terminate, we can make the status messages
simpler to find, and we don't need the "OK" output.

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9707)

Fix error handling in x509_lu.c

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9639)

Cleanup includes in rand_unix.c

Fixes #9757

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9761)

Change provider params from int to size_t

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9699)

Handle the renamed command POD files in find-doc-nits

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9729)

Move libapps.a source to apps/lib

This makes it clearer what's what.  The 'openssl' application and its
sub-commands remain in apps/

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9723)

Suppress 'No server certificate CA names sent' message

Fixes #9080

Signed-off-by: Billy Brawner <billy@wbrawner.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9710)

OSSL_PARAM_construct_utf8_string computes the string length.

If the passed string length is zero, the function computes the string length
from the passed string.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9760)

New function EVP_CIPHER_free()

This function re-implements EVP_CIPHER_meth_free(), but has a name that
isn't encumbered by legacy EVP_CIPHER construction functionality.

We also refactor most of EVP_CIPHER_meth_new() into an internal
evp_cipher_new() that's used when creating fetched methods.

EVP_CIPHER_meth_new() and EVP_CIPHER_meth_free() are rewritten in terms of
evp_cipher_new() and EVP_CIPHER_free().  This means that at any time, we can
deprecate all the EVP_CIPHER_meth_ functions with no harmful consequence.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9758)

New function EVP_MD_free()

This function re-implements EVP_MD_meth_free(), but has a name that
isn't encumbered by legacy EVP_MD construction functionality.

We also refactor most of EVP_MD_meth_new() into an internal
evp_md_new() that's used when creating fetched methods.

EVP_MD_meth_new() and EVP_MD_meth_free() are rewritten in terms of
evp_md_new() and EVP_MD_free().  This means that at any time, we can
deprecate all the EVP_MD_meth_ functions with no harmful consequence.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9758)

test/errtest.c: more conditions for checking __FILE__ and __LINE__

When at least one of OPENSSL_NO_ERR or OPENSSL_NO_FILENAMES is
defined, __FILE__ and __LINE__ are not saved with the error record.
This test only checked OPENSSL_NO_FILENAMES.  Now fixed.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9755)

Refactor how KEYMGMT methods get associated with other methods

KEYMGMT methods were attached to other methods after those were fully
created and registered, thereby creating a potential data race, if two
threads tried to create the exact same method at the same time.

Instead of this, we change the method creating function to take an
extra data parameter, passed all the way from the public fetching
function.  In the case of EVP_KEYEXCH, we pass all the necessary data
that evp_keyexch_from_dispatch() needs to be able to fetch the
appropriate KEYMGMT method on the fly.

Fixes #9592

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9678)

Fix Coverity 1453452: Control flow issues (DEADCODE)

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9749)

Add CPU info to the speed command summary

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9669)

Diverse ERR fixes

1.  There are still references to the removed ERR_put_func_error().
2.  ERR_put_error() is deprecated as off version 3.0, so should
    ERR_PUT_error().
3.  'no-err' didn't affect what was passed to ERR_set_debug().

Fixes #9522

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9747)

Rework the documentation of our individual MAC implementations

We now describe parameters instead of controls.

Also, since macros like EVP_MAC_CMAC do not exist any more, we rename
the pod files from EVP_MAC_{algo}.pod to EVP_MAC-{algo}.pod.  This
allows getting the documentation like this:

    man EVP_MAC CMAC

[skip ci]
Fixes #9709

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9713)

Fix function name typo in MAC documentation.

A spurious CTX crept into one of the function names.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9750)

Remove extern declarations of OPENSSL_ia32cap_P

Use the header file internal/cryptlib.h instead.
Remove checks for OPENSSL_NO_ASM and I386_ONLY
in cryptlib.c, to match the checks in other
places where OPENSSL_ia32cap_P is used and
assumed to be initialized.

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/9688)

Fix platform specific issues with provider ciphers

s390_aes naming issues with ofb128 and cfb128
Solaris missing include for camellia.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9702)

Fix pkeyutl -verifyrecover

When performing a pkeyutl -verifyrecover operation the input file is not
a hash - it is the signature itself. Therefore don't do the check to make
sure it looks like a hash.

Fixes #9658

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9731)

Don't include the DEVRANDOM being seeded logic on Android.

It lacks exposure of the `shm*` functions and should prefer the GETRANDOM
source.

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9735)

Fix NITs in comments and CHANGES for DEVRANDOM seeded check.

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/9734)

fix ERR_add_error_vdata() for use with multiple args/calls

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9558)

Use ENGINE_get_id() instead of ENGINE_get_name()

ENGINE_get_name() actually returns more of a long description of the
engine, whilst ENGINE_get_id() returns a shorter id.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9720)

Fix no-engine

Make sure references to ENGINE functions are appropriately guarded.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9720)

Fix no-cmac

Don't include files that we don't want to build

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9718)

Fix no-poly1305, no-siphash and no-blake2

Make sure we don't include files that we don't need if we've disabled
them.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9717)

Fix a documentation bug in ec.h

The EC_GROUP_new() function does not take a libctx parameter

Reviewed-by: Paul Yang <kaishen.yy@antfin.com>
(Merged from https://github.com/openssl/openssl/pull/9715)

Fix no-ec

A test in sslapitest.c was failing in a no-ec build because we were using
an EC based ciphersuite. That particular test doesn't require EC
specifically, so we swap to a non EC based ciphersuite.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9716)

Fix no-des

Don't refer to cipher functions in the providers that have been compiled
out.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9719)

Update fuzz README.md

Building with enable-fuzz-afl has always required no-shared. We now also
need no-module for a successful build. Therefore update the README
accordingly.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9721)

Fix data races in EVP_CIPHER_fetch and EVP_MD_fetch

Don't modify the cipher/md we just fetched - it could be shared by multiple
threads.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9590)

Make sure we pre-initialise properties

Simplify the initialisation of the core by pre-initialising properties.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9590)

Fix context locking

Some parts of OPENSSL_CTX intialisation can get quite complex (e.g. RAND).
This can lead to complex interactions where different parts of the library
try to initialise while other parts are still initialising. This can lead
to deadlocks because both parts want to obtain the init lock.

We separate out the init lock so that it is only used to manage the
dynamic list of indexes. Each part of the library gets its own
initialisation lock.

Fixes #9454

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9590)

Allow an output indentation of zero in apps.

Previously, it would indent one space even if zero were specified.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9725)

openssl provider: New sub-command, for provider discovery

This command is somewhat similar to 'openssl engine', but displays
what it can about the given providers.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9697)

Move print_param_types() to libapps, and give it indent argument

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9697)

Coverty fixes for MACs

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9700)

OPENSSL_info(): add the item OPENSSL_INFO_SEED_SOURCE and use it

'openssl version -r' prints the seed source based on compiler macros.
This does not necessarily reflect the library's idea of what seed
sources to use, so we reimplement the list of seed sources as a
OPENSSL_info() item and display that instead.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9689)

Fix bogus check for EVP_PKEY_supports_digest_nid() in check_cert_usable()

In commit 2d263a4a73 ("Honour mandatory digest on private key in
has_usable_cert()" I added two checks for the capabilities of the
EVP_PKEY being used. One of them was wrong, as it should only be
checking the signature of the X.509 cert (by its issuer) against the
sigalgs given in a TLS v1.3 signature_algorithms_cert extension.

Remove it and provide the code comments which, if they'd been present
in the first place, would hopefully have prevented the mistake.

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9672)

testing: set OPENSSL_MODULES to the providers directory by default

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9618)

Fix SCA vulnerability when using PVK and MSBLOB key formats

This commit addresses a side-channel vulnerability present when
PVK and MSBLOB key formats are loaded into OpenSSL.
The public key was not computed using a constant-time exponentiation
function.

This issue was discovered and reported by the NISEC group at TAU Finland.

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9587)

Do not have duplicate section heads

Change find-doc-nits to complain if a section header is repeated,
within a parent header (i.e., duplicate =head2 within a =head1).
In almost all cases, we just remove the duplicate header, as
it was a "continuation" of the =head1 that was already in affect.
In some cases, just remove "=head1 NOTES", possibly moving text
around, because the "NOTES" were really important parts of the
DESCRIPTION section.

No =headX sections should end with a period.
All =head1 labels should be in all uppercase.
No sub-head (=head2, etc) should be in all uppercase.
Update find-doc-nits to reject the above.

Fixup an internal POD link

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Yang <kaishen.yy@antfin.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9631)

Include mac_meth and mac_lib in the FIPS provider

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9694)

Cleanup ciphers and Add 3des ciphers.

Moved the relevant ciphers into default and restructed headers to allow the move.
This removed most of the cases of #ifdef NO_XXX (which are now specified in build.info)

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9482)

Fix Issue OSS-Fuzz: Branch on uninitialized memory (in ccm code).

This would also happen for aes-ccm. There was one branch path where it just returned 1
without setting *padlen, It now branches so that the value is set to 0.

Fixes #9691

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9692)

Update EVP_MAC.pod

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9667)

Get rid of the diversity of names for MAC parameters

The EVP_PKEY MAC implementations had a diversity of controls that were
really the same thing.  We did reproduce that for the provider based
MACs, but are changing our minds on this.  Instead of that, we now use
one parameter name for passing the name of the underlying ciphers or
digests to a MAC implementation, "cipher" and "digest", and one
parameter name for passing the output size of the MAC, "size".

Then we leave it to the EVP_PKEY->EVP_MAC bridge to translate "md"
to "digest", and "digestsize" to "size".

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9667)

Remove MAC cruft

A few declarations and static functions / arrays that are no longer
used were left behind when MACs moved to be implemented by providers.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9667)

Add app for fips installation

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9634)

Avoid overflowing FDSET when using select(2).

There is a problem in the rand_unix.c code when the random seed fd is greater
than or equal to FD_SETSIZE and the FDSET overruns its limit and walks the
stack.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9686)

test/evp_test.c: distinguish parsing errors from processing errors

Parsing functions are at liberty to return:

1:  when parsing on processing of the parsed value succeeded
0:  when the parsed keyword is unknown
-1: when the parsed value processing failed

Some parsing functions didn't do this quite right, they returned 0
when they should have returned -1, causing a message like this:

    Line 123: unknown keyword PeerKey

When this message (which is displayed when the parsing function
returns -1) would have been more appropriate:

    Line 123: error processing keyword PeerKey = ffdhe2048-2-pub

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9682)

crypto/bn/build.info: define OPENSL_IA32_SSE2 globally when needed

This is done conditionally.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9679)

Configure: Allow 'DEFINE[]=def'

DEFINE[] definitions end up pushed in @{$config{defines}} instead of
being added to the output file list of defines.  This allows for the
unusual case where we need something to be defined globally, so it
gets picked up by anything using $(CPPFLAGS).

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9679)

Deprecate unprefixed manual entries for openssl commands

Initially, the manual page entry for the 'openssl cmd' command used
to be available at 'cmd(1)'. Later, the aliases 'openssl-cmd(1)' was
introduced, which made it easier to group the openssl commands using
the 'apropos(1)' command or the shell's tab completion.

In order to reduce cluttering of the global manual page namespace,
the manual page entries without the 'openssl-' prefix have been
deprecated in OpenSSL 3.0 and will be removed in OpenSSL 4.0.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9666)

openssl dgst, openssl enc: check for end of input

The input reading loop in 'openssl dgst' and 'openssl enc' doesn't
check for end of input, and because of the way BIO works, it thereby
won't detect that the end is reached before the read is an error.
With the FILE BIO, an error occurs when trying to read past EOF, which
is fairly much ok, except when the command is used interactively, at
least on Unix.  The result in that case is that the user has to press
Ctrl-D twice for the command to terminate.

The issue is further complicated because both these commands use
filter BIOs on top of the FILE BIO, so a naïve attempt to check
BIO_eof() doesn't quite solve it, since that only checks the state of
the source/sink BIO, and the filter BIO may have some buffered data
that still needs to be read.  Fortunately, there's BIO_pending() that
checks exactly that, if any filter BIO has pending data that needs to
be processed.

We end up having to check both BIO_pending() and BIO_eof().

Thanks to Zsigmond Lőrinczy for the initial effort and inspiration.

Fixes #9355

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9668)

Support parsing of SM2 ID in hexdecimal

The current EVP_PEKY_ctrl for SM2 has no capability of parsing an ID
input in hexdecimal.

The newly added ctrl string is called: sm2_hex_id

Test cases and documentation are updated.

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9584)

Add basic aria and camellia ciphers modes to default provider

The aes code has been refactored into generic and algorithn specific parts,
so that most of the code can be shared.
The cipher related files have been broken up into smaller parts.
Add chunked variant of mode ciphers - aria uses this (many other ciphers will use this new code instead of the
generic code used by aes).

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9451)

Fix drbg_ossl_ctx_free() and drbg_nonce_ossl_ctx_free() to handle NULL

If these were passed NULL, the crashed with a SIGSEGV, when they
should do like all other freeing functions and become a no-op.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9650)

Modify ossl_method_store_add() to accept an OSSL_PROVIDER and check for it

If ossl_method_store_add() gets called with a method that already exists
(i.e. the store has one with matching provider, nid and properties), that
method should not be stored.  We do this check inside ossl_method_store_add()
because it has all the locking required to do so safely.

Fixes #9561

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9650)

Modify ossl_method_store_add() to handle reference counting

Because this function affects the reference count on failure (the call
to impl_free() does this), it may as well handle incrementing it as
well to indicate the extra reference in the method store.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9650)

Fix ossl_param_bld_push_{utf8,octet}_string() / param_bld_convert()

ossl_param_bld_push_{utf8,octet}_string() saved the constant string
pointer to a non-constant structure field, so we change that field to
a pointer to a constant.  We also modify param_bld_convert() to
pretend the resulting pointer for PTR types points to a constant as
well.

Completes #9649

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9651)

Constify param builder string functions.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9649)

Correct documented return value for BIO_get_mem_data()

CLA: trivial

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9643)

Start up DEVRANDOM entropy improvement for older Linux devices.

Improve handling of low entropy at start up from /dev/urandom by waiting for
a read(2) call on /dev/random to succeed.  Once one such call has succeeded,
a shared memory segment is created and persisted as an indicator to other
processes that /dev/urandom is properly seeded.

This does not fully prevent against attacks weakening the entropy source.
An attacker who has control of the machine early in its boot sequence
could create the shared memory segment preventing detection of low entropy
conditions.  However, this is no worse than the current situation.

An attacker would also be capable of removing the shared memory segment
and causing seeding to reoccur resulting in a denial of service attack.
This is partially mitigated by keeping the shared memory alive for the
duration of the process's existence.  Thus, an attacker would not only need
to have called call shmctl(2) with the IPC_RMID command but the system
must subsequently enter a state where no instances of libcrypto exist in
any process.  Even one long running process will prevent this attack.

The System V shared memory calls used here go back at least as far as
Linux kernel 2.0.  Linux kernels 4.8 and later, don't have a reliable way
to detect that /dev/urandom has been properly seeded, so a failure is raised
for this case (i.e. the getentropy(2) call has already failed).

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9595)

prevent endless recursion when trace API is used within OPENSSL_init_crypto()

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Yang <kaishen.yy@antfin.com>
(Merged from https://github.com/openssl/openssl/pull/9567)

cleanup provider digests

Added some missing #ifdef NO_XXX around some of the digest functions.
Renamed core_mkdigest.h to digestcommon.h
Added ERR_raise() to set/get params for digest.
Moved common code for get_params/gettable_params into digest_common.c
Renamed #defines in digestcommon.
Removed null_prov.c (It should not be needed)

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9625)

Add aes_ccm to provider

Add Cleanups for gcm - based on the changes to ccm.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Patrick Steuer <patrick.steuer@de.ibm.com>
(Merged from https://github.com/openssl/openssl/pull/9280)

Get rid of using deprecated function isascii

Avoid using functions deprecated in some libcs (e.g. uClibc-ng).
Fixes #9557

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9628)

Add a fallback definition for __NR_getrandom for x86 linux

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9639)

Use macros internally for algorithm names

The macros are defined in include/openssl/core_names.h and follow the
naming standard OSSL_{OPNAME}_NAME_{ALGONAME}, where {OPNAME} is the
name of the operation (such as MAC) and {ALGONAME} is the name of the
algorithm.  Example: OSSL_MAC_NAME_HMAC

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9635)

doc: fix link in BN_new.pod

Fixes #9622

CLA: trivial

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9627)

Untangle / retangle opensslv.h, openssslconf.h and macros.h

When openssl/macros.h is included without openssl/opensslv.h, it can't
define OPENSSL_API_4 properly (with sufficient warnings enabled, the
compiler will complain about OPENSSL_VERSION_MAJOR not being defined).
The quick fix could have been to include openssl/opensslv.h in
openssl/macros.h, but that would create a nasty include loop, since
openssl/opensslv.h includes openssl/opensslconf.h, which includes
openssl/macros.h, in an order that leads back to macro check errors.

The objective is to make these headers more independent:

- openssl/opensslconf.h should really be completely independent, as it
  only defines macros for configuration values.  However, it needs to
  include openssl/macros.h for backward compatibility reasons.  We do
  this at the very end, under inclusion guards.
- openssl/macros.h is changed to include openssl/opensslconf.h, so it
  gets necessary configuration values to build some macros.  This will
  not cause an endless inclusion loop, since opensslconf.h's inclusion
  of macros.h is under guard.
- openssl/opensslv.h is changed to include openssl/macros.h instead of
  openssl/opensslconf.h.

Only one last piece needs to be done to make openssl/macros.h
independent from openssl/opensslv.h.  We can realise that the
definition of OPENSSL_API_4 doesn't need to depend on the current
version number.  There's nothing in our configuration that  would have
OPENSSL_API_4 defined to 1, and if the user sets OPENSSL_API_COMPAT or
OPENSSL_API_LEVEL to a high enough value, we consider that a
deliberate and knowledgable action on their part.

Fixes #7874
Fixes #9601

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/9626)

Add a fallback definition for __NR_getrandom for ARM linux

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9619)

cipher cleanups.

Add test to evp_test_extra for ciphers (that is similiar to the digest_fetch).
Move some of the aes and gcm methods that can be shared with other ciphers into ciphers_common.c

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9580)

Add fips provider code for handling self test data

More PR's related to self test will be derived from this PR.

Note: the code removed in core_get_params() was causing a freeze since the
fips module was being loaded from a config file, which then called core_get_params()
which then tried to init the config fle again...

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9596)

Use WARNINGS heading not WARNING

Also update find-doc-nits to reject "=head1 WARNING"

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/9602)

Use EXAMPLES not EXAMPLE for section title

And update find-doc-nits to complain if "=head1 EXAMPLE" is found.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/9602)

Fix some pod-page ordering nits

The find-doc-nits script only looked for EXAMPLES, not EXAMPLE.
Fix the pattern and then fix the errors that resulted.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/9602)

Fix --strict-warnings build

Appease -Wstring-plus-int.

Signed-off-by: Patrick Steuer <patrick.steuer@de.ibm.com>
Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9608)

Test for out-of-bounds write when requesting zero bytes from shake

Signed-off-by: Patrick Steuer <patrick.steuer@de.ibm.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9433)

Directly return from final sha3/keccak_final if no bytes are requested

Requesting zero bytes from shake previously led to out-of-bounds write
on some platforms.

Signed-off-by: Patrick Steuer <patrick.steuer@de.ibm.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9433)

.travis.yml: Use travis_terminate on failure

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9620)

Fix error handling in X509_chain_up_ref

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9614)

Windows UWP builds: determine automatically if asm should be disabled

Earlier Windows SDK versions lack the necessary support for our ASM
builds, so we check for the SDK version that has the support.

Information on exactly what registry key to check was found here:

https://stackoverflow.com/questions/2665755/how-can-i-determine-the-version-of-the-windows-sdk-installed-on-my-computer

Ref: #9125

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9440)

Fix 9bf682f which broke nistp224_method

Signed-off-by: Patrick Steuer <patrick.steuer@de.ibm.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9607)

Rename ctx_{get,set}_params to {get,set}_ctx_params

Recently, we added dispatched functions to get parameter descriptions,
and those for operation context parameters ended up being called
something_gettable_ctx_params and something_settable_ctx_params.

The corresponding dispatched functions to actually perform parameter
transfers were previously called something_ctx_get_params and
something_ctx_set_params, which doesn't quite match, so we rename them
to something_get_ctx_params and something_set_ctx_params.

An argument in favor of this name change is English, where you'd
rather say something like "set the context parameters".

This only change the libcrypto <-> provider interface.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9612)

Ignore function code collisions

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9600)

Re-implement 'openssl list -mac-algorithms'

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8877)