Configure: ensure empty arrays aren't created inadvertently

Just refering to a hash table element as an array reference will
automatically create that element.  Avoid that by defaulting to
a separate empty array reference.

Fixes #7543

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7544)

(cherry picked from commit 3bed01a09071fb289484dfd265f0a8a991537282)

Test use of a brainpool ECDSA certificate

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7442)

(cherry picked from commit 24ae00388fb9e25af8f94d36b7c191ae90061586)

Add some test brainpool certificates

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7442)

(cherry picked from commit 83c81eebed52aa84b6b34d26e984c859158ca1c0)

Don't negotiate TLSv1.3 if our EC cert isn't TLSv1.3 capable

TLSv1.3 is more restrictive about the curve used. There must be a matching
sig alg defined for that curve. Therefore if we are using some other curve
in our certificate then we should not negotiate TLSv1.3.

Fixes #7435

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7442)

(cherry picked from commit de4dc598024fd0a9c2b7a466fd5323755d369522)

Fix SipHash init order.

Setting the SipHash hash size and setting its key is done with two
independent functions...  and yet, the internals depend on both.

Unfortunately, the function to change the size wasn't adapted for the
possibility that the key was set first, with a different hash size.

This changes the hash setting function to fix the internal values
(which is easy, fortunately) according to the hash size.

evpmac.txt value for digestsize:8 is also corrected.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7613)

(cherry picked from commit 425036130dfb3cfbef5937772f7526ce60133264)

Unbreak SECLEVEL 3 regression causing it to not accept any ciphers.

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #7391
(cherry picked from commit 75b68c9e4e8591a4ebe083cb207aeb121baf549f)

Check return value of EVP_PKEY_new

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7427)

(cherry picked from commit d896b79b0994a35ecfd1c8e729d348d67236150e)

[crypto/bn] swap BN_FLG_FIXED_TOP too

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/7599)

(cherry picked from commit dd41956d80686638d74fd203bd67060f90966280)

Fix issues with do_rand_init/rand_cleanup_int

Fixes #7022

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7588)

(cherry picked from commit e2d227bb4a25bb75354a40816439630a8162f073)

VMS build: colon after target must be separated with a space

... otherwise, it's taken to be part of a device name.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7602)

(cherry picked from commit e9994901f835420764d020968d4588fc09ec74c3)

Have install targets depend on more precise build targets

We only had the main 'install' target depend on 'all'.  This changes
the dependencies so targets like install_dev, install_runtime_libs,
install_engines and install_programs depend on build targets that are
correspond to them more specifically.  This increases the parallel
possibilities.

Fixes #7466

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7583)

(cherry picked from commit e8d01a608705e4320082a11a3870aa7e19c7290f)

Allow parallel install

When trying 'make -j{n} install', you may occasionally run into
trouble because to sub-targets (install_dev and install_runtime) try
to install the same shared libraries.  That makes parallel install
difficult.

This is solved by dividing install_runtime into two parts, one for
libraries and one for programs, and have install_dev depend on
install_runtime_libs instead of installing the shared runtime
libraries itself.

Fixes #7466

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7583)

(cherry picked from commit c1123d9f7efb005a109aeccaba82c40bf9bd4c1d)

Fix a doc-nit in EVP_PKEY_CTX_ctrl.pod

[skip-ci]

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7546)

(cherry picked from commit e5a8712d03334c4b7cb9f29d6d1daee399c1223e)

rand_unix.c: open random devices on first use only

Commit c7504aeb640a (pr #6432) fixed a regression for applications in
chroot environments, which compensated the fact that the new OpenSSL CSPRNG
(based on the NIST DRBG) now reseeds periodically, which the previous
one didn't. Now the reseeding could fail in the chroot environment if the
DEVRANDOM devices were not present anymore and no other entropy source
(e.g. getrandom()) was available.

The solution was to keep the file handles for the DEVRANDOM devices open
by default. In fact, the fix did more than this, it opened the DEVRANDOM
devices early and unconditionally in rand_pool_init(), which had the
unwanted side effect that the devices were opened (and kept open) even
in cases when they were not used at all, for example when the getrandom()
system call was available. Due  to a bug (issue #7419) this even happened
when the feature was disabled by the application.

This commit removes the unconditional opening of all DEVRANDOM devices.
They will now only be opened (and kept open) on first use. In particular,
if getrandom() is available, the handles will not be opened unnecessarily.

This change does not introduce a regression for applications compiled for
libcrypto 1.1.0, because the SSLEAY RNG also seeds on first use. So in the
above constellation the CSPRNG will only be properly seeded if it is happens
before the forking and chrooting.

Fixes #7419

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7437)

(cherry picked from commit 8cfc19716c22dac737ec8cfc5f7d085e7c37f4d8)

Test: enable internal tests for shared Windows builds

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7462)

(cherry picked from commit 1901516a4ba909fff12e0e7815aa2d499f4d6d67)

Test: link drbgtest statically against libcrypto

and remove duplicate rand_drbg_seedlen() implementation again.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7462)

(cherry picked from commit 1c615e4ce97715ae3af9255bc57be32a49687966)

Give a better error if an attempt is made to set a zero length groups list

Previously we indicated this as a malloc failure which isn't very
helpful.

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/7479)

(cherry picked from commit 680bd131b69d57e891888ab70d300176a5a16617)

Ignore disabled ciphers when deciding if we are using ECC

use_ecc() was always returning 1 because there are default (TLSv1.3)
ciphersuites that use ECC - even if those ciphersuites are disabled by
other options.

Fixes #7471

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/7479)

(cherry picked from commit 589b6227a85ea0133fe91d744b16dd72edee929a)

Add missing RAND initialisation call.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/7587)

(cherry picked from commit ac765685d4b08a48cefffc71c434760045154dad)

Rename the rand_drbg_st data member "pool" to "seed_pool"

... to make the intended use more clear and differentiate
it from the data member "adin_pool".

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7575)

(cherry picked from commit 31f32abb8eb2a3ebd8500e6e0460b4a6791e5ed7)

Initialize reseed_gen_counter to 1, like it is done in master

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7532)

util/add-depends.pl: go through shared_sources too

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7545)

(cherry picked from commit 3866b2247fb7904a4e660593a16365147f479415)

Fix a race condition in drbgtest.c

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7531)

(cherry picked from commit 2bb1b5ddd12c23bbfa7fb60ee3296612ca943fef)

Avoid two memory allocations in each RAND_DRBG_bytes

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7518)

Fix error handling in RAND_DRBG_uninstantiate

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7519)

Fix error handling in drbgtest.c

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7519)

Fix error handling in rand_drbg_new

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7519)

Fix error handling in RAND_DRBG_set

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7519)

Fix return formatting.

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/7564)

(cherry picked from commit 2087028612027368e9508e1b253aab715a5a35d6)

Cleanse the key log buffer.

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/7564)

(cherry picked from commit e931f370aa38d8645b35fb8d6260cb44d37b6b61)

Restore sensible "sess_accept" counter tracking

Commit 9ef9088c1585e13b9727796f15f77da64dbbe623 switched the SSL/SSL_CTX
statistics counters to using Thread-Sanitizer-friendly primitives.
However, it erroneously converted an addition of -1
(for s->session_ctx->stats.sess_accept) to an addition of +1, since that
is the only counter API provided by the internal tsan_assist.h header
until the previous commit.  This means that for each accepted (initial)
connection, the session_ctx's counter would get doubly incremented, and the
(switched) ctx's counter would also get incremented.

Restore the counter decrement so that each accepted connection increments
exactly one counter exactly once (in net effect).

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7464)

(cherry picked from commit 2aaa0b146b967397a6e61fa8df969e7847f82086)

Add tsan_decr() API, counterpart of tsan_counter()

The existing tsan_counter() API increments a reference counter.
Provide a new API, tsan_decr(), to decrement such a reference counter.
This can be used, for example, when a reference is added to the session_ctx's
sess_accept stats but should more properly be tracked in the regular ctx's
statistics.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7464)

(cherry picked from commit 95658c32436017aeeef3d8598957071baf6769a9)

crypto/engine/eng_devcrypto.c: ensure we don't leak resources

If engine building fails for some reason, we must make sure to close
the /dev/crypto handle.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/7506)

(cherry picked from commit 681e8cacdbdc44ac00af29b6656fc52745a9baa2)

crypto/engine/eng_devcrypto.c: open /dev/crypto only once

We opened /dev/crypto once for each session, which is quite unnecessary.
With this change, we open /dev/crypto once at engine init, and close
it on unload.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/7506)

(cherry picked from commit 458c7dad9e3f59490fb0908c1a27ca39e4cf38dd)

crypto/engine/eng_devcrypto.c: new compilers are strict on prototypes

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/7506)

(cherry picked from commit 28ac1bd9a97d9725273956e26d89ccfa5b4de67b)

crypto/engine/eng_devcrypto.c: add digest copy

Copying an EVP_MD_CTX, including the implementation local bits, is a
necessary operation.  In this case, though, it's the same as
initializing the local bits to be "copied to".

Fixes #7495

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/7506)

(cherry picked from commit 36af124bfb209b49cb92a5fb9fab627d9cd4a44b)

apps: Stop pretending to care about Netscape keys

The documentation says some commands care, but the code says differently.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7440)

(cherry picked from commit d91d443f0d26262148d1dc9d29f9fdf025b958ca)

Add a constant time flag to one of the bignums to avoid a timing leak.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7549)

(cherry picked from commit 00496b6423605391864fbbd1693f23631a1c5239)

Configuration: when building the dirinfo structure, include shared_sources

This makes sure that any resulting directory target in the build files
also depend on object files meant for shared libraries.

As a side effect, we move the production of the dirinfo structure from
common.tmpl to Configure, to make it easier to check the result.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7452)

(cherry picked from commit b6e660754c2e799cffe4906269fcace0e07c73bc)

Configurations/15-android.conf: detect NDK llvm-ar.

This excluses user from additional PATH adjustments in case NDK has
llvm-ar.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7443)

(cherry picked from commit ddf2e8c8ccad49cff0d6a0a0b26ca892ec4ce33a)

Configurations/15-android.conf: fix implicit __ANDROID_API__ handling.

03ad7c009e16a233c733098db3169c560142ccd3 failed if one didn't pass
explicit -D__ANDROID_API__=N :-(

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7443)

(cherry picked from commit 71f2b3171e8b94b0fbdc7fab2d7ca4ea66edec2b)

Fix a race condition in drbg_add

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7523)

(cherry picked from commit 4011bab1f85d4429bad1e9388bed90a8d0da5639)

Add a client_cert_cb test

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/7413)

(cherry picked from commit 6e46c065b9b97212d63ef1f321b08fb7fa6b320d)

Don't call the client_cert_cb immediately in TLSv1.3

In TLSv1.2 and below a CertificateRequest is sent after the Certificate
from the server. This means that by the time the client_cert_cb is called
on receipt of the CertificateRequest a call to SSL_get_peer_certificate()
will return the server certificate as expected. In TLSv1.3 a
CertificateRequest is sent before a Certificate message so calling
SSL_get_peer_certificate() returns NULL.

To workaround this we delay calling the client_cert_cb until after we
have processed the CertificateVerify message, when we are doing TLSv1.3.

Fixes #7384

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/7413)

(cherry picked from commit e45620140fce22c3251440063bc17440289d730c)

rand_drbg.h: include <openssl/obj_mac.h>

The RAND_DRBG_TYPE preprocessor define depends on a NID, so  we have
to include obj_mac.h to make the header selfcontained.

Fixes #7521

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7524)

(cherry picked from commit cf3d6ef7af7d6c47f5ccd4ce58d822972018b21c)

Fix MSVC warning C4819

CLA: trivial

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7444)

(cherry picked from commit cf4eea12046445fc418507d2d5e14956b4353495)

test/evp_test.c: don't misuse pkey_test_ctrl() in mac_test_run()

pkey_test_ctrl() was designed for parsing values, not for using in
test runs.  Relying on its returned value when it returned 1 even for
control errors made it particularly useless for mac_test_run().

Here, it gets replaced with a MAC specific control function, that
parses values the same way but is designed for use in a _run() rather
than a _parse() function.

This uncovers a SipHash test with an invalid control that wasn't
caught properly.  After all, that stanza is supposed to test that
invalid control values do generate an error.  Now we catch that.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7500)

(cherry picked from commit ce5d64c79c4d809ece8fe28a5b62915467a1c0e1)

Add a test where we reuse the EVP_PKEY_CTX for two HKDF test runs

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7501)

(cherry picked from commit 10d5b415f9e973f44f18eeaf2713868ec813e1d7)

Reset the HKDF state between operations

Fixes #7497

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7501)

(cherry picked from commit ca55d70be031746daddd8bd0611db54ed81f1737)

Windows build: build foo.d after foo.obj

We made the build of foo.obj depend on foo.d, meaning the latter gets
built first.  Unfortunately, the way the compiler works, we are forced
to redirect all output to foo.d, meaning that if the source contains
an error, the build fails without showing those errors.

We therefore remove the dependency and force the build of foo.d to
always happen after build of foo.obj.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7469)

(cherry picked from commit ecc347f5f53a9f2edc2805d50cba07db64267e8a)

ssl/statem: Don't compare size_t with less than zero

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7194)

(cherry picked from commit 60690b5b8396d7d5234cd067206190fb8aca78d2)

VMS & cryptoerr.h: include symhacks.h

Needed to clear a clash between ERR_load_CRYPTO_strings and
ERR_load_crypto_strings

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7194)

(cherry picked from commit cc3a2e4f51679d99507a979f9a920f6068473aa2)

apps/rehash.c: Convert ISO-8859-1 to UTF-8

Believe it or not, the VMS C compiler is remarking on this

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7194)

(cherry picked from commit 4602cc85aff35b7aa5e27dc57ead89f0867637f9)

Rework and simplify resource flow in drbg_add

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7504)

(cherry picked from commit f9e43929c46b38667f67e02765fe0f1c0d3061d6)

randfile.c: fix a Coverity warning

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7510)

(cherry picked from commit 040a03470c7c5bf95fe8e6143db7bef357a22833)

Timing vulnerability in ECDSA signature generation (CVE-2018-0735)

Preallocate an extra limb for some of the big numbers to avoid a reallocation
that can potentially provide a side channel.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/7486)

(cherry picked from commit 99540ec79491f59ed8b46b4edf130e17dc907f52)

Timing vulnerability in DSA signature generation (CVE-2018-0734).

Avoid a timing attack that leaks information via a side channel that
triggers when a BN is resized.  Increasing the size of the BNs
prior to doing anything with them suppresses the attack.

Thanks due to Samuel Weiser for finding and locating this.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/7486)

(cherry picked from commit a9cfb8c2aa7254a4aa6a1716909e3f8cb78049b6)

DSA mod inverse fix

There is a side channel attack against the division used to calculate one of
the modulo inverses in the DSA algorithm.  This change takes advantage of the
primality of the modulo and Fermat's little theorem to calculate the inverse
without leaking information.

Thanks to Samuel Weiser for finding and reporting this.

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/7487)

(cherry picked from commit 415c33563528667868c3c653a612e6fc8736fd79)

drbg_lib: avoid NULL pointer dereference in drbg_add

Found by Coverity Scan

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7511)

(cherry picked from commit 59f90557dd6e35cf72ac72016609d759ac78fcb9)

iOS config targets: disable "async" by default

This also gave enough reason to collect the stuff that's common for
all iOS config targets into the template "ios-common".

Fixes #7318

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7403)

(cherry picked from commit 04e3f9a114c2c142356ef1639d68397a72e0c7f8)

Configure: Improve warning if no random seed source was configured

The new Configure summary box (41349b5e6db) now hides the warning
about the missing seed source (2805ee1e095) too much. To make it
more visible again, add warning markers.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7509)

(cherry picked from commit caa8595276c60f009c8621ad466338d2ae39fb86)

Fix data race in RAND_DRBG_generate

Fixes #7394

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7399)

(cherry picked from commit a83dc59afa2e0207180d7218efed19b20d48de95)

RAND_add()/RAND_seed(): fix failure on short input or low entropy

Commit 5b4cb385c18a (#7382) introduced a bug which had the effect
that RAND_add()/RAND_seed() failed for buffer sizes less than
32 bytes. The reason was that now the added random data was used
exlusively as entropy source for reseeding. When the random input
was too short or contained not enough entropy, the DRBG failed
without querying the available entropy sources.

This commit makes drbg_add() act smarter: it checks the entropy
requirements explicitely. If the random input fails this check,
it won't be added as entropy input, but only as additional data.
More precisely, the behaviour depends on whether an os entropy
source was configured (which is the default on most os):

- If an os entropy source is avaible then we declare the buffer
  content as additional data by setting randomness to zero and
  trigger a regular   reseeding.

- If no os entropy source is available, a reseeding will fail
  inevitably. So drbg_add() uses a trick to mix the buffer contents
  into the DRBG state without forcing a reseeding: it generates a
  dummy random byte, using the buffer content as additional data.

Related-to: #7449

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7456)

(cherry picked from commit 8817215d5c52a76f2b184b624bde4df8556dee6d)

Backport some DRBG renamings and typo fixes

In commit 8bf366519661 some renamings andd typo fixes were made
while adding back the DRBG-HMAC and DRBG-HASH implementation.
Since the commit could not be backported, a lot of unnecessary
differences between master and 1.1.1 were introduced.

These differences result in tiresome merge conflicts when
cherry-picking. To minimize these merge-conflicts, this patch
ports all 'non-feature' changes of commit 8bf366519661
(e.g., renamings of private variables, fixes of typographical
errors, comment changes) manually back to 1.1.1.

The commits a83dc59afa2e (#7399) and 8817215d5c52 (#7456)
failed to cherry-pick previously to 1.1.1, with this patch
they both cherry-pick without conflicts.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/7505)

Add a test for duplicated DTLS records

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7414)

(cherry picked from commit f1358634af5b84be22cb20fff3dcb613f5f8c978)

Properly handle duplicated messages from the next epoch

Since 1fb9fdc30 we may attempt to buffer a record from the next epoch
that has already been buffered. Prior to that this never occurred.

We simply ignore a failure to buffer a duplicated record.

Fixes #6902

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7414)

(cherry picked from commit 840facc3ccab481e1a0bc2cb6e7740e362df7422)

Fix a possible crash in rand_drbg_get_entropy

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7474)

(cherry picked from commit 21311777ad8b8c2f2fb8eb7a4f9618ceae1e043d)

Configure: Reword the summary output

In commit 820e414d2830 (pr #5247) the summary output of the
Configure command was optimized towards instructing people
how to create issue reports.

It turned out that the wording of this message can confuse new
OpenSSL users and make them think that they are seeing an error
message. This commit makes the summary output start with a success
to prevent a misunderstanding. Also it gives more hints to new
OpenSSL users.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7499)

(cherry picked from commit 41349b5e6dbd72bfbeaf3cf189d64914240628e3)

RAND_load_file(): avoid adding small chunks to RAND_add()

Increase the load buffer size such that it exceeds the chunk
size by a comfortable amount. This is done to avoid calling
RAND_add() with a small final chunk. Instead, such a small
final chunk will be added together with the previous chunk
(unless it's the only one).

Related-to: #7449

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7456)

RAND_load_file(): return error if reseeding failed

The failure of RAND_load_file was only noticed because of the
heap corruption which was reported in #7499 and fixed in commit
5b4cb385c18a. To prevent this in the future, RAND_load_file()
now explicitly checks RAND_status() and reports an error if it
fails.

Related-to: #7449

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7456)

Windows: Produce a static version of the public libraries, always

When building shared libraries on Windows, we had a clash between
'libcrypto.lib' the static routine library and 'libcrypto.lib' the
import library.

We now change it so the static versions of our libraries get '_static'
appended to their names.  These will never get installed, but can
still be used for our internal purposes, such as internal tests.

When building non-shared, the renaming mechanism doesn't come into
play.  In that case, the static libraries 'libcrypto.lib' and
'libssl.lib' are installed, just as always.

Fixes #7492

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7496)

(cherry picked from commit b3023ced6b6a4aece6f4d4ec1f6a93b1c03712b6)

RAND: ensure INT32_MAX is defined

This value is used to set DRBG_MAX_LENGTH

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/7467)

(cherry picked from commit f81b043ad856d8b9af5239a4978f8bd4b965dab9)

RAND_add(): fix heap corruption in error path

This bug was introduced by #7382 which enhanced RAND_add() to
accept large buffer sizes. As a consequence, RAND_add() now fails
for buffer sizes less than 32 bytes (i.e. less than 256 bits).
In addition, rand_drbg_get_entropy() forgets to reset the attached
drbg->pool in the case of an error, which leads to the heap corruption.

The problem occurred with RAND_load_file(), which reads the file in
chunks of 1024 bytes each. If the size of the final chunk is less than
32 bytes, then RAND_add() fails, whence RAND_load_file() fails
silently for buffer sizes n = k * 1024 + r with r = 1,...,31.

This commit fixes the heap corruption only. The other issues will
be addressed in a separate pull request.

Thanks to Gisle Vanem for reporting this issue.

Fixes #7449

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7455)

(cherry picked from commit 5b4cb385c18a5bb4e118e300f1c746bf7c2a5628)

build file templates: have targets for all shared library names

We only had targets for the "simple" shared library names (libfoo.so
and not libfoo.so.x.y on Unix, import library libfoo.lib but not
libfoo.dll on Windows).  This has created some confusion why it wasn't
possible to rebuild the less "simple" name directly (just as an
example, someone who mistook the import library on Windows for a
static library, removed the DLL and then found it was difficult to
rebuild directly), so we change the target to include all possible
names.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7451)

(cherry picked from commit d8cac50b023be249cdaba054f43acecf17025ce4)

Don't complain and fail about unknown TLSv1.3 PSK identities in s_server

An unknown PSK identity could be because its actually a session resumption
attempt. Sessions resumptions and external PSKs are indistinguishable so
the callbacks need to fail gracefully if they don't recognise the identity.

Fixes #7433

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7434)

(cherry picked from commit 2d015189b97c60b67e10aed320230357bf6b200f)

Buffer a ClientHello with a cookie received via DTLSv1_listen

Previously when a ClientHello arrives with a valid cookie using
DTLSv1_listen() we only "peeked" at the message and left it on the
underlying fd. This works fine for single threaded applications but for
multi-threaded apps this does not work since the fd is typically reused for
the server thread, while a new fd is created and connected for the client.
By "peeking" we leave the message on the server fd, and consequently we
think we've received another valid ClientHello and so we create yet another
fd for the client, and so on until we run out of fds.

In this new approach we remove the ClientHello and buffer it in the SSL
object.

Fixes #6934

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/7375)

(cherry picked from commit 079ef6bd534d2f708d8013cfcd8ea0d2f600c788)

Use the read and write buffers in DTLSv1_listen()

Rather than using init_buf we use the record layer read and write buffers
in DTLSv1_listen(). These seem more appropriate anyway and will help with
the next commit.

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/7375)

(cherry picked from commit 2fc4c77c3f06443f4c476f6f58d83e5e108d1dce)

Test DTLS cookie generation and verification

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7431)

(cherry picked from commit edcd29efd32c51f298ad5ab438e2d4cc5411e9a9)

Fix a DTLS memory leak

Fixes #7428

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7431)

(cherry picked from commit 01666a8c1db3ecfb999e1a8f2c5436d114f95681)

Configurations/15-android.conf: add support for "standalone toolchain".

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7400)

(cherry picked from commit 03ad7c009e16a233c733098db3169c560142ccd3)

arch/async_posix.h: improve portability.

{make|swap|get|set}context are removed in POSIX.1-2008, but glibc
apparently keeps providing it.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7420)

(cherry picked from commit 9d71a24ebf57e7157888af1ca587eafe914bf96f)

Fix tls_cbc_digest_record is slow using SHA-384 and short messages

The formula used for this is now

kVarianceBlocks = ((255 + 1 + md_size + md_block_size - 1) / md_block_size) + 1

Notice that md_block_size=64 for SHA256, which results on the
magic constant kVarianceBlocks = 6.
However, md_block_size=128 for SHA384 leading to kVarianceBlocks = 4.

CLA:trivial

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7342)

(cherry picked from commit cb8164b05e3bad5586c2a109bbdbab1ad65a1a6f)

Apply self-imposed path length also to root CAs

Also, some readers of the code find starting the count at 1 for EE
cert confusing (since RFC5280 counts only non-self-issued intermediate
CAs, but we also counted the leaf).  Therefore, never count the EE
cert, and adjust the path length comparison accordinly.  This may
be more clear to the reader.

Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit dc5831da59e9bfad61ba425d886a0b06ac160cd6)

Only CA certificates can be self-issued

At the bottom of https://tools.ietf.org/html/rfc5280#page-12 and
top of https://tools.ietf.org/html/rfc5280#page-13 (last paragraph
of above https://tools.ietf.org/html/rfc5280#section-3.3), we see:

   This specification covers two classes of certificates: CA
   certificates and end entity certificates.  CA certificates may be
   further divided into three classes: cross-certificates, self-issued
   certificates, and self-signed certificates.  Cross-certificates are
   CA certificates in which the issuer and subject are different
   entities.  Cross-certificates describe a trust relationship between
   the two CAs.  Self-issued certificates are CA certificates in which
   the issuer and subject are the same entity.  Self-issued certificates
   are generated to support changes in policy or operations.  Self-
   signed certificates are self-issued certificates where the digital
   signature may be verified by the public key bound into the
   certificate.  Self-signed certificates are used to convey a public
   key for use to begin certification paths.  End entity certificates
   are issued to subjects that are not authorized to issue certificates.

that the term "self-issued" is only applicable to CAs, not end-entity
certificates.  In https://tools.ietf.org/html/rfc5280#section-4.2.1.9
the description of path length constraints says:

   The pathLenConstraint field is meaningful only if the cA boolean is
   asserted and the key usage extension, if present, asserts the
   keyCertSign bit (Section 4.2.1.3).  In this case, it gives the
   maximum number of non-self-issued intermediate certificates that may
   follow this certificate in a valid certification path.  (Note: The
   last certificate in the certification path is not an intermediate
   certificate, and is not included in this limit.  Usually, the last
   certificate is an end entity certificate, but it can be a CA
   certificate.)

This makes it clear that exclusion of self-issued certificates from
the path length count applies only to some *intermediate* CA
certificates.  A leaf certificate whether it has identical issuer
and subject or whether it is a CA or not is never part of the
intermediate certificate count.  The handling of all leaf certificates
must be the same, in the case of our code to post-increment the
path count by 1, so that we ultimately reach a non-self-issued
intermediate it will be the first one (not zeroth) in the chain
of intermediates.

Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit ed422a2d0196ada0f5c1b6e296f4a4e5ed69577f)

EVP module documentation pass

Replace ECDH_KDF_X9_62() with internal ecdh_KDF_X9_63()

Signed-off-by: Antoine Salon <asalon@vmware.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/7345)

(cherry picked from commit ffd89124bdfc9e69349492c3f15383bb35520a11)

Fix: 'openssl ca' command crashes when used with 'rand_serial' option

Commit ffb46830e2df introduced the 'rand_serial' option. When it is used,
the 'serialfile' does not get initialized, i.e. it remains a NULL pointer.
This causes a crash when the NULL pointer is passed to the rotate_serial()
call.

This commit fixes the crash and unifies the pointer checking before
calling the rotate_serial() and save_serial() commands.

Fixes #7412

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7417)

(cherry picked from commit aeec793b4bee929cef8ae35ec4b5a783a6e1d7ed)

Build file templates: look at *all* defines

When looking at configured macro definitions, we must look at both
what comes from the config target AND what comes from user
configuration.

Fixes #7396

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/7402)

(cherry picked from commit 92ebf6c4c21ff4b41ba1fd69af74b2039e138114)

Add a missing check on s->s3->tmp.pkey

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7405)

(cherry picked from commit 61bef9bde09dc6099a7c59baa79898e3b003fec3)

print() is a function in Python 3

CLA: trivial

Discovered via #7410 @ https://travis-ci.org/openssl/openssl/jobs/442003489#L440

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/7403)

(cherry picked from commit 83e4533a71c5c78278e9763552a5e5f1806473ee)

DRBG: fix reseeding via RAND_add()/RAND_seed() with large input

In pull request #4328 the seeding of the DRBG via RAND_add()/RAND_seed()
was implemented by buffering the data in a random pool where it is
picked up later by the rand_drbg_get_entropy() callback. This buffer
was limited to the size of 4096 bytes.

When a larger input was added via RAND_add() or RAND_seed() to the DRBG,
the reseeding failed, but the error returned by the DRBG was ignored
by the two calling functions, which both don't return an error code.
As a consequence, the data provided by the application was effectively
ignored.

This commit fixes the problem by a more efficient implementation which
does not copy the data in memory and by raising the buffer the size limit
to INT32_MAX (2 gigabytes). This is less than the NIST limit of 2^35 bits
but it was chosen intentionally to avoid platform dependent problems
like integer sizes and/or signed/unsigned conversion.

Additionally, the DRBG is now less permissive on errors: In addition to
pushing a message to the openssl error stack, it enters the error state,
which forces a reinstantiation on next call.

Thanks go to Dr. Falko Strenzke for reporting this issue to the
openssl-security mailing list. After internal discussion the issue
has been categorized as not being security relevant, because the DRBG
reseeds automatically and is fully functional even without additional
randomness provided by the application.

Fixes #7381

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7382)

(cherry picked from commit 3064b55134434a0b2850f07eff57120f35bb269a)

Fix no-engine

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/7365)

(cherry picked from commit 5f9f67b9d494e26941aa8d66d28a6a993c557643)

Indentation fixes.

The PR #7329 left some indentation slightly off.  This fixes it.

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
(Merged from https://github.com/openssl/openssl/pull/7360)

(cherry picked from commit 5b639d4cb3b9a33536e2ebadf6a03149ea26ba32)

Remove useless check.
Hash can be longer than EC group degree and it will be truncated.

CLA: trivial

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7329)

(cherry picked from commit b770a80f6d0b08659cd03f26388b45512adb84f3)

Fix no-psk

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7306)

(cherry picked from commit 734af93a278a7a06710167219e1f05e525c9dd49)

Fix no-tls1_2

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7308)

(cherry picked from commit 7f1d923aa9dc55dd23a7741e4341ec421c683941)

Safer memory cleanup in (crypto/rsa/rsa_lib.c)

We don't need to use secure clean for public key.

CLA: trivial

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
(Merged from https://github.com/openssl/openssl/pull/7363)

(cherry picked from commit c033101db39c93cf41d80dfee4357f2617ede1a0)

Fix copy&paste error found in Coverity scan

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7377)

(cherry picked from commit 628ee796389b555ddb5fc28355e16e9417ab1724)

rsa/rsa_ossl.c: fix and extend commentary [skip ci].

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/7123)

(cherry picked from commit d1c008f66bad435b18aa45aa59f72bed7c682849)

ssl/s3_enc.c: fix logical errors in ssl3_final_finish_mac.

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7085)

(cherry picked from commit 7d0effeacbb50b12bfc24df7614d7cf5c8686f51)

sha/asm/keccak1600-s390x.pl: resolve -march=z900 portability issue.

Negative displacement in memory references was not originally specified,
so that for maximum coverage one should abstain from it, just like with
any other extension. [Unless it's guarded by run-time switch, but there
is no switch in keccak1600-s390x.]

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7239)

(cherry picked from commit fc97c882f443060dffd8eb56a6b8784e52096c86)