CID 1442836:  Resource leaks

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/8318)

Windows: Call TerminateProcess, not ExitProcess

Ty Baen-Price explains:

> Problem and Resolution:
> The following lines of code make use of the Microsoft API ExitProcess:
>
> ```
> Apps\Speed.c line 335: ExitProcess(ret);
> Ms\uplink.c line 22: ExitProcess(1);
> ```
>
> These function calls are made after fatal errors are detected and
> program termination is desired. ExitProcess(), however causes
> _orderly_ shutdown of a process and all its threads, i.e. it unloads
> all dlls and runs all destructors. See MSDN for details of exactly
> what happens
> (https://msdn.microsoft.com/en-us/library/windows/desktop/ms682658(v=vs.85).aspx).
> The MSDN page states that ExitProcess should never be called unless
> it is _known to be safe_ to call it. These calls should simply be
> replaced with calls to TerminateProcess(), which is what should be
> called for _disorderly_ shutdown.
>
> An example of usage:
>
> ```
> TerminateProcess(GetCurrentProcess(), exitcode);
> ```
>
> Effect of Problem:
> Because of a compilation error (wrong c++ runtime), my program
> executed the uplink.c ExitProcess() call. This caused the single
> OpenSSL thread to start executing the destructors of all my dlls,
> and their objects. Unfortunately, about 30 other threads were
> happily using those objects at that time, eventually causing a
> 0xC0000005 ACCESS_VIOLATION. Obviously an ACCESS_VIOLATION is the
> best case scenario, as I'm sure you can imagine at the consequences
> of undiscovered memory corruption, even in a terminating process.

And on the subject of `TerminateProcess()` being asynchronous:

> That is technically true, but I think it's probably synchronous
> "enough" for your purposes, since a call to TerminateProcess
> suspends execution of all threads in the target process. This means
> it's really only asynchronous if you're calling TerminateProcess one
> some _other_ process. If you're calling TerminateProcess on your own
> process, you'll never return from the TerminateProcess call.

Fixes #2489
Was originally RT-4526

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8301)

Don't restrict the number of KeyUpdate messages we can process

Prior to this commit we were keeping a count of how many KeyUpdates we
have processed and failing if we had had too many. This simplistic approach
is not sufficient for long running connections. Since many KeyUpdates
would not be a particular good DoS route anyway, the simplest solution is
to simply remove the key update count.

Fixes #8068

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/8299)

engines/dasync: add explaining comments about AES-128-CBC-HMAC-SHA1

Fixes #7950

It was reported that there might be a null pointer dereference in the
implementation of the dasync_aes_128_cbc_hmac_sha1() cipher, because
EVP_aes_128_cbc_hmac_sha1() can return a null pointer if AES-NI is
not available. It took some analysis to find out that this is not
an issue in practice, and these comments explain the reason to comfort
further NPD hunters.

Detected by GitHub user @wurongxin1987 using the Sourcebrella Pinpoint
static analyzer.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8305)

engines/e_devcrypto: set digest input_blocksize

This restores the behavior of previous versions of the /dev/crypto
engine, in alignment with the default implementation.

Reported-by: Gerard Looije <lglooije@hotmail.com>
Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8213)

engines/e_devcrypto: fixes logic in close_devcrypto

Call close(cfd) before setting cfd = -1.

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8213)

engines/e_devcrypto.c: fix cipher_ctrl function

This fixes commit c703a80, which had a mistake in cipher_ctrl function.

Move the /dev/crypto session cleanup code to its own function.

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8213)

engines/build.info: fix devcrypto MODULES entry

The devcrypto MODULES line was missing the "engine" attribute.

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8213)

Fix a grammar nit in CRYPTO_get_ex_new_index.pod

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8303)

Make sure that generated POD files are actually created before we run doc-nits

Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #8285

Indent with 4

doc-nits says that over needs a parameter

Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #8285

Fix dasync engine

The aes128_cbc_hmac_sha1 cipher in the dasync engine is broken. Probably
by commit e38c2e8535 which removed use of the "enc" variable...but not
completely.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/8291)

SSL_CONF_cmd: fix doc for NoRenegotiation

The option is a flag for Options, not a standalone setting.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8292)

MIPS32R3 provides the EXT instruction to extract bits from
registers. As the AES table is already 1K aligned we can
use it everywhere and speedup table address calculation by
10%. Performance numbers:

decryption         16B       64B      256B     1024B     8192B
-------------------------------------------------------------------
aes-256-cbc   5636.84k  6443.26k  6689.02k  6752.94k  6766.59k bef.
aes-256-cbc   6200.31k  7195.71k  7504.30k  7585.11k  7599.45k aft.
-------------------------------------------------------------------
aes-128-cbc   7313.85k  8653.67k  9079.55k  9188.35k  9205.08k bef.
aes-128-cbc   7925.38k  9557.99k 10092.37k 10232.15k 10272.77k aft.

encryption         16B       64B      256B     1024B     8192B
-------------------------------------------------------------------
aes-256 cbc   6009.65k  6592.70k  6766.59k  6806.87k  6815.74k bef.
aes-256 cbc   6643.93k  7388.69k  7605.33k  7657.81k  7675.90k aft.
-------------------------------------------------------------------
aes-128 cbc   7862.09k  8892.48k  9214.04k  9291.78k  9311.57k bef.
aes-128 cbc   8639.29k  9881.17k 10265.86k 10363.56k 10392.92k aft.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8206)

cfi build fixes in x86-64 ghash assembly

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8281)

Clear BN_FLG_CONSTTIME on BN_CTX_get()

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8253)

Test for constant-time flag leakage in BN_CTX

This commit adds a simple unit test to make sure that the constant-time
flag does not "leak" among BN_CTX frames:

- test_ctx_consttime_flag() initializes (and later frees before
  returning) a BN_CTX object, then it calls in sequence
  test_ctx_set_ct_flag() and test_ctx_check_ct_flag() using the same
  BN_CTX object. The process is run twice, once with a "normal"
  BN_CTX_new() object, then with a BN_CTX_secure_new() one.
- test_ctx_set_ct_flag() starts a frame in the given BN_CTX and sets the
  BN_FLG_CONSTTIME flag on some of the BIGNUMs obtained from the frame
  before ending it.
- test_ctx_check_ct_flag() then starts a new frame and gets a number of
  BIGNUMs from it. In absence of leaks, none of the BIGNUMs in the new
  frame should have BN_FLG_CONSTTIME set.

In actual BN_CTX usage inside libcrypto the leak could happen at any
depth level in the BN_CTX stack, with varying results depending on the
patterns of sibling trees of nested function calls sharing the same
BN_CTX object, and the effect of unintended BN_FLG_CONSTTIME on the
called BN_* functions.

This simple unit test abstracts away this complexity and verifies that
the leak does not happen between two sibling functions sharing the same
BN_CTX object at the same level of nesting.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8253)

test/context_internal_test.c: don't initialize as a separate test

Because test order can be randomized, running foo_init() as a separate
test is unsafe practice.  Instead, we make it possible to call it
multiple times, and call it at the start of each separate test.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8288)

sha/keccak1600.c: subscribe more platforms for "complementing" optimization.

E.g. on MIPS64 it gives >20% improvement...

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8261)

Don't set SNI by default if hostname is not dNS name

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8175)

Fix reference to symbol 'main'.

The AIX binder needs to be instructed that the output will have no entry
point (see AIX' ld manual: -e in the Flags section; autoexp and noentry
in the Binder section).

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8282)

Add a test for interleaving app data with handshake data in TLSv1.3

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/8191)

Don't interleave handshake and other record types in TLSv1.3

In TLSv1.3 it is illegal to interleave handshake records with non handshake
records.

Fixes #8189

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/8191)

Fixup internal documentation

There were some faults that got caught by the updated doc-nits

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8270)

Fix a test ordering issue.

A randomised order causes failure due to unintentional dependencies between
two of the test cases.

[extended tests]

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/8279)

util/find-docs-nits: Recognise SPARSE_ARRAY_OF

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8269)

util/find-docs-nits: Extend to handle internal documentation

While we're at it, we also check for names that contain white-space,
as they are invalid.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8269)

cygwin: drop explicit O_TEXT

Cygwin binaries should not enforce text mode these days, just
use text mode if the underlying mount point requests it

CLA: trivial

Signed-off-by: Corinna Vinschen <vinschen@redhat.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8248)

Conform to proper NAME section format

The NAME section format is comma separated names to the left of the
left of the dash, free form on the right.  If we don't follow that
form, programs like apropos(1) and whatis(1) can't do their job
properly.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8267)

Property: naming and manual clarifiations

- Add a bit more text about that is expected of the user or
  OSSL_METHOD_STOREs.
- Clarify what a method and what a numeric identity are.
- Change all mentions of 'implementation' and 'result' to 'method'.

To clarify further: OpenSSL has used the term 'method' for structures
that mainly contains function pointers.  Those are the methods that
are expected to be stored away in OSSL_METHOD_STOREs.  In the end,
however, it's the caller's responsibility to define exactly what they
want to store, as long as its 'methods' are associated with a numeric
identity and properties.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8265)

Don't leak EVP_KDF_CTX on error

Found by Coverity

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/8260)

generated files

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8224)

Properties for implementation selection.

Properties are a sequence of comma separated name=value pairs.  A name
without a corresponding value is assumed to be a Boolean and have the
true value 'yes'.  Values are either strings or numbers.  Strings can be
quoted either _"_ or _'_ or unquoted (with restrictions).  There are no
escape characters inside strings.  Number are either decimal digits or
'0x' followed by hexidecimal digits.  Numbers are represented internally
as signed sixty four bit values.

Queries on properties are a sequence comma separated conditional tests.
These take the form of name=value (equality test), name!=value (inequality
test) or name (Boolean test for truth).  Queries can be parsed, compared
against a definition or merged pairwise.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8224)

Add missing dots in dgst man page

CLA: trivial

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
GH: #8142

Fixed typo

CLA: trivial

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
GH: #8121

Check for unpaired .cfi_remember_state

Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #8109

Fix some CFI issues in x86_64 assembly

The add/double shortcut in ecp_nistz256-x86_64.pl left one instruction
point that did not unwind, and the "slow" path in AES_cbc_encrypt was
not annotated correctly. For the latter, add
.cfi_{remember,restore}_state support to perlasm.

Next, fill in a bunch of functions that are missing no-op .cfi_startproc
and .cfi_endproc blocks. libunwind cannot unwind those stack frames
otherwise.

Finally, work around a bug in libunwind by not encoding rflags. (rflags
isn't a callee-saved register, so there's not much need to annotate it
anyway.)

These were found as part of ABI testing work in BoringSSL.

Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #8109

[test] unit test for field_inv function pointer in EC_METHOD

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/8254)

SCA hardening for mod. field inversion in EC_GROUP

This commit adds a dedicated function in `EC_METHOD` to access a modular
field inversion implementation suitable for the specifics of the
implemented curve, featuring SCA countermeasures.

The new pointer is defined as:
`int (*field_inv)(const EC_GROUP*, BIGNUM *r, const BIGNUM *a, BN_CTX*)`
and computes the multiplicative inverse of `a` in the underlying field,
storing the result in `r`.

Three implementations are included, each including specific SCA
countermeasures:
  - `ec_GFp_simple_field_inv()`, featuring SCA hardening through
    blinding.
  - `ec_GFp_mont_field_inv()`, featuring SCA hardening through Fermat's
    Little Theorem (FLT) inversion.
  - `ec_GF2m_simple_field_inv()`, that uses `BN_GF2m_mod_inv()` which
    already features SCA hardening through blinding.

From a security point of view, this also helps addressing a leakage
previously affecting conversions from projective to affine coordinates.

This commit also adds a new error reason code (i.e.,
`EC_R_CANNOT_INVERT`) to improve consistency between the three
implementations as all of them could fail for the same reason but
through different code paths resulting in inconsistent error stack
states.

Co-authored-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/8254)

ARM64 assembly pack: make it Windows-friendly.

"Windows friendliness" means a) unified PIC-ification, unified across
all platforms; b) unified commantary delimiter; c) explicit ldur/stur,
as Visual Studio assembler can't automatically encode ldr/str as
ldur/stur when needed.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8256)

ARM assembly pack: make it Windows-friendly.

"Windows friendliness" means a) flipping .thumb and .text directives,
b) always generate Thumb-2 code when asked(*); c) Windows-specific
references to external OPENSSL_armcap_P.

(*) so far *some* modules were compiled as .code 32 even if Thumb-2
was targeted. It works at hardware level because processor can alternate
between the modes with no overhead. But clang --target=arm-windows's
builtin assembler just refuses to compile .code 32...

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8252)

s390x assembly pack: fix formal interface bug in chacha module

Signed-off-by: Patrick Steuer <patrick.steuer@de.ibm.com>
Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8257)

Add an OpenSSL library context

The context builds on CRYPTO_EX_DATA, allowing it to be dynamically
extended with new data from the different parts of libcrypto.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8225)

Add CRYPTO_alloc_ex_data()

This allows allocation of items at indexes that were created after the
CRYPTO_EX_DATA variable was initialized, using the exact same method
that was used then.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8225)

Add  EC_GROUP_get0_field

New function to return internal pointer for field.

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8195)

Mark generated functions unused (applies to safestack, lhash, sparse_array)

safestack.h, lhash.h and sparse_array.h all define macros to generate
a full API for the containers as static inline functions.  This
potentially generates unused code, which some compilers may complain
about.

We therefore need to mark those generated functions as unused, so the
compiler knows that we know, and stops complaining about it.

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/8246)

Add option to disable Extended Master Secret

Add SSL_OP64_NO_EXTENDED_MASTER_SECRET, that can be set on either
an SSL or an SSL_CTX. When processing a ClientHello, if this flag
is set, do not indicate that the EMS TLS extension was received in
either the ssl3 object or the SSL_SESSION.  Retain most of the
sanity checks between the previous and current session during
session resumption, but weaken the check when the current SSL
object is configured to not use EMS.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3910)

Use order not degree to calculate a buffer size in ecdsatest

Otherwise this can result in an incorrect calculation of the maximum
encoded integer length, meaning an insufficient buffer size is allocated.

Thanks to Billy Brumley for helping to track this down.

Fixes #8209

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8237)

Fix no-stdio

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8238)

Configure: make --strict-warnings a regular user provided compiler option

This makes `--strict-warnings` into a compiler pseudo-option, i.e. it
gets treated the same way as any other compiler option given on the
configuration command line, but is retroactively replaced by actual
compiler warning options, depending on what compiler is used.

This makes it easier to see in what order options are given to the
compiler from the configuration command line, i.e. this:

    ./config -Wall --strict-warnings

would give the compiler flags in the same order as they're given,
i.e.:

    -Wall -Werror -Wno-whatever ...

instead of what we got previously:

    -Werror -Wno-whatever ... -Wall

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8239)

Fix -verify_return_error in s_client

The "verify_return_error" option in s_client is documented as:

 Return verification errors instead of continuing. This will typically
 abort the handshake with a fatal error.

In practice this option was ignored unless also accompanied with the
"-verify" option. It's unclear what the original intention was. One fix
could have been to change the documentation to match the actual behaviour.
However it seems unecessarily complex and unexpected that you should need
to have both options. Instead the fix implemented here is make the option
match the documentation so that "-verify" is not also required.

Note that s_server has a similar option where "-verify" (or "-Verify") is
still required. This makes more sense because those options additionally
request a certificate from the client. Without a certificate there is no
possibility of a verification failing, and so "-verify_return_error" doing
nothing seems ok.

Fixes #8079

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/8080)

Don't signal SSL_CB_HANDSHAKE_START for TLSv1.3 post-handshake messages

The original 1.1.1 design was to use SSL_CB_HANDSHAKE_START and
SSL_CB_HANDSHAKE_DONE to signal start/end of a post-handshake message
exchange in TLSv1.3. Unfortunately experience has shown that this confuses
some applications who mistake it for a TLSv1.2 renegotiation. This means
that KeyUpdate messages are not handled properly.

This commit removes the use of SSL_CB_HANDSHAKE_START and
SSL_CB_HANDSHAKE_DONE to signal the start/end of a post-handshake
message exchange. Individual post-handshake messages are still signalled in
the normal way.

This is a potentially breaking change if there are any applications already
written that expect to see these TLSv1.3 events. However, without it,
KeyUpdate is not currently usable for many applications.

Fixes #8069

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8096)

Ignore cipher suites when setting cipher list

set_cipher_list() sets TLSv1.2 (and below) ciphers, and its success or
failure should not depend on whether set_ciphersuites() has been used to
setup TLSv1.3 ciphers.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Ben Kaduk <kaduk@mit.edu>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7759)

Configure: stop forcing use of DEFINE macros in headers

There are times when one might want to use something like
DEFINE_STACK_OF in a .c file, because it defines a stack for a type
defined in that .c file.  Unfortunately, when configuring with
`--strict-warnings`, clang aggressively warn about unused functions in
such cases, which forces the use of such DEFINE macros to header
files.

We therefore disable this warning from the `--strict-warnings`
definition for clang.

(note for the curious: `-Wunused-function` is enabled via `-Wall`)

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8234)

Sparse array iterators include index position.

Iterators over the sparse array structures have gained an initial argument
which indicates the index into the array of the element.  This can be used,
e.g., to delete or modify the associated value.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8229)

Windows/Cygwin dlls need the executable bit set

CLA: trivial

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8226)

test/build.info: add missing ../apps/include

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/8227)

Added new EVP/KDF API.
Changed PKEY/KDF API to call the new API.
Added wrappers for PKCS5_PBKDF2_HMAC() and EVP_PBE_scrypt() to call the new EVP KDF APIs.
Documentation updated.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6674)

Sparse array limit testing: reduce the range limit for the number of bits
in a sparse array pointer block.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8221)

Fix null pointer dereference in cms_RecipientInfo_kari_init

CLA: trivial

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8137)

Move libapps headers into their own directory

This got triggered by test/testutil.h including ../apps/opt.h.

Some compilers do all inclusions from the directory of the C file
being compiled, so when a C file includes a header file with a
relative file spec, and that header file also includes another header
file with a relative file spec, the compiler no longer follows.

As a specific example, test/testutil/basic_output.c included
../testutil.h.  Fine so far, but then, test/testutil.h includes
../apps/opt.h, and the compiler ends up trying to include (seen from
the source top) test/apps/opt.h rather than apps/opt.h, and fails.

The solution could have been to simply add apps/ as an inclusion
directory.  However, that directory also has header files that have
nothing to do with libapps, so we take this a bit further, create
apps/include and move libapps specific headers there, and then add
apps/include as inclusion directory in the build.info files where
needed.

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
(Merged from https://github.com/openssl/openssl/pull/8210)

Fix master build.
The recent change from ENGINES to MODULES broke the configure it seems.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8219)

Macro typo fix not propagated

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8217)

Fix typo in comment

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
(Merged from https://github.com/openssl/openssl/pull/8218)

To use BN_BITS2, we'd better include openssl/bn.h

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8212)

eng_devcrypto.c: close open session on init

cipher_init may be called on an already initialized context, without a
necessary cleanup.  This separates cleanup from initialization, closing
an eventual open session before creating a new one.

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7859)

CHANGES: add note about building devcrypto dynamic

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7859)

e_devcrypto: make the /dev/crypto engine dynamic

Engine has been moved from crypto/engine/eng_devcrypto.c to
engines/e_devcrypto.c.

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7859)

AArch64 assembly pack: authenticate return addresses.

ARMv8.3 adds pointer authentication extension, which in this case allows
to ensure that, when offloaded to stack, return address is same at return
as at entry to the subroutine. The new instructions are nops on processors
that don't implement the extension, so that the vetification is backward
compatible.

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8205)

Add sparse array data type.

This commit adds a space and time efficient sparse array data structure.
The structure's raw API is wrapped by inline functions which provide type
safety.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/8197)

Rework build: small correction in unix-Makefile.tmpl

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8162)

Rework build: add special cases for AIX

When reworking the way library file names and extensions were formed,
AIX was lost in the process.  This restores the previous
functionality.

Fixes #8156

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8162)

apps/ocsp.c Use the same HAVE_FORK / NO_FORK as in speed.c

This allows the user to override our defaults if needed, and in a
consistent manner.

Partial fix for #7607

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7624)

test/recipes/02-err_errstr: skip errors that may not be loaded on Windows

Fixes #8091

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8094)

(cherry picked from commit 0e1b0e510dfe078b3fb2586d987d7b49ff8ef0b2)

Build: correct BASE shlib_version_as_filename

This function is designed to use $config{shlib_version} directly
instead of taking an input argument, yet the BASE variant didn't do
this.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8146)

ENGINE modules aren't special, so call them MODULES

The only thing that makes an ENGINE module special is its entry
points.  Other than that, it's a normal dynamically loadable module,
nothing special about it.  This change has us stop pretending anything
else.

We retain using ENGINE as a term for installation, because it's
related to a specific installation directory, and we therefore also
mark ENGINE modules specifically as such with an attribute in the
build.info files.

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/8147)

Updated test command line parsing to support commmon commands

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6975)

Allow the syntax of the .include directive to optionally have '='

If the old openssl versions not supporting the .include directive
load a config file with it, they will bail out with error.

This change allows using the .include = <filename> syntax which
is interpreted as variable assignment by the old openssl
config file parser.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8141)

Fix comment typo

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8196)

Fix null pointer dereference in ssl_module_init

CLA: Trivial

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8183)

Update d2i_PrivateKey documentation

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8168)

Fix d2i_PublicKey() for EC keys

o2i_ECPublicKey() requires an EC_KEY structure filled with an EC_GROUP.

o2i_ECPublicKey() is called by d2i_PublicKey(). In order to fulfill the
o2i_ECPublicKey()'s requirement, d2i_PublicKey() needs to be called with
an EVP_PKEY with an EC_KEY containing an EC_GROUP.

However, the call to EVP_PKEY_set_type() frees any existing key structure
inside the EVP_PKEY, thus freeing the EC_KEY with the EC_GROUP that
o2i_ECPublicKey() needs.

This means you can't d2i_PublicKey() for an EC key...

The fix is to check to see if the type is already set appropriately, and
if so, not call EVP_PKEY_set_type().

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8168)

Address a bug in the DRBG tests where the reseeding wasn't properly
reinstantiating the DRBG.

Bug reported by Doug Gibbons.

Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
(Merged from https://github.com/openssl/openssl/pull/8184)

test/drbgtest.c: call OPENSSL_thread_stop() explicitly

The manual says this in its notes:

    ... and therefore applications using static linking should also call
    OPENSSL_thread_stop() on each thread. ...

Fixes #8171

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/8173)

Make OPENSSL_malloc_init() a no-op

Making this a no-op removes a potential infinite loop than can occur in
some situations.

Fixes #2865

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8167)

Add CHANGES entry for blake2mac

Signed-off-by: Antoine Salon <asalon@vmware.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8172)

blake2: avoid writing to output buffer when using default digest length

Signed-off-by: Antoine Salon <asalon@vmware.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7726)

blake2: add evpmac test vectors

Signed-off-by: Antoine Salon <asalon@vmware.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7726)

blake2: backport changes to blake2s

Signed-off-by: Antoine Salon <asalon@vmware.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7726)

blake2: add EVP_MAC man page

Signed-off-by: Antoine Salon <asalon@vmware.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7726)

blake2: register MAC objects

Signed-off-by: Antoine Salon <asalon@vmware.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7726)

blake2b: add EVP_MAC API

Signed-off-by: Antoine Salon <asalon@vmware.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7726)

blake2b: add support for parameter setting and keyed hash

The param block structure is used as a container for parameter values
Added blake2b keyed init

Signed-off-by: Antoine Salon <asalon@vmware.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7726)

blake2: add implementation support for variable digest length

Signed-off-by: Antoine Salon <asalon@vmware.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7726)

Remove unnecessary trailing whitespace

Trim trailing whitespace. It doesn't match OpenSSL coding standards,
AFAICT, and it can cause problems with git tooling.

Trailing whitespace remains in test data and external source.

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8092)

crypto/poly1305/asm/poly1305-s390x.pl: add vx code path.

Signed-off-by: Patrick Steuer <patrick.steuer@de.ibm.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7991)

Make some simple getters take const SSL/SSL_CTX

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8145)

Fix Invalid Argument return code from IP_Factory in connect_to_server().

Fixes #7732

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8158)

Android build: fix usage of NDK home variable ($ndk_var)

CLA: trivial

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8153)

Build: correct assembler generation in crypto/rc4/build.info

In the removal of BEGINRAW / ENDRAW, attention to the difference
between capital .S and lowercase .s wasn't duly paid.  This corrects
the error.

Fixes #8155

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8157)