modes/ocb128.c: Reset nonce-dependent variables on setiv

Upon a call to CRYPTO_ocb128_setiv, either directly on an OCB_CTX or
indirectly with EVP_CTRL_AEAD_SET_IVLEN, reset the nonce-dependent
variables in the OCB_CTX.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/6420)

chacha/asm/chacha-ppc.pl: fix big-endian build.

It's kind of a "brown-bag" bug, as I did recognize the problem and
verified an ad-hoc solution, but failed to follow up with cross-checks
prior filing previous merge request.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6435)

Add a test for the raw private/public key getters

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6394)

Add function for setting the EVP_PKEY_ASN1_METHOD raw key getter functions

EVP_PKEY_asn1_set_get_priv_key() and EVP_PKEY_asn1_set_get_pub_key()

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6394)

Document the raw key getter functions

EVP_PKEY_get_raw_private_key() and EVP_PKEY_get_raw_public_key()

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6394)

Add support getting raw private/public keys

Only applies to algorithms that support it. Both raw private and public
keys can be obtained for X25519, Ed25519, X448, Ed448. Raw private keys
only can be obtained for HMAC, Poly1305 and SipHash

Fixes #6259

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6394)

Do not free a session before calling the remove_session_cb

If the remove_session_cb accesses the session's data (for instance,
via SSL_SESSION_get_protocol_version), a potential use after free
can occur. For this, consider the following scenario when adding
a new session via SSL_CTX_add_session:

- The session cache is full
  (SSL_CTX_sess_number(ctx) > SSL_CTX_sess_get_cache_size(ctx))
- Only the session cache has a reference to ctx->session_cache_tail
  (that is, ctx->session_cache_tail->references == 1)

Since the cache is full, remove_session_lock is called to remove
ctx->session_cache_tail from the cache. That is, it
SSL_SESSION_free()s the session, which free()s the data. Afterwards,
the free()d session is passed to the remove_session_cb. If the callback
accesses the session's data, we have a use after free.

The free before calling the callback behavior was introduced in
commit e4612d02c53cccd24fa97b08fc01250d1238cca1 ("Remove sessions
from external cache, even if internal cache not used.").

CLA: trivial

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6222)

Document UTF-8 expectation for pass phrases passed to OSSL_STORE

After some discussion, it was concluded that the better idea is to
stipulate that the pass phrases passed to the OSSL_STORE API are
expected to be UTF-8 encoded, and that all objects made accessible
through OSSL_STORE URIs should adhere to this expectation (at the
discretion of the loaders).

Email ref:
https://mta.openssl.org/pipermail/openssl-project/2018-June/000771.html

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6416)

Reformulate the if condition in tls_process_new_session_ticket

Improves readability

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6415)

Don't store the ticket nonce in the session

We generate the secrets based on the nonce immediately so there is no
need to keep the nonce.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6415)

Fix TLSv1.3 ticket nonces

All tickets on a connection need to have a unique nonce. When this was
originally implemented we only ever sent one ticket on the conneciton so
this didn't matter. We were just using the value 0. Now we can get multiple
tickets to we need to start doing the ticket nonce properly.

Fixes #6387

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6415)

Fix EAP-FAST

Commit 61fb59238d broke EAP-FAST support. This fixes it.

Fixes #6395

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6428)

Fix no-dsa

Broken by 0336df2fa.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6427)

crypto/ppccap.c: wire new ChaCha20_ctr32_vsx.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6419)

chacha/asm/chacha-ppc.pl: improve performance by 40/80% on POWER8/9.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6419)

perlasm/ppc-xlate.pl: add vmrg[eo]w instructions.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6419)

sha/asm/sha512p8-ppc.pl: fix build on Mac OS X.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6419)

Issue warnings for large DSA and RSA keys

Issue a warning when generating DSA or RSA keys of size greater than
OPENSSL_DSA_MAX_MODULUS_BITS resp. OPENSSL_RSA_MAX_MODULUS_BITS.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/6380)

Improve wording

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6413)

crypto/sm2/sm2_za.c: include internal/numbers.h

Needed for the platforms that don't define UINT16_MAX.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6414)

Rebuild error codes

There were a large number of error codes that were unused (probably a
copy&paste from somewhere else). Since these have never been made public
we should remove then and rebuild the error codes.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6386)

Use lowercase for internal SM2 symbols

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6386)

Remove non-existant functions from libcrypto.num

These were never made public so can just be deleted.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6386)

More more on SM2 error codes and tidy up

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6386)

Improve use of the test framework in the SM2 internal tests

Also general clean up of those tests

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6386)

Further work on SM2 error codes

Also does some reformatting to tidy things up

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6386)

Add test recipes for internal SM2 tests

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6386)

Set SM2 error codes

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6386)

Make SM2 functions private

Address issue #5670

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6386)

sha/asm/sha512p8-ppc.pl: improve POWER9 performance by ~10%.

Biggest part, ~7%, of improvement resulted from omitting constants'
table index increment in each round. And minor part from rescheduling
instructions. Apparently POWER9 (and POWER8) manage to dispatch
instructions more efficiently if they are laid down as if they have
no latency...

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6406)

chacha/asm/chacha-ppc.pl: improve POWER8 performance by 15%.

This comes at cost of minor 2.5% regression on G4, which is reasonable
trade-off. [Further improve compliance with ABI requirements.]

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6406)

PPC assembly pack: correct POWER9 results.

As it turns out originally published results were skewed by "turbo"
mode. VM apparently remains oblivious to dynamic frequency scaling,
and reports that processor operates at "base" frequency at all times.
While actual frequency gets increased under load.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6406)

{arm64|x86_64}cpuid.pl: add special 16-byte case to OPENSSL_memcmp.

OPENSSL_memcmp is a must in GCM decrypt and general-purpose loop takes
quite a portion of execution time for short inputs, more than GHASH for
few-byte inputs according to profiler. Special 16-byte case takes it off
top five list in profiler output.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6312)

modes/gcm128.c: coalesce calls to GHASH.

On contemporary platforms assembly GHASH processes multiple blocks
faster than one by one. For TLS payloads shorter than 16 bytes, e.g.
alerts, it's possible to reduce hashing operation to single call.
And for block lengths not divisible by 16 - fold two final calls to
one. Improvement is most noticeable with "reptoline", because call to
assembly GHASH is indirect.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6312)

evp/e_aes.c: replace calls to one-liners with references in GCM.

Even though calls can be viewed as styling improvement, they do come
with cost. It's not big cost and shows only on short inputs, but it is
measurable, 2-3% on some platforms.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6312)

Fix checking the return value of getentropy()

Reviewed-by: Andy Polyakov <appro@openssl.org>
GH: #6405

Remove support for calling getrandom(), we now always call getentropy()

Only Linux and FreeBSD provide getrandom(), but they both also provide
getentropy() since the same version and we already tried to call that.

Reviewed-by: Andy Polyakov <appro@openssl.org>
GH: #6405

Look up availability of getentropy() at runtime.

This will actually support most OSs, and at least adds support for
Solaris and OSX

Fixes: #6403
Reviewed-by: Andy Polyakov <appro@openssl.org>
GH: #6405

Add support for KERN_ARND to get random bytes on NetBSD

Reviewed-by: Andy Polyakov <appro@openssl.org>
GH: #6405

Document failure return for ECDSA_SIG_new

ECDSA_SIG_new() returns NULL on error.

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6398)

Make OS/X more explicit, to avoid questions

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6404)

STORE: split off the description of the 'file' scheme loader

This includes a quick recommendation on how to name loader docmentation.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6350)

ENGINE_pkey_asn1_find_str(): don't assume an engine implements ASN1 method

Just because an engine implements algorithm methods, that doesn't mean
it also implements the ASN1 method.  Therefore, be careful when looking
for an ASN1 method among all engines, don't try to use one that doesn't
exist.

Fixes #6381

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6383)

(cherry picked from commit 1ac3cd6277f880fac4df313702d5e3b3814e56e2)
(cherry picked from commit 13b578ada3106a4ce9b836f167ee520539f5fa8f)

Don't call setsockopt with an invalid fd

This is probably a "should not happen" scenario, but better check anyway.
Found by Coverity.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6373)

Fix a memory leak in an error path

Found by Coverity

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6373)

Free a variable

Fix a memory leak in storeutl.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6373)

Free a variable on an error path

Issue found by Coverity.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6373)

Only set TCP_NODELAY if the protocol is TCP

This doesn't apply if we're doing DTLS, or using UNIX domain sockets.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6373)

EVP_MD_size() can return an error

Fix some instances where we weren't checking the error return.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6373)

Remove some logically dead code

Found by coverity. This is an artifact left over from the original
decaf import which generated the source code for different curves. For
curve 448 this is dead.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6373)

Remove some logically dead code

Issues found by Coverity

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6373)

apps: when the 'compat' nameopt has been set, leave it be

XN_FLAG_COMPAT has a unique property, its zero for value.  This means
it needs special treatment; if it has been set (which can only be
determined indirectly) and set alone (*), no other flags should be
set.

(*) if any other nameopt flag has been set by the user, compatibility
mode is blown away.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6382)

Reduce minimal out length in CRYPTO_128_unwrap_pad

In `aes_wrap_cipher()`, the minimal out buff length is `(inlen - 8)`.
Since it calls `CRYPTO_128_unwrap_pad()` underneath, it makes sense to
reduce the minimal out length in `CRYPTO_128_unwrap_pad()` to align to
its caller.

Signed-off-by: Yihong Wang <yh.wang@ibm.com>
Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6266)

bn/bn_exp.c: mitigation of the One-and-Done side-channel attack.

The One&Done attack, which is described in a paper to appear in the
USENIX Security'18 conference, uses EM emanations to recover the values
of the bits that are obtained using BN_is_bit_set while constructing
the value of the window in BN_mod_exp_consttime. The EM signal changes
slightly depending on the value of the bit, and since the lookup of a
bit is surrounded by highly regular execution (constant-time Montgomery
multiplications) the attack is able to isolate the (very brief) part of
the signal that changes depending on the bit. Although the change is
slight, the attack recovers it successfully >90% of the time on several
phones and IoT devices (all with ARM processors with clock rates around
1GHz), so after only one RSA decryption more than 90% of the bits in
d_p and d_q are recovered correctly, which enables rapid recovery of
the full RSA key using an algorithm (also described in the paper) that
modifies the branch-and-prune approach for a situation in which the
exponents' bits are recovered with errors, i.e. where we do not know
a priori which bits are correctly recovered.

The mitigation for the attack is relatively simple - all the bits of
the window are obtained at once, along with other bits so that an
entire integer's worth of bits are obtained together using masking and
shifts, without unnecessarily considering each bit in isolation. This
improves performance somewhat (one call to bn_get_bits is faster than
several calls to BN_is_bit_set), so the attacker now gets one signal
snippet per window (rather than one per bit) in which the signal is
affected by all bits in the integer (rather than just the one bit).

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6276)

chacha/asm/chacha-ppc.pl: optimize AltiVec/VMX code path.

32-bit vector rotate instruction was defined from beginning, it
not being used from the start must be a brain-slip...

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6363)

perlasm/ppc-xlate.pl: add new instructions and clean up.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6363)

apps/speed.c: address asan failure with small -bytes and large -misalign.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6311)

apps/speed.c: add -aead flag.

Goal is to exercise AEAD ciphers in TLS-like sequence, i.e. 13-byte
AAD followed by payload. Update doc/man1/speed.pod accordingly.

[While we are at it, address even some styling and readability issues.]

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6311)

apps/speed.c: clean up SIGARM handling.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6311)

OSSL_STORE: don't test file: URIs on Mingw

Under a mingw shell, the command line path conversion either mangles
file: URIs to something useless (file;C:\...) or not at all (which
can't be opened by the Windows C RTL unless we're really lucky), so we
simply skip testing them in that environment.

Fixes #6369

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6376)

Add APIs for custom X509_LOOKUP_METHOD creation

OpenSSL 1.1.0 made the X509_LOOKUP_METHOD structure opaque, so
applications that were previously able to define a custom lookup method
are not able to be ported.

This commit adds getters and setters for each of the current fields of
X509_LOOKUP_METHOD, along with getters and setters on several associated
opaque types (such as X509_LOOKUP and X509_OBJECT).

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6152)

Text description of GOST signature algorithms

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6168)

Send GOST SignatureAlgorithms when TLS 1.2 in use

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6168)

Bugfix: GOST2012 certificates for GOST ciphersuites were broken.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6168)

Replace strdup() with OPENSSL_strdup()

It's freed with OPENSSL_free()

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6331)

Fix no-sm2 with clang

Return immediately upon discovery of bad message digest.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6298)

Test that a ^ 0 mod -1 is always 0

Check all functions that do this.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6355)

The result of a ^ 0 mod -1 is 0 not 1

Thanks to Guido Vranken and OSSFuzz for finding this issue.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6355)

Update the "Connected Commands" section of s_client/s_server docs

Fixes #6307

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/6330)

Prepare for 1.1.1-pre8-dev

Reviewed-by: Richard Levitte <levitte@openssl.org>

Prepare for 1.1.1-pre7 release

Reviewed-by: Richard Levitte <levitte@openssl.org>

Update copyright year

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6371)

ECDSA_SIG: restore doc comments which were deleted accidentally

amends 0396401d1c3f

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6366)

ECDSA_SIG: add simple getters for commonly used struct members

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6290)

RSA: add simple getters for commonly used struct members

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6290)

DSA: add simple getters for commonly used struct members

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6290)

DH: fix: add simple getters for commonly used struct members

amends 6db7fadf0975

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6290)

util/libcrypto.num: fix symbol collision between 1.1.0 and master

In commit 6decf9436f7, fourteen public symbols were removed from
util/libcrypto.num on the master branch and the following symbols
renumbered. Unfortunately, the symbols `OCSP_resp_get0_signer` and
`X509_get0_authority_key_id` were not adjusted accordingly on the
OpenSSL_1_1_0-stable branch. This commit fixes the collision by
doing a 'double swap'.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6278)

Try to work around ubuntu gcc-5 ubsan build failure

[extended tests]

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6360)

Improve the example getpass() implementation to show an error return

Also, modernize the code, so that it isn't trying to store a size_t
into an int, and then check the int's sign. :/

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6271)

Update documentation for PEM callback: error is now -1.

In previous versions of OpenSSL, the documentation for PEM_read_*
said:

   The callback B<must> return the number of characters in the
   passphrase or 0 if an error occurred.

But since c82c3462267afdbbaa5, 0 is now treated as a non-error
return value.  Applications that want to indicate an error need to
return -1 instead.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6271)

Fix array bounds violation in ssl_session_dup

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6353)

Add a sanity check on the length of pkeyutl inputs

When signing or verifying a file using pkeyutl the input is supposed to
be a hash. Some algorithms sanity check the length of the input, while
others don't and silently truncate. To avoid accidents we check that the
length of the input looks sane.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6284)

Revert "Support EVP_PKEY_sign() and EVP_PKEY_verify() for EdDSA"

This reverts commit a6f5b1163452a36da1c8378eab3f29ae234e5c7f.

The EVP_PKEY_sign() function is intended for pre-hashed input which is
not supported by our EdDSA implementation.

See the discussion in PR 5880

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6284)

Improve compatibility of point and curve checks

We check that the curve name associated with the point is the same as that
for the curve.

Fixes #6302

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6323)

Save and restore the Windows error around TlsGetValue.

TlsGetValue clears the last error even on success, so that callers may
distinguish it successfully returning NULL or failing. This error-mangling
behavior interferes with the caller's use of GetLastError. In particular
SSL_get_error queries the error queue to determine whether the caller should
look at the OS's errors. To avoid destroying state, save and restore the
Windows error.

Fixes #6299.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6316)

Use OPENSSL_EC_EXPLICIT_CURVE constant.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6314)

Fix explicit EC curve encoding.

Per SEC 1, the curve coefficients must be padded up to size. See C.2's
definition of Curve, C.1's definition of FieldElement, and 2.3.5's definition
of how to encode the field elements in http://www.secg.org/sec1-v2.pdf.

This comes up for P-521, where b needs a leading zero.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6314)

Skip CN DNS name constraint checks when not needed

Only check the CN against DNS name contraints if the
`X509_CHECK_FLAG_NEVER_CHECK_SUBJECT` flag is not set, and either the
certificate has no DNS subject alternative names or the
`X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT` flag is set.

Add pertinent documentation, and touch up some stale text about
name checks and DANE.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>

Limit scope of CN name constraints

Don't apply DNS name constraints to the subject CN when there's a
least one DNS-ID subjectAlternativeName.

Don't apply DNS name constraints to subject CN's that are sufficiently
unlike DNS names.  Checked name must have at least two labels, with
all labels non-empty, no trailing '.' and all hyphens must be
internal in each label.  In addition to the usual LDH characters,
we also allow "_", since some sites use these for hostnames despite
all the standards.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>

Use the client app traffic secret for PHA Finished message

The TLSv1.3 spec requires us to use the client application traffic secret
during generation of the Finished message following a post handshake
authentication.

Fixes #6263

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/6297)

INSTALL: Provide better documentation for enable-ec_nistp_64_gcc_128

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6328)

Better error code when lacking __SIZEOF_INT128__

Fixes #6327

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6328)

Fix no-ec, no-tls1_3 and no-tls

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6321)

Enable SSL_MODE_AUTO_RETRY by default

Because TLS 1.3 sends more non-application data records some clients run
into problems because they don't expect SSL_read() to return and set
SSL_ERROR_WANT_READ after processing it.

This can cause problems for clients that use blocking I/O and use
select() to see if data is available. It can be cleared using
SSL_CTX_clear_mode().

Reviewed-by: Matt Caswell <matt@openssl.org>
GH: #6260

Fix no-ec in combination with no-dh

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6321)

ssl/ssl_txt: fix NULL-check

NULL-check for cipher is redundant, instead check if cipher->name is NULL

While here fix formatting of BIO_printf calls as suggested by Andy Polyakov.

CLA: trivial

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6282)

apps/s_socket.c: address rare TLSProxy failures on Windows.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Check malloc failure via app_malloc

Thanks to GitHUb user murugesandins for reporting this.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6322)

Don't cache stateless tickets in TLSv1.3

In TLSv1.2 and below we always cache new sessions by default on the server
side in the internal cache (even when we're using session tickets). This is
in order to support resumption from a session id.

In TLSv1.3 there is no session id. It is only possible to resume using the
ticket. Therefore, in the default case,  there is no point in caching the
session in the internal store.

There is still a reason to call the external cache new session callback
because applications may be using the callbacks just to know about when
sessions are created (and not necessarily implementing a full cache). If
the application also implements the remove session callback then we are
forced to also store it in the internal cache so that we can create
timeout events. Otherwise the external cache could just fill up
indefinitely.

This mostly addresses the issue described in #5628. That issue also proposes
having an option to not create full stateless tickets when using the
internal cache. That aspect hasn't been addressed yet.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/6293)