document RSA-PSS algorithm options

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

add PSS key tests

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

print errors in pkey utility

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

make errors

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

add parameter error

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Set EVP_PKEY_CTX in SignerInfo

If we aren't setting public key parameters make EVP_PKEY_CTX available
in SignerInfo so PSS mode and parameters are automatically selected.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Only allow PSS padding for PSS keys.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Decode parameters properly.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Return errors PKCS#7/CMS enveloped data ctrls and PSS

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Add PSS parameter restrictions.

If a key contains any PSS parameter restrictions set them during
sign or verification initialisation. Parameters now become the
default values for sign/verify. Digests are fixed and any attempt
to change them is an error. The salt length can be modified but
must not be less than the minimum value.

If the key parameters are invalid then verification or signing
initialisation returns an error.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Initial parameter restrictions.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Add rsa_pss_get_param.

New function rsa_pss_get_param to extract and sanity check PSS parameters.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Don't allow PKCS#7/CMS encrypt with PSS.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Add macros to determine if key or ctx is PSS.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Support pad mode get/set for PSS keys.

Pad mode setting returns an error if the mode is anything other then PSS.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Key gen param support.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Set PSS padding mode for PSS keys.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Digest string helper function.

New function EVP_PKEY_CTX_md() which takes a string and passes a digest
to a ctrl.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Support RSA operations in PSS.

Add support for common operations in PSS by adding a new function
RSA_pkey_ctx_ctrl() which calls EVP_PKEY_CTX_ctrl if the key type
is RSA or PSS.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

PSS EVP_PKEY method

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

RSA-PSS key printing.

Print out RSA-PSS key parameters if present.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

PSS parameter encode and decode.

For RSA PSS keys encode and decode parameters when handling public
and private keys.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Split PSS parameter creation.

Split PSS parameter creation. This adds a new function rsa_pss_params_create
which creates PSS parameters from digest and salt values. This will be
used for PSS key generation.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Use method key type instead of EVP_PKEY_RSA

Make RSA method more flexible by using the key type from the
method instead of hard coding EVP_PKEY_RSA: by doing this the
same code supports both RSA and RSA-PSS.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

PSS ASN.1 method

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

add EVP_PKEY_RSA_PSS

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Add pss field to RSA structure and free it.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Cache maskHash parameter

Store hash algorithm used for MGF1 masks in PSS and OAEP modes in PSS and
OAEP parameter structure: this avoids the need to decode part of the ASN.1
structure every time it is used.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Update fuzz corpora

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

Make client and server fuzzer reproducible

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

Make the bignum fuzzer reproducible

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

Update fuzz documentation

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

Make rand_add predictable when fuzzing

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

server fuzzer: add support for DSA and ECDSA

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

Fix various style issues following feedback

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Rename the chain variable to chainidx

This variable represents the index of the cert within the chain, so give it
a name that better represents that.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Fix a double blank line style issue

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Initialise the al variable

al can be used uninitialised in an error path.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Update SSL_trace to understand TLSv1.3 Certificates

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Implement TLSv1.3 style CertificateStatus

We remove the separate CertificateStatus message for TLSv1.3, and instead
send back the response in the appropriate Certificate message extension.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Create Certificate messages in TLS1.3 format

Also updates TLSProxy to be able to understand the format and parse the
contained extensions.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Extends extension parsing to take the Certificate

Continuing from the previous commit we also need to extend the extensions
framework to supply the Certificate we just read during parsing.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Extend tls_construct_extensions() to enable passing of a certificate

The Certificate message in TLS1.3 has an extensions block for each
Certificate. Therefore we need to extend tls_construct_extensions() to pass
in the certificate we are working on. We also pass in the position in the
chain (with 0 being the first certificate).

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Remove BIO_seek/BIO_tell from evp_test.c

BIO_seek and BIO_tell can cause problems with evp_test.c on some platforms.
Avoid them by using a temporary memory BIO to store key PEM data.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2183)

Don't run MSBLOB conversion tests when RSA or DSA are disabled

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2174)

Don't run OCSP tests when OCSP is disabled

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2173)

Don't build OCSP stuff when OCSP is disabled

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2173)

Don't test SRP when it's disabled

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2172)

Don't run NPN tests when NPN is disabled

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2171)

Add RSA decrypt and OAEP tests.

Reviewed-by: Rich Salz <rsalz@openssl.org>

evptests.txt is not a shell script

Reviewed-by: Rich Salz <rsalz@openssl.org>

Don't run the sigalgs tests over a TLSv1.3 connection

We need a new API for TLSv1.3 sig algs

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2160)

Provide some tests for the sig algs API

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2160)

Fix the SSL_set1_sigalgs() macro

This macro has a typo in it which makes it unusable. This issue was already
fixed in 1.0.2 in commit 75fdee04827, but the same fix was not applied to
other branches.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2160)

70-test_sslvertol.t: skip test 1 and 2 if too few protocols are enabled

These tests depend on there being at least one protocol version below
TLSv1.3 enabled.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2144)

80-test_ssl_new.t: Make 19-mac-then-encrypt.conf work without TLSv1.2

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2144)

70-test_sslvertol.t: Make sure to check a max TLS version that matches configuration

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2144)

70-test_sslmessages.t: Don't check EXT_SIG_ALGS if TLS 1.2 is disabled

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2144)

Fix CT test_sslmessages hangs

The CT tests in test_sslmessages require EC to be available, therefore
we must skip these if no-ec

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2153)

Fix compilation with no-ec

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2153)

Fix extension for various no- options

Previously we were omitting the extension information from ext_defs if
the association no- option was defined. This doesn't work because the
indexes into the table are no longer valid.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2153)

Add a test to check the EC point formats extension appears when we expect

The previous commit fixed a bug where the EC point formats extensions did
not appear in the ServerHello. This should have been caught by
70-test_sslmessages but that test never tries an EC ciphersuite. This
updates the test to do that.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2153)

Fix the EC point formats extension

This should be sent in the ServerHello if a EC based ciphersuite is
negotiated. The relevant flag to do this was missed off in the recent
extensions refactor.

Fixes GitHub Issue #2133

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2153)

replace "will lookup up" by "will look up"

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
CLA: trivial
(Merged from https://github.com/openssl/openssl/pull/2145)

chacha/asm/chacha-x86_64.pl: add AVX512 path optimized for shorter inputs.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix EVP_MD_meth_get_flags

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2134)

Travis: The TLS 1.3 code isn't interoperable yet, move it to its own build

We should move it back to the BORINGTEST build when we are approaching
interoperability.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2123)

Reformat M_check_autoarg to match our coding style

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2121)

M_check_autoarg: sanity check the key

For now, checking that the size is non-zero will suffice.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2120)

Add bwrite_conv and bread_conv values to methods_dgramp_sctp

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2116)

Fix erroneous goto lable

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2116)

x86 assembly pack: update performance results.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Update fuzz corpora

Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #2090

Fix memory leak in tls_parse_stoc_key_share

Found by oss-fuzz

Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #2102

Fix typo.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
CLA: trivial
(Merged from https://github.com/openssl/openssl/pull/2086)

test/ssl_test: give up if both client and server wait on read

In some cases, both client and server end of the test can end up in
SSL_ERROR_WANT_READ and never get out of it, making the test spin.
Detect it and give up instead of waiting endlessly.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2096)

Fix no-ct, skip tests recipes that try to test CT

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2096)

e_afalg: Don't warn about kernel version when pedantic

When built with --strict-warnings and the Linux kernel headers don't
match the kernel version, the preprocessor warnings in
engines/afalg/e_afalg.c cause compilation errors.  Use the macro
PEDANTIC to avoid those warnings in that case.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2095)

evp_test: when function and reason strings aren't available, just skip

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2093)

HP-UX doesn't have hstrerror(), so make our own for that platform

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2092)
(cherry picked from commit 46766d003666da5f90346da7e6d09e109355f5c6)

Make client and server fuzzer support all ciphers

Also send a SNI extension in the client so the fuzzer can react to it.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2088

Document the recommended parameters for fuzzing

We use those parameters for calculating the coverage.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2088

Enable TLS1.3 and PEDANTIC in the coverage target

This make sure that the coverage is the same for the fuzzers and this
coverage target

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2088

Don't call memcpy with NULL as source

Calling it with lenght 0 and NULL as source is undefined behaviour.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2089

poly1305/asm/poly1305-x86_64.pl: allow nasm to assemble AVX512 code.
chacha/asm/chacha-x86_64.pl: refine nasm version detection logic.

Reviewed-by: Richard Levitte <levitte@openssl.org>

perlasm/x86_64-xlate.pl: add support for AVX512 OPMASK-ing.

Reviewed-by: Richard Levitte <levitte@openssl.org>

man3/OPENSSL_ia32cap.pod: clarify AVX512 support in clang context.

Reviewed-by: Richard Levitte <levitte@openssl.org>

CRL critical extension bugfix

More importantly, port CRL test from boringSSL crypto/x509/x509_test.cc

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1775)

Add function and reason checking to evp_test

Add options to check the function and reason code matches expected values.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Add X509_VERIFY_PARAM inheritance flag set/get

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2079)

Fix various doc nits.

find-doc-nits warns if you don't give a "what to do flag"
Don't use regexps for section names, just strings:  More consistency.
Rename "COMMAND OPTIONS" to OPTIONS.
Fix a couple of other nit-level things.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2076)

Remove ENGINE_load_dasync() (no OPENSSL_INIT_ENGINE_DASYNC already)

Fixes: 8d00e30f96fb86b20bc992f626b188c3548fc58c ("Don't try to init
dasync internally")

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
CLA: trivial

Typo fixed

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2075)

updated macro spacing for styling purposes

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
CLA: trivial

fix undoes errors introduced by https://github.com/openssl/openssl/commit/fc6076ca272f74eb1364c29e6974ad5da5ef9777?diff=split#diff-1014acebaa2c13d44ca196b9a433ef2eR184

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
CLA: trivial

Restore the ERR_FATAL_ERROR() macro

Commit 0cd0a820abc6124cf8e176fa92d620a2abf9e419 removed this macro
along with many unused function and reason codes; ERR_FATAL_ERROR()
was not used in the tree, but did have external consumers.

Add it back to restore the API compatibility and avoid breaking
applications for no internal benefit.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2049)

Fix a leak in SSL_clear()

SSL_clear() was resetting numwpipes to 0, but not freeing any allocated
memory for existing write buffers.

Fixes #2026

Reviewed-by: Rich Salz <rsalz@openssl.org>

perlasm/x86_64-xlate.pl: refine sign extension in ea package.

$1<<32>>32 worked fine with either 32- or 64-bit perl for a good while,
relying on quirk that [pure] 32-bit perl performed it as $1<<0>>0. But
this apparently changed in some version past minimally required 5.10,
and operation result became 0. Yet, it went unnoticed for another while,
because most perl package providers configure their packages with
-Duse64bitint option.

Reviewed-by: Rich Salz <rsalz@openssl.org>

x86_64 assembly pack: add AVX512 ChaCha20 and Poly1305 code paths.

Reviewed-by: Rich Salz <rsalz@openssl.org>

poly1305/poly1305_base2_44.c: add reference base 2^44 implementation.

Reviewed-by: Rich Salz <rsalz@openssl.org>