Fix a memory leak in RSA_padding_add_PKCS1_OAEP_mgf1

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Matt Caswell <matt@openssl.org>
GH: #2140

Rename "verify_cb" to SSL_verify_cb

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2151)

Doc nits: callback function typedefs

Enhance find-doc-nits to be better about finding typedefs for
callback functions.  Fix all nits it now finds.  Added some new
typedef names to ssl.h some of which were documented but did not
exist

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2151)

Add server temp key type checks

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2191)

Add new ssl_test option.

Add option ExpectedTmpKeyType to test the temporary key the server
sends is of the correct type.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2191)

fix a few more style issues

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Documentation clarification and fixes.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Remove unnecessary frees and style fixes.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

fix typo and remove duplicate macro

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Add documentation for PSS control operations.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Use more desciptive macro name rsa_pss_restricted()

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

style issues

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

free str on error

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

clarify comment

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

fix various style issues

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

make update

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

add test for invalid key parameters

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

document RSA-PSS algorithm options

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

add PSS key tests

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

print errors in pkey utility

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

make errors

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

add parameter error

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Set EVP_PKEY_CTX in SignerInfo

If we aren't setting public key parameters make EVP_PKEY_CTX available
in SignerInfo so PSS mode and parameters are automatically selected.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Only allow PSS padding for PSS keys.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Decode parameters properly.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Return errors PKCS#7/CMS enveloped data ctrls and PSS

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Add PSS parameter restrictions.

If a key contains any PSS parameter restrictions set them during
sign or verification initialisation. Parameters now become the
default values for sign/verify. Digests are fixed and any attempt
to change them is an error. The salt length can be modified but
must not be less than the minimum value.

If the key parameters are invalid then verification or signing
initialisation returns an error.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Initial parameter restrictions.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Add rsa_pss_get_param.

New function rsa_pss_get_param to extract and sanity check PSS parameters.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Don't allow PKCS#7/CMS encrypt with PSS.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Add macros to determine if key or ctx is PSS.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Support pad mode get/set for PSS keys.

Pad mode setting returns an error if the mode is anything other then PSS.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Key gen param support.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Set PSS padding mode for PSS keys.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Digest string helper function.

New function EVP_PKEY_CTX_md() which takes a string and passes a digest
to a ctrl.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Support RSA operations in PSS.

Add support for common operations in PSS by adding a new function
RSA_pkey_ctx_ctrl() which calls EVP_PKEY_CTX_ctrl if the key type
is RSA or PSS.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

PSS EVP_PKEY method

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

RSA-PSS key printing.

Print out RSA-PSS key parameters if present.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

PSS parameter encode and decode.

For RSA PSS keys encode and decode parameters when handling public
and private keys.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Split PSS parameter creation.

Split PSS parameter creation. This adds a new function rsa_pss_params_create
which creates PSS parameters from digest and salt values. This will be
used for PSS key generation.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Use method key type instead of EVP_PKEY_RSA

Make RSA method more flexible by using the key type from the
method instead of hard coding EVP_PKEY_RSA: by doing this the
same code supports both RSA and RSA-PSS.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

PSS ASN.1 method

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

add EVP_PKEY_RSA_PSS

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Add pss field to RSA structure and free it.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Cache maskHash parameter

Store hash algorithm used for MGF1 masks in PSS and OAEP modes in PSS and
OAEP parameter structure: this avoids the need to decode part of the ASN.1
structure every time it is used.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2177)

Update fuzz corpora

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

Make client and server fuzzer reproducible

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

Make the bignum fuzzer reproducible

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

Update fuzz documentation

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

Make rand_add predictable when fuzzing

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

server fuzzer: add support for DSA and ECDSA

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2182

Fix various style issues following feedback

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Rename the chain variable to chainidx

This variable represents the index of the cert within the chain, so give it
a name that better represents that.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Fix a double blank line style issue

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Initialise the al variable

al can be used uninitialised in an error path.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Update SSL_trace to understand TLSv1.3 Certificates

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Implement TLSv1.3 style CertificateStatus

We remove the separate CertificateStatus message for TLSv1.3, and instead
send back the response in the appropriate Certificate message extension.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Create Certificate messages in TLS1.3 format

Also updates TLSProxy to be able to understand the format and parse the
contained extensions.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Extends extension parsing to take the Certificate

Continuing from the previous commit we also need to extend the extensions
framework to supply the Certificate we just read during parsing.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Extend tls_construct_extensions() to enable passing of a certificate

The Certificate message in TLS1.3 has an extensions block for each
Certificate. Therefore we need to extend tls_construct_extensions() to pass
in the certificate we are working on. We also pass in the position in the
chain (with 0 being the first certificate).

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2020)

Remove BIO_seek/BIO_tell from evp_test.c

BIO_seek and BIO_tell can cause problems with evp_test.c on some platforms.
Avoid them by using a temporary memory BIO to store key PEM data.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2183)

Don't run MSBLOB conversion tests when RSA or DSA are disabled

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2174)

Don't run OCSP tests when OCSP is disabled

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2173)

Don't build OCSP stuff when OCSP is disabled

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2173)

Don't test SRP when it's disabled

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2172)

Don't run NPN tests when NPN is disabled

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2171)

Add RSA decrypt and OAEP tests.

Reviewed-by: Rich Salz <rsalz@openssl.org>

evptests.txt is not a shell script

Reviewed-by: Rich Salz <rsalz@openssl.org>

Don't run the sigalgs tests over a TLSv1.3 connection

We need a new API for TLSv1.3 sig algs

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2160)

Provide some tests for the sig algs API

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2160)

Fix the SSL_set1_sigalgs() macro

This macro has a typo in it which makes it unusable. This issue was already
fixed in 1.0.2 in commit 75fdee04827, but the same fix was not applied to
other branches.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2160)

70-test_sslvertol.t: skip test 1 and 2 if too few protocols are enabled

These tests depend on there being at least one protocol version below
TLSv1.3 enabled.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2144)

80-test_ssl_new.t: Make 19-mac-then-encrypt.conf work without TLSv1.2

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2144)

70-test_sslvertol.t: Make sure to check a max TLS version that matches configuration

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2144)

70-test_sslmessages.t: Don't check EXT_SIG_ALGS if TLS 1.2 is disabled

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2144)

Fix CT test_sslmessages hangs

The CT tests in test_sslmessages require EC to be available, therefore
we must skip these if no-ec

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2153)

Fix compilation with no-ec

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2153)

Fix extension for various no- options

Previously we were omitting the extension information from ext_defs if
the association no- option was defined. This doesn't work because the
indexes into the table are no longer valid.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2153)

Add a test to check the EC point formats extension appears when we expect

The previous commit fixed a bug where the EC point formats extensions did
not appear in the ServerHello. This should have been caught by
70-test_sslmessages but that test never tries an EC ciphersuite. This
updates the test to do that.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2153)

Fix the EC point formats extension

This should be sent in the ServerHello if a EC based ciphersuite is
negotiated. The relevant flag to do this was missed off in the recent
extensions refactor.

Fixes GitHub Issue #2133

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2153)

replace "will lookup up" by "will look up"

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
CLA: trivial
(Merged from https://github.com/openssl/openssl/pull/2145)

chacha/asm/chacha-x86_64.pl: add AVX512 path optimized for shorter inputs.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix EVP_MD_meth_get_flags

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2134)

Travis: The TLS 1.3 code isn't interoperable yet, move it to its own build

We should move it back to the BORINGTEST build when we are approaching
interoperability.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2123)

Reformat M_check_autoarg to match our coding style

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2121)

M_check_autoarg: sanity check the key

For now, checking that the size is non-zero will suffice.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2120)

Add bwrite_conv and bread_conv values to methods_dgramp_sctp

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2116)

Fix erroneous goto lable

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2116)

x86 assembly pack: update performance results.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Update fuzz corpora

Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #2090

Fix memory leak in tls_parse_stoc_key_share

Found by oss-fuzz

Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #2102

Fix typo.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
CLA: trivial
(Merged from https://github.com/openssl/openssl/pull/2086)

test/ssl_test: give up if both client and server wait on read

In some cases, both client and server end of the test can end up in
SSL_ERROR_WANT_READ and never get out of it, making the test spin.
Detect it and give up instead of waiting endlessly.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2096)

Fix no-ct, skip tests recipes that try to test CT

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2096)

e_afalg: Don't warn about kernel version when pedantic

When built with --strict-warnings and the Linux kernel headers don't
match the kernel version, the preprocessor warnings in
engines/afalg/e_afalg.c cause compilation errors.  Use the macro
PEDANTIC to avoid those warnings in that case.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2095)

evp_test: when function and reason strings aren't available, just skip

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2093)

HP-UX doesn't have hstrerror(), so make our own for that platform

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2092)
(cherry picked from commit 46766d003666da5f90346da7e6d09e109355f5c6)

Make client and server fuzzer support all ciphers

Also send a SNI extension in the client so the fuzzer can react to it.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2088

Document the recommended parameters for fuzzing

We use those parameters for calculating the coverage.

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2088

Enable TLS1.3 and PEDANTIC in the coverage target

This make sure that the coverage is the same for the fuzzers and this
coverage target

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2088