bn: fix BN_DEBUG + BN_DEBUG_RAND support

Couple of updates to make this code work properly again;
* use OPENSSL_assert() instead of assert() (and #include <assert.h>)
* the circular-dependency-avoidance uses RAND_bytes() (not pseudo)

Change-Id: Iefb5a9dd73f71fd81c1268495c54a64378955354
Signed-off-by: Geoff Thorpe <geoff@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1672)

bntests.txt: add a couple of checks of possibly negative zero

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2335)

bntest: do not stop on first fautl encountered

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2335)

bntest: make sure file_rshift tests BN_rshift1 as well when appropriate

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2335)

bntest: make sure that equalBN takes note of negative zero

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2335)

Add -Wno-parentheses-equality to dev-warnings.

Also fix a block comment formatting glitch.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2327)

Tidy up Suite B logic

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2324)

Remove peer_md and use peer_sigalg instead.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2324)

Simplify sigalgs code.

Remove unnecessary lookup operations: use the indices and data in the
lookup table directly.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2324)

Add digest and key indices to table.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2324)

Don't check certificate type against ciphersuite for TLS 1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2324)

Add TLS 1.3 signing curve check

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2324)

Only allow PSS signatures with RSA keys and TLS 1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2324)

Store table entry to peer signature algorithm.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2324)

Free up the memory for the NewSessionTicket extensions

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2326)

Make sure we free and cleanse the pms value in all code paths

Otherwise we get a memory leak.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2326)

Update macros.

Use TLS_MAX_SIGALGCNT for the maximum number of entries in the
signature algorithms array.

Use TLS_MAX_SIGSTRING_LEN for the maxiumum length of each signature
component instead of a magic number.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

fix style issues

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Fix TLS 1.2 and no sigalgs.

For TLS 1.2 if we have no signature algorithms extension then lookup
using the complete table instead of (empty) shared signature algorithms
list so we pick up defaults.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Use PSS for simple test so TLS 1.3 handhake is successful.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Add tests for client and server signature type

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Add test support for TLS signature types.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Update documentation

Add details of the use of PSS for signature algorithms.

Document SSL_get_peer_signature_nid() and SSL_get_peer_signature_type_nid().

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

make update

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Replace TLS_SIGALGS with SIGALG_LOOKUP

Since every supported signature algorithm is now an entry in the
SIGALG_LOOKUP table we can replace shared signature algortihms with
pointers to constant table entries.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Support TLS 1.3 signature scheme names.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Extend TLS 1.3 signature table.

Add additional entries in the TLS 1.2 signature table to include
the name, sig and hash NID (if any) and required curve (if any).

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Use shared signature algorithm list to find type.

Lookup the signature type in the shared list: we can use this to
use PSS if the peer supports it for TLS 1.2.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Add SSL_get_peer_signature_type_nid() function.

Add function to retrieve signature type: in the case of RSA
keys the signature type can be EVP_PKEY_RSA or EVP_PKEY_RSA_PSS.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Store peer signature type.

Store peer signature type in s->s3->tmp.peer_sigtype and check it
to see if the peer used PSS.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

More complete PSS support.

Extend support for PSS key signatures by using the EVP_PKEY_RSA_PSS type
to distinguish them from PKCS1 signature types.

Allow setting of PSS signature algorithms using the string "PSS" or
"RSA-PSS".

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Use uint16_t for signature scheme.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

use RSA_PSS_SALTLEN_DIGEST constant

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2301)

Expand comment in tls_process_hello_req()

Following review feedback.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Add a TODO around validating the ticket age

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Various style fixes following review feedback

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Remove unneccessary comments

Now we're using an enum the values themselves are self explanatory

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Use for loop in WPACKET_fill_lengths instead of do...while

Based on review feedback

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Move the SSL3_CK_CIPHERSUITE_FLAG out of public header

The newly added SSL3_CK_CIPHERSUITE_FLAG shouldn't be in a public header
file

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Add a test for the PSK kex modes extension

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

If we have no suitable PSK kex modes then don't attempt to resume

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Fix <= TLS1.2 break

Changing the value of SSL_MAX_MASTER_KEY_LENGTH had some unexpected
side effects in the <=TLS1.2 code which apparently relies on this being
48 for interoperability. Therefore create a new define for the TLSv1.3
resumption master secret which can be up to 64 bytes.

Found through the boring test suite.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Add a test for WPACKET_fill_lengths()

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Enable wpacket test on shared builds

Now that we support internal tests properly, we can test wpacket even in
shared builds.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Make calls to SSL_renegotiate() error out for TLSv1.3

When we have support for KeyUpdate we might consider doing that instead.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Make the "ticket" function return codes clearer

Remove "magic" return values and use an enum instead.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Miscellaneous style tweaks based on feedback received

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Ensure the al variable is properly ininitialised in all code paths

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Ensure the age_add variable is properly initialised

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Re-enable resumption for TLS1.3 CT tests

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Add testing of TLSv1.3 resumption in test_tls13messages

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Add a capability for TLSProxy to wait for a session before killing s_client

TLSProxy normally fires off s_client, which creates a connection to the
server. TLSProxy also pipes some data to send to the process and s_client
automatically exits when the pipe hits eof. Unfortunately this means that
it sends the data and closes before it has processed the NewSessionTicket
returned from the server in TLSv1.3. This commits adds an option for
s_client to stay loaded until the sesion has been processed. A side effect
of this is that s_client never sends a close_notify in this mode, so we
count success as seeing that data has been transferred.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Re-enable test_ssl_new resumption tests for TLSv1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Remove a TLS1.3 TODO that is now completed

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Re-enable TLSv1.3 session resumption related tests in sslapitest

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Update create_ssl_connection() to make sure its gets a session

In TLSv1.3 the connection will be created before the session is
established. In OpenSSL we send the NewSessionTicket message immediately
after the client finished has been received. Therefore we change
create_ssl_connection() to attempt a read of application data after the
handshake has completed. We expect this to fail but it will force the
reading of the NewSessionTicket and the session to be set up.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Re-enable ALPN resumption tests where we are using TLSv1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Always ensure that session->cipher is set

If we have deserialized the SSL_SESSION then in some circumstances the
session->cipher value is NULL. We were patching up in some places but not
in others. We should just do it as part of loading the SSL_SESSION.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Tweak a comment

psk_kex_mode stores the available modes - not the one which we selected.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Use the correct session resumption mechanism

Don't attempt to add a TLS1.3 session to a TLS1.2 ClientHello session
ticket extensions. Similarly don't add a TLS1.2 session to a TLS1.3
psk extension.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Move session version consistency check

Make sure the session version consistency check is inside
ssl_get_prev_session(). Also fixes a bug where an inconsistent version can
cause a seg fault in TLSv1.3.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Set the kex modes on the client too.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Implement server side of PSK extension construction

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Implement Server side of PSK extension parsing

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Make sure we also cleanse the finished key

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Provide a key_share extension finaliser

This mops up various edge cases with key_shares and makes sure we still
generate the handshake secret if we haven't been provided with one but we
have a PSK.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Add support for client side parsing of the PSK extension

Requires a refactor of the ServerHello parsing, so that we parse first and
then subsequently process. This is because the resumption information is
held in the extensions block which is parsed last - but we need to know that
information earlier.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Add support for the age_add field

Update SSL_SESSION to store the age_add and use it where needed.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Construct the client side psk extension for TLSv1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Provide a new WPACKET function for filling in all the lengths

For the psk extension we need to fill in all the lengths of the message so
far, even though we haven't closed the WPACKET yet. This provides a function
to do that.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Update the s_client -sess_out feature to work for TLSv1.3

Previously "-sess_out" wrote out the session as soon as the handshake
finished. In TLSv1.3 this won't work because the NewSessionTicket message
arrives post-handshake. Instead we use the session callback mechanism to
do this.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Never send a session id in TLS1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Add a TODO around handling of SSL_get_session() and SSL_get1_session()

These functions are problematic in TLSv1.3 because the server sends the
NewSessionTicket message after the handshake has finished.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Process incoming NewSessionTicket messages on the client side

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Create the NewSessionTicket message in TLSv1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Add support for the psk_key_exchange_modes extension

This is required for the later addition of resumption support.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Move TLSv1.3 Session Ticket processing into the state machine

We still ignore it for now, but at least its in the right place.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Disable requests for renegotiation in TLSv1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Move state machine knowledge out of the record layer

The record layer was making decisions that should really be left to the
state machine around unexpected handshake messages that are received after
the initial handshake (i.e. renegotiation related messages). This commit
removes that code from the record layer and updates the state machine
accordingly. This simplifies the state machine and paves the way for
handling other messages post-handshake such as the NewSessionTicket in
TLSv1.3.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Remove use of the SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS flag

This flag is never set by anything so remove it.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

ChangeCipherSpec is not allowed in TLSv1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

Remove annoying and pointless pause.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2167)

Fix faulty free

On error, i2o_SCT_signature() and i2o_SCT() free a pointer that may
have wandered off from the start of the allocated block (not currently
true for i2o_SCT_signature(), but has that potential as the code may
change.  To avoid this, save away the start of the allocated block and
free that instead.

Thanks to Guido Vranken for reporting this issue.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2323)

test/evp_test.c: If no algorithm was specified, don't try to check for DES

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2319)

Update client and server corpus

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #2318

Don't need the checkbox; the hook does it

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2317)

Add a couple of test to check CRL fingerprint

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2314)

Document what EXFLAG_SET is for in x509v3.h

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2314)

X509_CRL_digest() - ensure precomputed sha1 hash before returning it

X509_CRL_digest() didn't check if the precomputed sha1 hash was actually
present.  This also makes sure there's an appropriate flag to check.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2314)

Correct pointer to be freed

The pointer that was freed in the SSLv2 section of ssl_bytes_to_cipher_list
may have stepped up from its allocated position.  Use a pointer that is
guaranteed to point at the start of the allocated block instead.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2312)

Avoid over-long strings.  Fix a mem leak.

Reviewed-by: Stephen Henson <steve@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2299)

Add server signature algorithm bug test.

Add a client authentication signature algorithm to simple
ssl test and a server signature algorithm. Since we don't
do client auth this should have no effect. However if we
use client auth signature algorithms by mistake this will
abort the handshake with a no shared signature algorithms
error.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2290)

Use correct signature algorithm list when sending or checking.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2290)

Update CHANGES and NEWS for new release

Reviewed-by: Richard Levitte <levitte@openssl.org>

bn/asm/x86_64-mont5.pl: fix carry bug in bn_sqr8x_internal.

CVE-2017-3732

Reviewed-by: Rich Salz <rsalz@openssl.org>

test/bntest.c: regression test for carry bug in bn_sqr8x_internal.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Document DH_check_params()

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

Better check of DH parameters in TLS data

When the client reads DH parameters from the TLS stream, we only
checked that they all are non-zero.  This change updates the check to
use DH_check_params()

DH_check_params() is a new function for light weight checking of the p
and g parameters:

    check that p is odd
    check that 1 < g < p - 1

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

crypto/evp: harden AEAD ciphers.

Originally a crash in 32-bit build was reported CHACHA20-POLY1305
cipher. The crash is triggered by truncated packet and is result
of excessive hashing to the edge of accessible memory. Since hash
operation is read-only it is not considered to be exploitable
beyond a DoS condition. Other ciphers were hardened.

Thanks to Robert Święcki for report.

CVE-2017-3731

Reviewed-by: Rich Salz <rsalz@openssl.org>

crypto/evp: harden RC4_MD5 cipher.

Originally a crash in 32-bit build was reported CHACHA20-POLY1305
cipher. The crash is triggered by truncated packet and is result
of excessive hashing to the edge of accessible memory (or bogus
MAC value is produced if x86 MD5 assembly module is involved). Since
hash operation is read-only it is not considered to be exploitable
beyond a DoS condition.

Thanks to Robert Święcki for report.

CVE-2017-3731

Reviewed-by: Rich Salz <rsalz@openssl.org>