Fix a ssl session leak due to OOM in lh_SSL_SESSION_insert

- s == NULL can mean c is a new session *or* lh_insert was
  unable to create a hash entry.
- use lh_SSL_SESSION_retrieve to check for this error condition.
- If it happens simply remove the extra reference again.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2138)

RAND_egd_bytes: No need to check RAND_status on connection error.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1886)

Fixed the return code for RAND_egd_bytes.

According to the documentation, the return code should be -1 when
RAND_status does not return 1.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1886)

Fixed the return code of RAND_query_egd_bytes when connect fails.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1886)

Replace div-spoiler hack with simpler code

This comes from a comment in GH issue #1027. Andy wrote the code,
Rich made the PR.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2253)

Add documentation for the key logging callbacks

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1646)

Add tests for the key logging callbacks.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1646)

Add support for key logging callbacks.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1646)

Fix no-tls1_2

It seems that the ssl test 20-cert-select.conf dislikes the lack of TLSv1.2

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2268)

Fix use before assignment

 it was getting the SerialNumber of a previous cert.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2272)

Extend the test_ssl_new renegotiation tests to include client auth

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1982)

Stop server from expecting Certificate message when not requested

In a non client-auth renegotiation where the original handshake *was*
client auth, then the server will expect the client to send a Certificate
message anyway resulting in a connection failure.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1982)

Stop client from sending Certificate message when not requested

In a non client-auth renegotiation where the original handshake *was*
client auth, then the client will send a Certificate message anyway
resulting in a connection failure.

Fixes #1920

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1982)

Fix SSL_VERIFY_CLIENT_ONCE

The flag SSL_VERIFY_CLIENT_ONCE is documented as follows:

  B<Server mode:> only request a client certificate on the initial TLS/SSL
  handshake. Do not ask for a client certificate again in case of a
  renegotiation. This flag must be used together with SSL_VERIFY_PEER.

  B<Client mode:> ignored

But the implementation actually did nothing. After the server sends its
ServerKeyExchange message, the code was checking s->session->peer to see if
it is NULL. If it was set then it did not ask for another client
certificate. However s->session->peer will only be set in the event of a
resumption, but a ServerKeyExchange message is only sent in the event of a
full handshake (i.e. no resumption).

The documentation suggests that the original intention was for this to
have an effect on renegotiation, and resumption doesn't come into it.

The fix is to properly check for renegotiation, not whether there is already
a client certificate in the session.

As far as I can tell this has been broken for a *long* time.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1982)

fix a memory leak in ssl3_generate_key_block fix the error handling in ssl3_change_cipher_state

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2164)

Check the exit code from the server process

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2205)

poly1305/poly1305_base2_44.c: clarify shift boundary condition.

Reviewed-by: Matt Caswell <matt@openssl.org>

ec/asm/ecp_nistz256-ppc64.pl: minor POWER8-specific optimization.

Up to 4% depending on benchmark.

Reviewed-by: Rich Salz <rsalz@openssl.org>

'openssl passwd' command can now compute AIX MD5-based passwords hashes.

The difference between the AIX MD5 password algorithm and the standard MD5
password algorithm is that in AIX there is no magic string while in the
standard MD5 password algorithm the magic string is "$1$"

Documentation of '-aixmd5' option of 'openssl passwd' command is added.

1 test is added in test/recipes/20-test-passwd.t

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2251)

Fix DSA parameter generation control error

When setting the digest parameter for DSA parameter generation, the
signature MD was set instead of the parameter generation one.
Fortunately, that's also the one that was used for parameter
generation, but it ultimately meant the parameter generator MD and the
signature MD would always be the same.

Fixes github issue #2016

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2250)

Add signing hash tests

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2235)

Add options to check TLS signing hashes

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2235)

correct 3 mistakes

CLA: trivial

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2257)

Skip ECDH tests for SSLv3

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1597)

If client doesn't send curves list, don't assume all.

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1597)

Defines and strings for special salt length values, add tests

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2236)

additional PSS tests for -1 and invalid salt length

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2236)

Add support for -1, -2 salt lengths for PSS only keys.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2236)

Clean dead macros and defines

... mostly related to some old discarded modules .

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1906)

Clean one unused variable, plus an useless one.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1906)

Correct reference to SSL_get_peer_cert_chain().

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
CLA: trivial
(Merged from https://github.com/openssl/openssl/pull/2247)

Fix list -disabled for blake2 alg

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2241)

Fix man3 reference to CRYPTO_secure_used

CLA: trivial

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2238)

Add missing braces.

Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #2234

Fix undefined behaviour when printing the X509 and CRL version

Found by oss-fuzz

Reviewed-by: Andy Polyakov <appro@openssl.org>
GH: #2231

Fix VC warnings about unary minus to an unsigned type.

Reviewed-by: Andy Polyakov <appro@openssl.org>
GH: #2230

Fix undefined behaviour when printing the X509 serial

Found by afl

Reviewed-by: Andy Polyakov <appro@openssl.org>
GH: #2230

Add client cert type tests

Reviewed-by: Emilia Käsper <emilia@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2224)

Add certificate selection tests.

Add certifcate selection tests: the certificate type is selected by cipher
string and signature algorithm.

Reviewed-by: Emilia Käsper <emilia@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2224)

add ECDSA test server certificate

Reviewed-by: Emilia Käsper <emilia@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2224)

Add options to check certificate types.

Reviewed-by: Emilia Käsper <emilia@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2224)

Fix "failure rate" bugs

Reviewed-by: Emilia Käsper <emilia@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2228)

Clarify what RUN_ONCE returns

RUN_ONCE really just returns 0 on failure or whatever the init
function returned.  By convention, however, the init function must
return 0 on failure and 1 on success.  This needed to be clarified.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2225)

Fix no-ocsp

The use of EXFLAG_SET requires the inclusion of openssl/x509v3.h.
openssl/ocsp.h does that, except when OCSP is disabled.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2227)

UI: Use RUN_ONCE differently

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2222)

UI: Ensure there will be no race condition when getting the UI_METHOD ex_data

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2222)

Make X509_Digest,others public

Also, if want SHA1 then use the pre-computed value if there.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2223)

Rename file so "ls" works on 80 columns

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2221)

Add documentation

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1252)

Add "random malloc failure" tooling

Still needs to be documented, somehow/somewhere.

The env var OPENSSL_MALLOC_FAILURES controls how often malloc/realloc
should fail.  It's a set of fields separated by semicolons.  Each field
is a count and optional percentage (separated by @) which defaults to 100.
If count is zero then it lasts "forever."  For example: 100;@25 means the
first 100 allocations pass, then the rest have a 25% chance of failing
until the program exits or crashes.

If env var OPENSSL_MALLOC_FD parses as a positive integer, a record
of all malloc "shouldfail" tests is written to that file descriptor.
If a malloc will fail, and OPENSSL_NO_CRYPTO_MDEBUG_BACKTRACE is not set
(platform specific), then a backtrace will be written to the descriptor
when a malloc fails.  This can be useful because a malloc may fail but
not be checked, and problems will only occur later.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1252)

GH2176: Add X509_VERIFY_PARAM_get_time

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2208)

Review comments; fail build if nits found

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2202)

Run find-doc-nits in travis

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2202)

UI documentation fixup

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2218)

UI: fix uitest for VMS

- On VMS, apps/apps.c depends on apps/vms_term_sock.c, so add it to
  the build
- On VMS, apps/*.c are compiled with default symbol settings,
  i.e. uppercased and truncated symbols, which differs from test
  programs.  Make sure uitest.c knows that with a few pragmas.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2218)

UI: fix uitest for no-ui configuration

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2218)

Fix no-dh builds

One of the new tests uses a DH based ciphersuite. That test should be
disabled if DH is disabled.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2217)

Fix compilation with no-nextprotoneg

A guard was in the wrong place in the header file.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2212)

Add a test "uitest"

It tests both the use of UI_METHOD (through the apps/apps.h API) and
wrapping an older style PEM password callback in a UI_METHOD.

Replace the earlier UI test with a run of this test program

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2204)

Add a few documentation lines about UI_OpenSSL()

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2204)

Document the UI utility functions

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2204)

make update

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2204)

Add a UI utility function with which to wrap pem_callback_cb in a UI_METHOD

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2204)

Add an application data field in the UI_METHOD

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2204)

Constify the input parameter to UI_method_get_*

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2204)

Enable apps to get a UI_METHOD for the default prompter

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2204)

Fix no-ec following sigalgs refactor

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Fix tls1_set_sigalgs() length calculation

The length passed to tls1_set_sigalgs() is a multiple of two and there are
two char entries in the list for each sigalg. When we set
client_sigalgslen or conf_sigalgslen this is the number of ints in the list
where there is one entry per sigalg (i.e. half the length of the list passed
to the function).

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Add some missing sigalgs

The SHA1 sigalgs were inadvertently missed off in the sigalgs refactor.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Fix an uninit read picked up by Travis

The siglen value needs to be initialised prior to it being read in the
call to EVP_DigestSignFinal later in this function.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Fix a Travis failure

Declare a variable as static to silence the warning

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Add some signature tests

Check that signatures actually work, and that an incorrect signature
results in a handshake failure.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Teach TLSProxy how to re-encrypt a TLSv1.3 message after changes

This enables us to make changes to in-flight TLSv1.3 messages that appear
after the ServerHello.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Extend ServerKeyExchange parsing to work with a signature

Previously SKE in TLSProxy only knew about one anonymous ciphersuite so
there was never a signature. Extend that to include a ciphersuite that is
not anonymous. This also fixes a bug where the existing SKE processing was
checking against the wrong anon ciphersuite value. This has a knock on
impact on the sslskewith0p test. The bug meant the test was working...but
entirely by accident!

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Teach TLSProxy about the CertificateVerify message

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Teach SSL_trace about the new sigalgs

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Add a sigalg test to check we only allow sigalgs we sent

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Extend PSS signature support to TLSv1.2

TLSv1.3 introduces PSS based sigalgs. Offering these in a TLSv1.3 client
implies that the client is prepared to accept these sigalgs even in
TLSv1.2.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Fix test_sslversions to know that TLSv1.3 sets record version to TLSv1.0

This also acts as a test for the bug fixed in the previous commit.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Always use TLSv1.0 for record layer version in TLSv1.3

TLSv1.3 freezes the record layer version and ensures that it is always set
to TLSv1.0. Some implementations check this.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Add a TLS1.3 TODO for setting of sig algs

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Add some sig algs tests

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Ignore PKCS1 based sig algs in TLSv1.3

In TLSv1.3 we must use PSS based sig algs for RSA signing. Ignore any
shared sig algs which are PKCS1 based.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Verify that the sig algs extension has been sent for TLSv1.3

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Fix client application traffic secret

A misreading of the TLS1.3 spec meant we were using the handshake hashes
up to and including the Client Finished to calculate the client
application traffic secret. We should be only use up until the Server
Finished.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Temporarily ignore NewSessionTickets for TLS1.3

We can't handle these messages yet, so ignore them for now.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Use the correct size for TLSv1.3 finished keys

We need to use the length of the handshake hash for the length of the
finished key.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Ensure the record sequence number gets incremented

We were not incrementing the sequence number every time we sent/received
a record.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Remove some unneeded functions

The sigalgs work has made some old lookup tables and functions redundant
so remove them.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Use NIDs instead of the old TLSv1.2 sigalgs hash and sig ids

We had an extra layer of indirection in looking up hashes and sigs based
on sigalgs which is now no longer necessary. This removes it.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Remove a redundant function

The extensions refactor made this function redundant so we can remove it.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Convert Sigalgs processing to use ints

In TLSv1.2 an individual sig alg is represented by 1 byte for the hash
and 1 byte for the signature. In TLSv1.3 each sig alg is represented by
two bytes, where the two bytes together represent a single hash and
signature combination. This converts the internal representation of sigalgs
to use a single int for the pair, rather than a pair of bytes.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Sign CertificateVerify messages using PSS padding

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Make CertificateVerify TLS1.3 aware

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Move Certificate Verify construction and processing into statem_lib.c

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Add a TODO(TLS1.3) around certificate selection

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2157)

Fix typo in Blake2 function names

Fixes GitHub issue 2169.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2207)

Print the X509 version signed, and convert to unsigned for the hex version.

Found by tis-interpreter

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #1754

Only enable CRYPTO_3DES_ECB if that name is an existing macro

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2203)

Small fixes of cryptodev engine

- guard CRYPTO_3DES_CBC
- add a missing cast

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2203)