Provide documentation for the -psk_session option

The s_client/s_server docs were missing documentation for this option.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6215)

Prefer SHA-256 ciphersuites if using old style PSKs

If we have no certificate and we are using "old style" PSKs then we will
always default to using SHA-256 for that PSK. However we may have selected
a ciphersuite that is not based on SHA-256. Therefore if we see that there
are no certificates and we have been configured for "old style" PSKs then
we should prefer SHA-256 based ciphersuites during the selection process.

Fixes #6197

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6215)

Use void in all function definitions that do not take any arguments

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #6208

rsaz_avx2_eligible doesn't take parameters

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #6208

Set the ossl_shim to auto retry if not running asynchronously

In certain circumstances in the DTLS code we have to drop a record (e.g. if
it is a stale retransmit). We then have to move on to try and read the next
record. Some applications using blocking sockets (e.g. s_server/s_client
will hang if there isn't actually any data to be read from the socket yet).
Others can tolerate this. Therefore SSL_read()/SSL_write() can sometimes
return SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE even when using blocking
sockets. Applications can use the mode SSL_MODE_AUTO_RETRY, to switch this
behaviour off so that we never return unless we have read the data we
wanted to.

Commit ad96225285 fixed a DTLS problem where we always retried even if
SSL_MODE_AUTO_RETRY was not set. However that fix caused the Boring
ossl_shim to fail in some tests because it was relying on the previous
(buggy) behaviour. The ossl_shim should be set into SSL_MODE_AUTO_RETRY if
it is not operating asynchronously to avoid this problem.

[extended tests]

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6216)

PPC assembly pack: add POWER9 results.

Reviewed-by: Rich Salz <rsalz@openssl.org>

.travis.yml: add pair of linux-ppc64le targets.

One is clang --strict-warnings and one gcc sanitizer extended test.
Sanitizer build is quite expensive, can take >30 mins and is commented
for occasions when there is reason to believe that PPC-specific problem
can be diagnosed with sanitizer.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6192)

Fix no-cms

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6205)

Fix no-tls1_2, no-tls1_2-method, no-chacha and no-poly1305

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6205)

Fix typos in x509 documentation

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6207)

Add CHANGES entry for PR#6009

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6070)

Add blinding in BN_GF2m_mod_inv for binary field inversions

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6070)

ECC: unify generic ec2 and ecp scalar multiplication, deprecate ec2_mult.c

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6070)

ECDSA: remove nonce padding (delegated to EC_POINT_mul)

* EC_POINT_mul is now responsible for constant time point multiplication
  (for single fixed or variable point multiplication, when the scalar is
  in the range [0,group_order), so we need to strip the nonce padding
  from ECDSA.
* Entry added to CHANGES
* Updated EC_POINT_mul documentation
  - Integrate existing EC_POINT_mul and EC_POINTs_mul entries in the
    manpage to reflect the shift in constant-time expectations when
    performing a single fixed or variable point multiplication;
  - Add documentation to ec_method_st to reflect the updated "contract"
    between callers and implementations of ec_method_st.mul.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6070)

VMS rand: assign before check, not the other way around

items->ile3$w_code was checked before it was assigned its value...

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6200)

Fix --strict-warnings build of ppc-linux target

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6191)

ec/ec_mult.c: get BN_CTX_start,end sequence right.

Triggered by Coverity analysis.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6190)

Add a DTLS test for dropped records

Drop a record from a handshake and check that we can still complete the
handshake. Repeat for all records in the handshake.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6170)

Keep the DTLS timer running after the end of the handshake if appropriate

During a full handshake the server is the last one to "speak". The timer
should continue to run until we know that the client has received our last
flight (e.g. because we receive some application data).

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6170)

Only auto-retry for DTLS if configured to do so

Otherwise we may end up in a hang when using blocking sockets

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6170)

Fix s_client and s_server so that they correctly handle the DTLS timer

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6170)

Don't fail on an out-of-order CCS in DTLS

Fixes #4929

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6170)

Add a CMS API test

Previous tests only invoked CMS via the command line app. This test uses
the CMS API directly to do and encrypt and decrypt operation. This test
would have caught the memory leak fixed by the previous commit (when
building with enable-crypto-mdebug).

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6142)

Fix a mem leak in CMS

The function CMS_RecipientInfo_set0_pkey() is a "set0" and therefore
memory management passes to OpenSSL. If the same function is called again
then we should ensure that any previous value that was set is freed first
before we set it again.

Fixes #5052

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6142)

windows-makefile.tmpl: rearrange cleanup commands to avoid ...

deletion of *.exp files in krb5 sub-module.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6186)

.travis.yml: temporarily mask gcc-5 ubsan build.

Linking fails with "unrecognized option '--push-state--no-as-needed'",
which is beyond our control.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6185)

.travis.yml: minor facelift

Apparently trusty image has newer clang, there is no need to pull
clang-3.9 packages. It's clang-5.0.0, installation is a bit quirky,
as it fails to compile for example strcmp(s,"-") without warning,
and complains about unreferred -I flags. But it's argued that benefits
of exercising newer sanitizer outweights the inconvenience of
additional -D__NO_STRING_INLINE and -Wno-unused-command-line-argument.

Also pull golang when actually needed.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6185)

Cleanup

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6183)

Configure: fix Mac OS X builds that still require makedepend

Earlier Apple Xcode compilers, e.g. one targeting Mac OS X 10.7, don't
support dependency generation and one still has to use makedepend.  It's
unclear when it was fixed, but all clang-based Apple compilers seem to
support -M options.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6169)

Configure: pass more suitable argument to compiler_predefined().

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6174)

Configure: move --noexecstack probe to Configure.

config probe doesn't work in cross-compile scenarios or with clang.
In addition consolidate -Qunused-arguments handling.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6174)

BIO_s_mem() write: Skip early when input length is zero

When the input length is zero, just return zero early.  Otherwise,
there's a small chance that memory allocation is engaged, fails and
returns -1, which is a bit confusing when nothing should be written.

Fixes #4782 #4827

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/6175)

docs: Fix typo EVP_PKEY_new_id -> EVP_PKEY_CTX_new_id

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6177)

v3_purp.c: add locking to x509v3_cache_extensions()

Fixes #6121

Thanks to Mingtao Yang for reporting this bug.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6162)

VMS: modernise rand_pool_acquire_entropy, step 2

Add more items that could serve as entropy source.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6151)

VMS: modernise rand_pool_acquire_entropy, step 1

Stop redefining structures that are already defined in system
headers.  This also means we can stop setting the pointer size
globally, because the system structures will have the correct pointer
sizes either way.  The only exception is passing the right pointer
size to a function.

Stop trying to twist things around with rand(), that's the job of the
DRBG that we feed.

Stop assuming the location of the JPI$_FINALEXC item, look it up
instead.

Signal an exception if the sys$getjpiw call fails (it means the item
list isn't set up right, so works as an assertion, but using VMS
methodology).

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6151)

Return an error from BN_mod_inverse if n is 1 (or -1)

Calculating BN_mod_inverse where n is 1 (or -1) doesn't make sense. We
should return an error in that case. Instead we were returning a valid
result with value 0.

Fixes #6004

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6119)

Make X509_VERIFY_PARAM_get_hostflags() take a const arg

Commit 5b748dea5 added this function which should have taken a const
argument.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6154)

Add a test for SSL_get_shared_ciphers()

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6113)

Fix a bug in create_ssl_ctx_pair()

The max protocol version was only being set on the server side. It should
have been done on both the client and the server.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6113)

Add some documentation for SSL_get_shared_ciphers()

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6113)

Fix comment in ssl_locl.h

The ciphers field in a session contains the stack of ciphers offered by
the client.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6113)

Fix SSL_get_shared_ciphers()

The function SSL_get_shared_ciphers() is supposed to return ciphers shared
by the client and the server. However it only ever returned the client
ciphers.

Fixes #5317

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6113)

opensslconf.h inclusion cleanup
No need to buildtest on opensslconf.h

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6149)

bn/asm/*-mont.pl: harmonize with BN_from_montgomery_word.

Montgomery multiplication post-conditions in some of code paths were
formally non-constant time. Cache access pattern was result-neutral,
but a little bit asymmetric, which might have produced a signal [if
processor reordered load and stores at run-time].

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6141)

apps/speed.c: merge parameters defining EC curves to test ...

... and unify 'bits' declarations and printing format.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6132)

a_strex.c: prevent out of bound read in do_buf()

which is used for ASN1_STRING_print_ex*() and X509_NAME_print_ex*().

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6105)

Change rand_pool_bytes_needed to handle less entropy than 1 per 8 bits

rand_pool_bytes_needed() was constructed in such a way that the
smallest acceptable entropy factor was 1 entropy bits per 8 bits of
data.  At the same time, we have a DRBG_MINMAX_FACTOR that allows
weaker source, as small as 1 bit of entropy per 128 bits of data.
The conclusion is that rand_pool_bytes_needed() needs to change to
support weaker entropy sources.  We therefore change the input of
entropy per byte to be an entropy factor instead.  This entropy factor
expresses how many bits of data it takes (on average) to get 1 bit of
entropy.

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/6150)

Add a note about Nagle's algorithm on the SSL_connect man page

Fixes #4237

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/6143)

Fix memory leaks in CA related functions.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4700)

Fix regression with session cache use by clients

Commit d316cdcf6d8d6934663278145fe0a8191e14a8c5 introduced some extra
checks into the session-cache update procedure, intended to prevent
the caching of sessions whose resumption would lead to a handshake
failure, since if the server is authenticating the client, there needs to
be an application-set "session id context" to match up to the authentication
context.  While that change is effective for its stated purpose, there
was also some collatoral damage introduced along with the fix -- clients
that set SSL_VERIFY_PEER are not expected to set an sid_ctx, and so
their usage of session caching was erroneously denied.

Fix the scope of the original commit by limiting it to only acting
when the SSL is a server SSL.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5967)

Improve error handling in rand_init function

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6124)

Add getter for X509_VERIFY_PARAM_get_hostflags

Fixes #5061

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6139)

Clarify BN_mod_exp docs

Specifically this is not supported with an even modulus and
BN_FLG_CONSTTIME.

Fixes #5082

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6137)

Fix some errors and missing info in the CMS docs

Fixes #5063

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6134)

Prepare for 1.1.1-pre7-dev

Reviewed-by: Rich Salz <rsalz@openssl.org>

Prepare for 1.1.1-pre6 release

Reviewed-by: Rich Salz <rsalz@openssl.org>

Update copyright year

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6145)

Use the config file from the source not the host for the tests

Fixes: #6046

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #6125

Configurations/unix-Makefile.tmpl: harmonize with no-engine.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6106)

Configurations/10-main.conf: force no-engine on ios targets.

Rationale for enforcing no-engine is because of disconnect between
compile-time config and run-time, which is a per-application sandbox
directory which one can't predict in advance. Besides, none of the
bundled engines actually give an edge on iOS...

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6106)

Don't cleanup uninitialized thread local slots

Fixes: #6120

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/6123)

Fix drbg thread cleanup and error handling

Fixes: #6081

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6089)

Update version docs

Make it clear that you should not attempt to get the version before the
first handshake is complete.

Fixes #2893

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6013)

apps/s_server.c: Avoid unused variable due to 'no-dtls'

Fixes #6098

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/6104)

fix: BN_swap mishandles flags

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6099)

Fix mixed indentation (and other whitespace issues)

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6094)

15-test_out_option: Refactor and don't test directory write on VMS

To my surprise, it turns out that on OpenVMS, opening './' (which
is translated to '[]') for writing actually creates a file, '[].'.
On OpenVMS, this is a perfectly valid file with no name or extension,
just the delimiter between the two.

Because of the mess the exception would generate in the test recipe,
it gets refactored again, to clearly separate each test inside it,
and use skips to avoid some of them (that makes it clear that they are
skipped and why, when running the recipe).

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6100)

Fix last(?) batch of malloc-NULL places

Add a script to find them in the future

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/6103)

Fix memleaks in async api

Fixes: #5950

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6038)

Wait max. 60 seconds for s_client to connect

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5964)

Add missing error code when alloc-return-null

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6085)

Update the *use_certificate* docs

Note that calling the *use_certificate* functions will replace any existing
certificate of the same type. The same thing applies for private keys.

Fixes #2147

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6092)

Fix typo in the definition of tls13_ciphers

SSL_kANY, and SSL_aANY were placed in the wrong fields. It makes no
functional difference since these macros evaluate to 0 anyway, which is
the correct value for these fields.

Fixes #6048

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6095)

Use  get_last_sys_error() instead of get_last_rtl_error()

get_last_sys_error() already exists, so there's no need for yet
another macro that fulfills the same purpose.

Fixes #4120

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6088)

PEM_def_callback(): use same parameter names as for pem_password_cb

Add a bit more commentary to explain what's going on.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6080)

PEM_def_callback(): don't loop because of too short password given

That error is already caught by EVP_read_pw_string_min, and causes
this function to return -1, so the code detecting too short passwords
in this function is practically dead.

Fixes #5465

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6080)

openssl rehash: exit 0 on warnings, same as c_rehash

Fixes #6083

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/6084)

ms/uplink-x86.pl: close the file handle that was opened

Fixes #5656

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6079)

Fix documentation for the -showcerts s_client option

This option shows the certificates as sent by the server. It is not the
full verified chain.

Fixes #4933

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6067)

apps/opt.c: Remove the access checks of input and output files

open() will take care of the checks anyway

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6033)

Better check of return values from app_isdir and app_access

[extended tests]

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6033)

Revert "Check directory is able to create files for various -out option"

This reverts commit 555c94a0db9661428da0a45cb32b9f002324eefd.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6033)

Revert "Add VMS version of app_dirname()"

This reverts commit 215a6730f1eaf53b01a4eb10d75bd09fd74f70cc.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6033)

Revert "Check on VMS as well"

This reverts commit f6d765988f37c43edb1056ab83165f2569182e9d.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6033)

test/recipes/15-test_out_option.t: refine tests

Test writing to the null device.  This should be successful.

Also, refactor so the planned number of tests is calculated.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6033)

[SM2_sign] add minimal EVP_PKEY functionality testing

The actual functionality of generating signatures through the `EVP_PKEY`
API is completely untested.
Current tests under the `EVP_PKEY` API
(`test/recipes/30-test_evp_data/evppkey.txt`) only cover `Verify` and
`Decrypt`, while encryption and signature generation are tested with
ad-hoc clients (`test/sm2crypttest.c`, `test/sm2signtest.c`) that do not
call the `EVP_PKEY` interface at all but soon-to-be private functions
that bypass it (cf. PR#5895 ).

It is my opinion that an ideal solution for the future would consist on
enhancing the `test/evp_pkey` facility and syntax to allow tests to take
control of the PRNG to inject known nonces and validate the results of
`EVP_PKEY` implementations against deterministic known answer tests, but
it is probably too late to work on this feature in time for next release.

Given that commit b5a85f70d8 highlights some critical bugs in the hook
between the `EVP_PKEY` interface and SM2 signature generation and that
these defects escaped testing and code review, I think that at least for
now it is beneficial to at least add the kind of "bogus" testing
provided by this patch:
this is a "fake" test as it does only verify that the SM2 `EVP_PKEY`
interface is capable of creating a signature without failing, but it
does not say anything about the generated signature being valid, nor
does it test the functional correctness of the cryptosystem.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6066)

[SM2_sign] fix double free and return value

Currently, critical bugs prevent using SM2 signatures through the
`EVP_PKEY` interface: any application that managed to satisfy the
requirement of forcing SM3 as the message digest – even if this is
currently not possible transparently through the `EVP_PKEY` interface
and requires manually forcing the MD selection – would crash with a
segmentation fault upon calling the `SM2_sign()` function.

This is easily verified using the OpenSSL CLI to execute this critical
code path under the right conditions:
`openssl dgst -sm3 -hex -sign sm2.eckey /path/to/file/to/sign`

The issue is caused by a double free at the end of `SM2_sign()` in
`crypto/sm2/sm2_sign.c` in case of successful signature generation.
In addition, even if the double free was not causing segfaults,
the function returns the wrong return value in case of success (it
would return 0 rather than 1).

This patch fixes both problems.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6066)

Fix the MAX_CURVELIST definition

The MAX_CURVELIST macro defines the total number of in-built SSL/TLS curves
that we support. However it has not been updated as new curves are added.

Fixes #5232

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/6065)

Add missing malloc-return-null instance

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6071)

x509/by_dir.c: Remove dead code

Noticed in #5837

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6064)

Add a test to verify the ClientHello version is the same in a reneg

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6059)

In a reneg use the same client_version we used last time

In 1.0.2 and below we always send the same client_version in a reneg
ClientHello that we sent the first time around, regardless of what
version eventually gets negotiated. According to a comment in
statem_clnt.c this is a workaround for some buggy servers that choked if
we changed the version used in the RSA encrypted premaster secret.

In 1.1.0+ this behaviour no longer occurs. This restores the original
behaviour.

Fixes #1651

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6059)

Don't build AFALG on android

This didn't get built anyway for gcc because it was detected as a cross
compile. But it did get built for clang - even though this is still a cross
compile build. This disables it in all cases for Android.

Fixes #5748

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/6057)

Allow TLSv1.3 EC certs to use compressed points

The spec does not prohib certs form using compressed points. It only
requires that points in a key share are uncompressed. It says nothing
about point compression for certs, so we should not fail if a cert uses a
compressed point.

Fixes #5743

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/6055)

Document when a session gets removed from cache

Document the fact that if a session is not closed down cleanly then the
session gets removed from the cache and marked as non-resumable.

Fixes #4720

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/6053)

Remove some logically dead code

This dead code should have been removed as part of #5874 but got missed.

Found by Coverity.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/6049)

Improve backwards compat with 1.0.2 for ECDHParameters

In 1.0.2 you could configure automatic ecdh params by using the
ECDHParameters config directive and setting it to the value
"+Automatic" or just "Automatic". This is no longer required in 1.1.0+
but we still recognise the "+Automatic" keyword for backwards compatibility.
However we did not recognise just "Automatic" without the leading "+" which
is equally valid. This commit fixes that omission.

Fixes #4113

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6035)

X509: add more error codes on malloc or sk_TYP_push failure

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5837)

a_strex.c: improve documentation of 'tag2nbyte' lookup table

The 'tag2nbyte' lookup table maps the tags of ASN1 string types
to their respective character widths. It is used for example by
ASN1_STRING_to_UTF8(). This commit adds the tag names as comments.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6062)