NEWS: DH, DSA, ECDH, ECDSA and RSA public key function deprecation note

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11063)

rsa.h: fix preprocessor indentation

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11063)

Remove unneeded switch statement to fix warning

https://github.com/openssl/openssl/issues/10958

CLA: trivial

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11125)

Handle max_fragment_length overflow for DTLS

Allow for encryption overhead in early DTLS size check
and send overflow if validated record is too long

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/11096)

[PROV][EC] Update documentation

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10631)

[BN] harden `BN_copy()` against leaks from memory accesses

`BN_copy()` (and indirectly `BN_dup()`) do not propagate the
`BN_FLG_CONSTTIME` flag: the propagation has been turned on and off a
few times in the past years, because in some conditions it has shown
unintended consequences in some code paths.

Without turning the propagation on once more, we can still improve
`BN_copy()` by avoiding to leak `src->top` in case `src` is flagged with
`BN_FLG_CONSTTIME`.
In this case we can instead use `src->dmax` as the number of words
allocated for `dst` and for the `memcpy` operation.

Barring compiler or runtime optimizations, if the caller provides `src`
flagged as const time and preallocated to a public size, no leak should
happen due to the copy operation.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10631)

[EC] harden EC_KEY against leaks from memory accesses

We should never leak the bit length of the secret scalar in the key,
so we always set the `BN_FLG_CONSTTIME` flag on the internal `BIGNUM`
holding the secret scalar.

This is important also because `BN_dup()` (and `BN_copy()`) do not
propagate the `BN_FLG_CONSTTIME` flag from the source `BIGNUM`, and
this brings an extra risk of inadvertently losing the flag, even when
the called specifically set it.

The propagation has been turned on and off a few times in the past
years because in some conditions has shown unintended consequences in
some code paths, so at the moment we can't fix this in the BN layer.

In `EC_KEY_set_private_key()` we can work around the propagation by
manually setting the flag after `BN_dup()` as we know for sure that
inside the EC module the `BN_FLG_CONSTTIME` is always treated
correctly and should not generate unintended consequences.

Setting the `BN_FLG_CONSTTIME` flag alone is never enough, we also have
to preallocate the `BIGNUM` internal buffer to a fixed public size big
enough that operations performed during the processing never trigger
a realloc which would leak the size of the scalar through memory
accesses.

Fixed Length
------------

The order of the large prime subgroup of the curve is our choice for
a fixed public size, as that is generally the upper bound for
generating a private key in EC cryptosystems and should fit all valid
secret scalars.

For preallocating the `BIGNUM` storage we look at the number of "words"
required for the internal representation of the order, and we
preallocate 2 extra "words" in case any of the subsequent processing
might temporarily overflow the order length.

Future work
-----------

A separate commit addresses further hardening of `BN_copy()` (and
indirectly `BN_dup()`).

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10631)

[PROV][KEYMGMT][EC] Import/export of priv_key as padded const time BN

For EC keys it is particularly important to avoid leaking the bit length
of the secret scalar.

Key import/export should never leak the bit length of the secret
scalar in the key.

For this reason, on export we use padded BIGNUMs with fixed length,
using the new `ossl_param_bld_push_BN_pad()`.

When importing we also should make sure that, even if short lived,
the newly created BIGNUM is marked with the BN_FLG_CONSTTIME flag as
soon as possible, so that any processing of this BIGNUM might opt for
constant time implementations in the backend.

Setting the BN_FLG_CONSTTIME flag alone is never enough, we also have
to preallocate the BIGNUM internal buffer to a fixed size big enough
that operations performed during the processing never trigger a
realloc which would leak the size of the scalar through memory
accesses.

Fixed length
------------

The order of the large prime subgroup of the curve is our choice for
a fixed public size, as that is generally the upper bound for
generating a private key in EC cryptosystems and should fit all valid
secret scalars.

For padding on export we just use the bit length of the order
converted to bytes (rounding up).

For preallocating the BIGNUM storage we look at the number of "words"
required for the internal representation of the order, and we
preallocate 2 extra "words" in case any of the subsequent processing
might temporarily overflow the order length.

Future work
-----------

To ensure the flag and fixed size preallocation persists upon
`EC_KEY_set_private_key()`, we need to further harden
`EC_KEY_set_private_key()` and `BN_copy()`.
This is done in separate commits.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10631)

[PROV][KMGMT][KEXCH][EC] Implement EC keymgtm and ECDH

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10631)

[CMS] Test decryption of a ciphertext encrypted from 1.1.1

Current CMS en/decryption tests only validate that our current decyption
and encryption algorithms are compatible, but they say nothing about
correctness of the output for the given set of parameters.

As a partial fix in absence of proper KAT tests, we decrypt ciphertexts
generated with OpenSSL 1.1.1.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10631)

[EC] Constify internal EC_KEY pointer usage

A pair of internal functions related to EC_KEY handling could benefit
from declaring `EC_KEY *` variables as `const`, providing clarity for
callers and readers of the code, in addition to enlisting the compiler
in preventing some mistakes.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10631)

[PROV][KEYMGMT][DH][DSA] use BN_clear_free for secrets

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10631)

x86_64: Don't assume 8-byte pointer size

Since pointer in x32 is 4 bytes, add x86_64-support.pl to define
pointer_size and pointer_register based on flavour to support
stuctures like:

struct {  void *ptr; int blocks;  }

This fixes 90-test_sslapi.t on x32.  Verified with

$ ./Configure shared linux-x86_64
$ make
$ make test

and

$ ./Configure shared linux-x32
$ make
$ make test

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/10988)

DOC:Fix typos in x509v3_config.pod

CLA: trivial

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11104)

DOC:Fix typos in man5/config.pod

CLA: trivial

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11104)

DOC:Fix typos in openssl-enc.pod.in + openssl.pod

CLA: trivial

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11104)

DOC:Fix typos in ossl_cmp_print_log.pod

CLA: trivial

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11104)

DOC:Fix typos in ossl_param_bld_init

CLA: trivial

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11104)

Deprecate EC command line apps.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11106)

TEST: Optionally silence OpenSSL::Test::setup()

test/generate_ssl_tests.pl uses OpenSSL::Test to get to some of its
practical location functions.  A recent note in the setup() code made
its result not quite match the original (we do check that), so there's
a need to silence setup(), which we do with a simple optional argument.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/tconversion.pl to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/80-test_ssl_old.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/80-test_ssl_new.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/80-test_ocsp.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/80-test_cms.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/80-test_ca.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/20-test_pkeyutl.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/20-test_enc_more.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/25-test_x509.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/25-test_req.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/25-test_crl.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/20-test_enc.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/20-test_dgst.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/15-test_rsapss.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Modify test/recipes/15-test_mp_rsa.t to leave artifacts behind

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

TEST: Create test specific output directories

We had all tests run with test/test-runs/ as working directory, and
tests cleaned up after themselves...  which is well and good, until
you want to have a look at what went wrong when a complex test fails,
and you have to recreate everything it does manually.

To remedy this, we have OpenSSL::Test create the result directory
dynamically (and cleaning it up first if it's already there) and let
the test recipe have that as working directory.

Test recipes are now encouraged to name their diverse output files
uniquely, and not to clean them up, to allow a developer to have a
look at the files that were produced.

With continuous integration that allows this, the result directories
could also be archived and be left as a build artifact.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11080)

DOC: Add documentation related to X509_LOOKUPs

Most of all, the base X509_LOOKUP functionality is now documented.
Furthermore, the names X509_LOOKUP_METHOD and X509_STORE are added for
reference.

Some functions were moved from X509_LOOKUP_meth_new.pod

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/10986)

Fix podchecker warnings

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11045)

Make util/find-doc-nits runnable from the build tree

Because we generate an increasing number of POD files, some of them
end up in the build tree.  This makes it difficult for find-doc-nits
to work as desired when the build tree is separate from the source
tree.

The best supported way to make it work in such an environment is to
run it from the build tree and let it use the build information from
configdata.pm to find all the POD files.  To make this smooth enough,
we add a function 'files' that returns an array of file names
corresponding to criteria from the caller.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11045)

ffc: use sizeof(*pointer) instead of sizeof(struct) in memset(3) call.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11101)

Coverity 1458438: fix uninitialised memory access.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11101)

Coverity 1458439: fix resource leak

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11101)

chunk 7 of CMP contribution to OpenSSL

add CMP message validation and related tests; while doing so:
* add ERR_add_error_mem_bio() to crypto/err/err_prn.c
* move ossl_cmp_add_error_txt() as ERR_add_error_txt() to crypto/err/err_prn.c
* add X509_STORE_CTX_print_verify_cb() to crypto/x509/t_x509.c,
  adding internally x509_print_ex_brief(), print_certs(), and print_store_certs()
* move {ossl_cmp_,}X509_STORE_get1_certs() to crypto/x509/x509_lu.c

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/10620)

fix various formatting nits in CMP contribution chunks 1-6 found by the new util/check-format.pl

in addition:
correct wording in doc, comments, and parameter names: self-signed -> self-issued where appropriate

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/10620)

improve CMP logging according to comments on CMP chunk 7 preview

in particular:
consolidate documentation of CMP logging and error reporting functions
fix compilation problem with clang on some platforms
rename OSSL_CMP_log etc. to ossl_cmp_log etc. since these macros are CMP-internal
move chopping of trailing separator to ossl_cmp_add_error_txt(), also fix handling of leading separator
internalize X509_print_ex_brief() as x509_print_ex_brief()

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/10620)

add internal doc files actually belonging to CMP contribution chunk 6

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/10620)

Also check for errors in x86_64-xlate.pl.

In https://github.com/openssl/openssl/pull/10883, I'd meant to exclude
the perlasm drivers since they aren't opening pipes and do not
particularly need it, but I only noticed x86_64-xlate.pl, so
arm-xlate.pl and ppc-xlate.pl got the change.

That seems to have been fine, so be consistent and also apply the change
to x86_64-xlate.pl. Checking for errors is generally a good idea.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: David Benjamin <davidben@google.com>
(Merged from https://github.com/openssl/openssl/pull/10930)

Add $debug variable and use it

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10852)

Fix a bug for aarch64 BigEndian
FIXES #10692 #10638
a bug for aarch64 bigendian with instructions 'st1' and 'ld1' on AES-GCM mode.

CLA: trivial

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10751)

crypto/bn: fix debug build

Fixes:

    gcc  -I. -Iinclude -Iproviders/common/include -Iproviders/implementations/include -Icrypto/include  -DSTATIC_LEGACY -DBN_DEBUG -DREF_DEBUG     -DCONF_DEBUG -DBN_CTX_DEBUG -DOPENSSL_NO_ASM -ggdb -g2 -Wformat -Wshadow -Wmissing-prototypes -Wmissing-declarations -Werror      -DOPENSSL_BUILDING_OPENSSL -DOPENSSLDIR="\"/home/wsh/opt/openssl/ssl\"" -DENGINESDIR="\"/home/wsh/opt/openssl/lib/engines-3\""     -DMODULESDIR="\"/home/wsh/opt/openssl/lib/ossl-modules\""   -MMD -MF crypto/bn/libcrypto-lib-bn_conv.d.tmp -MT crypto/bn/libcrypto-lib-bn_conv.o -c -o crypto/bn/libcrypto-lib-bn_conv.o crypto/bn/bn_conv.c
    In file included from crypto/bn/bn_conv.c:12:
    crypto/bn/bn_conv.c: In function 'BN_hex2bn':
    crypto/bn/bn_local.h:178:31: error: implicit declaration of function 'ossl_assert'; did you mean 'ossl_ssize_t'? [-Werror=implicit-function-declaration]
                             (void)ossl_assert((_top == 0 && !_bnum2->neg) || \
                                   ^~~~~~~~~~~
    crypto/bn/bn_conv.c:188:5: note: in expansion of macro 'bn_check_top'
         bn_check_top(ret);
         ^~~~~~~~~~~~

CLA: trivial

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10993)

Add FFC param/key validation

Embed libctx in dsa and dh objects and cleanup internal methods to not pass libctx (This makes it consistent with the rsa changes)

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10910)

x86_64: Add endbranch at function entries for Intel CET

To support Intel CET, all indirect branch targets must start with
endbranch.  Here is a patch to add endbranch to function entries
in x86_64 assembly codes which are indirect branch targets as
discovered by running openssl testsuite on Intel CET machine and
visual inspection.

Verified with

$ CC="gcc -Wl,-z,cet-report=error" ./Configure shared linux-x86_64 -fcf-protection
$ make
$ make test

and

$ CC="gcc -mx32 -Wl,-z,cet-report=error" ./Configure shared linux-x32 -fcf-protection
$ make
$ make test # <<< passed with https://github.com/openssl/openssl/pull/10988

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10982)

include/openssl/whrlpool.h: correct unbalanced deprecation guards

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11087)

DOC: document in more detail what a BIO_read_ex() via BIO_f_buffer() does

The BIO_f_buffer() documentation tells in enough detail how it affects
BIO_gets(), but not how it affects BIO_read_ex().  This change
remedies that.

Fixes #10859

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10890)

Document OSSL_SERIALIZER_PUBKEY_TO_DER_PQ and friends

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11071)

PROV: Build the main FIPS module code with FIPS_MODE defined

Without that, its main source wasn't compiled correctly.

Fixes #11089

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/11090)

apps x509: passing PKCS#11 URL as -signkey

OpenSSL 1.1.0 has extended option checking, and rejects passing a PKCS#11
engine URL to "-signkey" option. The actual code is ready to take it.

Change the option parsing to allow an engine URL to be passed and modify
the manpage accordingly.

CLA: trivial

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/11086)

Make secure-memory be a config option

Adding support for "no-secure-memory" was simple, a one-liner.
Fixing all the "ifdef OPENSSL_SECURE_MEMORY" to be "ifndef NO_xxx"
was a bit more work.  My original goof, for not following the OpenSSL
pattern "ifndef NO_" used everywhere else.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/11023)

Adapt i2d_PUBKEY for provider only keys

It uses EVP_PKEY serializers to get the desired results.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11078)

Adapt i2d_PrivateKey for provider only keys

It uses EVP_PKEY serializers to get the desired results.

One might think that ddoing this might make things a bit dicy for
existing serializers, as they should obviously use i2d functions.
However, since our serializers use much more primitive functions
such as i2d_ASN1_INTEGER(), or keytype specific ones such as
i2d_RSAPrivateKey(), there is no clash.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11056)

Forgotten GOST2012 support in non-vital places

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/11072)

dsatest: fix the non-deprecated version of the test run.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11081)

Apps: don't build deprecated DH and DSA apps.

This also means that there doesn't need to be any conditional checks in the .c
files to avoid deprecated builds.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11067)

Fix coverity issues

CID : 1458169 RESOURCE LEAK ffc_params_generate.c - False positive, but addressed another Leak in adjacent code
CID : 1458171 UNCHECKED RET apps/pkeyutl.c
CID : 1458173 DEAD CODE ffc_params_generate.c
CID : 1458174 RESOURCE LEAK ssl_lib.c

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/11048)

x86_64: Always generate .note.gnu.property section for ELF outputs

We should always generate .note.gnu.property section in x86_64 assembly
codes for ELF outputs to mark Intel CET support since all input files
must be marked with Intel CET support in order for linker to mark output
with Intel CET support.  Also .note.gnu.property section in x32 should
be aligned to 4 bytes, not 8 bytes and .p2align should be used
consistently.

Verified with

$ CC="gcc -Wl,-z,cet-report=error" ./Configure shared linux-x86_64 -fcf-protection
$ make
$ make test

and

$ CC="gcc -mx32 -Wl,-z,cet-report=error" ./Configure shared linux-x32 -fcf-protection
$ make
$ make test # <<< 90-test_sslapi.t failed because 8-byte pointer size.

Fix #10896

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/10985)

Fix a mem leak in libssl

Make sure we free up any ENGINE references after we have finished using
them.

Fixes #11064

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11070)

Add Asymmetric RSA cipher tests in FIPS provider

We also add some DSA tests for code that is already in the FIPS provider.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10881)

Make the RSA ASYM_CIPHER implementation available inside the FIPS module

RSA ASYM_CIPHER was already available within the default provider. We
now make it also available from inside the FIPS module.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10881)

Add assembly config targets for UEFI build

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11054)

Fix no-ec build

Don't attempt to build ecx related source files in a "no-ec" build.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11065)

Add OSSL_SERIALIZER_PUBKEY_TO_DER_PQ and friends

It's already used internally, there's no reason the DER serializer
propqueries shouldn't be present alongside the PEM and TEXT ones.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11055)

fix build for new HTTP client in case OPENSSL_NO_CMP or OPENSSL_NO_OCSP

fix also formatting nits w.r.t. #if indentations in ocsp.h

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: David von Oheimb <david.von.oheimb@siemens.com>
(Merged from https://github.com/openssl/openssl/pull/11058)

Remove unused ossl_param_bld_to_param_ex() function.

The recently introduced ossl_param_bld_to_param_ex() function is only
called by the unit tests.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11053)

Remove unused OSSL_PARAM_construct_from_text() function.

This function is recently introduced and never called by the library or tests.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11053)

Add NEWS entry about deprecation of command line public tools

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10977)

dsa: deprecate applications that depend on the low level DSA functions.

speed is updated to not support DSA instead of being removed.

The dhparam, dsaparam, dsa and gendsa commands are deprecated but still
exist without NO_DEPRECATED defined.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10977)

app: add a deprecation warning to all deprecated commands.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10977)

test/recipes/80-test_ssl_old.t: Replace 'openssl gendsa'

Use 'openssl genpkey' instead.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10977)

test_dsa: fix deprecation logic

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10977)

test/recipes/15-test_dsa.t: Deal with deprecation of 'openssl dsa'

Do not run programs that depend on deprecated APIs when
'no-deprecated' is configured.

We still retain the conversion tests that use 'openssl pkey', and add
the one that's missing.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10977)

Deprecate the low level DSA functions.

Use of the low level DSA functions has been informally discouraged for a
long time. We now formally deprecate them.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10977)

dsa.h: fix preprocessor indentation

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10977)

DSA: fix the DSA parameter logic in test.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10977)

Add S390 support for provider based X25519/X448

Reviewed-by: Patrick Steuer <patrick.steuer@de.ibm.com>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10964)

Add X25519/X448 Key Exchange to the default provider

Reviewed-by: Patrick Steuer <patrick.steuer@de.ibm.com>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10964)

Implement Provider side Key Management for X25519 and X448

Reviewed-by: Patrick Steuer <patrick.steuer@de.ibm.com>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10964)

Implement a stricter ECX_KEY type

Add ref counting and control how we allocate storage for the private key.
We will need this type in following commits where we move the ecx code
to be provider aware.

Reviewed-by: Patrick Steuer <patrick.steuer@de.ibm.com>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10964)

Check that ed25519 and ed448 are allowed by the security level

Signature algorithms not using an MD weren't checked that they're
allowed by the security level.

Reviewed-by: Matt Caswell <matt@openssl.org>
GH: #10785

Generate new Ed488 certificates

Create a whole chain of Ed488 certificates so that we can use it at security
level 4 (192 bit). We had an 2048 bit RSA (112 bit, level 2) root sign the
Ed488 certificate using SHA256 (128 bit, level 3).

Reviewed-by: Matt Caswell <matt@openssl.org>
GH: #10785

Add a minimal build target for Travis and Appveyor

[extended tests]

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9982)

X509_PUBKEY_set(): Fix memory leak

With the provided method of creating the new X509_PUBKEY, an extra
EVP_PKEY is created and needs to be properly cleaned away.

(note: we could choose to keep it just as well, but there are
consequences, explained in a comment in the code)

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/11038)

Generalize the HTTP client so far implemented mostly in crypto/ocsp/ocsp_ht.c
The new client has become an independent libcrpyto module in crypto/http/ and
* can handle any types of requests and responses (ASN.1-encoded and plain)
* does not include potentially busy loops when waiting for responses but
* makes use of a new timeout mechanism integrated with socket-based BIO
* supports the use of HTTP proxies and TLS, including HTTPS over proxies
* supports HTTP redirection via codes 301 and 302 for GET requests
* returns more useful diagnostics in various error situations
Also adapts - and strongly simplifies - hitherto uses of HTTP in crypto/ocsp/,
crypto/x509/x_all.c, apps/lib/apps.c, and apps/{ocsp,s_client,s_server}.c

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: David von Oheimb <david.von.oheimb@siemens.com>
(Merged from https://github.com/openssl/openssl/pull/10667)

add BIO_socket_wait(), BIO_wait(), and BIO_connect_retry() improving timeout support

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: David von Oheimb <david.von.oheimb@siemens.com>
(Merged from https://github.com/openssl/openssl/pull/10667)

PROV: Ensure the AlgorithmIdentifier registers in DSA signature impl

When setting up the hash function for DSA signature, the encoded
AlgorithmIdentifier for the DSA+hash combination is queried, but not
stored, which leads to problems when signing ASN.1 items in libcrypto.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11037)

Memory allocator code cleanup

Typedefs of CRYPTO malloc, realloc and free.

MEM_CHECK "modes" are used only as a CRYPTO_mem_ctrl() parameter

The CRYPTO_mem_ctrl is defined only if OPENSSL_NO_CRYPTO_MDEBUG is
defined, thus define the MEM_CHECK modes under the same condition.

Maybe the macros can be removed at all since:
1. CRYPTO_mem_ctrl() just returns -1 and ignores the parameter
2. CRYPTO_mem_ctr() is declared as DEPRECATED by 3.0

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11042)

x86: Always generate .note.gnu.property section for ELF outputs

We should always generate .note.gnu.property section in x86 assembly
codes for ELF outputs to mark Intel CET support since all input files
must be marked with Intel CET support in order for linker to mark output
with Intel CET support.

Verified with

$ CC="gcc -Wl,-z,cet-report=error" ./Configure shared linux-x86 -fcf-protection
$ make
$ make test

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11044)

Don't compile AESNI code if we're not AESNI capable

Compile failures were occuring on systems that weren't AESNI capable
because the detection wasn't quite right in a couple of files.

This fixes a run-checker build failure for the 386 compile option.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11022)

Fix no-des

Don't attempt to self-test DES in the FIPS provider if we have been built
without FIPS support.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11034)

Fix no-engine

We don't need to check if an engine has a cipher/digest in a no-engine
build.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/11031)

Fix no-dh

Don't use DH specific macros that might need to be used in a no-dh build.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/11030)