doc: use /* ... */ comments in code examples

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1956)

doc/man3: fix SSL_SESSSION typos

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1956)

SSL_CTX_set_verify.pod: move a typedef in front of its first usage

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1956)

doc/man3: unindent a few unintended code blocks

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1956)

doc/man3: reformat the function prototypes in the synopses

I tried hard to keep the lines at 80 characters or less, but in a few
cases I had to punt and just indented the subsequent lines by 4 spaces.

A few well-placed typedefs for callback functions would really help, but
these would be part of the API, so that's probably for later.

I also took the liberty of inserting empty lines in overlong blocks to
provide some visual space.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1956)

ERR_put_error.pod: fix the name of function ERR_add_error_vdata()

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1956)

doc/man3: remove a duplicate BIO_do_accept() call

The SSL server example in BIO_f_ssl.pod contains two copies of the
BIO_do_accept() call. Remove the second one.

Signed-off-by: Beat Bolli <dev@drbeat.li>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1956)

doc/man3: use the documented coding style in the example code

Adjust brace placement, whitespace after keywords, indentation and empty
lines after variable declarations according to
https://www.openssl.org/policies/codingstyle.html.

Indent literal sections by exactly one space.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1956)

make error tables const and separate header file

Run perltidy on util/mkerr
Change some mkerr flags, write some doc comments
Make generated tables "const" when genearting lib-internal ones.
Add "state" file for mkerr
Renerate error tables and headers
Rationalize declaration of ERR_load_XXX_strings
Fix out-of-tree build
Add -static; sort flags/vars for options.
Also tweak code output
Moved engines/afalg to engines (from master)
Use -static flag
Standard engine #include's of errors
Don't linewrap err string tables unless necessary

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3392)

Add a lock around the OBJ_NAME table

Various initialization functions modify this table, which can cause heap
corruption in the absence of external synchronization.

Some stats are modified from OPENSSL_LH_retrieve, where callers aren't
expecting to have to take out an exclusive lock. Switch to using atomic
operations for those stats.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3525)

Fix #2400 Add NO_RENEGOTIATE option

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3432)

Always flush the BIO when we send any alert

At the moment we flush the write BIO if we send a fatal alert, but not a
warning one. This can mean the warning is never sent if we never do another
write and subsequently flush the BIO. Instead we should just always flush
after writing an alert.

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3432)

Handle the server refusing to reneg in a reneg_setup

During setup of a reneg test the server can refuse to start reneg.
If that happens we should let the client continue and then fail.

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3432)

sha/asm/keccak1600-armv4.pl: add NEON code path.

Reviewed-by: Rich Salz <rsalz@openssl.org>

sha/asm/keccak1600-armv4.pl: add SHA3_absorb and SHA3_squeeze.

Reviewed-by: Rich Salz <rsalz@openssl.org>

sha/asm/keccak1600-armv4.pl: optimization based on profiler feedback.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add sha/asm/keccak1600-armv4.pl.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix coding style

Signed-off-by: Paul Yang <paulyang.inf@gmail.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3614)

Fix doc nits in X509_check_private_key.pod

remove the tailing dot

Signed-off-by: Paul Yang <paulyang.inf@gmail.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3614)

Set local variable to 0 to avoid build error

Signed-off-by: Paul Yang <paulyang.inf@gmail.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3614)

Add test cases for X509_check_private_key

To test X509_check_private_key and relatives.

Add a CSR and corresponding RSA private key to test
X509_REQ_check_private_key function.

Signed-off-by: Paul Yang <paulyang.inf@gmail.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3614)

Document X509_check_private_key and relative

Document two private key check functions:

X509_check_private_key
X509_REQ_check_private_key

Signed-off-by: Paul Yang <paulyang.inf@gmail.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3614)

Move and update RSA-PSS documentation.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3621)

Fix define to match documentation

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3621)

Document default client -psk_identity

Document that -psk is required to use PSK cipher
[skip ci]

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/3607)

Avoid failing s_server when client's psk_identity is unexpected

s_server has traditionally been very brittle in PSK mode.  If the
client offered any PSK identity other than "Client_identity" s_server
would simply abort.

This is breakage for breakage's sake, and unlike most other parts of
s_server, which tend to allow more flexible connections.

This change accomplishes two things:

 * when the client's psk_identity does *not* match the identity
   expected by the server, just warn, don't fail.

 * allow the server to expect instead a different psk_identity from
   the client besides "Client_identity"

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Reviewed-by: Ben Kaduk <kaduk@mit.edu>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3605)

Use common STANZA for bn and exp tests

Standardize file:line messages
Reduce buff size; move to end of STANZA
Add some Title entries (with blank line after)
Add Title to some BN test files.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3608)

sha/keccak1600.c: add #ifdef KECCAK1600_ASM.

Reviewed-by: Rich Salz <rsalz@openssl.org>

sha/keccak1600.c: reduce temporary storage utilization even futher.

Reviewed-by: Rich Salz <rsalz@openssl.org>

sha/keccak1600.c: add another 1x variant.

Reviewed-by: Rich Salz <rsalz@openssl.org>

sha/keccak1600.c: add ARM-specific "reference" tweaks.

Reviewed-by: Rich Salz <rsalz@openssl.org>

ec/asm/ecp_nistz256-x86_64.pl: minor sqr_montx cleanup.

Drop some redundant instructions in reduction in ecp_nistz256_sqr_montx.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix ex_data and session_dup issues

Code was added in commit b3c31a65 that overwrote the last ex_data value
using CRYPTO_dup_ex_data() causing a memory leak, and potentially
confusing the ex_data dup() callback.

In ssl_session_dup(), fix error handling (properly reference and up-ref
shared data) and new-up the ex_data before calling CRYPTO_dup_ex_data();
all other structures that dup ex_data have the destination ex_data new'd
before the dup.

Fix up some of the ex_data documentation.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3323)

Add text pointing to full change list.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3606)

Work around Travis "virtual memory exhausted" error

One particular build was running out of memory. By swapping to debug mode
we reduce the optimisation level which should reduce the amount of memory
required.

[extended tests]

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3601)

Left shift of a negative number is undefined behaviour

Fix undefined behaviour in curve25519.c. Prior to this running with
ubsan produces errors like this:

crypto/ec/curve25519.c:3871:18: runtime error: left shift of negative
value -22867

[extended tests]

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3600)

Added mysql as starttls protocol.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3456)

Remove unnecessary if condition from apps.c

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3596)

Add -p (public only) flag to find-doc-nits

Report if any non-public items are documented.
Add util/private.num that lists items that aren't in the public
(lib*.num) files that we do want to document.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3603)

Only release thread-local key if we created it.

Thanks to Jan Alexander Steffens for finding the bug and confirming the
fix.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3592)

Add some OID's from X.520

CLA: trivial

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3590)

Add UI functionality to duplicate the user data

This can be used by engines that need to retain the data for a longer time
than just the call where this user data is passed.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3575)

Fix coding style in apps/passwd file

Reformat some indents and braces based on OpenSSL coding style spec.

Signed-off-by: Paul Yang <paulyang.inf@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3586)

CLA: trivial

Fix coding style

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3581)

Wait longer for the server in TLSProxy to start

In a recent PR (#3566) it seems that TLSProxy gave up trying to connect to
the server process too quickly. This meant the test failed even though the
server *did* eventually start. Currently we try 3 times to connect with a
0.1 second pause between each attempt. That is probably too aggressive.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3587)

Clear sensitive data in ED25519_sign

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Add Ed25519 documentation

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

no-ec fix

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Add Ed25519 verify test.

Add Ed25519 certificate verify test using certificate from
draft-ietf-curdle-pkix-04 and custom generated root certificate.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Add custom sig_info_set for ED25519

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

make errors

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Add custom ASN.1 sign and verify

Since ED25519 doesn't have an associated digest it needs custom sign/verify
routines to handle ASN.1 signatures.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Add ED25519 as signature OID

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Add RFC8032 tests and additional cases

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

make errors

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

ED25519 public key method.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

add method

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Add ED25519 ASN.1 method

Make X25519 key method more flexible by removing hard coding of NID_X25519
OID. Since the parameters and key syntax between ED25519 and X25519 are
almost identical they can share a lot of common code.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Add EdDSA algorithm OIDs from draft-ietf-curdle-pkix-04

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Make Ed25519 consistent with X25519

Rename and change ED25519_keypair_from_seed to ED25519_public_from_private
to be consistent with X25519 API.

Modidy ED25519_sign to take separate public key argument instead of
requiring it to follow the private key.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Add Ed25519 algorithm.

Reinstate Ed25519 algorithm to curv25519.c this is largely just a copy of
the code from BoringSSL with some adjustments so it compiles under OpenSSL.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

make errors

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Allow NULL md for custom signing methods

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

Add support for custom digestsign/digestverify methods.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3503)

sha/keccak1600.c: implement lane complementing transform

...as discussed in section 2.2 of "Keccak implementation overview".

[skip ci]

Reviewed-by: Rich Salz <rsalz@openssl.org>

sha/keccak1600.c: implement bit interleaving optimization.

This targets 32-bit processors and is discussed in section 2.1 of
"Keccak implementation overview".

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add stricter checking in NAME section

Require a comma between every name and a single space before the dash

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3559)

fixing incorrect OID of signingCertificateV2
Reviewed-by: Kurt Roeckx <kurt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3538)

Fix certificate version number in test

The version number 3 means version 4, while 2 means version 3. Since this is the v3nametest, version 3 should be used.

CLA: Trivial

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3577)

Revert "Add internal functions to fetch a refcount"

It turned out to be a bad idea.

This reverts commits 6891a79da67ccd621b67e49b60ddc188d7864291
and c27bc74698ed043b7549d5637ec0a8cf65b39e59.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3576)

Correct small typo in CRYPTO_GET_REF

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3574)

Add internal functions to fetch a refcount

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3573)

Fix spelling errors in CMS.

Unfortunately it affects error code macros in public cms.h header, for
which reason misspelled names are preserved for backward compatibility.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3463)

Fix a Proxy race condition

Issue #3562 describes a problem where a race condition can occur in the
Proxy such that a test "ok" line can appear in the middle of other text
causing the test harness to miss it. The issue is that we do not wait for
the client process to finish after the test is complete, so that process may
continue to write data to stdout/stderr at the same time that the test
harness does.

This commit fixes TLSProxy so that we always wait for the client process to
finish before continuing.

Fixes #3562

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3567)

Update the pyca-cryptography version

Commit 9bfeeef made some function parameters const. This actually broke
the pyca-cryptography tests. The discussion in #3360 considers this to
actually be a problem with pyca-cryptography not an OpenSSL issue (they
replicate some of our header file contents which then causes function
prototype mismatches). This commit updates the pyca-cryptography version
to pull in their fix for this issue and make our external tests pass again.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3569)

Fix inconsistent check of UNSAFE_LEGACY_RENEGOTIATION

The check for SSL3_FLAGS_ALLOW_UNSAFE_LEGACY_RENEGOTIATION is
inconsistent. Most places check SSL->options, one place is checking
SSL_CTX->options; fix that.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
GH: #3523

Fix use of "can_load()" in run_tests.pl.

CLA: Trivial

Fixes #3563.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3564)

test/evp_test.c: improve output in case of errors.

Recently introduced TEST_* macros print variables' symbolic names.
In order to make error output more readable rename some variables.

Reviewed-by: Rich Salz <rsalz@openssl.org>

modes/ocb128.c: address undefined behaviour warning.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3526)

aria/aria.c: address undefined behaviour warning in small-footprint path.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3526)

Add missing commas in pod files

Reviewed-by: Rich Salz <rsalz@openssl.org>
GH: #3557

Fix BoringSSL alert related test failures

Commit bd990e2535 changed our handling of alerts. Some of the BoringSSl
tests were expecting specific errors to be created if bad alerts were sent.
Those errors have now changed as a result of that commit, so the BoringSSL
test config needs to be updated to match.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3549)

Document that HMAC() with a NULL md is not thread safe

Fixes #3541

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3553)

Updates CHANGES and NEWS for new release

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3546)

Document that PKCS#12 functions assume UTF-8 for passwords

Part of issue #3531

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3535)

Clarify what character encoding is used in the returned UI strings

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3532)

Fix va_arg all in test_error_c90

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3506)

Revise evp_test parser; make like bn_test

Re-order and comment on the functions

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3506)

Add titles to groups of EVP tests

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3529)

Fix line numbering for errors.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3528)

Don't use one shot API for SSLv3.

SSLv3 (specifically with client auth) cannot use one shot APIs: the digested
data and the master secret are handled in separate update operations. So
in the special case of SSLv3 use the streaming API.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3527)

Fix typo in doc/man3/EVP_EncrypInit.pod

In the example section.

CLA: trivial

Signed-off-by: Paul Yang <paulyang.inf@gmail.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3520)

Revert "Integration build a small memory image"

This reverts commit e2580e70d5c644c5dadd33ffa8bb23223959b87a.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
(Updates https://github.com/openssl/openssl/pull/3268)

Add an error to the stack on failure in dtls1_write_bytes()

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3496)

Convert existing usage of assert() to ossl_assert() in libssl

Provides consistent output and approach.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3496)

Fix some style issues in returns

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3496)

Replace instances of OPENSSL_assert() with soft asserts in libssl

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3496)

Add a macro for testing assertion in both debug and production builds

If we have an assert then in a debug build we want an abort() to occur.
In a production build we want the function to return an error.

This introduces a new macro to assist with that. The idea is to replace
existing use of OPENSSL_assert() with this new macro. The problem with
OPENSSL_assert() is that it aborts() on an assertion failure in both debug
and production builds. It should never be a library's decision to abort a
process (we don't get to decide when to kill the life support machine or
the nuclear reactor control system). Additionally if an attacker can
cause a reachable assert to be hit then this can be a source of DoS attacks
e.g. see CVE-2017-3733, CVE-2015-0293, CVE-2011-4577 and CVE-2002-1568.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3496)

Reformat the string output to be more in line with the decisions made in #3465

Don't highlight excess when comparing unequal length strings.

Clean up the NULL / empty string display.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3514)

Bring the memory output inline with the suggestions in #3465.

Excess bytes, when one block is longer than the other, are not explicitly
highlighted.

The NULL / zero length block output has been cleaned up.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3515)