Limit depth of nested sequences when generating ASN.1

Reported by Hanno Böck <hanno@hboeck.de>
PR#3800

Reviewed-by: Rich Salz <rsalz@openssl.org>

Remove obsolete options for debug-steve*

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add -Wtype-limits to strict warnings.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Initialize variable

newsig may be used (freed) uninitialized on a malloc error.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix ssl_get_prev_session overrun

If OpenSSL is configured with no-tlsext then ssl_get_prev_session can read
past the end of the ClientHello message if the session_id length in the
ClientHello is invalid. This should not cause any security issues since the
underlying buffer is 16k in size. It should never be possible to overrun by
that many bytes.

This is probably made redundant by the previous commit - but you can never be
too careful.

With thanks to Qinghao Tang for reporting this issue.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Check for ClientHello message overruns

The ClientHello processing is insufficiently rigorous in its checks to make
sure that we don't read past the end of the message. This does not have
security implications due to the size of the underlying buffer - but still
needs to be fixed.

With thanks to Qinghao Tang for reporting this issue.

Reviewed-by: Rich Salz <rsalz@openssl.org>

free NULL cleanup 9

Ongoing work to skip NULL check before calling free routine.  This gets:
    ecp_nistz256_pre_comp_free nistp224_pre_comp_free nistp256_pre_comp_free
    nistp521_pre_comp_free PKCS7_free PKCS7_RECIP_INFO_free
    PKCS7_SIGNER_INFO_free sk_PKCS7_pop_free PKCS8_PRIV_KEY_INFO_free
    PKCS12_free PKCS12_SAFEBAG_free PKCS12_free sk_PKCS12_SAFEBAG_pop_free
    SSL_CONF_CTX_free SSL_CTX_free SSL_SESSION_free SSL_free ssl_cert_free
    ssl_sess_cert_free

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Fix memory leak

It should have freed them when != NULL, not when == NULL.

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Viktor Dukhovni <openssl-users@dukhovni.org>

do_dirname: Don't change gen on failures

It would set gen->d.dirn to a freed pointer in case X509V3_NAME_from_section
failed.

Reviewed-by: Rich Salz <rsalz@openssl.org>

X509_VERIFY_PARAM_free: Check param for NULL

Reviewed-by: Viktor Dukhovni <openssl-users@dukhovni.org>

free NULL cleanup 10

Avoid checking for NULL before calling free functions.  This gets
ssl.*free:
    ssl_sess_cert_free ssl_free ssl_excert_free ssl_cert_free
    SSL_free SSL_SRP_CTX_free SSL_SESSION_free SSL_CTX_free
    SSL_CTX_SRP_CTX_free SSL_CONF_CTX_free

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

The wrong ifdef is used to guard usage of PSK code

PR#3790

Reviewed-by: Stephen Henson <steve@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>

Don't set *pval to NULL in ASN1_item_ex_new.

While *pval is usually a pointer in rare circumstances it can be a long
value. One some platforms (e.g. WIN64) where
sizeof(long) < sizeof(ASN1_VALUE *) this will write past the field.

*pval is initialised correctly in the rest of ASN1_item_ex_new so setting it
to NULL is unecessary anyway.

Thanks to Julien Kauffmann for reporting this issue.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix ECDH detection, add ECDH keyid test.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix ECDH key identifier support.

PR#3789

Reviewed-by: Rich Salz <rsalz@openssl.org>

Polish shell script to avoid needless complexity.

No need for here documents, just use "yes" or </dev/null.
No need for "|| exit 1" clauses, just use "set -e".

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

fix to "test script cleanup"

Fix commit 30f54ad295d58ff8c6d28c1fd612d23c2c343d19 which used
non-portable syntax for checking exit status.

Reviewed-by: Matt Caswell <matt@openssl.org>

Fix read_ahead issue

Fix a "&" that should have been "!" when processing read_ahead.

RT#3793

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

test script cleanup

Removed commented-out tests

Standardize on doing
        cmd ... || exit 1
instead of
        cmd ...
        if [ $? != 0] ; then
           exit 1
        fi
where that if statement has ben one, three, or four lines, variously.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Have mkerr.pl treat already existing multiline string defs properly

Since source reformat, we ended up with some error reason string
definitions that spanned two lines.  That in itself is fine, but we
sometimes edited them to provide better strings than what could be
automatically determined from the reason macro, for example:

    {ERR_REASON(SSL_R_NO_GOST_CERTIFICATE_SENT_BY_PEER),
     "Peer haven't sent GOST certificate, required for selected ciphersuite"},

However, mkerr.pl didn't treat those two-line definitions right, and
they ended up being retranslated to whatever the macro name would
indicate, for example:

    {ERR_REASON(SSL_R_NO_GOST_CERTIFICATE_SENT_BY_PEER),
     "No gost certificate sent by peer"},

Clearly not what we wanted.  This change fixes this problem.

Reviewed-by: Matt Caswell <matt@openssl.org>

Drop CA.sh for CA.pl

Remove CA.sh script and use CA.pl for testing, etc.

Reviewed-by: Richard Levitte <levitte@openssl.org>

consistent test-start logging

Output a consistent "start" marker for each test.
Remove "2>/dev/null" from Makefile command lines.
Add OPENSSL_CONFIG=/dev/null for places where it's needed, in
order to suppress a warning message from the openssl CLI.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Ignore the non-dll windows specific build directories

Reviewed-by: Rich Salz <rsalz@openssl.org>

Appease clang -Wshadow

The macros BSWAP4 and BSWAP8 have statetemnt expressions
implementations that use local variable names that shadow variables
outside the macro call, generating warnings like this

e_aes_cbc_hmac_sha1.c:263:14: warning: declaration shadows a local variable
      [-Wshadow]
    seqnum = BSWAP8(blocks[0].q[0]);
             ^
../modes/modes_lcl.h:41:29: note: expanded from macro 'BSWAP8'
                            ^
e_aes_cbc_hmac_sha1.c:223:12: note: previous declaration is here
    size_t ret = 0;
           ^

Have clang be quiet by modifying the macro variable names slightly
(suffixing them with an underscore).

Reviewed-by: Rich Salz <rsalz@openssl.org>

Appease clang -Wgnu-statement-expression

We use GNU statement expressions in crypto/md32_common.h, surrounded
by checks that GNU C is indeed used to compile.  It seems that clang,
at least on Linux, pretends to be GNU C, therefore finds the statement
expressions and then warns about them.

The solution is to have clang be quiet about it.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Appease clang -Wempty-translation-unit

ebcdic.c:284:7: warning: ISO C requires a translation unit to contain at least one
      declaration [-Wempty-translation-unit]
      ^
1 warning generated.

Reviewed-by: Rich Salz <rsalz@openssl.org>

update ordinals

Reviewed-by: Rich Salz <rsalz@openssl.org>

make depend

Reviewed-by: Rich Salz <rsalz@openssl.org>

remove asn1_mac.h

Reviewed-by: Rich Salz <rsalz@openssl.org>

Remove old ASN.1 functions.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Remove unnecessary use of ASN1_const_CTX

Reviewed-by: Rich Salz <rsalz@openssl.org>

Rewrite ssl_asn1.c using new ASN.1 code.

Complete reimplementation of d2i_SSL_SESSION and i2d_SSL_SESSION using
new ASN.1 code and eliminating use of old ASN.1 macros.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add macro to implement static encode functions.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fewer newlines in comp method output

Print "supported compression methods" all on one line.

Reviewed-by: Andy Polyakov <appro@openssl.org>

modes/asm/ghashv8-armx.pl: up to 90% performance improvement.

Reviewed-by: Matt Caswell <matt@openssl.org>

sha/asm/sha*-armv8.pl: add Denver and X-Gene esults.

Reviewed-by: Richard Levitte <levitte@openssl.org>

aes/asm/aesv8-armx.pl: optimize for Cortex-A5x.

ARM has optimized Cortex-A5x pipeline to favour pairs of complementary
AES instructions. While modified code improves performance of post-r0p0
Cortex-A53 performance by >40% (for CBC decrypt and CTR), it hurts
original r0p0. We favour later revisions, because one can't prevent
future from coming. Improvement on post-r0p0 Cortex-A57 exceeds 50%,
while new code is not slower on r0p0, or Apple A7 for that matter.

[Update even SHA results for latest Cortex-A53.]

Reviewed-by: Richard Levitte <levitte@openssl.org>

perlasm/arm-xlate.pl update (fix end-less loop and prepare for 32-bit iOS).

Reviewed-by: Rich Salz <rsalz@openssl.org>

Configure: android-arm facelift.

Reviewed-by: Rich Salz <rsalz@openssl.org>

make update

Reviewed-by: Matt Caswell <matt@openssl.org>

Remove SSL_TASK, the DECnet Based SSL Engine - addendum

A bit of cleanup was forgotten.

Reviewed-by: Tim Hudson <tjh@openssl.org>

Remove SSL_TASK, the DECnet Based SSL Engine

This engine is for VMS only, and isn't really part of the core OpenSSL
but rather a side project of its own that just happens to have tagged
along for a long time.  The reasons why it has remained within the
OpenSSL source are long lost in history, and there not being any real
reason for it to remain here, it's time for it to move out.

This side project will appear as a project in its own right, the
location of which will be announced later on.

Reviewed-by: Tim Hudson <tjh@openssl.org>

Remove old ASN.1 code from evp_asn1.c

Rewrite ASN1_TYPE_set_int_octetstring and ASN1_TYPE_get_int_octetstring
to use the new ASN.1 code instead of the old macros.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Now that we've removed the need for symlinks, we can safely remove util/mklinks.pl

Reviewed-by: Rich Salz <rsalz@openssl.org>

Remove remaining variables for symlinked/copied headers and tests

GitConfigure:   no more 'no-symlinks'

util/bat.sh, util/mk1mf.pl, util/pl/VC-32.pl, util/pl/unix.pl:
- Remove all uses of EXHEADER.
  That includes removing the use if INC_D and INCO_D.
- Replace the check for TEST with a check for [A-Z0-9_]*TEST.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Remove EXHEADER, TEST, APPS, links:, install: and uninstall: where relevant

With no more symlinks, there's no need for those variables, or the links
target.  This also goes for all install: and uninstall: targets that do
nothing but copy $(EXHEADER) files, since that's now taken care of by the
top Makefile.

Also, removed METHTEST from test/Makefile.  It looks like an old test that's
forgotten...

Reviewed-by: Rich Salz <rsalz@openssl.org>

Stop symlinking, move files to intended directory

Rather than making include/openssl/foo.h a symlink to
crypto/foo/foo.h, this change moves the file to include/openssl/foo.h
once and for all.

Likewise, move crypto/foo/footest.c to test/footest.c, instead of
symlinking it there.

Originally-by: Geoff Thorpe <geoff@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>

Ensure EC private keys retain leading zeros

RFC5915 requires the use of the I2OSP primitive as defined in RFC3447
for storing an EC Private Key. This converts the private key into an
OCTETSTRING and retains any leading zeros. This commit ensures that those
leading zeros are present if required.

Reviewed-by: Andy Polyakov <appro@openssl.org>

Clean up record layer

Fix up various things that were missed during the record layer work. All
instances where we are breaking the encapsulation rules.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix record layer "make clean"

The "clean" target in libssl has been updated to handle the new record
layer sub-directory.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix some faults in util/mk1mf.pl

When building on Unix, there are times when the 'EX_LIB' MINFO variable
contains valuable information.  Make sure to take care of it.

fixrules in util/pl/unix.pl was previously changed with a simpler fix of
rules, with a comment claiming that's compatible with -j.  Unfortunately,
this breaks multiline rules and doesn't change anything for single line
rules.  While at it, do not prefix pure echo lines with a 'cd $(TEST_D) &&',
as that's rather silly.

Reviewed-by: Andy Polyakov <appro@openssl.org>

Remove duplicate code.

Update code to use ASN1_TYPE_pack_sequence and ASN1_TYPE_unpack_sequence
instead of performing the same operation manually.

Reviewed-by: Rich Salz <rsalz@openssl.org>

New ASN1_TYPE SEQUENCE functions.

Add new functions ASN1_TYPE_pack_sequence and ASN1_TYPE_unpack_sequence:
these encode and decode ASN.1 SEQUENCE using an ASN1_TYPE structure.

Update ordinals.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Rewrite X509_PKEY_new to avoid old ASN1. macros.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Remove unnecessary asn1_mac.h includes.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Initialised 'ok' and redo the logic.

The logic with how 'ok' was calculated didn't quite convey what's "ok",
so the logic is slightly redone to make it less confusing.

Reviewed-by: Andy Polyakov <appro@openssl.org>

sha/asm/sha512-armv4.pl: adapt for use in Linux kernel context.

Follow-up to sha256-armv4.pl in cooperation with Ard Biesheuvel
(Linaro) and Sami Tolvanen (Google).

Reviewed-by: Rich Salz <rsalz@openssl.org>

sha/asm/sha256-armv4.pl: fix compile issue in kernel
and eliminate little-endian dependency.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Have a shared library version thats reasonable with our version scheme

The FAQ says this:

    After the release of OpenSSL 1.0.0 the versioning scheme changed. Letter
    releases (e.g. 1.0.1a) can only contain bug and security fixes and no
    new features. Minor releases change the last number (e.g. 1.0.2) and
    can contain new features that retain binary compatibility. Changes to
    the middle number are considered major releases and neither source nor
    binary compatibility is guaranteed.

With such a scheme (and with the thinking that it's nice if the shared
library version stays on track with the OpenSSL version), it's rather
futile to keep the minor release number in the shared library version.
The deed already done with OpenSSL 1.0.x can't be changed, but with
1.x.y, x=1 and on, 1.x as shared library version is sufficient.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Add private/public key conversion tests

Reviewed-by: Matt Caswell <matt@openssl.org>

Remove d2i_X509_PKEY and i2d_X509_PKEY

Remove partially implemented d2i_X509_PKEY and i2d_X509_PKEY: nothing
uses them and they don't work properly. Update ordinals.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

ec/asm/ecp_nistz256-x86_64.pl: update commentary with before-after performance data.

Reviewed-by: Richard Levitte <levitte@openssl.org>

free NULL cleanup

EVP_.*free; this gets:
        EVP_CIPHER_CTX_free EVP_PKEY_CTX_free EVP_PKEY_asn1_free
        EVP_PKEY_asn1_set_free EVP_PKEY_free EVP_PKEY_free_it
        EVP_PKEY_meth_free; and also EVP_CIPHER_CTX_cleanup

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Engage vpaes-armv8 module.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Add vpaes-amrv8.pl module.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Configure: remove unused variables.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Make asn1_ex_i2c, asn1_ex_c2i static.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Remove combine option from ASN.1 code.

Remove the combine option. This was used for compatibility with some
non standard behaviour in ancient versions of OpenSSL: specifically
the X509_ATTRIBUTE and DSAPublicKey handling. Since these have now
been revised it is no longer needed.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Simplify DSA public key handling.

DSA public keys could exist in two forms: a single Integer type or a
SEQUENCE containing the parameters and public key with a field called
"write_params" deciding which form to use. These forms are non standard
and were only used by functions containing "DSAPublicKey" in the name.

Simplify code to only use the parameter form and encode the public key
component directly in the DSA public key method.

Reviewed-by: Richard Levitte <levitte@openssl.org>

ASN1_TYPE documentation.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Add Record Layer documentation

Add some design documentation on how the record layer works to aid future
maintenance.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix formatting oddities

Fix some formatting oddities in rec_layer_d1.c.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix record.h formatting

Fix some strange formatting in record.h. This was probably originally
introduced as part of the reformat work.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Define SEQ_NUM_SIZE

Replace the hard coded value 8 (the size of the sequence number) with a
constant defined in a macro.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix compilation on windows for record layer

Reviewed-by: Richard Levitte <levitte@openssl.org>

Rename record layer source files

Reviewed-by: Richard Levitte <levitte@openssl.org>

Remove some unneccessary macros

Reviewed-by: Richard Levitte <levitte@openssl.org>

Renamed record layer header files

Reviewed-by: Richard Levitte <levitte@openssl.org>

Reorganise header files

Reviewed-by: Richard Levitte <levitte@openssl.org>

Remove last trace of non-record layer code reading and writing sequence
numbers directly

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move last_write_sequence from s->d1 to s->rlayer.d.
Also push some usage of last_write_sequence out of dtls1_retransmit_message
and into the record layer.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move ssl3_record_sequence_update into record layer

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move buffered_app_data from s->d1 to s->rlayer.d

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move handshake_fragment, handshake_fragment_len, alert_fragment and
alert_fragment_len from s->d1 to s->rlayer.d

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix seg fault in dtls1_new

Reviewed-by: Richard Levitte <levitte@openssl.org>

Moved processed_rcds and unprocessed_rcds from s->d1 to s->rlayer.d

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move bitmap and next_bitmap from s->d1 to s->rlayer.d.
Create dtls_bitmap.h and dtls_bitmap.c

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move r_epoch and w_epoch from s->d1 to s->rlayer.d

Reviewed-by: Richard Levitte <levitte@openssl.org>

Introduce a DTLS_RECORD_LAYER type for DTLS record layer state

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move DTLS1_RECORD_DATA into rec_layer.h

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move read_sequence and write_sequence from s->s3 to s->rlayer

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move s->s3->wpend_* to s->rlayer

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move handshake_fragment, handshake_fragment_len, alert_fragment and
alert_fragment_len from s->s3 into s->rlayer

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move s->s3->wnum to s->rlayer.wnum

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move s->rstate to s->rlayer.rstate

Reviewed-by: Richard Levitte <levitte@openssl.org>

Move s->packet and s->packet_length into s->rlayer

Reviewed-by: Richard Levitte <levitte@openssl.org>

Remove unneccessary use of accessor function now code is moved into record
layer

Reviewed-by: Richard Levitte <levitte@openssl.org>

Make rrec, wrec, rbuf and wbuf fully private to the record layer. Also, clean
up some access to them. Now that various functions have been moved into the
record layer they no longer need to use the accessor macros.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Removed dependency on rrec from heartbeat processing

Reviewed-by: Richard Levitte <levitte@openssl.org>

Introduce macro RECORD_LAYER_setup_comp_buffer

Reviewed-by: Richard Levitte <levitte@openssl.org>