VMS: add [.util]shlib_wrap.exe and its build instructions

This is a program for VMS that corresponds to util/shlib_wrap.sh.

Reviewed-by: Rich Salz <rsalz@openssl.org>

descrip.mms.tmpl: in obj2bin, make sure an empty @deps means no empty lines

Reviewed-by: Rich Salz <rsalz@openssl.org>

util/dofile.pl: report if a template couldn't be loaded

Reviewed-by: Rich Salz <rsalz@openssl.org>

Quiet compiler warning about uninitialised variable

Reviewed-by: Rich Salz <rsalz@openssl.org>

RT3669: dgst can only sign/verify one file.

Check arg count and print an error message.

Reviewed-by: Richard Levitte <levitte@openssl.org>

apps/apps.c: include sys/socket.h to declare recv()

Reviewed-by: Tim Hudson <tjh@openssl.org>

Documentation fixup; no more ECDHParameters

Reviewed-by: Rich Salz <rsalz@openssl.org>

Crude VMS build files for demos/bio/

Reviewed-by: Rich Salz <rsalz@openssl.org>

Crude Makefile for demos/bio/

Reviewed-by: Rich Salz <rsalz@openssl.org>

Don't ignore Makefiles in demos/

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fixup BIO demos for OpenSSL 1.1.x

Call SSL_CTX_new() before doing any configuration.
(or call OPENSSL_ssl_init())

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fixup BIO demos for OpenSSL 1.1.x

'ECDHParameters = Automatic' isn't accepted.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fixup BIO demos for OpenSSL 1.1.x

Note: server-cmod doesn't seem to do things right...  from loading
cmod.cnf, it tries to load libssl_conf.so.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Convert WPACKET_put_bytes to use convenience macros

All the other functions that take an argument for the number of bytes
use convenience macros for this purpose. We should do the same with
WPACKET_put_bytes().

Reviewed-by: Rich Salz <rsalz@openssl.org>

Convert CCS construction to WPACKET

Reviewed-by: Rich Salz <rsalz@openssl.org>

Convert Finished construction to WPACKET

Reviewed-by: Rich Salz <rsalz@openssl.org>

Style tweaks following review feedback

Reviewed-by: Rich Salz <rsalz@openssl.org>

Convert NextProto message construction to WPACKET

Reviewed-by: Rich Salz <rsalz@openssl.org>

Convert Certificate message construction to WPACKET

Reviewed-by: Rich Salz <rsalz@openssl.org>

Convert ClientVerify Construction to WPACKET

Reviewed-by: Rich Salz <rsalz@openssl.org>

Remove an option related to a deprecated flag

CMS_NOOLDMIMETYPE and PKCS7_NOOLDMIMETYPE  are unused in pkcs7/cms code.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1585)

Fix various missing option help messages ...

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1585)

Convert num_alloc to a size_t in stack.c and tweak style

We were casting num_alloc to size_t in lots of places, or just using it in
a context where size_t makes more sense - so convert it. This simplifies
the code a bit.

Also tweak the style in stack.c a bit following on from the previous
commit

Reviewed-by: Rich Salz <rsalz@openssl.org>

Prevent overflows in stack API

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

Update INSTALL about no-md5 removal

no-rsa is no longer an option since 7ec8de1

Fix a typo about poly1305

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1582)

Fix no-ocsp

Some compilers complain about unused variables, and some tests do
not run well without OCSP.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Constify code about X509_VERIFY_PARAM

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1594)

Have the configuration options 'no-err' and 'no-async' work again

In an earlier attempt to simplify the processing of disabled options,
'no-err' and 'no-async' stopped working properly.  'err' and 'async'
are directories under 'crypto/', but they are special insofar that
they can't be simply skipped, like all the algorithm directories can,
so they need special treatment among the disablable things.

Reviewed-by: Tim Hudson <tjh@openssl.org>

Simplify configudata.pm dependency generation

While we're at it, correct the fault in windows-makefile.tmpl

Reviewed-by: Andy Polyakov <appro@openssl.org>

Remove an useless definition.

APP_INFO is currently a field of MEM struct.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1583)

Use switch instead of multiple ifs

Makes the logic a little bit clearer.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1571)

Simplify the dependencies for remaking configdata.pm

Reviewed-by: Andy Polyakov <appro@openssl.org>

Rearrange the storage of build file template names in %config

They are now relative paths as well

Reviewed-by: Andy Polyakov <appro@openssl.org>

Use the registered conf file names as dependencies to build configdata.pm

Reviewed-by: Andy Polyakov <appro@openssl.org>

Register the name of the config file each config target was found in

Reviewed-by: Andy Polyakov <appro@openssl.org>

Revert "Abort on unrecognised warning alerts"

This reverts commit 77a6be4dfc2ecf406c2559a99bea51317ce0f533.

There were some unexpected side effects to this commit, e.g. in SSLv3 a
warning alert gets sent "no_certificate" if a client does not send a
Certificate during Client Auth. With the above commit this causes the
connection to abort, which is incorrect. There may be some other edge cases
like this so we need to have a rethink on this.

Reviewed-by: Tim Hudson <tjh@openssl.org>

Finally, make sure vms_term_sock.c is built on VMS

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>

Refactor to avoid unnecessary preprocessor logic

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>

Reformat to fit OpenSSL source code standards

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>

Add copyright and license on apps/vms_term_sock.[ch]

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>

VSI submission: redirect terminal input through socket

This is needed, because on VMS, select() can only be used on sockets.  being
able to use select() on all kinds of file descriptors is unique to Unix.

So, the solution for VMS is to create a layer that translates input from
standard input to socket communication.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>

Make reference to other manpage more explicit

Where -curves, etc., are defined: SSL_CONF_cmd

Reviewed-by: Andy Polyakov <appro@openssl.org>

GH1536: Install empty CT log list

Reviewed-by: Richard Levitte <levitte@openssl.org>

VMS: Really don't force symbol mixed case when building DSOs

This is an amendment to the september 8 commit titled "VMS: Don't
force symbol mixed case when building DSOs"

Reviewed-by: Rich Salz <rsalz@openssl.org>

Rather than one variable for each passwd type, use one enum variable

Reviewed-by: Rich Salz <rsalz@openssl.org>

Test the new SHA256 and SHA512 based password generation options

Reviewed-by: Rich Salz <rsalz@openssl.org>

Document the new SHA256 and SHA512 password generation options

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add SHA256 and SHA512 based output for 'openssl passwd'

RT#4674 issue 2

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add -h and -help for c_rehash script and app

Resolves GH1515 and GH1509.

Reviewed-by: Matt Caswell <matt@openssl.org>

Fix OCSP_RESPID processing bug introduced by WPACKET changes

An OCSP_RESPID in a status request extension has 2 bytes for the length
not 1.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add some sanity checks around usage of t_fromb64()

The internal SRP function t_fromb64() converts from base64 to binary. It
does not validate that the size of the destination is sufficiently large -
that is up to the callers. In some places there was such a check, but not
in others.

Add an argument to t_fromb64() to provide the size of the destination
buffer and validate that we don't write too much data. Also add some sanity
checks to the callers where appropriate.

With thanks to Shi Lei for reporting this issue.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Encourage use of the macros for the various "sub" functions

Don't call WPACKET_sub_memcpy(), WPACKET_sub_allocation_bytes() and
WPACKET_start_sub_packet_len() directly.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Make wpackettest conform to style rules

Remove extra indentation at the start of an "if".

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add a WPACKET_sub_allocate_bytes() function

Updated the construction code to use the new function. Also added some
convenience macros for WPACKET_sub_memcpy().

Reviewed-by: Rich Salz <rsalz@openssl.org>

Convert CKE construction to use the WPACKET API

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add a test for 'openssl passwd'

Also, enlarge test group 20 to include openssl commands that aren't
tested otherwise

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix 'openssl passwd' with arguments -1 or -apr1

RT#4674

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix some magic values about revocation info type...

Add comments, document -valid option.
Add some const qualifiers.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1560)

Add --missing-help to list command

Reviewed-by: Andy Polyakov <appro@openssl.org>

Remove -xtarget=ultra from solaris(64)-sparcv9-cc builds.

This flag got moved after -xarch=v9 in 1.1.0 and had the unexpected
side effect of the compiler building for 32-bit v8plusa instead of v9.

GH#1521
CLA: none; trivial

Signed-off-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>

bio.h: fix number of arguments passed to BIO_ptr_ctrl()
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1520)

s_client: avoid warning on Windows/MS-DOS systems

it appears when using gcc/mingw:
```
apps/s_client.c:815:9: warning: variable 'at_eof' set but not used [-Wunused-but-set-variable]
     int at_eof = 0;
         ^~~~~~
```
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1512)

Configure: detect gcc's dependency generation capability more accurately.

Reviewed-by: Rich Salz <rsalz@openssl.org>

bn/bn_lcl.h: improve interoperability with clang and Android NDK.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Configure: impose ^X on whole build procedure.

Traditionally Configure passed $ENV{PERL} to Makefile. But this
resulted in ambiguilty as Configure script could be executed by
interpreter different from one executing remaining scripts. Since
we separate compile- and run-time interpreters with HASHBANGPERL
variable, there is no reason to segment the build procedure.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Abort on unrecognised warning alerts

A peer continually sending unrecognised warning alerts could mean that we
make no progress on a connection. We should abort rather than continuing if
we receive an unrecognised warning alert.

Thanks to Shi Lei for reporting this issue.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix a few style nits in the wpacket code

Addressing more feedback comments.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Remove else after a return in packet code

Reviewed-by: Rich Salz <rsalz@openssl.org>

Pull out some common packet code into a function

Two locations had the same loop for writing out a value. Pull it out into
a function.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add some soft asserts where applicable

This is an internal API. Some of the tests were for programmer erorr and
"should not happen" situations, so a soft assert is reasonable.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Address WPACKET review comments

A few style tweaks here and there. The main change is that curr and
packet_len are now offsets into the buffer to account for the fact that
the pointers can change if the buffer grows. Also dropped support for the
WPACKET_set_packet_len() function. I thought that was going to be needed
but so far it hasn't been. It doesn't really work any more due to the
offsets change.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Simplify the overflow checks in WPACKET_allocate_bytes()

Reviewed-by: Rich Salz <rsalz@openssl.org>

Move the WPACKET documentation comments to packet_locl.h

The PACKET documentation is already in packet_locl.h so it makes sense to
have the WPACKET documentation there as well.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add tests for the WPACKET implementation

The tests will only work in no-shared builds because WPACKET is an
internal only API that does not get exported by the shared library.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Ensure the WPACKET gets cleaned up in the event of an error

Otherwise a mem leak can occur.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Various bug fixes and tweaks to WPACKET implementation

Also added the WPACKET_cleanup() function to cleanup a WPACKET if we hit
an error.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add WPACKET_sub_memcpy() function

Reviewed-by: Rich Salz <rsalz@openssl.org>

Move from explicit sub-packets to implicit ones

No need to declare an explicit sub-packet. Just start one.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Rename PACKETW to WPACKET

To avoid confusion with the read PACKET structure.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Convert tls_construct_client_hello() to use PACKETW

Reviewed-by: Rich Salz <rsalz@openssl.org>

First pass at writing a writeable packets API

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add enginesdir to libcrypto.pc pkg-config file

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

VMS: Don't force symbol mixed case when building DSOs

Reviewed-by: Rich Salz <rsalz@openssl.org>

VMS: be less picky when loading DSOs

The DSO API was picky about casing of symbol names on VMS.

There's really no reason to be that picky, it's mostly just annoying.
Therefore, we take away the possibility to flag for a choice, and will
instead first try to find a symbol with exact case, and failing that,
we try to find it in upper case.

Reviewed-by: Rich Salz <rsalz@openssl.org>

util/shlib_wrap.sh is now auto-generated so tell git to ignore it

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix EC_KEY_print so it prints out private key information
even when the public key is not present in an EC_KEY

Reviewed-by: Stephen Henson <steve@openssl.org>

Trim config output

With extensive help and feedback from Richard and Andy.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>

VMS: Use different C flags for programs that aren't to be installed

This is generalised by having the following macros for stuff that won't
be installed:

    NO_INST_LIB_CFLAGS, used instead of LIB_CFLAGS
    NO_INST_DSO_CFLAGS, used instead of DSO_CFLAGS
    NO_INST_BIN_CFLAGS, used instead of BIN_CFLAGS

They take values from corresponding target config fields if those are
defined, otherwise they take the respective values from LIB_CFLAGS,
DSO_CFLAGS and BIN_CFLAGS.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Build file templates: additional information to build file template functions

Send a bit information to the build file template functions.  For
src2obj(), the additional option 'product' holds the name of the final
file that the object file will go into.  Additionally, the diverse
functions will get the option 'installed', with a value that evaluates
true if the final product is to be installed, otherwise false.

Reviewed-by: Rich Salz <rsalz@openssl.org>

If errno is ENXIO in BSS_new_file(), set BIO_R_NO_SUCH_FILE

VMS sets that errno when the device part of a file spec is malformed
or a logical name that doesn't exist.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Travis: Do asan and msan with shared libraries

Reviewed-by: Emilia Käsper <emilia@openssl.org>

Fix mem leaks during auto-deinit

Certain functions are automatically called during auto-deinit in order
to deallocate resources. However, if we have never entered a function which
marks lib crypto as inited then they never get called. This can happen if
the user only ever makes use of a small sub-set of functions that don't hit
the auto-init code.

This commit ensures all such resources deallocated by these functions also
init libcrypto when they are initially allocated.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Ben Laurie <ben@openssl.org>

Ensure trace recognises X25519

Using the -trace option to s_server or s_client was incorrectly printing
UNKNOWN for the X25519 curve.

Reviewed-by: Rich Salz <rsalz@openssl.org>

engines/afalg: make it compile with backward compatibility headers.

Reviewed-by: Tim Hudson <tjh@openssl.org>

Configurations/10-main.cf: AIX "facelift".

Improve interchangeability of aix*-gcc targets by linking shared
libraries with -static-libgcc, and address linking problems with
vendor compiler.

Reviewed-by: Richard Levitte <levitte@openssl.org>

chacha/asm/chacha-ppc.pl: add missing .text directive.

RT#4667

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add missing debug strings.

Found by turning -Wswitch-enum on.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Allow asan, msan and ubsan to be configured with shared libraries

The background story is that util/shlib_wrap.sh was setting LD_PRELOAD
or similar platform dependent variables, just in case the shared
libraries were built with -rpath.  Unfortunately, this doesn't work
too well with asan, msan or ubsan.

So, the solution is to forbid the combination of shared libraries,
-rpath and any of the sanity analyzers we can configure.

This changes util/shlib_wrap.sh so it only contains the code that sets
LD_PRELOAD when -rpath has been used when configuring.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Avoid EVP_PKEY_cmp() crash on EC keys without public component

Some hardware devices don't provide the public EC_POINT data. The only
way for X509_check_private_key() to validate that the key matches a
given certificate is to actually perform a sign operation and then
verify it using the public key in the certificate.

Maybe that can come later, as discussed in issue 1532. But for now let's
at least make it fail gracefully and not crash.

GH: 1532

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1547)
(cherry picked from commit 92ed7fa575a80955f3bb6efefca9bf576a953586)

Configure: Reorganise the checking of disabled options

The way we figured out what options are crypto algorithms and what are
something other was somewhat sketchy.  This change bases the
distinction on available sdirs instead.

Reviewed-by: Rich Salz <rsalz@openssl.org>