trace: ensure correct grouping

trace: ensure correct grouping

It is important that output to the trace channels occurs only inside
a trace group. This precondtion is satisfied whenever the standard
TRACE macros are used. It can be violated only by a bad programming
mistake, like copying the 'trc_out' pointer and using it outside
the trace group.

This commit enforces correct pairing of the OSSL_TRACE_CTRL_BEGIN and
OSSL_TRACE_CTRL_END callbacks, and checks that OSSL_TRACE_CTRL_WRITE
callbacks only occur within such groups.

While implementing it, it turned out that the group assertion failed

  apps/openssl.c:152: OpenSSL internal error: \
                      Assertion failed: trace_data->ingroup

because the set_trace_data() function invokes some callbacks which
generate trace output, but the correct channel type was set only
after the set_trace_data() call.

To fix the failed assertions, the correct channel type is now set
inside the set_trace_data() call, instead of doing it afterwards.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8463)