bio printf: Avoid using rounding errors in range check
There is a problem casting ULONG_MAX to double which clang-10 is warning about.
ULONG_MAX typically cannot be exactly represented as a double. ULONG_MAX + 1
can be and this fix uses the latter, however since ULONG_MAX cannot be
represented exactly as a double number we subtract 65535 from this number,
and the result has at most 48 leading one bits, and can therefore be
represented as a double integer without rounding error. By adding
65536.0 to this number we achive the correct result, which should avoid the
warning.
The addresses a symptom of the underlying problem: we print doubles via an
unsigned long integer. Doubles have a far greater range and should be printed
better.
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11955)