Fix SSL_MODE_RELEASE_BUFFERS functionality

At some point in the past do_ssl3_write() used to return the number of
bytes written, or a value <= 0 on error. It now just returns a success/
error code and writes the number of bytes written to |tmpwrit|.

The SSL_MODE_RELEASE_BUFFERS code was still looking at the return code
for the number of bytes written rather than |tmpwrit|. This has the effect
that the buffers are not released when they are supposed to be.

Fixes #9490

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9505)

diff --git a/ssl/record/rec_layer_s3.c b/ssl/record/rec_layer_s3.c
index 9efaf54be29ada14ce9680042cca75f6b06673a8..df5f6812eb0635d3c0afe78cb788ff50ca1bff59 100644 (file)
--- a/ssl/record/rec_layer_s3.c
+++ b/ssl/record/rec_layer_s3.c
@@ -639,8 +639,9 @@ int ssl3_write_bytes(SSL *s, int type, const void *buf_, size_t len,
              */
             s->s3.empty_fragment_done = 0;
 
-            if ((i == (int)n) && s->mode & SSL_MODE_RELEASE_BUFFERS &&
-                !SSL_IS_DTLS(s))
+            if (tmpwrit == n
+                    && (s->mode & SSL_MODE_RELEASE_BUFFERS) != 0
+                    && !SSL_IS_DTLS(s))
                 ssl3_release_write_buffer(s);
 
             *written = tot + tmpwrit;