Fix type error in PEM processing

The get_name() helper was using a variable of type size_t to hold the
result of BIO_gets(), but BIO_gets() returns int and makes use of negative
values to indicate error conditions.

Change the type of the local variable to match, and propagate that
through to other places in the file to avoid -Wsign-compare issues.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5211)

diff --git a/crypto/pem/pem_lib.c b/crypto/pem/pem_lib.c
index 97b8a0d7c0ab8baa9107f873c18ff046c74102f6..1f0f8d6c8d319cc063b6ad3c510b12d64547e256 100644 (file)
--- a/crypto/pem/pem_lib.c
+++ b/crypto/pem/pem_lib.c
@@ -722,14 +722,14 @@ static int sanitize_line(char *linebuf, int len, unsigned int flags)
 static const char beginstr[] = "-----BEGIN ";
 static const char endstr[] = "-----END ";
 static const char tailstr[] = "-----\n";
-#define BEGINLEN (sizeof(beginstr) - 1)
-#define ENDLEN (sizeof(endstr) - 1)
-#define TAILLEN (sizeof(tailstr) - 1)
+#define BEGINLEN ((int)(sizeof(beginstr) - 1))
+#define ENDLEN ((int)(sizeof(endstr) - 1))
+#define TAILLEN ((int)(sizeof(tailstr) - 1))
 static int get_name(BIO *bp, char **name, unsigned int flags)
 {
     char *linebuf;
     int ret = 0;
-    size_t len;
+    int len;
 
     /*
      * Need to hold trailing NUL (accounted for by BIO_gets() and the newline