fix incorrect overflow check for allocation in fmemopen

when a null buffer pointer is passed to fmemopen, requesting it
allocate its own memory buffer, extremely large size arguments near
SIZE_MAX could overflow and result in underallocation. this results
from omission of the size of the cookie structure in the overflow
check but inclusion of it in the calloc call.

instead of accounting for individual small contributions to the total
allocation size needed, simply reject sizes larger than PTRDIFF_MAX,
which will necessarily fail anyway. then adding arbitrary fixed-size
structures is safe without matching up the expressions in the
comparison and the allocation.

diff --git a/src/stdio/fmemopen.c b/src/stdio/fmemopen.c
index 7c193a579ba4b5e5d780a63c2cce0e526276f8d3..2ce43d327f812145be057a4d61f684dfb615f844 100644 (file)
--- a/src/stdio/fmemopen.c
+++ b/src/stdio/fmemopen.c
@@ -81,7 +81,7 @@ FILE *fmemopen(void *restrict buf, size_t size, const char *restrict mode)
                return 0;
        }
 
-       if (!buf && size > SIZE_MAX-sizeof(FILE)-BUFSIZ-UNGET) {
+       if (!buf && size > PTRDIFF_MAX) {
                errno = ENOMEM;
                return 0;
        }