VMS open() doesn't take O_BINARY, but takes a context description

Tell open() O_BINARY on VMS doesn't make sense, as it's possible to
use more precise file attributes.  However, if we're still going to
fdopen() it in binary mode, we must set the fd in binary context.

Reviewed-by: Rich Salz <rsalz@openssl.org>

diff --git a/apps/apps.c b/apps/apps.c
index 43a51f63aed7f9b36a9d0ad3e6565dcd36972cda..bb47039ce091c3597d5a70927b8d7fb730cb13e0 100644 (file)
--- a/apps/apps.c
+++ b/apps/apps.c
@@ -2817,7 +2817,17 @@ BIO *bio_open_owner(const char *filename, int format, int private)
 #endif
     }
 
-    fd = open(filename, mode, 0600);
+#ifdef OPENSSL_SYS_VMS
+    /* VMS doesn't have O_BINARY, it just doesn't make sense.  But,
+     * it still needs to know that we're going binary, or fdopen()
+     * will fail with "invalid argument"...  so we tell VMS what the
+     * context is.
+     */
+    if (!textmode)
+        fd = open(filename, mode, 0600, "ctx=bin");
+    else
+#endif
+        fd = open(filename, mode, 0600);
     if (fd < 0)
         goto err;
     fp = fdopen(fd, modestr('w', format));