dtsession, DtSvc: fix CVE-2020-2696/VU#308289

dtsession, DtSvc: fix CVE-2020-2696/VU#308289

Marco Ivaldi <marco.ivaldi@mediaservice.net> has identified 3
vulnerabilities in CDE.

Two of them could affect our CDE (open-source version), while the 3rd
(sdtcm_convert) is Solaris specific.

The two vulnerabilities, both of which affect dtsession could allow a
local privilege escalation to root.  A POC exists for Solaris.  The
POC will not function on our CDE for two main reasons:

- the POC is Solaris specific
- The overflowed variables in question are allocated on the heap,
  whereas in Solaris these variables are located on the stack.

The first vulnerability allows an extra long palette name to be used
to cause a crash via insufficient validation in
SrvPalette.c:CheckMonitor().

The second, which has not yet been assigned a CERT CVE resides in
SmCreateDirs.c:_DtCreateDtDirs() in libDtSvc.  Due to insufficient
bounds checking, a crash or corruption can be achieved by using a very
long DISPLAY name.

This one is considered difficult to exploit, and no POC code is
available at this time.  CDE 2.x code-bases are also listed as not
vulnerable, however some work has been done anyway to do some proper
bounds checking in this function.

The following text portions are copied from the relevant advisories,
which have not been released as of this writing.

NOTE: Oracle CDE does NOT use CDE 2.3.0a or earlier as mentioned
below.  They are completely different code-bases):

Regarding CVE-2020-2692:

  A buffer overflow in the CheckMonitor() function in the Common
  Desktop Environment 2.3.0a and earlier, as distributed with Oracle
  Solaris 10 1/13 (Update 11) and earlier, allows local users to gain
  root privileges via a long palette name passed to dtsession in a
  malicious .Xdefaults file.

  Note that Oracle Solaris CDE is based on the original CDE 1.x train,
  which is different from the CDE 2.x codebase that was later open
  sourced. Most notably, the vulnerable buffer in the Oracle Solaris
  CDE is stack-based, while in the open source version it is
  heap-based.

Regarding the DtSvc bug, which does not currently have a CERT CVE:

  A difficult to exploit stack-based buffer overflow in the
  _DtCreateDtDirs() function in the Common Desktop Environment version
  distributed with Oracle Solaris 10 1/13 (Update 11) and earlier may
  allow local users to corrupt memory and potentially execute
  arbitrary code in order to escalate privileges via a long X11
  display name. The vulnerable function is located in the libDtSvc
  library and can be reached by executing the setuid program
  dtsession.

  The open source version of CDE (based on the CDE 2.x codebase) is
  not affected.