/* $XConsortium: README-DEVELOPER /main/3 1996/07/15 14:12:13 drk $ */ IMPORTANT!!! PLEASE READ ON!! ------------------------------ The intention of this file is to outline the procedure to use, anytime a new update of the ksh-93 source is received. When you receive the new source package, you will first need to do the following: 0) mkdir ksh93-base 1) copy the packing into this directory 2) uudecode the package 3) uncompress the package 4) un-tar the package into ksh93-base (you may need to use cpio, if that is what Korn used to create his package). Once the source ball has been un-tar'ed, go down into the hierarchy, until you see a directory called 'ship' ; don't go down into the 'ship' directory, simply stop at its parent. Dave Korn uses his own mechanism for creating the actual ksh-93 sources. It's not like you can simply look at what is in the 'ship' directory. Instead, you need to run his 'shipin' tool, which takes his base source ball, and applies all of his patches, thus generating the source tree. To start this process, which will result in the latest source files being generated, issue the following commands: /bin/ksh export SHELL=/bin/ksh nohup /bin/ksh ship/shipin -u -L& tail -f nohup.out When this completes, you will notice that the current directory now contains a series of other subdirectories; the only ones we really care about now are the 'man', 'ship' and the 'src' subdirectories. The trick now is to determine which of the files in these subdirectories have changed, so that you can update the real dtksh/ksh93 source tree. You will want to copy into the dtksh/ksh93/ship directory the newest versions of the following files: shipiffe.sh shipout shipin shipproto.c shipop.c Go into the 'shipin' file, and locate the several places where it attempts to mail out information (I look for the word 'mail', until I find what looks like the correct places). Remove this code, since we don't want a ton of mail messages happening each time we build. Within the 'ship' directory, you will notice a series of subdirectories; each one of these corresponds to one of the pieces making up the ksh93 client (i.e. there is one for each library: libast, libcmd, libvdelta, etc). Within each of these subdirectories is another subdirectory, whose name represents the date that source depot was created (i.e. 950619). You will need to recreate these dated directories, along with their contents, within your dtksh tree (under dtksh/ksh93/ship). Within each dated directory, copy over (and check in) the items, copyright, UNCRATED and owner files. Check in a dummy 'base' file also. Doing these steps will allow the ksh build to proceed, without it ever again attempting to uncrate new ksh93 sources. The 'items' file is particularly important, since it indicates which other libraries the indicated piece is dependent upon. ==================== In /clone/fred/DtkshTools I have a collection of scripts which make it easier to update a dtksh tree with new ksh sources. The steps for determining which files have changed are as follows: 1) First get a list of all of the files in the new source tree. I do this with the following steps: cd ksh93-base (This is the tree you just untar'ed) find src -print > srcList This gives you a complete list of all of the source files. 2) Use this list to determine which source files are new or have changed: /clone/fred/DtkshTools/CheckKshSrc dtksh ksh93-base srcList > diffList The output generated by the above script will tell you which files have changed, which have not changed, and which have been added. 3) You now need to take the diffList, and modify it so that it only contains the name of new or changed files within the ksh93 source tree: /clone/fred/DtkshTools/CreateTrimList diffList > trimList 4) Now, we will delete the files in the dtksh tree which we are going to replace; we do this before we copy over the new versions: /clone/fred/DtkshTools/TrimOldSrcs /clone/fred/dtksh trimList 5) We now need to check out (Rcs -l) the files we are going to replace: /clone/fred/DtkshTools/CheckOutSrcs /clone/fred/dtksh trimList If at any time you want to get a complete list of checked out files in your dtksh tree, issue the following commands: find /clone/fred/dtksh -type d > dirList /clone/fred/DtkshTools/RunRcslocks dirList 6) We are now ready to copy the new source files out of the ksh93-base directory, and into the dtksh directory: cd /clone/fred /clone/fred/DtkshTools/CopyNewSrcs ksh93-base dtksh trimList ==================== You're not done yet, however. We have found it necessary to modify several of the stock ksh sources, Makefiles and Mamfiles, to accomodate problems we have encountered, or to enable options not turned on by default, such as multibyte support. You will need to make sure that these changes get rolled back in, before checking in the new source. The following source files need to have changes added: src/cmd/ksh93/Makefile (*** THIS STEP IS OBSOLETE ***) ---------------------- Change: MULTIBYTE == To: MULTIBYTE == 1 src/cmd/ksh93/Mamfile (*** THIS STEP IS OBSOLETE ***) --------------------- Change: setv MULTIBYTE -UMULTIBYTE To: setv MULTIBYTE -DMULTIBYTE src/cmd/ksh93/OPTIONS (*** THIS STEP IS OBSOLETE ***) --------------------- Change: MULTIBYTE=0 To: MULTIBYTE=1 *** The following two files must change, to prevent the ksh source from linking in its own malloc package. This creates problems on the IBM platform, due to the way their libraries are built. As it turns out, the IBM Xt library already has all of its malloc/free/etc calls resolved to their standard libc malloc functions. If ksh defines its own malloc package, then dtksh ends of effectively linked simultaneously to two different malloc packages. This rapidly leads to a core dump, due to cross contamination of the memory pools (i.e. A value obtained by XtMalloc() passed to free() ). As it turns out, this also creates problems on the HP platform, as was recently discovered (Note that it appears that the problem which was seen on the HP platform was due to a defect in the ksh93 memory allocation code; this defect had not yet been fixed in the ksh93-d release. Until the time that this is fixed, we must continue to be sure to not link in ksh93's malloc package). src/lib/libast/Makefile ----------------------- Remove: calloc.c malloc.c from the list of sources src/lib/libast/Mamfile ---------------------- Remove: calloc.o malloc.o from the archive line; DO NOT REMOVE ANY OF THE OTHER REFERENCES TO malloc OR calloc, AS THIS WILL CAUSE THE BUILD TO BREAK. *** Another file which needs special handling is builtins.c, which is in the base dtksh directory. This file was first created by taking the file 'dtksh/ksh93/src/cmd/ksh93/data/builtins.c', and adding into its list of builtin commands, the list we added for X, Xt, Xm and Dt. The best way to update this file is to first goto 'dtksh/ksh93/src/cmd/ksh93/data' and do an Rcsdiff, to find out what changes Dave Korn made. You will then make those same changes to the dtksh/builtins.c file. dtksh/builtins.c ---------------- Fold in any changes made to dtksh/ksh93/src/cmd/ksh93/data/builtins.c *** Another file which needs special handling is aliases.c, which is in the base dtksh directory. This file was first created by taking the file 'dtksh/ksh93/src/cmd/ksh93/data/aliases.c', and adding into its list of commands aliases, the list we added for X, Xt, Xm and Dt. The best way to update this file is to first goto 'dtksh/ksh93/src/cmd/ksh93/data' and do an Rcsdiff, to find out what changes Dave Korn made. You will then make those same changes to the dtksh/aliases.c file. dtksh/aliases.c --------------- Fold in any changes made to dtksh/ksh93/src/cmd/ksh93/data/aliases.c HP-SPECIFIC Ksh-93 SOURCE MODIFICATIONS: ======================================== The dtksh code requires a handful of changes to be made to the ksh93 source itself. Most of these changes relate to either localization customizations, additional multibyte support, or fixes for environment variable mirroring. For each of the files listed below, two versions have been checked into the RCS vaults: the original (unaltered) version, and the version with our customizations. This will make it easy to determine the changes we have made, so that the next time we receive new code from Korn, it is a relatively simple task to again merge in the changes. Note that all of the changes relating to multibyte support have been passed onto David Korn, and that hopefully they will show up in the official source sometime in the future (after the ksh93-d release). Modification 1: --------------- The ksh (and dtksh) executables use a special client (suid_exec) when trying to execute an suid script; this special client makes sure that you really have permission to run the suid script. The client, as supplied by David Korn, has hardcoded assumptions about where it lives (/etc) and where the valid (and secure) places are where shells can live. We need to change this so that it knows the client lives in the cde tree, and that in addition to the normal 'safe' shell locations, /usr/dt/bin is also a safe location. To do this, the following changes must be made, each time new ksh source is received: dtksh/ksh93/src/cmd/ksh93/data/msg.c ------------------------------------ Change: e_sysprofile = "/etc/profile"; e_suidprofile = "/etc/suid_profile"; e_suidexec = "/etc/suid_exec"; To: e_sysprofile = PROFILEPATH; e_suidprofile = SUIDPROFILEPATH; e_suidexec = SUIDEXECPATH; dtksh/ksh93/src/cmd/ksh93/sh/suid_exec.c ---------------------------------------- Change: #define THISPROG "/etc/suid_exec" To: #define THISPROG SUIDEXECPATH Also, search for the block of code like the following (around line 256): if (shell == 0 || !endsh(shell) || ( !in_dir("/bin",shell) && !in_dir("/usr/bin",shell) && And change it as follows: if (shell == 0 || !endsh(shell) || ( !in_dir(CDE_INSTALLATION_TOP"/bin",shell) && !in_dir("/bin",shell) && !in_dir("/usr/bin",shell) && Modification 2: --------------- During CDE development, it was discovered that some shared library implementations (i.e. at least IBM) bound a shared library to all of its dependencies at the time that the library was build. This created problems for libraries which made getenv() or putenv() calls, because we expected those calls to be resolved against the getenv() and putenv() calls provided in the ksh93 libraries, not against the standard libc versions. This was a problem because ksh keeps track of its own environment, and does not pass the information on through to the standard libc environment. The result was that certain key pieces of information (i.e LANG) might be set by a shell script, but when calls where made to shared library functions, they would not see this setting. The fix is to modify the ksh93 code to *also* pass along any environment settings onto the libc functions; we call this 'environment variable mirroring'. We won't describe the exact changes here; you can find them by diff'ing the last two revisions for the following files: src/cmd/ksh93/include/nval.h (New field added to Namval structure) src/cmd/ksh93/sh/name.c (Mirroring code implemented here) Modification 3: --------------- There are a handful of multibyte and localization changes which we must make. Some of them relate to adding message catalog support to ksh, while others relate to making changes so that Japanese SJIS character will work. The files which must be modified are: src/cmd/ksh93/sh/init.c (Message cat and file desc locking/unlocking) src/cmd/ksh93/sh/lex.c (Japanese SJIS processing fixes) src/cmd/ksh93/sh/macro.c (Multibyte fixes) src/cmd/ksh93/bltins/print.c (SJIS multibyte fix to echo/print command) src/cmd/ksh93/sh/userinit.c (Add call to setlocale() in sh_userinit()) Modification 4: --------------- During CDE development, USL needed to add an 'ifndef __USLC__' around a certain define. The file which needs the change is: src/lib/libast/sfio/sfhdr.h Modification 5: --------------- There appears to be a bug in the ksh93 build process, which shows up on the HP platform. David Korn was unable to find the cause, so we need to modify one of his build scripts: ksh93/src/cmd/ksh93/features/options.sh --------------------------------------- Towards the top of the script, there is the following set of statements: : get the option settings from the options file . $OPTIONS You need to modify this so that it is like the following: : get the option settings from the options file if [ -z "$OPTIONS" ] ; then OPTIONS="OPTIONS" fi . $OPTIONS ======================= Each time a new release is received from Dave Korn, we also need to copy his latest ksh93 man page from src/cmd/ksh93/sh.1 into /x/cdesrc/doc/cde1/man/man1/ksh93. You will need to make one change to the man page, so that it prints out the ksh information, instead of sh information. The change is one approximately line 7. Change: ------- .nr Z 0 \" set to 1 when command name is ksh rather than sh To: --- .nr Z 1 \" set to 1 when command name is ksh rather than sh =================== One last note. There is a shell script called 'MakeClean' in the root directory of the dtksh tree. This script cleans out files in the ksh93 portion of the tree, which will force a complete build to occur. Since the ksh93 portion does not use Imake (it uses Mam or something bazarre like that), we needed to jury rig something which would act like our normal 'make clean' . If you discover any other ksh93 files which need to be removed during a 'make clean' operation, simply add another line to the MakeClean script. A good indication that a file needs to be added is if you make a clone from one of the source trees, do a 'make clean', start a 'make', and somewhere during the building of the ksh93 sources, you see an error message telling you that you did not have permission to overwrite something within your clone tree. The ksh93 build process builds alot of files on the fly, and moves things around to other directories, so these file typically need to be cleared out before doing a complete make.