New features with AN-2020-07-01: This is the first localization step for the schily source consolidation. Many programs now (hopefully) call gettext() for all strings that need localization. - The next step will include dgettext() calls for the libraries and the missing programs - The following step will include the extracted strings - The last step will include German translations and install support for the resulting binary message object files. ----------> Please test and report compilation problems! <--------- ***** NOTE: As mentioned since 2004, frontends to the tools should ***** ***** call all programs in the "C" locale ***** ***** by e.g. calling: LC_ALL=C cdrecord .... ***** ***** unless these frontends support localized strings ***** ***** used by the cdrtools with NLS support. ***** *** WARNING *** *** Need new smake *** *** Due to the fact that schily-tools 2014-04-03 introduced to use new macro *** expansions and a related bug fix in smake, you need a newer smake *** to compile this source. If your smake is too old and aborts, ensure to *** use the recent smake by calling: cd ./psmake ./MAKE-all cd .. psmake/smake psmake/smake install The new smake version mentioned above is smake-1.2.4 The recent smake version is smake-1.3 *** Due to the fact that schily-tools 2018-01-26 introduced *** optimizations for the Schily version of SunPro Make, you *** need at least the dmake version from 2018/01/11 with support *** for the "export" directive to compile with this makefile system. WARNING: the new version of the isoinfo program makes use of the *at() series of functions that have been introduced by Sun in August 2001 and added to POSIX.1-2008. For older platforms, libschily now includes emulations for these functions but these emulations have not yet been tested thoroughly. Please report problems! BUG WARNING: Please never report bugs only to Linux distributions as they usually do not forward these bug reports upstream and as the Linux distributions typically do not let skilled people check the bugs. We did not hear about a FIFO problem in star for a long time. Then a problem on Linux occurred once every 6000-10000 tries but it did not happen on Solaris after even 10 million tries, so it was not known besides Linux. BUG WARNING: GNU make release 4.3 starts too early with parallel execution (when reading Makefiles and evaluating rules for "include" statements already). Since GNU make does not support a concept for correct ordering of such actions, you need to be prepared to see gmake-4.3 fail in parallel mode. If you are interested in reliable parallel execution, it is recommended to use the included "dmake" program with a command line like: dmake -j10 -f SMakefile from the top level directory - star: Backslashes are now honored when looking for the substitution separators in options like -s /from/to/ or -pax-s /from/to/ Previously, -s '/xxx\/yy//' was interpreted in a way that made the **to string** "yy" and the **from string** "xxx\" instead of making the from string "xxx\/yy" and the to string empty. - Bourne Shell: A command like bosh -c 'var=$({ echo value >&3; } 3>&1); echo $var' Did not print "value" as expected, but rather bosh: 3: bad file number This is related to the fact that curly brackets do not have an own binary node type in the parsed output and the recursive parser from the $(cmd) execution needs to reconstruct the text form of a command tree. As a result, the command has been translated into: bosh -c 'var=$( echo value >&3 3>&1); echo $var' and this caused the observed problem. Note that this basic problem may not be finally fixed, so please report in case that other commands with curly brackets together with $(cmd) cause problems. Thanks to Koichi Nakashima for reporting. - Bourne Shell: Added a unit test for the bug above. - cdda2wav: The default for the CDDB database has been changed from "freedb.freedb.org" -> "gnudb.gnudb.org" because freedb.freedb.org has been shut down forerver. - cdrtools: Assigend a new version date to inform people about the new cdda2wav version with the updated CDDB default host. - SCCS: This is a huge step towards the planned project mode of SCCS. We now fully support SCCSv6 history files and and all tools now work in the "new mode" that is based on unified path name rewriting controlled by the option -NSCCS and by the directory $PROJECT_HOME/.sccs Please carefully read the following description: - SCCS: A new sccs(1) subcommand "histfile" has been implemented in order to convert g-filenames into s-filenames. This is needed as it is hard to find the related history file in case that the repository is an off-tree repository. A command like "sccs histfile xec.c" could e.g. result in: ../.sccs/data/sh/SCCS/s.xec.c So if you like to have the SCCS history file for the file xec.c in the curent directory in your editor, you could call: $EDITOR $(sccs histfile xec.c) regardless of which model to store SCCS history files is used. The argument to "sccs histfile" may be a directory. In this case the result is the directory that holds the correcponding SCCS history files. - SCCS: The sccs(1) man page now mentions the new command "histfile" - SCCS: The sccs(1) online help now mentions the new command "histfile" - SCCS: sccs(1) now calls the new function bulkchdir() when in NewMode after bulkprepare() has been used inside getN(). bulkchdir() performs an fchdir() back to the original "." dir before leaving the function getN(). - SCCS: admin(1) now correcly supports -version and --version. There was a problem because in the admin program, -V/-version is an option shared with -V4/-V6 (-V with argument) and we only distinct between both variants via the "option requires an argument" error, from getopt(). But getopt() translates "-version" into -V and for this reason, we need to distinct between an illegal -V parameter and the legal option -version. - SCCS: libcomobj the function bulkprepare() is now more accurate in case that the file argument is an existing directory. This is needed for the new "sccs histfile" command. - SCCS: libcomobj the function bulkprepare() now moved the previously static file descriptor for doing a fchdir() back to the directory "." into a new member of the structure Nparms. - SCCS: libcomobj the function bulkprepare() now supports a new flag NP_NOCHDIR to forbid a chdir() which is needed for "sccs histfile" in order to print the complete path name instead of doing a chdir to a part of the path and reporting back only the remaining path components. - SCCS: libcomobj: a new struct member n_error has been added to struct Nparms - SCCS: libcomobj: a new function bulkchdir() has been added. - SCCS: libcomobj: a new function bulkclosedir() has been added. - SCCS: libcomobj: a new function bulkerror() has been added. - SCCS: libcomobj: The function bulkprepare() now remembers whether it did perform a chdir(). - SCCS: libcomobj: The function bulkchdir() now only peforms a fchdir() back in case that struct Nparms remembered a previous chdir(). - SCCS: libcomobj: The function bulkprepare() now calls resolvenpath() a second time (this time with the complete resulting path). This prevents us from trying to use constructed and non-working path names like: ../.sccs/data/non-exist/../SCCS/s.foo by converting them to something like: ../.sccs/data/SCCS/s.foo before we use them. - SCCS: libcomobj: The function bulkprepare() for now no longer performs a chdir() at all. The related code has been commented out. We first need to make the code work correctly in all cases before we can think about performance or other goals. BTW: the reason why the chdir() has been implemented in 2015 was that the historic behavior of all SCCS low level commands was to use filenames relative to the current directory only. - SCCS: mpwlib::lockit.c some better comment has been added. - SCCS: mpwlib::lockit.c If a SCCS lock file is seen that is related to a process that no longer exists, lockit() no longer waits 10 seconds after the file has been removed, before retrying to create a new lock. - SCCS: mpwlib::lockit.c If a SCCS lock file is seen that is empty and older than at least 60 seconds, lockit() no longer waits for 10 seconds after the file has been removed, before retrying to create a new lock. - SCCS: mpwlib::lockit.c If a SCCS lock file seems to be empty when checking, the check for a potential later modification not also checks nanoseconds in the time stamp when deciding whether the file is currently not in an active creation procedure and thus must be a dead file. - SCCS: mpwlib::lockit.c No longer uses O_DSYNC since we believe that it does not give more security against locking problems but on the other side just slows down the write() call to the lock file. On Solaris, the related approx. performance loss is 18ms, but on Linux it is approx. 50ms. Since we need an additional global lock with the upcoming project mode, this would delay locked operations on Linux by 100ms. This is a lot of time, as a delta(1) command call for a small file takes less than 10ms otherwise. - SCCS: mpwlib::lockit.c ismylock() now checks whether a local process that holds the lock still exists. In this case, a new return code -2 is used. This is needed to reduce the number of cased that need a check on wether a recovery procedure is needed. - SCCS: mpwlib::lockit.c now implements a new function lockrefresh(). This is needed to keep locks over NFS active for more than 60 seconds. With the classic SCCS, this never was a problem, but the project mode may need longer lasting locks on the global changeset file. If we like to support concurrent attempts to update a project from different NFS based machines, this is an important enhancement. - SCCS: libcomobj: A new file lockchset.c has been added with functions to manage the project global lock file. The new functions are: lockchset() Lock the project. unlockchset() Unlock the project. refreshchsetlock() refresh the mtime of the global lock file to be able to hold the project lock for a time longer than 60 seconds even over NFS. - SCCS: The man page sccsfile.4 now includes a list of SCCS programs that create lock files for SCCS history files. - SCCS: The commands admin and delta now support a new option -Xuser=name that allows to specify a different user (programmer) name for a delta. By default, the logname is used as it has been always done before. - SCCS: admin fixed a potential buffer overflow related to the user (programmer) name that is used in the delta table. - SCCS: "sccs help Xopts" now mentions -Xuser=name - SCCS: all commands have been converted to use the new function bulkerror() from libcomobj::bulk.c - SCCS: A new flag SCCS_NEWMODE has ben addded to make the getenv("SCCS_NMODE") control from sccs(1) visible to all SCCS library code as well, without a need to access the variable NewMode from sccs.c - SCCS: "sccs root -v" added a new column NEWMODE to show whether the current mode in use is the NewMode. - SCCS: The commands admin, delta, get, unget, sccscvt, rmdel, cdc now all manage a project global lock file when a populated directory $PROJECTHOME/.sccs exists that is created via "sccs init dir". Since all needed commands now support the global lock, it is now safe to switch to SCCSv6 history files. - SCCS: The command "sccs cvt" now supports a new -o (old date) option that retains the original time stamp of the history file that has been present before conversion, -oo to make the new time stamp the old time stamp + 1ns, -ooo to make the new time stamp the old time stamp + 1us, -oooo to make the new time stamp the old time stamp + 1s. The reason for the increment variants is to make sure that when copying the SCCS history files via star, or other programs that only extract from the archive if the new file is newer than on disk, the converted variants overwrite the old ones in the filesystem. If it may be possible that the copy might be targeted to UFS as well, it is recommended to use -ooo to get a time stamp that is one microsecond newer than the original time. For historic UNIX versions with only 1s time stamp resolution, -oooo is recommended. - SCCS: The sccs command now explixitly disables chdir() for bulkprepare() in more subcommands. This is needed since otherwise these sub-commands would run from a different directory than expected. - SCCS: The command "sccs unedit" now works in NewMode with a directory parameter != ".". - SCCS: The unget command now correctly works when using the -N option together with a file that is not in the curent directory. - SCCS: The unit tests now added checks in the NewMode methods that run after a chdir to "subdir" and with a file path of "../tfile" in order to verify that the new path manipulating routines in NewMode not only work correctly for files in the current directory. - SCCS: The unit tests now copy the whole test suite to /tmp/sccstests/ before starting the tests. This is needed because we now may have a directory .sccs in our project root directory and this changes the behavior of SCCS. By copying the test suite to /tmp/sccstests/, there is no .sccs directory anymore in the path above our tests. Well, unless a bad guy created /tmp/.sccs or /.sccs WARNING: This new method for the SCCS unit tests require "star" to be installed and accessible from $PATH. - SCCS: The unit test commands now unset SCCS_NMODE before starting the tests to let SCCS work in default mode. If SCCS_NMODE is needed for a specific test in the test suite, it is set by that test. - SCCS: The unit tests for the three operation modes of sccs(1) now call "sccs init ." instead of just "sccs init" in order to make shure that this command does not apply to an already existing higher level .sccs directory. - SCCS: libcomobj: A new variable changesetgfile has been added with the path to the filename $PROJECT_HOME/.sccs/changeset - SCCS: "sccs root -v" now also prints the path for $PROJECT_HOME/.sccs/changeset - SCCS: The commands admin(1) and delta(1) now write changeset records into the changeset file in $PROJECT_HOME/.sccs/changeset if that file exists. This file is to collect the entries for a future project aware SCCS before they are prepended to the changeset history file $PROJECT_HOME/.sccs/SCCS/s.changeset with a commit operation. The new code is currently only used to debug the future interface used for "sccs commit". - SCCS: "sccs cvt" now is usable with directory arguments in NewMode. If you call "sccs cvt -V6 .", this first converts the current directory to the correcsponding directory with SCCS history files. This directory is then opened and scanned for s.files ant these files are converted back to plain file names in the current directory. - SCCS: "sccs log" command did incorrectly use a GMT offset of 1 second if it encountered a SCCSv6 delta entry with nanoseconds. This has been fixed and the GMT offset from the entry is used. - SCCS: "man sccs-cvt" now includes a better description on how directory type arguments are handled. - SCCS: "man sccs-cvt" now includes a hint on the name of the binary package that contains the software. - SCCS: "man sccs-cvt" now lists more utilities via "SEE ALSO" - SCCS: "man sccs-get" now includes a hint on the name of the binary package that contains the software. - SCCS: "man sccs-get" now lists more utilities via "SEE ALSO" - SCCS: "man sccs-init" now includes a hint on the name of the binary package that contains the software. - SCCS: "man sccs-init" now lists more utilities via "SEE ALSO" - SCCS: "man sccs-log" now includes a hint on the name of the binary package that contains the software. - SCCS: "man sccs-log" now lists more utilities via "SEE ALSO" - SCCS: "man sccs-root" now includes a hint on the name of the binary package that contains the software. - SCCS: "man sccs-root" now lists more utilities via "SEE ALSO" - SCCS: A new man page for "man sccs-edit" has been created. - SCCS: A new man page for "man sccs-editor" has been created. - SCCS: A new man page for "man sccs-check" has been created. - SCCS: A new man page for "man sccs-info" has been created. - SCCS: A new man page for "man sccs-tell" has been created. - SCCS: A new man page for "man sccs-delget" has been created. - SCCS: A new man page for "man sccs-deledit" has been created. - SCCS: A new man page for "man sccs-histfile" has been created. - SCCS: A new man page for "man sccs-istext" has been created. - SCCS: "man sccs-init" a typo in the man page has been fixed. We now see the expected sccs-init(1) instead of val(1). - SCCS: A typo in help.d/cmds has been fixed - SCCS: a new version date has been created to mark this new version. - SCCS: *** The SCCS history for the SCCS project has been converted *** *** into SCCSv6 history files. *** From the previous TODO list: - Find a way to decide when to use SCCS V6 type history files. For the project mode, this is needed in order to prevent historic SCCS implementations to believe they could modify files without knowing about project global locks. Done: The only program that creates new SCCS history files and thus controls the format used when they are created is "admin" and "admin" now by default creates an SCCSv6 history file if a valid .sccs directory exists in the project home directory. This helps to avoid a need for converting the files when the repository is later converted into a repository using the upcommig project mode. - Add code to all programs that modify the SCCS history files (currently admin, cdc, delta, rmdel, sccscvt to maintain a lock (for the file $PROJECTHOME/.sccs/SCCS/s.changeset, that would be in the file $PROJECTHOME/.sccs/SCCS/z.changeset) to maintain a project global lock. There is now a new global lock stratey implemented in all needed binaries, which are binaries that modify history files (s.files) or binaries that modify permssion lock files (p.files). Conclusions from the last changes: - SCCS: sccs(1) should now be finally usable in NewMode to check the correctness of the current state of the implementation beyond the existing unit tests. To switch into this test mode do this: cd path/to/project/root sccs init -i . SCCS_NMODE= export SCCS_NMODE export SCCS_NMODE= should be placed into a shell startup script. By calling: :> .sccs/changeset debugging for admin(1) and delta(1) is enabled and both programs will write changeset entries into that file. If this debugging is enabled, the file .sccs/changeset needs to be removed in the future before the final project mode support is used when it is ready. The commands above initialize the project with an in-tree repository that has the SCCS history files in the traditional location. The whole repository is initialized as a traditional repository for a collection of single files and an empty .sccs/SCCS/s.changeset is created. All newly created SCCS history files are created as SCCSv6 files in this mode. Note that these SCCSv6 history files are not understood by old SCCS implementations unless they are converted back to V4 via "sccs cvt". Since the initial .sccs/SCCS/s.changeset file is present, this enforces SCCS to create a global project lock for every operation. This helps to check for unexpected behavior with this new feature. As we removed O_DSYNC while writing the lock files, this additional lock does not make SCCS slower than it has been before. As this version implements the new global project locking rules, it is safe to start with creating SCCSv6 history files by default. The new enhanced SCCSv6 history file format has been designed to be intentionally incompatible to historic SCCS implementations, to make sure that all software that supports SCCSv6 is aware about the need for the project global lock file. *** The SCCSv6 hisory file format typically causes 3% larger SCCS history files and causes SCCS to consume 10% more CPU time. *** To convert a project into SCCSv6, the following procedure is currently recommended: 1) Set up the project home directory for an in-tree NewMode repository. 2) Call "sccs cvt -V6 -k -ooo ." for every directory in the project. Later, when the project mode is ready, it is recommended to convert the in-tree repository into an off-tree repository. NOTE: *** The SCCS sourcecode has been converted to use the *** *** SCCSv6 history file format last week *** IMPORTANT: *** the -R option has been introduced in 2008 as a hack *** This hack has been introduced in order to make the historic old mode usable for another few years. We decide later how the -R option will be integrated into the new mode in the future. The basic background is that the historic SCCS system did not deal with filenames that are not from the current working directory. As a workaround, we introduced -R, but this uses an internal option "-C dir" that does not work well together with the NewMode. In the old mode, "sccs delget ../data/file" does not work, but "sccs -R delget ../data/file" is needed. In NewMode, "sccs delget ../data/file" does work. - SCCS TODO: - Convert sccs.c to use -NSCCS in the back end programs instead of converting g-file names from the command line into s.file names in the frontend in order to forward s.file names to the backend programs. Doing it this way allows us to have the SCCS history files "off tree" in a separate directory tree starting with ".sccs/" at the "project root" directory, as the related code to convert path names is in libcomobj::bulkprepare(). *** This is mostly done for now *** - Add code to admin(1) and delta(1) to retrieve the list of new or modified files in order to have all important code for a "sccs commit" in a single program that does not need to deal with ARG_MAX limitations. *** Done, but sccs(1) should implement a counterpart. *** - Add code to sccs(1) to give the list of affected files to admin(1) and delta(1) via stdin to overcome ARG_MAX limitations. - Add code to admin(1), delta(1) and get(1) to maintain/understand the changeset file. This is mainly writing out the sccschangeset(4) entries to an imtermediate store if a single file has been treated succsessfully. - sccs -R tell (and probably other subcommands?) does not yet work in NewMode - Add code to libcomobj to understand the changeset file, which is needed in order to e.g. know the file names and file specific SIDs/state that correspond to a project global SID. - Add code to all programs that modify the SCCS history files (currently admin, cdc, delta, rmdel, sccscvt to maintain a lock (for the file $PROJECTHOME/.sccs/SCCS/s.changeset, that would be in the file $PROJECTHOME/.sccs/SCCS/z.changeset) to maintain a project global lock. *** Done, see above. *** - Find/verify a complete transactional model that allows to repair complex changes to the set of files for a project that have been aborted in the middle. The current idea is to create the file $PROJECTHOME/.sccs/changeset with the deltas to the changeset during a complex update operation. *** A concept for global locks has been implemented *** *** We still need the repair model *** - Find a way to decide when to use SCCS V6 type history files. For the project mode, this is needed in order to prevent historic SCCS implementations to believe they could modify files without knowing about project global locks. *** Done, see above. *** - Find a decision on how to deal with the admin flags that are currently implemented as global flags and thus do not depend on the SID (version) if the history file. - Aborting a transaction via ^C currently requires a manual removal of the global lock file. Find a way to avoid this in case that a commit has been aborted while being prompted for a commit message (which is before any real action happened). - Make the loking method more robust against long running programs. The challenge is a program on a different NFS connected computer and long delays, e.g. via a conguestion on the Wlan. The best way to do this is most likely to install a signal controlled timer that refreshes the lock file every 45 or every 30 seconds. - Enhance "sccs log" to let it ignore the nanoseconds and probably seconds as well when comparing the SCCS delta commants for initial SCCS history file creation. - Add the missing separate man pages: sccs-branch.1 sccs-clean.1 sccs-create.1 sccs-diffs.1 sccs-fix.1 sccs-print.1 sccs-unedit.1 - Bourne Shell Missing features for POSIX compliance: - Support for $'...' quoting (this is not needed for the current version of POSIX but for the next POSIX version that will be named SUSv8). The development of SUSv8 will start in late 2016. We are now expecting the Bourne Shell to be fully POSIX compliant. - Bourne Shell further TODO list: - Finish loadable builtin support. - POSIX does not allow us to implement ". -h", so we will add a "source" builtin to be able to implement "source -h" - The following builtins (that are available in bsh) are still missing in the Bourne Shell: err echo with output going to stderr glob echo with '\0' instead of ' ' between args env a builtin version of /usr/bin/env The following bsh intrinsics are still missing in the Bourne Shell: - the restricted bsh has restriction features that are missing in the Bourne shell. - source -h read file into history but do not execute and probably more features not yet identified to be bsh unique. Author: Joerg Schilling D-13353 Berlin Germany Email: joerg@schily.net, joerg.schilling@fokus.fraunhofer.de Please mail bugs and suggestions to me.