System Test Procedures

Please note: Sytem Test Procedures are being updated.
John, Serguei and Steve, 19th August 2002.

The System Testing Area at CERN

All source, binaries and output are kept under /afs/cern.ch/sw/geant4/stt/.  This area is read accessible to all Geant4 Collaborators (zb:collaborators rl) and write-accessible to the STT group (zb:sttmembers rlidwka).

Note that sometimes this area contains symbolic links to local discs of specific machines to improve performance.  E.g.:

/afs/cern.ch/user/s/stesting/stt/prod/SUN-CC/debug_NONISO -> /home3/stesting/prod/SUN-CC/debug_NONISO
so you need to be logged into the appropriate machine to see files under these links.

Production and Development Areas

Under /afs/cern.ch/sw/geant4/stt/ there are 3 areas, prod/, dev1/ and dev2/.  In each area there is src/, testtools/ and one subdirectory per system, e.g., DEC-cxx/, HP-aCC/, Linux-g++/, SUN-CC/, SUN-CC5/. The data directories (which are not in distribution!) are used from /afs/cern.ch/sw/geant4/dev/data : G4NDL3.5, G4EMLOW1.1.
  • src/: this contains geant4/.  This is checked out/updated from the geant4 CVS repository as defined by the *.sdb files (see Defining the Geant4 Source Tree).
  • testtools/: this contains a separately checked out version of geant4/tests/GNUmakefile, History, README, mailing_list, responsibles, results/ and tools/.  This is so that the scripts (see below) in testtools/geant4/tools/bin/ are decoupled from the versions under src/geant4/tests/tools/bin.  The former are the ones which are effective at run time.  This means one can change and develop them without affecting src/.  To avoid ambiguities with tag names, each of above files or directories under testtools/geant4/tests/, if changed, must be committed and tagged with distinctive tag names and proposed for testing in the normal way so that they get included in reference tags.
  • The rest contain binaries and output for each system, debug option and ISO option, e.g., SUN-CC/debug_NONISO/,  which themselves contain the directories for binaries - bin/, lib/ and tmp/ - for the current tests and stt/, a symbolic link to the current test output, e.g., stt -> stt.geant4-01-01-cand-00.  The latter are kept, the directories for binaries are re-used.
  • Scripts for System Testing

    These are kept in geant4/tests/tools/bin/.  Be aware that there are 6 checked out versions, namely in prod/src/, dev1/src/, dev2/src/, /prod/testtools/, dev1/testtools/ and dev2/testtools/ so please maintain them with CVS, checking in when you change something, updating in the other areas if necessary.  The current used scripts are in summary:
     
    g4status.sh Prints the status of currently running rsh-submitted jobs.
    g4allsys.sh Reads stt-hosts...data files, submits g4sbr.sh via rsh one per line.
    g4sbr.sh Submit, build and run.  Uses limit.sh and tmpenv.sh
    build.sh Builds libraries and/or test and examples excutables.
    run.sh Runs tests and examples.
    limit.sh Sets limits for some platforms.
    tmpenv.sh Sets TMPDIR for some platforms.
    setup.sh/csh "Source" script to set environment based on current platform and working directory.  Sources...
    specific.sh/csh Detailed platform specific environment setting.
    update.sh Reads stt...sdb files, updates directory one per line.  Uses...
    updt.sh Basic CVS update instructions.

    Some other "useful" scripts are also there:
     

    streamedit.sh Changes all occurences of given string in given file(s).
    rename.sh Changes names of files, replacing occurences of string in name.

    HTML files for Recording Activities

    The Tags Database and these Procedures are currently an HTML files in geant4/tests/results/.  Maintain them through CVS.  When you commit this directory they are posted on the web automatically.

    Notifying Tags for Testing

    When he/she wishes to have code tested, a category coordinators should notify the System Test Team (STT) using geant4-stt@listbox.cern.ch giving the tag name, its domain (sub-directory) of application and a short description of the salient features.  Co-working tags should also be specified.

    Recording a Notified Tag

    When a tag is notified to the STT, the on-standby STT member should update the Bonsai database, marking the tag "Proposed".

    Preparing a Tag for Testing

    Update the Bonsai database, marking the tag "Selected.".

    Submitting a Tags Set to Tests

    In the appropriate System Test Area:
  • Change to testtools/geant4/tests/tools/bin, set G4DEBUG and/or G4STTNONISO if required, and source setup.csh.
  • Prepare an update driver file, say bonsaiXXX.sdb, from the Bonsai database (by Hasami.plx).
  • Update the code, e.g., update.sh [-n] <bonsaiXXX.sdb >&bonsaiXXX.log &.(csh) or update.sh [-n] <bonsaiXXX.sdb >bonsaiXXX.log 2>&1 & (sh).
  • Check the log file for unexpected CVS flags C, M, or ?.
  • Prepare a file, say hostsXXX.data (by Hasami.plx), for piping into g4allsys.sh and run it, e.g., g4allsys.sh <hostsXXX.data.
  • Check with g4status.sh.  Check log files, e.g., sungeant.<area>.deb_NONISO.log for errors.
  • Inspecting the Output

    The output is in /afs/cern.ch/sw/geant4/stt/<area>/$G4SYSTEM/<debug-ISO-option>/stt/$G4SYSTEM as appropriate (the last stt is a symbolic link to the current results sub-directory name).  .diff files compare the .out files with the "reference" .out files in the source tree, e.g., test01.diff is diff -w test01.out $G4INSTALL/tests/test01/test01.out.

    We are using Tinderbox to check gmake.log and output/diff files: http://pcitapiww/tinderbox/tinderbox/html/

    If all's well, update the Bonsai database and mark the tag(s) "Accepted" or "Rejected".  Notify the category coordinator.

    After a few tags have built up, or for the monthly reference tag, consider placing a global (geant4-wide) reference tag on the geant4 source tree - see below.

    Large N Tests

    Periodically, say every other weekend, submit "large N" tests.  This is done by setting the "action" parameter on each line of the job submission driver file bonsaiXXX.large_N.data. to large_N.  Note that the binaries must already be in place from an earlier normal build.

    This option runs jobs in the following cases: (a) if a test or example has a file ending in large_N.in, in which case this file is taken instead of the normal .in file, or (b) if a job's input is generated by an "hadronic exerciser", in which case the exerciser is run with the large N option.

    Making a Global (geant4-wide) Reference Tag

    Make sure src/geant/tests is up to date.  In src/geant4/ of the relevant test area, cvs tag <tag-name> >& cvs_log_<tag-name>.log &, where <tag-name> is geant4-xx-yy-ref-zz, where xx-yy refer to the most recent past Geant4 release and zz is a serial number.

    Announce on the Tags Blackboard and invite category coordinators to add further information.

    Candidate Tags

    These are simply global tags used internally by the STT to define a version of geant4 which has not yet been tested.  The format is geant4-xx-yy-cand-zz, where zz refers to the next global reference tag.


    John Allison , Serguei Sadilov  and Steve O'Neale  19th August 2002