last
updated: 27.06.2005, 15:00 CEST
CMS Offline Repository and Releases
Contents
- Basic glossary and work model
- Framework and standard utilities
- Repository structure and management
- External software
- Builds, releases and distribution on supported platforms
- Quality assurance
- Physics validation
- Technical validation
- Naming conventions, coding and style rules
- Logging and monitoring
- Dependency analysis and metrics
- Unit testing
- Memory and performance monitoring
- Problem tracking and change request management
- Documentation
- Appendix A: Repository and SCRAM Project Structure
- Appendix B: Logging
Basic glossary and work model
- bill of materials = source file names and tags, build product
file names, package names and tags, configuration parameters, external
software versions, tool versions and system parameters
- build products = products derived from source files
- developer = a user with the right to commit package changes to
the repository
- external software = software over which the project has no
control
- frozen = immutable (of releases)
- package coordinator = developer with overall responsibility for a
package
- package = the smallest releasable software item or collections
of such software items
- platform = combination of cpu (and if relevant the supplier of
the motherboard), operating system and version and the compiler and
version on which the software is built
- release = 1) ability to deploy a given component for the end
user in his/her current set-up
2) set of frozen versions of packages and their build products for a
given set of target platforms, identified by a unique tag
- site = set of instances of platforms under the control of a site
manager and frequently co-located geographically e.g. the CERN site,
the FNAL site
- site manager = someone who mirrors and/or installs software to a
site for the benefit
of users at that site
- software item = any file or set of files
- source directory = directory which holds source files
- source file = a file which, in the build process, is not derived
from another file by some transformation
- tag = a label given to a version of a software item
- target platform = platform for which software is being built
- user = person wishing to use or build packages, without the right
to commit modifications back to the repository
- validate (v) = to submit a software item, package, collection of
packages or a release to a set of acceptance or quality assurance
tests, including unit, integration and system level tests
- version = a revision of a software item identified by a base
name and a distinguishing identifier
Basic release types in work model
- developer release (or prerelease) = produced by developers
for developers - no
quality requirements - frozen and installed in public release area
- public release = produced by package coordinators - validated
(technical validation), released, frozen and installed in public
release area
- production release = produced by package coordinators - as above
but also validated
(physics validation) by the CMS PRS groups
User scenarios: a user may
- query release database and/or release
configurations (scram list, scram tool info etc)
- install
released software locally (scram project) and run
standard applications
- create and run user applications: modify and/or add to checked
out items; build using released build
products; run; validate
- submit error reports following standard procedure
Developer scenarios: a
developer is a user who may also
- commit software items to the repository
- tag software items following
versioning conventions
- announce updates to package
coordinator
Package coordinator scenarios:
a package coordinator is a developer who may also
- create (sub)package(s)
- grant/remove package-wide write/commit and/or tag access to/from
developers
- produce a developer or public release of a package: check out or update, check, build, validate,
write release notes, release and announce the release
Site manager scenarios: a site
manager may
- install external software locally
- mirror or distribute a release: pull
and open distribution file(s) of release in remote site release area
- copy release sources to remote site release area and rebuild
release products locally on one or more (possibly new) platforms
Repository manager scenarios:
the repository manager may
- grant/remove any or all access to
repository and/or release areas to/from package coordinators and/or
developers and/or users
Release manager scenarios: the
release manager may
- select or accept from corresponding coordinators packages and
package tags for public release of large set
of packages or for a production release
- request validation of a production
release following the project policy and QA plan
- remove a release: announce
scheduled removal to all users then delete
the release from the public release area (but not from the repository)
Framework and standard utilities
For all items below, it is assumed that all CMS offline software
domains shall use the standard CMS offline framework and the utilities
endorsed and supported by it. Such utilities comprise threading,
logging and monitoring etc
Logging and monitoring
in
preparation (E. Meschi)
Repository structure and
management
General organization
The CMS offline software shall reside in a single CVS repository to be
organised in two levels, following the standard SCRAM project structure:
+Project/+SubSystem/+Package
where
- Project = src
- SubSystem = directory grouping of packages
- Package = directory mapping to a fully-bound shared library
Package internal
structure:
- interface/
public interface files only i.e. header files that can be exposed to
other packages; recommended
(?) extension: .h
- src/
source and package-internal header files
; recommended (?) extensions: .cc and .h
respectively
- test/
sources, BuildFiles, scripts etc for unit tests
;
recommended (?) extensions
C++ sources: .cpp
- shell scripts: .sh, .csh
- python scripts: .py
doc/
If subsystems produce one or more
standard executables that serve specific purposes (i.e. they are
"applications" available to end-users as opposed to "tests"), e.g. the oscar simulation executable, the sources and
BuildFiles for these should (?) reside in an appropriate bin/ directory.
Repository
breakdown
see Appendix A
- to be updated with final
tree...
Repository
management
The CMS offline software repository
is hosted on the CMS CVS server.
Direct access to it is restricted to
the expert(s) responsible for its maintenance and support.
The repository is browsable using
standard WWW tools such as LXR.
Developer access via CVS commands
(cvs commit, cvs tag etc) is given by the repository
manager(s) upon request of domain and/or subsystem manager(s). This is
done using the module
responsible ("admin") and "developers" mechanisms.
The admins are assigned by the project leaders and are responsible for
the units and the developers of their domain.
Wildcard write access is not permitted except for very few rare cases
(if any at all) to be discussed and documented.
External software
- document policy for adding dependencies on new external software
and procedure for version upgrades
- document integration testing of external software in CMS offline
Builds, releases and distribution
Build and release tool
CMS offline software is built and
released using SCRAM.
Amongst possible SCRAM improvements, affecting developers and users
concerns the ease of migration of SCRAM working areas to latest
(pre)releases.
Build and release scope and types
releasable unit (or release unit) =
defined by a well identified function and usually mapped to one or more
binaries
Releases
differ from builds in the sense that they are frozen; released software
is publicly installed and distributed in RPM or other forms in Tier N
centres.
Regular builds
An automated regular build system is to
be put in place, utilizing trivially customizable tools such as nicos or Mozilla Tinderbox. Not to
be confused with nightly builds, the system aims to improve and
accelerate the release procedure and allow early integration tests by
the release manager. For that purpose, it will also cover quality
assurance tasks such as style and code checking, dependency analysis
and unit testing as well as documentation generation.
Project releases will be made based on working subsystem
release-candidate tags (e.g. Digitization_RC) that have been built and
tested in this system well in advance (or as soon as code is ready for
release). The build and test status will be summarized on a WWW page,
automatically updated for every build-and-test cycle. The system will
also support automatic e-mail notification in case of problems.
Production (major)
releases
A major release contains
- the basic underlying tools: POOL, ROOT etc
for example, build EDM Object Browser
(data structures)
- the digitisation executable
- the (Geant4) simulation (oscar) executable
- the HLT "executable"
- the production executable
- the calibration/monitoring executable
- the analysis executable(s)
For each executable the behaviour is
defined by the "job" configuration. The executable consists of the
"main" plus the complete set of basic libraries and the scheduler. The
scheduler loas the modules according to the configuration. The modules
come with standardized "hooks". One such hook is for explicit
scheduling, for example an EDProducer module is invoked through its
"produce" method. Another is for implicit scheduling; this corresponds
in functionality to the PkBuilder and are based on SEAL plug-ins (?)
Key steps for a major release
- identify release purpose (function) and scope
- identify required subsystems/packages and binaries
(libraries and executables) to be released by corresponding coordinators
- establish delivery schedule and build sequence, using recent
release history and overall status as indicated by daily build WWW form
- assemble release in common structure (to be defined),
including results of QA procedures (code checking, dependency analysis,
memory and performance profiling, unit testing
- publish release notes and update relevant documents
- apply global release tag (probably requires new functionality in
SCRAM)
- validate as needed
- depending on validation outcome, promote (apply CMS-standard tag
- to be defined) or deprecate
Developer releases
and prereleases
These roughly correspond to the current
scheme.
Distribution
in preparation (N. Ratnikova)
Supported platforms
It is crucial that CMS offline software
be built and tested on at least
one more platform other than the CERN-standard Scientific Linux
(slc3). CMS is represented in relevant IT and LCG
meetings where next targets are identified and priorities are set.
- now: Scientific Linux (slc3)
- next: 64-bit architectures
- timescale to be defined
- hardware for building available (CERN and other sites)
- status of external software?
- SCRAM-support?
- medium-to-longer term: MacOS? other?
Quality assurance
I. Physics
validation
Physics validation, which is
essentially a process to be defined and managed by the Physics
Coordination and the PRS, is not discussed here.
However, one important requirement that also affects the technical
validation issues addressed below, is that physics validation (at a level to
be defined) be performed on all T1 centres.
II. Technical
validation
Naming
conventions, coding and style
rules
Logging and monitoring
- use standard framework utilities and tools as described in
Framework and Standard Utilities Section
- note: this is especially important for the Production
infrastructure
Dependency analysis
and metrics
- policy: dependency
analysis shall be run and metrics collected for every public/production
release and the results (WWW browsable) shall be posted under
corresponding doc/ directory (note: this is an automated SCRAM
procedure anyway); success criteria
to be established
- tools: IgNominy
Unit testing
- policy: shall be carried
out (automated, under the responsibility of package coordinators and
the release manager) for
every release and, if feasible, for every build including the regular
one; reports of unit test failures should be automatically e-mailed to
developers concerned
- tools: CppUnit, PyUnit, OVAL,...
Memory and
performance monitoring
- policy: to be defined
- tools: valgrind, profiling tools
(IgProf, oprofile etc)
Problem tracking and change request management
- use Savannah
- to schedule and track change (feature) requests
- to generate summary reports as part of release notes
Documentation
- use meaningful cvs commit comments rather than maintain ChangeLog
proposals
being considered:
- automatic
e-mail notification for all commits incl. commit comments to be sent to
all developers of a package
- enforcing
cvs tag with comments
- use
of Savannah: cvs commits should require a Savannah report number, where
the relevant bug fix, change request or new feature is described and
justified
notes:
- a ChangeLog consisting of the cvs commits comments can be
automatically generated at any time
- IGUANA(CMS) developers use clcommit and
thus maintain ChangeLog; according to their experience,
it makes it easier to entermore meaningful comments
- use Doxygen to generate Reference Manuals
- Doxygen comments must include
- summary description of class, author name etc
- description of public methods
- WWW pages for ORCA, OSCAR, COBRA etc
- several improvements may be envisaged
- proposal:
maintain centrally and allow
regular updates by release manager
Appendix A: Repository and SCRAM Project Structure
The issue
Should we adopt +Project/+Domain/+SubSystem/+Package
thus introducing an extra level to the structure of a SCRAM project?
The Current Scheme
Standard SCRAM project structure: +Project/+SubSystem/+Package
where
"Project" = src
"SubSystem" = directory which groups packages together
"Package" = directory which maps to a fully-bound shared library
Current CMS repository structure contains project sources in a
directory PROJECT under repository/<PROJECT> which, when checked
out, populate the
src directory.
E.g. repository/COBRA/COBRA/CARF becomes <cobra project>/src/CARF
All dependencies are defined in terms of packages.
The Proposed Scheme
New repository structure, one project - CMSSW - where each of the
current projects becomes a domain (current names left for clarity):
+Project/+Domain/+SubSystem/+Package
This implies a source tree like:
<cmssw project>/src/COBRA/CARF/...
<cmssw project>/src/ORCA/Muon/...
<cmssw project>/src/OSCAR/Mantis/...
Domain is a grouping of subsystems (not a functional grouping).
SCRAM Build Mechanism
Build can be split into two distinct parts (oversimplified...):
- Parsing all BuildFiles in the source tree to collect metadata,
determine build ordering of packages and generate project Makefile
- Run "gmake -f Makefile <TARGET>"
Time taken for step 1 is proportional to number of BuildFiles (every
package needs a BuildFile to define it and express dependencies).
SCRAM was designed for projects with sizes like ORCA (~300 BuildFiles).
With all project sources combined (based on last releases) there are
~660 BuildFiles.
A slowdown of step 1 can be expected with or without change in project
structure.
Support for a Domain Level
Support for this structure added to SCRAM for testing and seems to
work.
Timings for step 1 slightly slower with new structure but not a
significant effect.
But significant disadvantages...
Disadvantages
- Separation serves no functional purpose (cf. subsystems which are
"collections of packages" with a well-defined interface, namely the
group).
- The domain must be added to all #include in source files
(consequently #includes become more difficult to read and to type).
Alternatively, with "subsystem/package/interface/header.h" format, the
project "Self" tool must define extra INCLUDEs (one for each extra
domain).
- Added confusion for users: <use name=subsystem/package> now
becomes
<use name=domain/subsystem/package>.
- The structure is more likely to result in a proliferation of
packages leading to slowdown of SCRAM:
- the deeper the source tree, the more time taken to iterate
and collect timestamps of BuildFiles and directories (at first glance,
less serious than previously thought)
- for every extra package a BuildFile is added.
Appendix B: Logging
in preparation (E. Meschi)