last updated: 27.06.2005, 15:00 CEST

CMS Offline Repository and Releases


Contents


Basic glossary and work model

Basic release types in work model

developer release  (or prerelease) = produced by developers for developers - no quality requirements - frozen and installed in public release area
public release = produced by package coordinators - validated (technical validation), released, frozen and installed in public release area
production release = produced by package coordinators - as above but also validated (physics validation) by the CMS PRS groups
User scenarios: a user may
Developer scenarios: a developer is a user who may also

Package coordinator scenarios: a package coordinator is a developer who may also
Site manager scenarios: a site manager may
Repository manager scenarios: the repository manager may
Release manager scenarios: the release manager may

Framework and standard utilities

For all items below, it is assumed that all CMS offline software domains shall use the standard CMS offline framework and the utilities endorsed and supported by it. Such utilities comprise threading, logging and monitoring etc

Logging and monitoring
in preparation (E. Meschi)


Repository structure and management

General organization

The CMS offline software shall reside in a single CVS repository to be organised in two levels, following the standard SCRAM project structure:
+Project/+SubSystem/+Package
where
Package internal structure:
If subsystems produce one or more standard executables that serve specific purposes (i.e. they are "applications" available to end-users as opposed to "tests"), e.g. the oscar simulation executable, the sources and BuildFiles for these should (?) reside in an appropriate bin/ directory.

Repository breakdown

see Appendix A

Repository management

The CMS offline software repository is hosted on the CMS CVS server.
Direct access to it is restricted to the expert(s) responsible for its maintenance and support.
The repository is browsable using standard WWW tools such as LXR.
Developer access via CVS commands (cvs commit, cvs tag etc)  is given by the repository manager(s) upon request of domain and/or subsystem manager(s). This is done using the module responsible ("admin") and "developers" mechanisms.
The admins are assigned by the project leaders and are responsible for the units and the developers of their domain.
Wildcard write access is not permitted except for very few rare cases (if any at all) to be discussed and documented.

External software


Builds, releases and distribution

Build and release tool

CMS offline software is built and released using SCRAM.

Amongst possible SCRAM improvements, affecting developers and users concerns the ease of migration of SCRAM working areas to latest (pre)releases.

Build and release scope and types

releasable unit (or release unit) = defined by a well identified function and usually mapped to one or more binaries

Releases differ from builds in the sense that they are frozen; released software is publicly installed and distributed in RPM or other forms in Tier N centres.

Regular builds

An automated regular build system is to be put in place, utilizing trivially customizable tools such as nicos or Mozilla Tinderbox. Not to be confused with nightly builds, the system aims to improve and accelerate the release procedure and allow early integration tests by the release manager. For that purpose, it will also cover quality assurance tasks such as style and code checking, dependency analysis and unit testing as well as documentation generation.
Project releases will be made based on working subsystem release-candidate tags (e.g. Digitization_RC) that have been built and tested in this system well in advance (or as soon as code is ready for release). The build and test status will be summarized on a WWW page, automatically updated for every build-and-test cycle. The system will also support automatic e-mail notification in case of problems.

Production (major) releases

A major release contains
For each executable the behaviour is defined by the "job" configuration. The executable consists of the "main" plus the complete set of basic libraries and the scheduler. The scheduler loas the modules according to the configuration. The modules come with standardized "hooks". One such hook is for explicit scheduling, for example an EDProducer module is invoked through its "produce" method. Another is for implicit scheduling; this corresponds in functionality to the PkBuilder and are based on SEAL plug-ins (?)

Key steps for a major release

Developer releases and prereleases

These roughly correspond to the current scheme.

Distribution 

in preparation (N. Ratnikova)

Supported platforms

It is crucial that CMS offline software be built and tested on at least one more platform other than the CERN-standard Scientific Linux (slc3).  CMS is represented in relevant IT and LCG meetings where next targets are identified and priorities are set.

Quality assurance

I. Physics validation

Physics validation, which is essentially a process to be defined and managed by the Physics Coordination and the PRS, is not discussed here.
However, one important requirement that also affects the technical validation issues addressed below, is that physics validation (at a level to be defined) be performed on all T1 centres.

II. Technical validation

Naming conventions, coding and style rules

Logging and monitoring

Dependency analysis and metrics

Unit testing

Memory and performance monitoring


Problem tracking and change request management


Documentation


Appendix A: Repository and SCRAM Project Structure

The issue

Should we adopt  +Project/+Domain/+SubSystem/+Package thus introducing an extra level to the structure of a SCRAM project?

The Current Scheme

Standard SCRAM project structure:  +Project/+SubSystem/+Package
where
"Project" = src
"SubSystem" = directory which groups packages together
"Package" = directory which maps to a fully-bound shared library

Current CMS repository structure contains project sources in a directory PROJECT under repository/<PROJECT> which, when checked out, populate the
src directory.
E.g. repository/COBRA/COBRA/CARF becomes <cobra project>/src/CARF
All dependencies are defined in terms of packages.

The Proposed Scheme

New repository structure, one project - CMSSW - where each of the current projects becomes a domain (current names left for clarity):
+Project/+Domain/+SubSystem/+Package

This implies a source tree like:
     <cmssw project>/src/COBRA/CARF/...
     <cmssw project>/src/ORCA/Muon/...
     <cmssw project>/src/OSCAR/Mantis/...

Domain is a grouping of subsystems (not a functional grouping).

SCRAM Build Mechanism

Build can be split into two distinct parts (oversimplified...):
  1. Parsing all BuildFiles in the source tree to collect metadata, determine build ordering of packages and generate project Makefile
  2. Run "gmake -f Makefile <TARGET>"
Time taken for step 1 is proportional to number of BuildFiles (every package needs a BuildFile to define it and express dependencies).
SCRAM was designed for projects with sizes like ORCA (~300 BuildFiles). With all project sources combined (based on last releases) there are ~660 BuildFiles.
A slowdown of step 1 can be expected with or without change in project structure.

Support for a Domain Level

Support for this structure added to SCRAM for testing and seems to work.
Timings for step 1 slightly slower with new structure but not a significant effect.
But significant disadvantages...

Disadvantages
  1. Separation serves no functional purpose (cf. subsystems which are "collections of packages" with a well-defined interface, namely the group).
  2. The domain must be added to all #include in source files (consequently #includes become more difficult to read and to type). Alternatively, with "subsystem/package/interface/header.h" format, the project "Self" tool must define extra INCLUDEs (one for each extra domain).
  3. Added confusion for users: <use name=subsystem/package> now becomes   
    <use name=domain/subsystem/package>.
  4. The structure is more likely to result in a proliferation of packages leading to slowdown of SCRAM:

Appendix B: Logging

in preparation (E. Meschi)