Pylcgdict Overview


Some parts of this document are still under development. Text appearing in this colour highlights that which is known to be missing.

Setting up the environment


Pylcgdict provides a shell script which prepares the environment for its use. This script relies on access to the cern.ch AFS cell. If this script does not work for you, please refer to the detailed environment setup instructions.

Set up the environment by sourcing the env.sh script provided by Pylcgidct:
$> source /afs/cern.ch/sw/lcg/app/releases/SEAL/SEAL_HEAD/src/Scripting/PyLCGDict2/env.sh
In short, this ensures that the correct compiler version is used, that the correct version of Python is used, that the Seal products (lcgdict, Reflection) are available ... see detailed environment setup instructions for the gory details.

Generating your own dictionary


One of Pylcgdict's chief advantages over other Python/C++ gateways is the ease with which a user can gain access to C++ libraries from Python, even when no Python binding for that library is available. This is achieved by using LCG dictionaries as the interface between Python an C++. Creating a PyLCGDict binding to a C++ library is as simple as generating the LCG Dictionary for that library. In this example we show how to generate the dictionary for a small library. For more information about LCG Dictionaries, refer to the Reflection documentation.
The dictionary we are about to generate is used in the interactive example that follows.

LCG dictionaries are built from information found in C++ header files. Seal provides the lcgdict tool for generating the dictionary source. To generate the dictionary for the overview library, invoke lcgdict like this:
$> lcgdict overview.h
Parsing file overview.h with GCC_XML OK Generating LCG Dictionary class DynamicDerived class DynamicBase class Expose class Derived class Base class Staticmethods class Overload class MyFirstClass class Templates::Nestme<Templates::Parametrized<Outer::Inner::AClass,MyFirstClass> > class Templates::Parametrized<Outer::Inner::AClass,MyFirstClass> class Templates::Parametrized<int,float> class Templates::Parametrized<int,double> class Outer::Inner::AClass class Outer::Inner::AClass::InnerClass

Note that lcgdict lists all the classes it adds to the dictionary. It places the dictionary source in a file called overview_dict.cpp:
$> head -30 overview_dict.cpp
// Generated at Tue Jul 20 15:59:13 2004. Do not modify it ... //------Dictionary for class DynamicDerived ------------------------------- class DynamicDerived_dict { public: DynamicDerived_dict(); static int tobase_8(void*); static void* constructor_1008( void*, const std::vector<void*>& ); static void* constructor_1009( void* ); static void destructor( void* o ) { ((DynamicDerived*)o)->~DynamicDerived(); } static void* method_1011( void* ); static void* method_1012( void* ); static void method_1013( void*, const std::vector<void*>& ); }; ...

The dictionary source must now be compiled into a shared library. The dictionary must be linked with Seal's ReflectionBuilder library:
$> g++ -shared -o libOverviewDict.so -fpic overview_dict.cpp -llcg_ReflectionBuilder -I$SEAL/include -L$SEAL/$SCRAM_ARCH/lib

$> ls -l libOverviewDict.so
-rwxr-xr-x 1 jacek zb 87041 Jul 20 15:59 libOverviewDict.so

The dictionary libOverviewDict.so will be used by the Pylcgdict runtime. The dynamic library loader looks for libraries in the locations listed in the LD_LIBRARY_PATH environment variable. Make sure that the location of any dictionaries you build is included in LD_LIBRARY_PATH:
$> export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH

Using Pylcgdict itself


The interactive Python session shown below, describes how Pylcgdict interacts with the library defined in the file PyLCGDict2/doc/overview/overview.h, for which the dictionary was generated above, and with the C++ STL dictionary which is made available in Seal releases. You will need to refer to overview.h to fully understand the output shown.

start Python:
$> python
Python 2.3.4 (#1, Jun 7 2004, 11:39:13) [GCC 3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.

Import the PyLCGDict package
>>> import pylcgdict

Load the dictionary we created (on the basis of the header file overview.h):
>>> pylcgdict.loadDictionary('OverviewDict')
Loaded dictionary OverviewDict

Bind the C++ global namespace to the Python variable g:
>>> g = pylcgdict.Namespace('')

You can use Python's dir function to inspect the namespace:
>>> dir(g)
['Base', 'Derived', 'DynamicBase', 'DynamicDerived', 'Expose', 'MyFirstClass', 'Outer', 'Overload', 'Staticmethods', 'Templates', '_C_metanamespace', '__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', '_add_attribute', 'std']

Pylcgdict provides a more C++ friendly version (pylcgdict.dir). NOTE this is still experimental.
>>> pylcgdict.dir(g)
['Base', 'Derived', 'DynamicBase', 'DynamicDerived', 'Expose', 'MyFirstClass', 'Outer', 'Overload', 'Staticmethods', 'Templates', 'std']

Classes


This section relates to the following portion of overview.h:

class MyFirstClass { public: MyFirstClass(): datum() {} MyFirstClass(int i): datum(i) {} int datum; int get_datum() { return datum; } void set_datum(int i) { datum = i; } };
Instantiating a class looks similar to the way it would be done in C++:
>>> mc = g.MyFirstClass(1)

mc is an object of type pylcgdict.MyFirstClass:
>>> mc
<pylcgdict.MyFirstClass object at 0x401e8a6c>
>>> type(mc)
<class 'pylcgdict.MyFirstClass'>

Note that dir reports the existence of the constructor (MyFirstClass), the field (datum), and the two methods (get_datum, set_datum):
>>> pylcgdict.dir(mc)
['MyFirstClass', 'datum', 'get_datum', 'set_datum']

Methods are invoked just like they would be in C++:
>>> mc.get_datum()
1
>>> mc.set_datum(2)

Fields are accessed just like they would be in C++:
>>> mc.datum
2
>>> mc.datum = 3
>>> mc.get_datum()
3

Namespaces


This section relates to the following portion of overview.h:

namespace Outer { namespace Inner { class AClass { public: class InnerClass { // A pretty boring one }; }; } }
Pylcgdict reflects the C++ namespace hierarchy found in the loaded dictionaries. The C++ scope resolution operator (::) maps to Python's attribute access syntax (.). g.Outer is a pylcgdict.Namespace whose name is "Outer":
>>> g.Outer
<pylcgdict.Namespace Outer object at 0x401ee2cc>

It contains the Inner namespace:
>>> pylcgdict.dir(g.Outer)
['Inner']
>>> g.Outer.Inner
<pylcgdict.Namespace Inner object at 0x401ee38c>

... which, in turn, contains AClass
>>> pylcgdict.dir(g.Outer.Inner)
['AClass']
>>> g.Outer.Inner.AClass
<class 'pylcgdict.AClass'>

... and the latter contains an inner class
>>> pylcgdict.dir(g.Outer.Inner.AClass)
['AClass', 'InnerClass']
>>> g.Outer.Inner.AClass.AClass
<pylcgdict.Constructor object at 0x401ee4cc>
>>> g.Outer.Inner.AClass.InnerClass
<class 'pylcgdict.InnerClass'>

Templates


This section relates to the following portion of overview.h:

namespace Templates { template<class T, class U> class Parametrized { public: T t; U u; Parametrized() {} Parametrized(T tt, U uu): t(tt), u(uu) {} T get_tea() { return t; } U get_you() { return u; } }; // The compiler must be instructed to generate code for the // templates you will want to use: the templates must be // instantiatied. Parametrized<int, double> pid; Parametrized<int, float> pif; Parametrized<Outer::Inner::AClass, MyFirstClass> pam; template<class T> class Nestme { public: std::string whoami() { return typeid(*this).name(); } }; // Force compiler to instantiate. Nestme<Parametrized<Outer::Inner::AClass, MyFirstClass> > npa; }
The g.Templates namespace contains a template called "Parametrized":
>>> g.Templates.Parametrized
<Template 'Templates::Parametrized<>'>

We will store it in a Python variable to save typing:
>>> gTP = g.Templates.Parametrized
>>> gTP
<Template 'Templates::Parametrized<>'>

The instantiations which can be made are shown by dir:
>>> pylcgdict.dir(gTP)
[]
There is a bug here: the instantiations do to show up in dir until after they have been instantiated for the first time. The new dictionary will allow this to be fixed trivially, so the bug will probably remain until the new dictionary is available.

The angled brackets which enclose template arguments in C++, map to parentheses in Python: Parametrized<int, double> becomes Parametrized(int, double).
The instantiated templates appear as Python classes:
>>> gTP(int, float)
<class 'pylcgdict.Parametrized<int,float>'>
>>> gTP(int, double)
<class 'pylcgdict.Parametrized<int,double>'>
>>> gTP(g.Outer.Inner.AClass, g.MyFirstClass)
<class 'pylcgdict.Parametrized<Outer::Inner::AClass,MyFirstClass>'>

They can be instantiated:
>>> Pif = gTP(int, float)(1, 2.5)
>>> Pif
<pylcgdict.Parametrized<int,float> object at 0x401ee8ac>

... and their instances can be used just like those of any other class:
>>> Pif.get_tea()
1
>>> Pif.get_you()
2.5

Note that although Python has only one precision of floating point numbers (equivalent to C's double), pylcgdict provides both float and double as symbols for use in template arguments:
>>> gTP(int, float )(1, 0.8).get_you(); type(_)
0.80000001192092896 <type 'float'>
>>> gTP(int, double)(1, 0.8).get_you(); type(_)
0.80000000000000004 <type 'float'>
(In Python's interactive shell, the variable _ is bound to the result of the last command, so type(_), in this context, gives us the Python type of the value returned by the preceding calls to the get_you methods.)
Note the loss of precision in the case of the float template argument in the above. Even though the result is stored in a Python (double precision) float in both cases, in the first case the underlying C++ method returned a single precision C++ float which was converted to Python's double precision float.

Templates can be nested:
>>> g.Templates.Nestme(g.Templates.Parametrized(g.Outer.Inner.AClass, g.MyFirstClass))()
<pylcgdict.Nestme<Templates::Parametrized<Outer::Inner::AClass,MyFirstClass> > object at 0x401f42ac>

Sometimes the template parameter is a type for which no dictionary is available. In such cases you can pass the type name as a string: ... to be done ... FIXME

Overloaded methods


This section relates to the following portion of overview.h:
class Overload { public: std::string overloaded() { return "void"; } std::string overloaded(int) { return "int"; } std::string overloaded(double) { return "double"; } std::string overloaded(MyFirstClass) { return "MyFirstClass"; } std::string overloaded(int, int) { return "int, int"; } std::string overloaded(int, MyFirstClass) { return "int, MyFirstClass"; } };
You can use Python's help to see the signatures of methods:
>>> help(g.Overload.overloaded)
Help on method dispatching_method in module pylcgdict: dispatching_method(self, *args) unbound pylcgdict.Overload method std::basic_string<char> () std::basic_string<char> (int) std::basic_string<char> (double) std::basic_string<char> (MyFirstClass) std::basic_string<char> (int, int) std::basic_string<char> (int, MyFirstClass)
We see in the above that g.Overload.overloaded is an overloaded method. There are six different methods by this name. Each returns std::basic_string<char> (for which std::string is a typedef). The first method takes zero arguments, the second takes a single int ... the last takes int and MyFirstClass as arguments.

The overloaded methods can be called just like they would be called in C++; Pylcgdict takes care of invoking the one which matches the actual types of the arguments:
>>> ov = g.Overload()
>>> ov.overloaded()
'void'
>>> ov.overloaded(1)
'int'
>>> ov.overloaded(1.0)
'double'
>>> ov.overloaded(g.MyFirstClass())
'MyFirstClass'
>>> ov.overloaded(1, 1)
'int, int'
>>> ov.overloaded(1, g.MyFirstClass())
'int, MyFirstClass'

If the arguments passed do not match any of the signatures, then a pylcgdict.NoMatchingMethod exception is raised:
>>> ov.overloaded(1, g.MyFirstClass(), 2.3)
Traceback (most recent call last): ... NoMatchingMethod: Overload::overloaded(<type 'int'>, <class 'pylcgdict.MyFirstClass'>, <type 'float'>) Candidates: () (int) (double) (MyFirstClass) (int, int) (int, MyFirstClass)
Note how Pylcgdict reports the types which were passed as arguments, as well as list of the sets of argument types which are acceptable.

NoMatchingMethod is a subclass of TypeError.

Static methods


This section relates to the following portion of overview.h:

class Staticmethods { public: static std::string staticmethod() { return "void"; } static std::string staticmethod(int) { return "int"; } };
Static methods can be called through an instance of the class:
>>> sm = g.Staticmethods()
>>> sm.staticmethod()
'void'
>>> sm.staticmethod(1)
'int'
... or without any instance, through the class:
>>> g.Staticmethods.staticmethod()
'void'
>>> g.Staticmethods.staticmethod(1)
'int'

Sequence protocol.


Load the dictionary for the C++ STL, which is provided in SEAL releases
>>> pylcgdict.loadDictionary('SealSTLDict')
Loaded dictionary SealSTLDict

Note that this has added list and vector into the std namespace:
>>> pylcgdict.dir(g.std)
['basic_string', 'list', 'vector']

Let us consider vector<double>:
>>> g.std.vector(double)
<class 'pylcgdict.vector<double>'>

Instantiate one with 3 (default) elements:
>>> vd = g.std.vector(double)(3)
By default, Pylcgdict displays the contents of the vector in the printed representation of the instances:
>>> vd
<vector<double>:[0.0, 0.0, 0.0]>

The std::vector methods are usable from Python:
>>> pylcgdict.dir(vd)
['=', '[]', 'append', 'assign', 'at', 'back', 'begin', 'capacity', 'clear', 'empty', 'end', 'erase', 'front', 'get_allocator', 'insert', 'max_size', 'pop_back', 'push_back', 'readData', 'reserve', 'resize', 'size', 'swap', 'vector', 'writeData', '~vector']
>>> vd.push_back(1); vd.push_back(2)

The Python sequence protocol is automatically supported:
>>> vd[3]
1.0
>>> len(vd)
5

... and so is the iterator protocol. Python for loops can loop over the object:
>>> [x*x for x in vd]
[0.0, 0.0, 0.0, 1.0, 4.0]
... and iterators can be acquired explicitly:
>>> vd2 = g.std.vector(double)()
>>> vd2.push_back(15)
>>> vd2.push_back(16)
>>> ivd = iter(vd2)
>>> ivd.next()
15.0
>>> ivd.next()
16.0
>>> ivd.next()
Traceback (most recent call last): File "<console>", line 1, in ? StopIteration
Note: the StopIteration exception is Python's standard mechanism for signalling that an iterator has been exhausted.

Inheritance


This section relates to the following portion of overview.h:

class Base { public: virtual std::string whoami() { return "Base"; } virtual std::string inheritme() { return "This was defined in Base"; } }; class Derived : public Base { std::string whoami() { return "Derived"; } };
Pylcgdict reflects the class hierarchies it finds in the original library:
>>> g.Derived.__bases__
(<class 'pylcgdict.Base'>,)
>>> isinstance(g.Derived(), g.Base)
True
Methods (and data) are inherited and overridden as one expects in C++:
>>> b = g.Base()
>>> d = g.Derived()
>>> b.whoami()
'Base'
>>> d.whoami()
'Derived'
>>> b.inheritme()
'This was defined in Base'
>>> d.inheritme()
'This was defined in Base'

Pointers, values and references


This section relates to the following portion of overview.h:

class Expose { private: std::string text; public: Expose(std::string t): text(t) {} std::string passvalue (Expose e) { return e.text; } std::string passreference(Expose& e) { return e.text; } std::string passpointer (Expose* e) { if (e) {return e->text;} return "NULL pointer"; } void outputparameter(std::string t, Expose*& e) { e = new Expose(t); } Expose getvalue() { return *this; } Expose& getreference() { return *this; } Expose* getpointer(bool wantNULL=false) { if (wantNULL) { return 0; } else { return this; } } std::string whoami() { return text; } };
There are no pointers in Python and, ideally, users should be protected from ever having to handle a raw pointer. Unfortunately very many C++ library interfaces do expose pointers and, sometimes, it is impossible to avoid handling them in Python. Pylcgdict has pointer types which try to behave, as much as possible, as the underlying type referenced by the pointer.
>>> e1 = g.Expose('one')
>>> e2 = g.Expose('two')
>>> type(e1)
<class 'pylcgdict.Expose'>
>>> e1.passvalue(e2)
'two'
>>> e1.passreference(e2)
'two'

Pylcgdict considers an instance of a type to be a valid argument even if the pointer to the type is expected:
>>> e1.passpointer(e2)
'two'

Sometimes, however, the interface really requires a pointer. Pylcgdict provides a Pointer metatype, which can create pointer types, or pointers to existing instances. One can create the Python proxy for the type Expose* by passing the desired type to pylcgdict.Pointer:
>>> pylcgdict.Pointer(g.Expose)
<class 'pylcgdict.Pointer(Expose)'>

A pointer to an existing object can be made by instantiating an appropriate Pointertype with the existing object as an argument:
>>> ep2 = pylcgdict.Pointer(g.Expose)(e2)
>>> ep2
<pylcgdict.Pointer(Expose) object at 0x4022decc>

Such pointers can be passed to functions expecting Expose*:
>>> e1.passpointer(ep2)
'two'
... and they can also be passed to functions expecting Expose values:
>>> e1.passvalue(ep2)
'two'
>>> e1.passreference(ep2)
'two'

The pointer types try to behave as the underlying type whenever possible:
>>> ep2.passvalue(e1)
'one'
Note that the above is equivalent to ep2->passvalue(e1) in C++. Consequently the C++ dereference operator "->" is mapped to Python's attribute access syntax "." (This has not been implemented yet: FIXME).

Objects returned from methods always appear as values, regardless of whether the formal return type is a value, reference or pointer:
>>> type(e1.getvalue())
<class 'pylcgdict.Expose'>
>>> type(e1.getreference())
<class 'pylcgdict.Expose'>
>>> type(e1.getpointer())
<class 'pylcgdict.Expose'>

pylcgdict.Deref

NULL Pointers


This section relates to the following portion of overview.h:

class Expose { private: std::string text; public: Expose(std::string t): text(t) {} std::string passvalue (Expose e) { return e.text; } std::string passreference(Expose& e) { return e.text; } std::string passpointer (Expose* e) { if (e) {return e->text;} return "NULL pointer"; } void outputparameter(std::string t, Expose*& e) { e = new Expose(t); } Expose getvalue() { return *this; } Expose& getreference() { return *this; } Expose* getpointer(bool wantNULL=false) { if (wantNULL) { return 0; } else { return this; } } std::string whoami() { return text; } };
A NULL pointer of a given type, can be created by instantiating the appropriate Pointer type with 0 as argument:
>>> epNULL = pylcgdict.Pointer(g.Expose)(0)
Note that pylcgdict notices that it has a NULL pointer, and stores it in a special NULLPointer(...) type:
>>> epNULL
<pylcgdict.NULLPointer(Expose) object at 0x4027466c>

Alternatively, a NULL pointer may be returned by some function:
>>> e1.getpointer(True); e1.getpointer(False)
<pylcgdict.NULLPointer(Expose) object at 0x402747ec> <pylcgdict.Expose object at 0x402747ac>
Note that pylcgdict recognizes when a function returns a NULL pointer, and stores it in the NULLPointer type.

The NULLPointer type protects you against memory access violations by raising a NullPointer exception when you try perform an operation which would result in the NULL pointer being dereferenced. This could happen when you try to access attributes of the object which is pointed to:
>>> epNULL.whoami()
Traceback (most recent call last): ... NullPointer: You are trying to invoke a method via a NULL pointer.
... or when you pass the NULL pointer where a vaule or reference is expected:
>>> e1.passvalue(epNULL)
Traceback (most recent call last): ... NoMatchingMethod: Expose::passvalue(<class 'pylcgdict.NULLPointer(Expose)'>,) Candidates: (Expose)
When the formal argument type is a pointer, it is entirely up to the caller to decide whether it is safe to do so; pylcgdict will not intervene.

Raw pointers are made unavoidable by interfaces which require pointers as output parameters. The NULL pointer we created above might be just what you need:
>>> e1.outputparameter('returned via output parameter', epNULL)
This makes the pointer point to a valid object. Pylcgdict does not, initially, notice this, and continues to believe that it is a NULL pointer:
>>> epNULL
<pylcgdict.NULLPointer(Expose) object at 0x4027466c>
... but it does notice as soon as you try to use the object ...
>>> epNULL.whoami() # previously this raised NullPointer
'returned via output parameter'
... and automatically changes its type to be a non-NULL pointer:
>>> epNULL
<pylcgdict.Pointer(Expose) object at 0x4027466c>
Note that the object's IDENTITY (address) remains unchanged, even though the type is modified.

Automatic Dynamic Casting


This section relates to the following portion of overview.h:

class DynamicBase { public: void fubar() {} virtual ~DynamicBase() {} }; class DynamicDerived : public DynamicBase { public: DynamicBase* getBase() { return new DynamicBase; } DynamicBase* getDerived() { return new DynamicDerived; } void modify(int select, DynamicBase*& it) { if (select == 0) { it = new DynamicBase; } else { it = new DynamicDerived; } } };
In C++ the actual (dynamic) type of an object may differ from its formal (static) type. It is up to the programmer to discover the true dynamic type by guessing it, performing a dynamic_cast and checking the validity of the resulting pointer. Such low-level details are none of the programmer's concern in Python, so Pylcgdict downcasts all returned values to their true dynamic type:
>>> dd = g.DynamicDerived()
>>> help(dd.getBase)
... DynamicBase* ()
>>> type(dd.getBase())
<class 'pylcgdict.DynamicBase'>
... note how the dynamic type matches the formal return type (as shown by help) of the method in the above, but not in the following ...
>>> help(dd.getDerived)
... DynamicBase* ()
>>> type(dd.getDerived())
<class 'pylcgdict.DynamicDerived'>

If, however, the dynamic type of an existing Python proxy object is changed as follows:
>>> db = g.DynamicBase()
>>> dd.modify(1, db)
such changes are too subtle for Pylcgdict to handle efficiently, and it does not perform a type change automatically:
>>> type(db)
<class 'pylcgdict.DynamicBase'>
In such cases the conversion to the new dynamic type must be requested explicitly:
>>> pylcgdict.dynamic_downcast(db)
>>> type(db)
<class 'pylcgdict.Pointer(DynamicDerived)'>
NOTE ... the dynamic casting interface and behaviour will change (improve!) in the future.

Subclassing C++ classes in Python: callbacks
Contact: Jacek Generowicz