Passing python functions to I3Tray.AddModule()

:py:I3Tray.AddModule() has typically accepted only strings in its first argument. These strings are used to look up C++ classes that inherit from I3Module, like so:

tray.AddModule('Dump', 'dump')

where the C++ module Dump has been registered via the C++ I3_MODULE() macro.

I3Tray.AddModule() now additionally accepts python functions in its first argument, for instance:

tray = I3Tray()

def frame_printer(frame):
    print "Frame is:\n", frame

tray.AddModule(frame_printer, 'printer')

When icetray receives a python function as the first argument to AddModule, it constructs a special I3Module of type PythonFunction which forwards the frames that it receives to the python function passed. By default it does this for the Physics stream only.

Warning

it is not

AddModule('frame_printer', ...

it is

AddModule(frame_printer, ...

If you see a message like

RuntimeError: Module/service “frame_printer” not registered with I3_MODULE() or I3_SERVICE_FACTORY()

it is because you’ve put the python function into quotes, making it a string, and icetray is failing to find that string in its registry of available C++ I3Modules.

If you put that function between a BottomlessSource, (which just pushed empty physics frames) and a TrashCan and run a couple of frames, the output should look like this:

Frame is:
[ I3Frame :
]

Frame is:
[ I3Frame :
]

Let’s add a second function that puts something into the frame, and modify the frame_printer function to get and print it.

def int_putter(frame):
    frame['some_int'] = icetray.I3Int(777)

tray.AddModule(int_putter, 'putter')


def frame_printer(frame):
    print "Frame is:\n", frame
    value = frame['some_int'].value
    print "Value of int at some_int is", value

tray.AddModule(frame_printer, 'printer')

Here the function int_putter() puts I3Int with the value 777 in into the frames as they go by. This is reflected in the table of contents printed by the frame_printer() function.

Output:

Frame is:
[ I3Frame :
  'some_int' ==> I3Int
]
Value of int at some_int is 777

Frame is:
[ I3Frame :
  'some_int' ==> I3Int
]
Value of int at some_int is 777

Functions with parameters

To be useful, reusable and modular, such functions need to take parameters such as the location in the frame of useful frame objects, values, thresholds, etc. The hardcoded values 777 and some_int just make our code brittle.

Functions passed to AddModule() may take more than one parameter (the first parameter is always the I3Frame that is flowing through the framework). The parameter values passed to AddModule() will be delivered (along with the current I3Frame, of course) to the keyword parameters of the associated python function passed each time the function is executed.

We modify the function int_putter() to accept parameters that specify what value to put inside the I3Int, and where in the frame to put them:

def int_putter(frame, where = 'someplace', value = -1):
    frame[where] = icetray.I3Int(value)

tray.AddModule(int_putter, 'putter',
               where = 'some_int',
               value = 777)

def frame_printer(frame, whatvalue):
    print "Frame is:\n", frame
    value = frame[whatvalue].value
    print "Value of int at", whatvalue, "is", value

tray.AddModule(frame_printer, 'printer',
               whatvalue = 'some_int')

Note the default parameter values for the function int_putter().

Choosing streams the functions should run on

The underlying PythonFunction module also takes a parameter Streams, which is a list of stream types that the function should run on. By default this list is [icetray.I3Frame.Physics]. To e.g. cause a python function foo() to run on Calibration and Geometry streams, configure as follows:

from icecube import icetray

def foo(frame):
    ...  # do something physicsy here

tray.AddModule(foo, 'foofunc',
               Streams = [icetray.I3Frame.Geometry,
                          icetray.I3Frame.Calibration])

Functions as filters

The functions passed to AddModule() may return None (i.e. never call return at all), or a boolean. The PythonFunction module examines the return values of these functions and if the value is None or True, the module will call PushFrame(): modules further down the chain will see the frame. If the function returns False, the module will drop the frame.

Note

The rationale for having None and True correspond to the same action (typically None is taken to be False), is so that the ‘default’ behavior (when nothing is returned) is reasonable. Otherwise one- or two-line functions that just check or print data would need to have lines return True added. The thinking is that this extra work to provoke behavior that should be default isn’t so elegant. So the rule of thumb is, if you want to drop the frame, return False, otherwise don’t bother returning anything (or return True if it is clearer to do so).

For instance, the following code would cause frames that contain an I3Int with value less than 80 to be dropped:

def ints_are_greater_than(frame,  key,  threshold):
    frameval = frame[key].value
    return frameval > threshold

tray.AddModule(ints_are_greater_than,
               key = 'intlocation',
               threshold = 80)

Passing python functions to I3ConditionalModules

The old way

Recall that an I3ConditionalModule looks for an I3IcePick in its I3Context, indexed by string. So the user must configure an I3IcePickInstaller (where T is the class containing the desired pick logic) and the name given by the user to the instance of this pick logic must match the name that the using module accesses it by.:

tray.AddService('I3IcePickInstaller<I3FrameObjectFilter>', 'fofilter')(
    ("FrameObjectKey", 'some_int')
    )

tray.AddModule('AddNulls', 'adder')(
    ('IcePickServiceKey', 'fofilter'),
    ('where', ['x1', 'x2', 'x3'])
    )

Here the module AddNulls, being an I3ConditionalModule, will add nulls named ‘x1’, ‘x2’, and ‘x3’ to the frame when its icepick, located in its context via the string ‘fofilter’, returns true.

This has several disadvantages:

• The logic that triggers the AddNulls module is separated from the configuration of the module itself
• There is the possibility for name collisions in the various I3Contexts.

If the condition is complicated, for instance the disjunction of two other conditions, the syntax gets yet more verbose.

The new way

As of icetray v3, one can pass a python function to the parameter If of I3ConditionalModules. Identical to the above is the following:

tray.AddModule('AddNulls', 'adder',
               Where = ['x1', 'x2', 'x3'],
               If    = lambda frame: 'some_int' in frame)

Here we use a lambda, (nameless inline) function. Check google for more information on this standard python construct.

Another example: run the reconstruction LineFit if the I3Int at ‘where’ is greater than 80:

def ints_are_greater_than_80(frame):
    frameval = frame['where'].value
    return frameval > 80

tray.AddModule('LineFit', 'linefit',
               HitSeries = 'WhereTheIntIs',
               If = ints_are_greater_than_80)

Note that in this case the key in the frame and the value ‘80’ are hardcoded inside the python function we pass. Not so good: we want to reuse the functions we wrote in previous sections. To do so we use a small python forwarding function:

def fwd(fn, **kwargs):
    def wrap(frame):
        return fn(frame, **kwargs)
    return wrap

Which captures the values of parameters passed to it and passes them on to the function fn. You would use this like this:

def ints_are_greater_than(frame,  key,  value):
    frameval = frame[key].value
    return frameval > value

tray.AddModule('LineFit', 'linefit',
               If = fwd(ints_are_greater_than,
                        key = 'WhereTheIntIs',
                        value = 80))

A forwarding function is necessary here, but not when passing a python function directly to AddModule(). This asymmetry is unfortunate but presently unavoidable.

Source code organization

You may want to store your useful functions in their own file, say my_utils.py:

#
#   my_utils.py
#
#   My useful stuff
#

def ints_are_greater_than(frame,  key,  value):
    frameval = frame[key].value
    return frameval > value

Which should be located somewhere along your PYTHONPATH or in the current working directory. To use them from your python scripts simply:

#!/usr/bin/env python

from my_utils import ints_are_greater_than
from I3Tray import *

tray = I3Tray()

...

tray.AddModule(ints_are_greater_than, 'igt',
               key = 'where',
               value = 30)