API Reference

Data Handler Class

class dnn_reco.data_handler.DataHandler(config)[source]

Data handler for IceCube simulation data.

The DataHandler class manages the loading of IceCube hdf5 files. It can read DOM data, labels and misc data from while, while also able to filter events.

In addition, the DataHandler manages the meta data of the input data into the network and also provides the placeholder variables for the tensorflow graph.

Attributes

label_nameslist of str: Names of labels.
label_shapelist of int: Shape of label/y tensor without batch dimension.
misc_data_existsbool: If true, misc data exists and is != None.
misc_nameslist of str: Names of misc names. If no misc data exists, this is an empty list.
misc_shapelist of int: Shape of misc data without batch dimension.
num_binsint: Number of bins for each DOM input.
num_labelsint: Number of labels.
num_miscint: Number of misc values that will be loaded per event.
relative_time_keysTYPE: Description
test_input_datalist of str: List of files that is used to obtain meta data.

get_batch_generator(input_data, batch_size, sample_randomly=True, pick_random_files_forever=True, file_capacity=1, batch_capacity=5, num_jobs=1, num_add_files=0, num_repetitions=1, init_values=0.0, num_splits=None, nan_fill_value=None, verbose=False, *args, **kwargs)[source]

Get an IceCube data batch generator.

This is a multiprocessing data iterator. There are 3 levels:

A number of ‘num_jobs’ workers load files from the file list into memory and extract the DOM input data, labels, and misc data if defined. The events (input data, labels, misc data) of the loaded file is then queued onto a multiprocessing queue named ‘data_batch_queue’.

Another worker aggregates the events of several files (number of files defined by ‘num_add_files’) together by dequeuing elements from the ‘data_batch_queue’. It then creates batches from these events (randomly if sample_randomly == True ). These batches are then put onto the ‘final_batch_queue’. Elements in the ‘final_batch_queue’ now include ‘batch_size’ many events ( tuples of dom_responses, cascade_parameters).

The third level consists of the actual generator object. It pops elements off of the ‘final_batch_queue’ and yields these as the desired batches of (input data, labels, misc data).

Parameters

input_datastr or list of str

File name pattern or list of file patterns which define the paths to input data files.

batch_sizeint

Number of events per batch.

sample_randomlybool, optional

If True, random files and events will be sampled. If False, file list and events will not be shuffled.

Although the order will most likely stay the same, this can not be guaranteed, since batches are queued as soon as the workers finish loading and processing the files.

pick_random_files_foreverbool, optional

If True, random files are sampled from the file list in an infinite: loop.
If False, all files will only be loaded once. The ‘num_repetitions’: key defines how many times the events of a file will be used.

file_capacityint, optional

Defines the maximum size of the queue which holds the loaded and processed events of a whole file.

batch_capacityint, optional

Defines the maximum size of the batch queue which holds the batches of size ‘batch_size’. This queue is what is used to obtain the final batches, which the generator yields.

num_jobsint, optional

Number of jobs to run in parallel to load and process input files.

num_add_filesint, optional

Defines how many files are additionally loaded at once. Batches will be generated among events of these (1 + num_add_files) files

num_repetitionsint, optional

Number of times the events in a loaded file are to be used, before new files are loaded.

init_valuesfloat, optional

The x_ic78 array will be initialized with these values via: np.zeros_like(x_ic78) * np.array(init_values)

num_splitsint, optional

If num_splits is given, the loaded file will be divided into num_splits chunks of about equal size. This can be useful when the input files contain a lot of events, since the multiprocessing queue can not handle elements of arbitrary size.

nan_fill_valuefloat, optional

Fill value for nan values in loaded data. Entries with nan values will be replaced by this value. If None, no replacement will be performed.

verbosebool, optional

If True, verbose output with additional information on queues.

*args

Variable length argument list.

**kwargs

Arbitrary keyword arguments.

Returns

generator: A generator object which yields batches of: np.ndarry, np.ndarray

dom_responses: [batch_size, x_dim, y_dim, z_dim, num_bins] cascade_parameters: [batch_size, num_cascade_parameters]

Raises

ValueError: Description

get_label_index(label_name)[source]

Get index of a label.

Parameters

label_namestr: Name of label.

Returns

int: Index.

get_misc_index(misc_name)[source]

Get index of a misc variable.

Parameters

misc_namestr: Name of misc variable.

Returns

int: Index.

kill()[source]: Kill Multiprocessing queues and workers

read_icecube_data(input_data, nan_fill_value=None, init_values=0.0, verbose=False)[source]

Read IceCube hdf5 data files

Parameters

input_datastr: Path to input data file.
nan_fill_valuefloat, optional: Fill value for nan values in loaded data. Entries with nan values will be replaced by this value. If None, no replacement will be performed.
init_valuesfloat, optional: The x_ic78 array will be initialized with these values via: np.zeros_like(x_ic78) * np.array(init_values)
verbosebool, optional: Print out additional information on runtimes for loading and processing of files.

Returns

x_ic78numpy.ndarray: DOM input data of main IceCube array. shape: [batch_size, 10, 10, 60, num_bins]
x_deepcorenumpy.ndarray: DOM input data of DeepCore array. shape: [batch_size, 8, 60, num_bins]
labelsnumpy.ndarray: Labels. shape: [batch_size] + label_shape
miscnumpy.ndarray: Misc variables. shape: [batch_size] + misc_shape

Raises

ValueError: Description

setup_with_config(config_file)[source]

Setup the datahandler with settings from a yaml configuration file.

Parameters

config_filestr: The path to the configuration file

Raises

NotImplementedError: Description

setup_with_test_data(test_input_data)[source]

Setup the datahandler with a test input file.

Parameters

test_input_datastr or list of str: File name pattern or list of file patterns which define the paths to input data files. The first of the specified files will be read in to obtain meta data.

Data Transformation Class ========================- .. automodule:: dnn_reco.data_trafo

members:

IceCube Detector Specifics

IceCube detector constants such as the DOM positions are hardcoded here.

dnn_reco.detector.get_dom_coords(string, dom)[source]

Get Coordinates of a DOM specified by the string and dom number.

Parameters

stringint: String number (between 1 and 86)
domint: DOM number (between 1 and 60)

Returns

tuple(float, float, float): The x, y, z coordinates of the DOM.

Miscellaneous Module

dnn_reco.misc.get_full_class_string_of_object(object_instance)[source]

Get full class string of an object’s class.

o.__module__ + “.” + o.__class__.__qualname__ is an example in this context of H.L. Mencken’s “neat, plausible, and wrong.” Python makes no guarantees as to whether the __module__ special attribute is defined, so we take a more circumspect approach. Alas, the module name is explicitly excluded from __qualname__ in Python 3.

Adopted from:: https://stackoverflow.com/questions/2020014/ get-fully-qualified-class-name-of-an-object-in-python

Parameters

object_instanceobject: The object of which to obtain the full class string.

Returns

str: The full class string of the object’s class

dnn_reco.misc.load_class(full_class_string)[source]

dynamically load a class from a string

Parameters

full_class_stringstr: The full class string to the given python clas. Example:

my_project.my_module.my_class

Returns

python class: PYthon class defined by the ‘full_class_string’

dnn_reco.misc.print_warning(msg)[source]

Print Warning in yellow color.

Parameters

msgstr: String to print.

Main NN Model class

class dnn_reco.model.BaseNNModel(is_training, config, data_handler, data_transformer, dtype: str = 'float32', logger=None, name: str | None = None)[source]

Base class for neural network architecture

Derived classes must implement the __call__ method, create all necessary variables in the __init__ method, and set the following attributes in the __init__ method:

self.vars_pred: a list of all variables used for the prediction

self.vars_unc: a list of all variables used for the uncertainty

Attributes

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
is_trainingbool: True if model is in training mode, false if in inference mode.
savertensorflow.train.Saver: A tensorflow saver used to save and load model weights.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.

compile()[source]: Compile the model and create optimizers.

fit(num_training_iterations, train_data_generator, val_data_generator, evaluation_methods=None, *args, **kwargs)[source]

Trains the NN model with the data provided by the data iterators.

Parameters

num_training_iterationsint: The number of training iterations to perform.
train_data_generatorgenerator object: A python generator object which generates batches of training data.
val_data_generatorgenerator object: A python generator object which generates batches of validation data.
evaluation_methodsNone, optional: Description
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Raises

ValueError: Description

get_loss(data_batch, is_training=True, summary_writer=None)[source]

Get optimizers and loss terms as defined in config.

Parameters

data_batchlist

A list containing the input data, containing:: x_ic78, x_deepcore, labels, misc_data

is_trainingbool, optional

True if model is in training mode, false if in inference mode.

summary_writertf.summary.SummaryWriter, optional

A summary writer to write summaries to.

Returns

dict: A dictionary containing the model predictions and auxiliary tensors.

get_tensors(data_batch, is_training=True, summary_writer=None)[source]

Get result tensors from the model.

Performs a forward pass through the model and adds additional auxiliary tensors to the result dictionary.

Parameters

data_batchlist: A list containing the input data. This is typically: x_ic78, x_deepcore, labels, misc_data
is_trainingbool, optional: True if model is in training mode, false if in inference mode.
summary_writertf.summary.SummaryWriter, optional: A summary writer to write summaries to.

Returns

dict: A dictionary containing the model predictions and auxiliary tensors.

perform_training_step(data_batch, summary_writer=None)[source]

Perform a single training step.

Parameters

data_batchlist: A list containing the input data. This is typically: x_ic78, x_deepcore, labels, misc_data
summary_writertf.summary.SummaryWriter, optional: A summary writer to write summaries to.

Returns

dict: A dictionary containing the model predictions and auxiliary tensors.

predict(x_ic78, x_deepcore, transformed=False, is_training=False)[source]

Reconstruct events.

Parameters

x_ic78float, list or numpy.ndarray: The input data for the main IceCube array.
x_deepcorefloat, list or numpy.ndarray: The input data for the DeepCore array.
transformedbool, optional: If true, the normalized and transformed values are returned.
is_trainingbool, optional: True if model is in training mode, false if in inference mode.

Returns

np.ndarray, np.ndarray: The prediction and estimated uncertainties

predict_batched(x_ic78, x_deepcore, max_size, transformed=False, *args, **kwargs)[source]

Reconstruct events in multiple batches.

Parameters

x_ic78float, list or numpy.ndarray: The input data for the main IceCube array.
x_deepcorefloat, list or numpy.ndarray: The input data for the DeepCore array.
transformedbool, optional: If true, the normalized and transformed values are returned.
max_sizeint, optional: The maximum number of events to predict at once in a batch.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

np.ndarray, np.ndarray: The prediction and estimated uncertainties

restore(is_training=True)[source]: Restore model weights from checkpoints

Setup Manager

Utilities Module

Version Control

Loss Functions

All defined models must have the following signature:

configdict
Dictionary containing all settings as read in from config file.

data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.

data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.

shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.

*args
Variable length argument list.

**kwargs
Arbitrary keyword arguments.

tf.Tensor
A tensorflow tensor containing the loss for each label. A weighted sum with weights as defined in the config will be performed over these loss terms to obtain a scalar loss. Shape: label_shape (same shape as labels)

dnn_reco.modules.loss.default_loss.abs(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

Absolute error of transformed prediction and true values.

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

dnn_reco.modules.loss.default_loss.gaussian_likelihood(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

Gaussian likelhood of transformed prediction and true values.

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

dnn_reco.modules.loss.default_loss.mse(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

Mean squared error of transformed prediction and true values.

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

dnn_reco.modules.loss.default_loss.mse_and_cross_entropy(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

Mean squared error of transformed prediction and true values. Cross entropy loss will be applied to labels for which logit tensors are defined in shared_objects[logit_tensors]. These logit tensors must be added to the shared_objects during building of the NN model. This is necessary since using the logits directly is more numerically stable than reverting the sigmoid function on the output of the model.

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

dnn_reco.modules.loss.default_loss.mse_and_weighted_cross_entropy(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

Mean squared error of transformed prediction and true values. Weighted cross entroy loss will be applied to labels for which logit tensors are defined in shared_objects[logit_tensors]. These logit tensors must be added to the shared_objects during building of the NN model. This is necessary since using the logits directly is more numerically stable than reverting the sigmoid function on the output of the model.

Events will be weighted according to how many signal events [pid value == 1] have a lower classification score. This aims to ignore abundant background events in a highly imbalanced classification task. Generally we only care about correctly modeling the signal region near 1. Hence, here we downweight the background region which is defined to be at smaller classification scores than the signal events.

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

dnn_reco.modules.loss.default_loss.opening_angle(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

Error of opening angle between true and predicted angle.

The MSE of the opening angle will be predicted as uncertainty. This loss only applies to label_azimuth_key and label_zenith_key!

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

dnn_reco.modules.loss.default_loss.opening_angle_raleigh(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

Raleigh loss of opening angle between true and predicted angle.

This loss only applies to label_azimuth_key and label_zenith_key!

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

dnn_reco.modules.loss.default_loss.pull_distribution_scale(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

This loss penalized the standard deviation of the pull distribution.

This is meant to run for a few steps with very high batch size at the very end of the training procedure of a model in order to correct the scale of the uncertainty estimates such that the pull distribution has a standard deviation of 1.

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

dnn_reco.modules.loss.default_loss.tukey(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

Tukey loss of transformed prediction and true values. A robust loss measure that is equivalent to MSE for small residuals, but has constant loss for very large residuals. This reduces the effect of outliers.

From Paper: ‘Robust Optimization for Deep Regression’

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

dnn_reco.modules.loss.default_loss.weighted_mse(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

Weighted mean squared error of transformed prediction and true values.

The MSE is weighted by the per event uncertainty estimate.

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

This file contains commonly used utility functions to compute loss.

dnn_reco.modules.loss.utils.loss_utils.correct_azimuth_residual(y_diff_trafo, config, data_handler, data_transformer, name_pattern)[source]

Correct azimuth residuals for two pi periodicity.

Parameters

y_diff_trafotf.Tensor: The residuals between the transformed prediction and true values.
configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
name_patternstr: A name pattern specifying the labels that should be corrected.

Returns

tf.Tensor: The residual tensor with corrected azimuth residual. Same shape and type as y_diff_trafo

dnn_reco.modules.loss.utils.loss_utils.get_y_diff_trafo(config, data_handler, data_transformer, shared_objects)[source]

Get corrected transformed residuals.

Parameters

configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.

Returns

tf.Tensor: A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)

Evaluation Functions

All defined evaluation functions must have the following signature:

batch_traindict
The input data batch used for the evaluation on training data.

batch_valdict
The input data batch used for the evaluation on validation data.

results_traindict
A dictionary with the results of the tensorflow operations for the training data.

results_valdict
A dictionary with the results of the tensorflow operations for the validation data.

configdict
Dictionary containing all settings as read in from config file.

data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.

data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.

shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.

*args
Variable length argument list.

**kwargs
Arbitrary keyword arguments.

dnn_reco.modules.evaluation.default_evaluation.eval_direction(batch_train, batch_val, results_train, results_val, config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]

Evaluation method to compute angular resolution.

Parameters

batch_traindict: The input data batch used for the evaluation on training data.
batch_valdict: The input data batch used for the evaluation on validation data.
results_traindict: A dictionary with the results of the tensorflow operations for the training data.
results_valdict: A dictionary with the results of the tensorflow operations for the validation data.
configdict: Dictionary containing all settings as read in from config file.
data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
data_transformer:obj: of class DataTransformer: An instance of the DataTransformer class. The object is used to transform data.
shared_objectsdict: A dictionary containing settings and objects that are shared and passed on to sub modules.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Label Data Loading

All label functions must have the following parameters and return values:

input_datastr
Path to input data file.

configdict
Dictionary containing all settings as read in from config file. Label function specific settings can be passed via the config file.

label_namesNone, optional
The names of the labels. This defines which labels to include as well as the ordering. If label_names is None (e.g. first call to initiate name list), then a list of label names needs to be created and returned.

*args
Variable length argument list.

**kwargs
Arbitrary keyword arguments.

np.ndarray
The numpy array containing the labels. Shape: [batch_size] + label_shape

list of str
The names of the labels

dnn_reco.modules.data.labels.default_labels.simple_label_loader(input_data, config, label_names=None, *args, **kwargs)[source]

Simple Label Loader.

Will load variables contained in the hdf5 field specified via config[‘data_handler_label_key’].

Parameters

input_datastr

Path to input data file.

configdict

Dictionary containing all settings as read in from config file. Must contain:

‘data_handler_label_key’: str
The hdf5 key from which the labels will be loaded.

‘label_add_dir_vec’: bool
If true, the direction vector components will be calculated on the fly and added to the labels. For this, the keys ‘label_azimuth_key’ and ‘label_zenith_key’ have to be provided.

label_namesNone, optional

The names of the labels. This defines which labels to include as well as the ordering. If label_names is None, then all keys except event specificers will be used.

*args

Variable length argument list.

**kwargs

Arbitrary keyword arguments.

Returns

list of str: The names of the labels

Misc Data Loading

All misc functions must have the following parameters and return values:

input_datastr
Path to input data file.

configdict
Dictionary containing all settings as read in from config file. Misc function specific settings can be passed via the config file.

misc_namesNone, optional
The names of the misc variables. This defines which variables to include as well as the ordering. If misc_names is None (e.g. first call to initiate name list), then a list of misc names needs to be created and returned.

*args
Variable length argument list.

**kwargs
Arbitrary keyword arguments.

np.ndarray
The numpy array containing the misc variables. Shape: [batch_size] + misc_shape

list of str
The names of the misc variables.

dnn_reco.modules.data.misc.default_misc.dummy_misc_loader(input_data, config, misc_names=None, *args, **kwargs)[source]

Dummy misc loader.

Dummy function that does not do anything.

Parameters

input_datastr: Path to input data file.
configdict: Dictionary containing all settings as read in from config file.
misc_namesNone, optional: The names of the misc variables. This defines which variables to include as well as the ordering. If misc_names is None (e.g. first call to initiate name list), then a list of misc names needs to be created and returned.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

None: The misc variables.
empty list: The names of the misc variables.

dnn_reco.modules.data.misc.default_misc.general_misc_loader(input_data, config, misc_names=None, *args, **kwargs)[source]

A general misc variable loader.

Loads values as specified in config.

Parameters

input_datastr: Path to input data file.
configdict: Dictionary containing all settings as read in from config file.
misc_namesNone, optional: The names of the misc variables. This defines which variables to include as well as the ordering. If misc_names is None (e.g. first call to initiate name list), then a list of misc names needs to be created and returned.
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

None: The misc variables.
empty list: The names of the misc variables.

Event Filter

All filter functions must have the following parameters and return values:

data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.

input_datastr
Path to input data file.

configdict
Dictionary containing all settings as read in from config file.

x_IC78numpy.ndarray
DOM input data of main IceCube array. shape: [batch_size, 10, 10, 60, num_bins]

x_deepcorenumpy.ndarray
DOM input data of DeepCore array. shape: [batch_size, 8, 60, num_bins]

labelsnumpy.ndarray
Labels. shape: [batch_size] + label_shape

misc_datanumpy.ndarray
Misc variables. shape: [batch_size] + misc_shape

time_range_startnumpy.ndarray
Time offset for relative timing for each event. shape: [batch_size, 1]

*args
Variable length argument list.

**kwargs
Arbitrary keyword arguments.

numpy.ndarray of type bool
The boolean mask to filter out events. shape: [batch_size]

dnn_reco.modules.data.filter.default_filter.dummy_filter(data_handler, input_data, config, x_ic70, x_deepcore, labels, misc_data, time_range_start, *args, **kwargs)[source]

Dummy filter that accepts all events.

Parameters

data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
input_datastr: Path to input data file.
configdict: Dictionary containing all settings as read in from config file.
x_ic70numpy.ndarray: DOM input data of main IceCube array. shape: [batch_size, 10, 10, 60, num_bins]
x_deepcorenumpy.ndarray: DOM input data of DeepCore array. shape: [batch_size, 8, 60, num_bins]
labelsnumpy.ndarray: Labels. shape: [batch_size] + label_shape
misc_datanumpy.ndarray: Misc variables. shape: [batch_size] + misc_shape
time_range_startnumpy.ndarray: Time offset for relative timing for each event. shape: [batch_size, 1]
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

numpy.ndarray of type bool: The boolean mask to filter out events. shape: [batch_size]

dnn_reco.modules.data.filter.default_filter.general_filter(data_handler, input_data, config, x_ic70, x_deepcore, labels, misc_data, time_range_start, *args, **kwargs)[source]

A general filter.

Filters events according to the key value pairs defined in the config:

filter_equal:

For events to pass this filter, the following must be True:
misc_data[key] == value

filter_greater_than:

For events to pass this filter, the following must be True:
misc_data[key] > value

filter_less_than:

For events to pass this filter, the following must be True:
misc_data[key] < value

Optionally, a list of MC Primary PDG encodings may be provided to limit the application of the filter to events with the provided PDG encodings. The PDG encodings are provided via config[‘filter_apply_on_pdg_encodings’].

Parameters

data_handler:obj: of class DataHandler: An instance of the DataHandler class. The object is used to obtain meta data.
input_datastr: Path to input data file.
configdict: Dictionary containing all settings as read in from config file.
x_ic70numpy.ndarray: DOM input data of main IceCube array. shape: [batch_size, 10, 10, 60, num_bins]
x_deepcorenumpy.ndarray: DOM input data of DeepCore array. shape: [batch_size, 8, 60, num_bins]
labelsnumpy.ndarray: Labels. shape: [batch_size] + label_shape
misc_datanumpy.ndarray: Misc variables. shape: [batch_size] + misc_shape
time_range_startnumpy.ndarray: Time offset for relative timing for each event. shape: [batch_size, 1]
*args: Variable length argument list.
**kwargs: Arbitrary keyword arguments.

Returns

numpy.ndarray of type bool: The boolean mask to filter out events. shape: [batch_size]

Raises

NotImplementedError: Description