API Reference
Data Handler Class
- class dnn_reco.data_handler.DataHandler(config)[source]
Data handler for IceCube simulation data.
The DataHandler class manages the loading of IceCube hdf5 files. It can read DOM data, labels and misc data from while, while also able to filter events.
In addition, the DataHandler manages the meta data of the input data into the network and also provides the placeholder variables for the tensorflow graph.
Attributes
- label_nameslist of str
Names of labels.
- label_shapelist of int
Shape of label/y tensor without batch dimension.
- misc_data_existsbool
If true, misc data exists and is != None.
- misc_nameslist of str
Names of misc names. If no misc data exists, this is an empty list.
- misc_shapelist of int
Shape of misc data without batch dimension.
- num_binsint
Number of bins for each DOM input.
- num_labelsint
Number of labels.
- num_miscint
Number of misc values that will be loaded per event.
- relative_time_keysTYPE
Description
- test_input_datalist of str
List of files that is used to obtain meta data.
- get_batch_generator(input_data, batch_size, sample_randomly=True, pick_random_files_forever=True, file_capacity=1, batch_capacity=5, num_jobs=1, num_add_files=0, num_repetitions=1, init_values=0.0, num_splits=None, nan_fill_value=None, verbose=False, *args, **kwargs)[source]
Get an IceCube data batch generator.
This is a multiprocessing data iterator. There are 3 levels:
A number of ‘num_jobs’ workers load files from the file list into memory and extract the DOM input data, labels, and misc data if defined. The events (input data, labels, misc data) of the loaded file is then queued onto a multiprocessing queue named ‘data_batch_queue’.
Another worker aggregates the events of several files (number of files defined by ‘num_add_files’) together by dequeuing elements from the ‘data_batch_queue’. It then creates batches from these events (randomly if sample_randomly == True ). These batches are then put onto the ‘final_batch_queue’. Elements in the ‘final_batch_queue’ now include ‘batch_size’ many events ( tuples of dom_responses, cascade_parameters).
The third level consists of the actual generator object. It pops elements off of the ‘final_batch_queue’ and yields these as the desired batches of (input data, labels, misc data).
Parameters
- input_datastr or list of str
File name pattern or list of file patterns which define the paths to input data files.
- batch_sizeint
Number of events per batch.
- sample_randomlybool, optional
If True, random files and events will be sampled. If False, file list and events will not be shuffled.
Although the order will most likely stay the same, this can not be guaranteed, since batches are queued as soon as the workers finish loading and processing the files.
- pick_random_files_foreverbool, optional
- If True, random files are sampled from the file list in an infinite
loop.
- If False, all files will only be loaded once. The ‘num_repetitions’
key defines how many times the events of a file will be used.
- file_capacityint, optional
Defines the maximum size of the queue which holds the loaded and processed events of a whole file.
- batch_capacityint, optional
Defines the maximum size of the batch queue which holds the batches of size ‘batch_size’. This queue is what is used to obtain the final batches, which the generator yields.
- num_jobsint, optional
Number of jobs to run in parallel to load and process input files.
- num_add_filesint, optional
Defines how many files are additionally loaded at once. Batches will be generated among events of these (1 + num_add_files) files
- num_repetitionsint, optional
Number of times the events in a loaded file are to be used, before new files are loaded.
- init_valuesfloat, optional
The x_ic78 array will be initialized with these values via: np.zeros_like(x_ic78) * np.array(init_values)
- num_splitsint, optional
If num_splits is given, the loaded file will be divided into num_splits chunks of about equal size. This can be useful when the input files contain a lot of events, since the multiprocessing queue can not handle elements of arbitrary size.
- nan_fill_valuefloat, optional
Fill value for nan values in loaded data. Entries with nan values will be replaced by this value. If None, no replacement will be performed.
- verbosebool, optional
If True, verbose output with additional information on queues.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- generator
A generator object which yields batches of: np.ndarry, np.ndarray
dom_responses: [batch_size, x_dim, y_dim, z_dim, num_bins] cascade_parameters: [batch_size, num_cascade_parameters]
Raises
- ValueError
Description
- get_label_index(label_name)[source]
Get index of a label.
Parameters
- label_namestr
Name of label.
Returns
- int
Index.
- get_misc_index(misc_name)[source]
Get index of a misc variable.
Parameters
- misc_namestr
Name of misc variable.
Returns
- int
Index.
- read_icecube_data(input_data, nan_fill_value=None, init_values=0.0, verbose=False)[source]
Read IceCube hdf5 data files
Parameters
- input_datastr
Path to input data file.
- nan_fill_valuefloat, optional
Fill value for nan values in loaded data. Entries with nan values will be replaced by this value. If None, no replacement will be performed.
- init_valuesfloat, optional
The x_ic78 array will be initialized with these values via: np.zeros_like(x_ic78) * np.array(init_values)
- verbosebool, optional
Print out additional information on runtimes for loading and processing of files.
Returns
- x_ic78numpy.ndarray
DOM input data of main IceCube array. shape: [batch_size, 10, 10, 60, num_bins]
- x_deepcorenumpy.ndarray
DOM input data of DeepCore array. shape: [batch_size, 8, 60, num_bins]
- labelsnumpy.ndarray
Labels. shape: [batch_size] + label_shape
- miscnumpy.ndarray
Misc variables. shape: [batch_size] + misc_shape
Raises
- ValueError
Description
Data Transformation Class ========================- .. automodule:: dnn_reco.data_trafo
- members:
IceCube Detector Specifics
IceCube detector constants such as the DOM positions are hardcoded here.
Miscellaneous Module
Main NN Model class
- class dnn_reco.model.NNModel(is_training, config, data_handler, data_transformer, sess=None)[source]
Base class for neural network architecture
Attributes
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- is_trainingbool
True if model is in training mode, false if in inference mode.
- savertensorflow.train.Saver
A tensorflow saver used to save and load model weights.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- count_parameters(var_list=None)[source]
Count number of trainable parameters
Parameters
- var_listlist of tf.Tensors, optional
A list of tensorflow tensors for which to calculate the number of trainable parameters. If None, then all trainable parameters available will be counted.
Returns
- int
Number of trainable parameters
- fit(num_training_iterations, train_data_generator, val_data_generator, evaluation_methods=None, *args, **kwargs)[source]
Trains the NN model with the data provided by the data iterators.
Parameters
- num_training_iterationsint
The number of training iterations to perform.
- train_data_generatorgenerator object
A python generator object which generates batches of training data.
- val_data_generatorgenerator object
A python generator object which generates batches of validation data.
- evaluation_methodsNone, optional
Description
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Raises
- ValueError
Description
- predict(x_ic78, x_deepcore, transformed=False, *args, **kwargs)[source]
Reconstruct events.
Parameters
- x_ic78float, list or numpy.ndarray
The input data for the main IceCube array.
- x_deepcorefloat, list or numpy.ndarray
The input data for the DeepCore array.
- transformedbool, optional
If true, the normalized and transformed values are returned.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- np.ndarray, np.ndarray
The prediction and estimated uncertainties
- predict_batched(x_ic78, x_deepcore, max_size, transformed=False, *args, **kwargs)[source]
Reconstruct events in multiple batches.
Parameters
- x_ic78float, list or numpy.ndarray
The input data for the main IceCube array.
- x_deepcorefloat, list or numpy.ndarray
The input data for the DeepCore array.
- transformedbool, optional
If true, the normalized and transformed values are returned.
- max_sizeint, optional
The maximum number of events to predict at once in a batch.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- np.ndarray, np.ndarray
The prediction and estimated uncertainties
Setup Manager
- class dnn_reco.setup_manager.SetupManager(config_files)[source]
Setup Manager for DNN reco project
Handles loading and merging of yaml config files, sets up directories.
Config Keys
Automatically created settings
- config_namestr
An automatically created config name. Base names of yaml config files are concatenated with ‘__’. This can be used to create subdirectories for logging and checkpoints.
General settings
- test_data_filestr
Path to test data file.
- training_data_filestr
Path to training data file.
- validation_data_filestr
Path to validation data file.
- num_jobsint
Relevant for IceCubeDataHandler Defines number of workers (background processes) which load data.
- file_capacityint
Relevant for IceCubeDataHandler Defines size of queue which stores events of loaded files.
- batch_capacity
Relevant for IceCubeDataHandler Defines size of final batch queue. This queue is directly used by the data iteratotor to yield batches of data.
- num_add_files
Relevant for IceCubeDataHandler Defines number of files from which the batches are generated.
- num_repetitions
Relevant for IceCubeDataHandler Defines how many times events of the loaded files are used before new files are loaded.
- dom_response_shapelist of int
The shape of the DOM response tensor excluding the batch dimension. E.g.: [x_dim, y_dim, z_dim, num_bins]
- DOM_init_values: float or array-like
The x_ic78 and deepcore array will be initialized with these values via: np.zeros_like(x_ic78) * np.array(init_values)
- batch_sizeint
The batch size which will be used by the data generators.
- log_pathstr
Path to the directory in which the logs are to be stored.
General Training settings
- num_training_iterationsint
Number of training iterations to perform.
- validation_frequencyint
Defines the interval at which validation results should be calculated and logged.
- save_frequencyint
Defines the interval at which the model parameters will be stored to file.
- keep_global_countbool
If true, a global count of training iterations is performed. The amount of previous training iterations is inferred from the latest checkpoint file. If false, counting of training iterations will start at zero.
- keep_probability_listlist of float
A list of keep probabilities for dropout layers. A tensorflow placeholder is created for each float given in this list. These placeholders can then be used in the generator or discriminator networks by defining the index of the correct keep probability. During training these placeholders will be set to the values as defined in ‘keep_probability_list’. During testing and validation these will be set to 1.0, e.g. no dropout is applied.
- hypothesis_smearingNone or list of float
Smearing to be used for cascade hypothesis during training. ‘hypothesis_smearing’ is a list of floats with the length equal to the number of cascade parameters. The ith value in ‘smearing’ resembles the std deviation of the gaussian that will be added as a smearing. Cascade Parameters: x, y, z, zenith, azimuth, energy, t If smearing is None, no smearing will be applied.
Trafo settings
- trafo_model_pathstr
Path to trafo model file.
- trafo_data_filestr
Path to trafo data file.
- trafo_load_modelbool
If true, the transformation model will be loaded from file. If false, a new transorfmation model will be created from the data
specified by the ‘trafo_data’ key.
- trafo_save_modelbool
- If true, the transformation model will be saved to the file
specified by the ‘trafo_model_path’ key. Note: This will overwrite the file!
- trafo_normalizebool
If true, data will be normalized to have a mean of 0 and a variance of 1.
- trafo_log_binsbool, list of bool
The natural logarithm is applied to the hits prior to normalization. If a list is given, the length of the list must match the number of bins ‘num_bins’. The logarithm is applied to bin i if the ith entry of the log_bins list is True.
- trafo_log_energybool, list of bool
The natural logarithm is applied to the cascade energy.
- trafo_treat_doms_equallybool
All DOMs are treated equally, e.g. the mean and variance is calculated over all DOMs and not individually.
- trafo_norm_constantfloat
A small constant that is added to the denominator during normalization to ensure finite values.
NN Model Architecture
- generator_model_filestr
Name of python file in dnn_reco/modules/model/ directory in which the generator is defined.
- generator_model_namestr
Name of function in the ‘generator_model_file’ that is used to define the generator.
NN Model Training
- model_checkpoint_pathstr
Path to directory in which the generator checkpoints are stored.
- generator_restore_modelbool
If true, generator parameters are restored from file. If false, generator parameters are randomly initialized.
- generator_save_modelbool
- If true, the generator parameters are saved to file.
Note: This will overwrite possible existing files.
- generator_perform_trainingbool
- If true, the generator will be trained in a perform_training call
if it has trainable parameters.
- generator_loss_filestr
Name of python file in dnn_reco/modules/loss/ directory in which the loss function for the generator optimizer is defined.
- generator_loss_namestr
Name of function in the ‘generator_loss_file’ that is used to define the loss function for the generator optimizer.
- generator_optimizer_namestr
Name of the tensorflow optimizer to use for the generator training.
- generator_optimizer_settingsdict
Settings for the chosen generator optimizer.
Attributes
- configdictionary
Dictionary with defined settings.
- shared_objectsdictionary
Dictionary with additional objects that are available in all modules. Keys:
‘data_transformer’ : DataTransformer object used to transform data. ‘keep_prob_list’ : Tensorflow placeholders for keep probabilities
for dropout layers
Utilities Module
Version Control
Loss Functions
All defined models must have the following signature:
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
- tf.Tensor
A tensorflow tensor containing the loss for each label. A weighted sum with weights as defined in the config will be performed over these loss terms to obtain a scalar loss. Shape: label_shape (same shape as labels)
- dnn_reco.modules.loss.default_loss.abs(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
Absolute error of transformed prediction and true values.
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
- dnn_reco.modules.loss.default_loss.gaussian_likelihood(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
Gaussian likelhood of transformed prediction and true values.
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
- dnn_reco.modules.loss.default_loss.mse(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
Mean squared error of transformed prediction and true values.
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
- dnn_reco.modules.loss.default_loss.mse_and_cross_entropy(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
Mean squared error of transformed prediction and true values. Cross entropy loss will be applied to labels for which logit tensors are defined in shared_objects[logit_tensors]. These logit tensors must be added to the shared_objects during building of the NN model. This is necessary since using the logits directly is more numerically stable than reverting the sigmoid function on the output of the model.
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
- dnn_reco.modules.loss.default_loss.mse_and_weighted_cross_entropy(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
Mean squared error of transformed prediction and true values. Weighted cross entroy loss will be applied to labels for which logit tensors are defined in shared_objects[logit_tensors]. These logit tensors must be added to the shared_objects during building of the NN model. This is necessary since using the logits directly is more numerically stable than reverting the sigmoid function on the output of the model.
Events will be weighted according to how many signal events [pid value == 1] have a lower classification score. This aims to ignore abundant background events in a highly imbalanced classification task. Generally we only care about correctly modeling the signal region near 1. Hence, here we downweight the background region which is defined to be at smaller classification scores than the signal events.
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
- dnn_reco.modules.loss.default_loss.opening_angle(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
Error of opening angle between true and predicted angle.
The MSE of the opening angle will be predicted as uncertainty. This loss only applies to label_azimuth_key and label_zenith_key!
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
- dnn_reco.modules.loss.default_loss.opening_angle_raleigh(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
Raleigh loss of opening angle between true and predicted angle.
This loss only applies to label_azimuth_key and label_zenith_key!
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
- dnn_reco.modules.loss.default_loss.pull_distribution_scale(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
This loss penalized the standard deviation of the pull distribution.
This is meant to run for a few steps with very high batch size at the very end of the training procedure of a model in order to correct the scale of the uncertainty estimates such that the pull distribution has a standard deviation of 1.
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
- dnn_reco.modules.loss.default_loss.tukey(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
Tukey loss of transformed prediction and true values. A robust loss measure that is equivalent to MSE for small residuals, but has constant loss for very large residuals. This reduces the effect of outliers.
From Paper: ‘Robust Optimization for Deep Regression’
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
- dnn_reco.modules.loss.default_loss.weighted_mse(config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
Weighted mean squared error of transformed prediction and true values.
The MSE is weighted by the per event uncertainty estimate.
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
This file contains commonly used utility functions to compute loss.
- dnn_reco.modules.loss.utils.loss_utils.add_logging_info(data_handler, shared_objects)[source]
Add some TensorBoard logging info for the labels.
Adds Tf.summary.scalars of the RMSE for each label.
Parameters
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- dnn_reco.modules.loss.utils.loss_utils.correct_azimuth_residual(y_diff_trafo, config, data_handler, data_transformer, name_pattern)[source]
Correct azimuth residuals for two pi periodicity.
Parameters
- y_diff_trafotf.Tensor
The residuals between the transformed prediction and true values.
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- name_patternstr
A name pattern specifying the labels that should be corrected.
Returns
- tf.Tensor
The residual tensor with corrected azimuth residual. Same shape and type as y_diff_trafo
- dnn_reco.modules.loss.utils.loss_utils.get_y_diff_trafo(config, data_handler, data_transformer, shared_objects)[source]
Get corrected transformed residuals.
Parameters
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
Returns
- tf.Tensor
A tensorflow tensor containing the loss for each label. Shape: label_shape (same shape as labels)
Evaluation Functions
All defined evaluation functions must have the following signature:
- feed_dict_traindict
The feed_dict used to feed tensorflow placeholder for the evaluation on training data.
- feed_dict_valdict
The feed_dict used to feed tensorflow placeholder for the evaluation on validation data.
- results_traindict
A dictionary with the results of the tensorflow operations for the training data.
- results_valdict
A dictionary with the results of the tensorflow operations for the validation data.
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
- dnn_reco.modules.evaluation.default_evaluation.eval_direction(feed_dict_train, feed_dict_val, results_train, results_val, config, data_handler, data_transformer, shared_objects, *args, **kwargs)[source]
Evaluation method to compute angular resolution.
Parameters
- feed_dict_traindict
The feed_dict used to feed tensorflow placeholder for the evaluation on training data.
- feed_dict_valdict
The feed_dict used to feed tensorflow placeholder for the evaluation on validation data.
- results_traindict
A dictionary with the results of the tensorflow operations for the training data.
- results_valdict
A dictionary with the results of the tensorflow operations for the validation data.
- configdict
Dictionary containing all settings as read in from config file.
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- data_transformer:obj: of class DataTransformer
An instance of the DataTransformer class. The object is used to transform data.
- shared_objectsdict
A dictionary containing settings and objects that are shared and passed on to sub modules.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Label Data Loading
All label functions must have the following parameters and return values:
- input_datastr
Path to input data file.
- configdict
Dictionary containing all settings as read in from config file. Label function specific settings can be passed via the config file.
- label_namesNone, optional
The names of the labels. This defines which labels to include as well as the ordering. If label_names is None (e.g. first call to initiate name list), then a list of label names needs to be created and returned.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
- np.ndarray
The numpy array containing the labels. Shape: [batch_size] + label_shape
- list of str
The names of the labels
- dnn_reco.modules.data.labels.default_labels.simple_label_loader(input_data, config, label_names=None, *args, **kwargs)[source]
Simple Label Loader.
Will load variables contained in the hdf5 field specified via config[‘data_handler_label_key’].
Parameters
- input_datastr
Path to input data file.
- configdict
Dictionary containing all settings as read in from config file. Must contain:
- ‘data_handler_label_key’: str
The hdf5 key from which the labels will be loaded.
- ‘label_add_dir_vec’: bool
If true, the direction vector components will be calculated on the fly and added to the labels. For this, the keys ‘label_azimuth_key’ and ‘label_zenith_key’ have to be provided.
- label_namesNone, optional
The names of the labels. This defines which labels to include as well as the ordering. If label_names is None, then all keys except event specificers will be used.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- list of str
The names of the labels
Misc Data Loading
All misc functions must have the following parameters and return values:
- input_datastr
Path to input data file.
- configdict
Dictionary containing all settings as read in from config file. Misc function specific settings can be passed via the config file.
- misc_namesNone, optional
The names of the misc variables. This defines which variables to include as well as the ordering. If misc_names is None (e.g. first call to initiate name list), then a list of misc names needs to be created and returned.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
- np.ndarray
The numpy array containing the misc variables. Shape: [batch_size] + misc_shape
- list of str
The names of the misc variables.
- dnn_reco.modules.data.misc.default_misc.dummy_misc_loader(input_data, config, misc_names=None, *args, **kwargs)[source]
Dummy misc loader.
Dummy function that does not do anything.
Parameters
- input_datastr
Path to input data file.
- configdict
Dictionary containing all settings as read in from config file.
- misc_namesNone, optional
The names of the misc variables. This defines which variables to include as well as the ordering. If misc_names is None (e.g. first call to initiate name list), then a list of misc names needs to be created and returned.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- None
The misc variables.
- empty list
The names of the misc variables.
- dnn_reco.modules.data.misc.default_misc.general_misc_loader(input_data, config, misc_names=None, *args, **kwargs)[source]
A general misc variable loader.
Loads values as specified in config.
Parameters
- input_datastr
Path to input data file.
- configdict
Dictionary containing all settings as read in from config file.
- misc_namesNone, optional
The names of the misc variables. This defines which variables to include as well as the ordering. If misc_names is None (e.g. first call to initiate name list), then a list of misc names needs to be created and returned.
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- None
The misc variables.
- empty list
The names of the misc variables.
Event Filter
All filter functions must have the following parameters and return values:
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- input_datastr
Path to input data file.
- configdict
Dictionary containing all settings as read in from config file.
- x_IC78numpy.ndarray
DOM input data of main IceCube array. shape: [batch_size, 10, 10, 60, num_bins]
- x_deepcorenumpy.ndarray
DOM input data of DeepCore array. shape: [batch_size, 8, 60, num_bins]
- labelsnumpy.ndarray
Labels. shape: [batch_size] + label_shape
- misc_datanumpy.ndarray
Misc variables. shape: [batch_size] + misc_shape
- time_range_startnumpy.ndarray
Time offset for relative timing for each event. shape: [batch_size, 1]
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
- numpy.ndarray of type bool
The boolean mask to filter out events. shape: [batch_size]
- dnn_reco.modules.data.filter.default_filter.dummy_filter(data_handler, input_data, config, x_ic70, x_deepcore, labels, misc_data, time_range_start, *args, **kwargs)[source]
Dummy filter that accepts all events.
Parameters
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- input_datastr
Path to input data file.
- configdict
Dictionary containing all settings as read in from config file.
- x_ic70numpy.ndarray
DOM input data of main IceCube array. shape: [batch_size, 10, 10, 60, num_bins]
- x_deepcorenumpy.ndarray
DOM input data of DeepCore array. shape: [batch_size, 8, 60, num_bins]
- labelsnumpy.ndarray
Labels. shape: [batch_size] + label_shape
- misc_datanumpy.ndarray
Misc variables. shape: [batch_size] + misc_shape
- time_range_startnumpy.ndarray
Time offset for relative timing for each event. shape: [batch_size, 1]
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- numpy.ndarray of type bool
The boolean mask to filter out events. shape: [batch_size]
- dnn_reco.modules.data.filter.default_filter.general_filter(data_handler, input_data, config, x_ic70, x_deepcore, labels, misc_data, time_range_start, *args, **kwargs)[source]
A general filter.
Filters events according to the key value pairs defined in the config:
- filter_equal:
- For events to pass this filter, the following must be True:
misc_data[key] == value
- filter_greater_than:
- For events to pass this filter, the following must be True:
misc_data[key] > value
- filter_less_than:
- For events to pass this filter, the following must be True:
misc_data[key] < value
Optionally, a list of MC Primary PDG encodings may be provided to limit the application of the filter to events with the provided PDG encodings. The PDG encodings are provided via config[‘filter_apply_on_pdg_encodings’].
Parameters
- data_handler:obj: of class DataHandler
An instance of the DataHandler class. The object is used to obtain meta data.
- input_datastr
Path to input data file.
- configdict
Dictionary containing all settings as read in from config file.
- x_ic70numpy.ndarray
DOM input data of main IceCube array. shape: [batch_size, 10, 10, 60, num_bins]
- x_deepcorenumpy.ndarray
DOM input data of DeepCore array. shape: [batch_size, 8, 60, num_bins]
- labelsnumpy.ndarray
Labels. shape: [batch_size] + label_shape
- misc_datanumpy.ndarray
Misc variables. shape: [batch_size] + misc_shape
- time_range_startnumpy.ndarray
Time offset for relative timing for each event. shape: [batch_size, 1]
- *args
Variable length argument list.
- **kwargs
Arbitrary keyword arguments.
Returns
- numpy.ndarray of type bool
The boolean mask to filter out events. shape: [batch_size]
Raises
- NotImplementedError
Description