Storage requirements ====================== Summary --------- Depending on how much information is kept from the L2 files, there are three possibile total storage requirements for all experimental data and simulation produced to date: Bare-bones: 6 TB This is the current baseline and throws away a number of reconstructions from earlier processing steps as well as NFE pulses and IceTop information. Keep everything: 13 TB Delete nothing, making the L3 files a strict superset of L2a. Middle way: 7.5 TB Delete only the NFE recopulses from the frame. .. note:: For comparison, the total size of the muon-channel L3 data is 20 TB. Detailed file sizes for various simulation types ------------------------------------------------- .. note:: These requirements are estimated from the size of a single file from each set. The estimates for 5-component Corsika may be off by large integer factors. .. table:: Best-case scenario: `keep-list from rc3 `_ +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | Dataset | Description | L2 size [MB] | L3 size [MB] | Compression | Est. total [GB] | +=========+=====================================================+==============+==============+=============+=================+ | 6451 | unweighted Hoerandel, polygonato (SPICEMie ice) | 171.5 | 1.8 | 96.68 | 17 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6454 | E^-1, NuMu (SPICEMie ice) | 576.7 | 50.6 | 11.41 | 504 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6461 | E^-1, NuE (SPICEMie ice) | 473.6 | 123.4 | 3.84 | 1228 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6466 | E^-1, NuTau (SPICEMie ice) | 322.9 | 60.6 | 5.33 | 604 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6467 | E^-2, NuMu (PPC:SPICEMIE ice) | 853.7 | 38.7 | 22.08 | 11 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6514 | E-2, 5-component 5-component (SPICEMie ice) | 123.6 | 4.2 | 29.69 | 416 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6593 | E^-2, NuTau (SPICEMie ice) | 1445.2 | 258.1 | 5.60 | 258 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6725 | E^-2, NuE (SPICEMie ice) | 135.6 | 36.3 | 3.74 | 36 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6726 | E^-2, NuMu (SPICEMie ice) | 94.6 | 4.3 | 21.94 | 4 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6937 | of NuMu+BG (SPICEMie ice) | 101.7 | 3.3 | 31.05 | 3 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6939 | unweighted Hoerandel, polygonato (PPC:SPICEMie ice) | 166.1 | 1.8 | 94.76 | 17 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 7017 | E-2, 5-component 5-component (SPICEMie ice) | 146.8 | 13.7 | 10.75 | 1365 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 7260 | spectrum coincident coincident (SPICEMie ice) | 671.2 | 34.5 | 19.44 | 1035 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 7785 | E^-2, NuE (PPC:SPICEMie ice) | 128.7 | 33.2 | 3.87 | 33 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | Data | | 519.6 | 2.5 | 207.14 | 376 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | | | | | Total | 5907 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ .. table:: Worst-case scenario: delete nothing +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | Dataset | Description | L2 size [MB] | L3 size [MB] | Compression | Est. total [GB] | +=========+=====================================================+==============+==============+=============+=================+ | 6451 | unweighted Hoerandel, polygonato (SPICEMie ice) | 171.5 | 2.3 | 76.00 | 22 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6454 | E^-1, NuMu (SPICEMie ice) | 576.7 | 150.2 | 3.84 | 1499 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6461 | E^-1, NuE (SPICEMie ice) | 473.6 | 490.3 | 0.97 | 4879 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6466 | E^-1, NuTau (SPICEMie ice) | 322.9 | 210.4 | 1.53 | 2097 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6467 | E^-2, NuMu (PPC:SPICEMIE ice) | 853.7 | 82.6 | 10.34 | 23 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6514 | E-2, 5-component 5-component (SPICEMie ice) | 123.6 | 4.7 | 26.20 | 471 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6593 | E^-2, NuTau (SPICEMie ice) | 1445.2 | 1107.5 | 1.31 | 1107 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6725 | E^-2, NuE (SPICEMie ice) | 135.6 | 80.2 | 1.69 | 80 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6726 | E^-2, NuMu (SPICEMie ice) | 94.6 | 9.9 | 9.55 | 9 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6937 | of NuMu+BG (SPICEMie ice) | 101.7 | 5.9 | 17.26 | 5 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6939 | unweighted Hoerandel, polygonato (PPC:SPICEMie ice) | 166.1 | 2.2 | 77.24 | 21 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 7017 | E-2, 5-component 5-component (SPICEMie ice) | 146.8 | 15.3 | 9.59 | 1531 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 7260 | spectrum coincident coincident (SPICEMie ice) | 671.2 | 39.4 | 17.04 | 1181 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 7785 | E^-2, NuE (PPC:SPICEMie ice) | 128.7 | 74.3 | 1.73 | 74 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | Data | | 519.6 | 4.4 | 118.52 | 657 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | | | | | Total | 13656 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ .. table:: Middle way: delete only NFE pulses +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | Dataset | Description | L2 size [MB] | L3 size [MB] | Compression | Est. total [GB] | +=========+=====================================================+==============+==============+=============+=================+ | 6451 | unweighted Hoerandel, polygonato (SPICEMie ice) | 171.5 | 1.8 | 93.51 | 18 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6454 | E^-1, NuMu (SPICEMie ice) | 576.7 | 68.4 | 8.43 | 682 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6461 | E^-1, NuE (SPICEMie ice) | 473.6 | 183.8 | 2.58 | 1828 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6466 | E^-1, NuTau (SPICEMie ice) | 322.9 | 88.4 | 3.65 | 881 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6467 | E^-2, NuMu (PPC:SPICEMIE ice) | 853.7 | 49.7 | 17.16 | 14 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6514 | E-2, 5-component 5-component (SPICEMie ice) | 123.6 | 4.2 | 29.14 | 424 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6593 | E^-2, NuTau (SPICEMie ice) | 1445.2 | 658.5 | 2.19 | 658 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6725 | E^-2, NuE (SPICEMie ice) | 135.6 | 47.8 | 2.84 | 47 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6726 | E^-2, NuMu (SPICEMie ice) | 94.6 | 5.6 | 17.03 | 5 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6937 | of NuMu+BG (SPICEMie ice) | 101.7 | 3.9 | 25.90 | 3 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 6939 | unweighted Hoerandel, polygonato (PPC:SPICEMie ice) | 166.1 | 1.8 | 92.62 | 17 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 7017 | E-2, 5-component 5-component (SPICEMie ice) | 146.8 | 14.0 | 10.47 | 1401 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 7260 | spectrum coincident coincident (SPICEMie ice) | 671.2 | 35.7 | 18.79 | 1071 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | 7785 | E^-2, NuE (PPC:SPICEMie ice) | 128.7 | 44.3 | 2.90 | 44 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | Data | | 519.6 | 2.9 | 176.85 | 440 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ | | | | | Total | 7533 | +---------+-----------------------------------------------------+--------------+--------------+-------------+-----------------+ .. note:: The L2a takes about 200 GB per day, or roughly 75 TB for the entire year. .. note:: For comparison, the muon-channel L3 appears to reduce the file size by roughly a factor of 7. For example, /data/exp/IceCube/2010/filtered/level3-mu/sim/6568/Level3a_IC79_corsika.006568.00001x.i3.bz2 is 230 MB in size and was made from 10 L2a files with a total size of 1.6 GB.