Storage requirements

Summary

Depending on how much information is kept from the L2 files, there are three possibile total storage requirements for all experimental data and simulation produced to date:

Bare-bones: 6 TB

This is the current baseline and throws away a number of reconstructions from earlier processing steps as well as NFE pulses and IceTop information.

Keep everything: 13 TB

Delete nothing, making the L3 files a strict superset of L2a.

Middle way: 7.5 TB

Delete only the NFE recopulses from the frame.

Note

For comparison, the total size of the muon-channel L3 data is 20 TB.

Detailed file sizes for various simulation types

Note

These requirements are estimated from the size of a single file from each set. The estimates for 5-component Corsika may be off by large integer factors.

Best-case scenario: keep-list from rc3

Dataset	Description	L2 size [MB]	L3 size [MB]	Compression	Est. total [GB]
6451	unweighted Hoerandel, polygonato (SPICEMie ice)	171.5	1.8	96.68	17
6454	E^-1, NuMu (SPICEMie ice)	576.7	50.6	11.41	504
6461	E^-1, NuE (SPICEMie ice)	473.6	123.4	3.84	1228
6466	E^-1, NuTau (SPICEMie ice)	322.9	60.6	5.33	604
6467	E^-2, NuMu (PPC:SPICEMIE ice)	853.7	38.7	22.08	11
6514	E-2, 5-component 5-component (SPICEMie ice)	123.6	4.2	29.69	416
6593	E^-2, NuTau (SPICEMie ice)	1445.2	258.1	5.60	258
6725	E^-2, NuE (SPICEMie ice)	135.6	36.3	3.74	36
6726	E^-2, NuMu (SPICEMie ice)	94.6	4.3	21.94	4
6937	of NuMu+BG (SPICEMie ice)	101.7	3.3	31.05	3
6939	unweighted Hoerandel, polygonato (PPC:SPICEMie ice)	166.1	1.8	94.76	17
7017	E-2, 5-component 5-component (SPICEMie ice)	146.8	13.7	10.75	1365
7260	spectrum coincident coincident (SPICEMie ice)	671.2	34.5	19.44	1035
7785	E^-2, NuE (PPC:SPICEMie ice)	128.7	33.2	3.87	33
Data		519.6	2.5	207.14	376
				Total	5907

Worst-case scenario: delete nothing

Dataset	Description	L2 size [MB]	L3 size [MB]	Compression	Est. total [GB]
6451	unweighted Hoerandel, polygonato (SPICEMie ice)	171.5	2.3	76.00	22
6454	E^-1, NuMu (SPICEMie ice)	576.7	150.2	3.84	1499
6461	E^-1, NuE (SPICEMie ice)	473.6	490.3	0.97	4879
6466	E^-1, NuTau (SPICEMie ice)	322.9	210.4	1.53	2097
6467	E^-2, NuMu (PPC:SPICEMIE ice)	853.7	82.6	10.34	23
6514	E-2, 5-component 5-component (SPICEMie ice)	123.6	4.7	26.20	471
6593	E^-2, NuTau (SPICEMie ice)	1445.2	1107.5	1.31	1107
6725	E^-2, NuE (SPICEMie ice)	135.6	80.2	1.69	80
6726	E^-2, NuMu (SPICEMie ice)	94.6	9.9	9.55	9
6937	of NuMu+BG (SPICEMie ice)	101.7	5.9	17.26	5
6939	unweighted Hoerandel, polygonato (PPC:SPICEMie ice)	166.1	2.2	77.24	21
7017	E-2, 5-component 5-component (SPICEMie ice)	146.8	15.3	9.59	1531
7260	spectrum coincident coincident (SPICEMie ice)	671.2	39.4	17.04	1181
7785	E^-2, NuE (PPC:SPICEMie ice)	128.7	74.3	1.73	74
Data		519.6	4.4	118.52	657
				Total	13656

Middle way: delete only NFE pulses

Dataset	Description	L2 size [MB]	L3 size [MB]	Compression	Est. total [GB]
6451	unweighted Hoerandel, polygonato (SPICEMie ice)	171.5	1.8	93.51	18
6454	E^-1, NuMu (SPICEMie ice)	576.7	68.4	8.43	682
6461	E^-1, NuE (SPICEMie ice)	473.6	183.8	2.58	1828
6466	E^-1, NuTau (SPICEMie ice)	322.9	88.4	3.65	881
6467	E^-2, NuMu (PPC:SPICEMIE ice)	853.7	49.7	17.16	14
6514	E-2, 5-component 5-component (SPICEMie ice)	123.6	4.2	29.14	424
6593	E^-2, NuTau (SPICEMie ice)	1445.2	658.5	2.19	658
6725	E^-2, NuE (SPICEMie ice)	135.6	47.8	2.84	47
6726	E^-2, NuMu (SPICEMie ice)	94.6	5.6	17.03	5
6937	of NuMu+BG (SPICEMie ice)	101.7	3.9	25.90	3
6939	unweighted Hoerandel, polygonato (PPC:SPICEMie ice)	166.1	1.8	92.62	17
7017	E-2, 5-component 5-component (SPICEMie ice)	146.8	14.0	10.47	1401
7260	spectrum coincident coincident (SPICEMie ice)	671.2	35.7	18.79	1071
7785	E^-2, NuE (PPC:SPICEMie ice)	128.7	44.3	2.90	44
Data		519.6	2.9	176.85	440
				Total	7533

Note

The L2a takes about 200 GB per day, or roughly 75 TB for the entire year.

Note

For comparison, the muon-channel L3 appears to reduce the file size by roughly a factor of 7. For example, /data/exp/IceCube/2010/filtered/level3-mu/sim/6568/Level3a_IC79_corsika.006568.00001x.i3.bz2 is 230 MB in size and was made from 10 L2a files with a total size of 1.6 GB.