Slides and Notes 11-July-2011

I used a different ingest scheme for 2006 AMANDA data. Half of it I ran using the system rsync, and half using a more recent version that does the initial file map differently. filtered and unbiased have about 700K files each, and should be a good tool for measuring the difference.

rsync -rLptgoDvWP --inplace /Lexp/2006/AMANDA/A+ICoinc  /hsm/EXP/2006/AMANDA/ 
  >& /data/exp/hsmlogs/AMANDA/2006/ingestamanda01.log
rsync -rLptgoDvWP --inplace /Lexp/2006/AMANDA/filtered  /hsm/EXP/2006/AMANDA/
  >& /data/exp/hsmlogs/AMANDA/2006/ingestamanda02.log
rsync -rLptgoDvWP --inplace /Lexp/2006/AMANDA/internal-system  /hsm/EXP/2006/AMANDA/
  >& /data/exp/hsmlogs/AMANDA/2006/ingestamanda03.log
/data/exp/hsmlogs/rsync -rLptgoDvWP --inplace /Lexp/2006/AMANDA/monitoring  /hsm/EXP/2006/AMANDA/
  >& /data/exp/hsmlogs/AMANDA/2006/ingestamanda04.log
/data/exp/hsmlogs/rsync -rLptgoDvWP --inplace /Lexp/2006/AMANDA/unbiased  /hsm/EXP/2006/AMANDA/
  >& /data/exp/hsmlogs/AMANDA/2006/ingestamanda05.log
/data/exp/hsmlogs/rsync -rLptgoDvWP --inplace /Lexp/2006/AMANDA/upmu  /hsm/EXP/2006/AMANDA/
  >& /data/exp/hsmlogs/AMANDA/2006/ingestamanda06.log

The difference is substantial: the huge memory bump (and delay!) while creating the table is gone.

The CPU usage and ethernet usage are similar:

Chuma doesn't look much different.

Sam shows consistent behavior.

Note that the RFS software has a memory leak.

Followup on reading from /hsm

I wrote a script to read the rsync log file every minute as it tried to copy from the /hsm to a different place on the /hsm. This time the slowdown is very easy to see. It starts quite early, and within a few hours reaches a rate of less than 1 new file located per second. The effect on the CPU can be seen on the samCPU plot above: it doubles the 'User' time on the 8'th of July, but has no other effect. Even chuma shows no noticeable effect. The early collection of file information looks OK, but after about 16K files it changes behavior.

Summary:

New rsync with incremental file list starts without delay and uses much less memory.
Trying to read files from the /hsm hits a serious slowdown after 16K.
rfsd has a memory leak.

Modified 11-July-2011 at 09:46

http://icecube.wisc.edu/~jbellinger/StorHouse/11Jul2011
Previous notes Next notes Main slide directory

Slides and Notes 11-July-2011

Followup on reading from /hsm

Please contact jnbt@hep.physics.wisc.edu if you have trouble accessing the information on this page.