How we started UW FAT 09

This page was last updated on August 16, 2005

How we started UW FAT 09

First, generate a list of DOMs which includes old and any new DOMs in the DFL

1) plug in all DOMs into the DFL

2) on HUB: make sure /usr/local/python/dtsxall will pick up all DOMs

3) on HUB: make sure /usr/local/python/domnames.dat is up-to-date

4) on HUB: edit ~/bin/monitor so that it will ask for info on all DOMs

5) on HUB: move /home/testdaq/monitor/monitor.dat so we start fresh

6) on HUB: "on all", "gotoiceboot", "dtsxall", run "monitor" script

"cat" all monitor files together (eg cat monitor1.dat >> monitor.dat, cat monitor2.dat >> monitor.dat)

Scan monitor.dat for DOMs which have pressure leaks

Scan monitor.dat for DOMs which have zero scaler rates (indicates HV problem)

The file monitor.dat will be used later.

Second, make a list of domids and mbids

1) If STF was run in production room, then do this
cat prod_id.sql | mysql -u penguin domprodtest > newdoms.txt
2) If the domids are not in the database yet, then make a file called newdoms.txt which has a list of domids and mainboard ids such as (note that the FIRST LINE NEEDS TO BE SOME KIND OF HEADER LINE):

This is what newdoms.txt looks like (remember, you need a header line):
TP5P0581 347fd0110547
UP5P0582 ec54677bbb9e
TP5P0583 a3ae4b646a70

The file newdoms.txt will be used later (note, in this example, two of the fields were NULL because the production room typed in the domids incorrectly during testing and overwrote already existing DOMs - it is very common for people to remember domids incorrectly).

Third, initialize the doms and domtune tables with the new DOMs

get file monitor.dat

get file newdoms.txt

run " StartFAT.pl step1" from the directory with the above two files

this will print out a summary like this:

total number of unknown (monitor) doms = 42
total number of known (monitor) doms = 9
total number of found unknown (newdoms.txt) doms = 42
total number of found unknown T doms = 18
total number of found unknown U doms = 14
total number of found unknown A doms = 10

MAKE SURE THE SUMMARY ABOVE IS CORRECT! THIS IS VERY IMPORTANT, OTHEWISE YOUR DFL MIGHT NOT BE LOADED CORRECTLY.

The initialization code will recognize DOMs present in the monitor.dat file but not present in the newdoms.txt file and generate a message like this:

NUMBER OF unkown new (monitor) DOMs DOES NOT MATCH number of new DOMS in newdoms.txt !!!!!

It is up to the FAT initializer to determine whether or not the distribution of DOMs is correct (ie the total number of new and old DOMs, as well as the distribution of new T DOMs, U DOMs and A DOMs must match the FAT loader's statistics. Reused mainboards mean the database has to be reinitialized for these DOMs (the old DOM mainboard ids should get a "-x" on the end, and the new names should get an "ish" at the end).

Please do this: make fat.domtune.domhub = NULL for all hubs you want to initialize (only reinit the hubs you are initializing - the database has several active systems in it!). An example command would be "update domtune set domhub="" where domhub="fathub1";

Option 1: Assign the DOM names by hand

run " StartFAT.pl step2"

this time you will be asked the NAME of each new DOM. Names are best if they are short in length and have themes whereby you can associate NAMEs with particular FATs later on (for example, at pole we wanted to put DOMs from particular FATs last year near the bottom of the string)

a file called InitializeNewDoms.SQL will be created

to enter the info into the database you do this

mysql fat -u tester -h fat-database-machine -p < InitializeNewDoms.SQL (enter appropriate database machine name)

Look over the file before you enter the info into the database

Option 2: Assign the DOM names automatically

run " StartFAT.pl step2 names.txt"

names.txt is a list of DOM names and explanations and the theme, from the pirate1 database, see make-database-insertion-file.pl and isms.txt for how to insert info easily into the pirate1 database, from which you can make a simple query to create the file names.txt.

a file called InitializeNewDoms.SQL will be created

to enter the info into the database you do this

mysql fat -u tester -h fat-database-machine -p < InitializeNewDoms.SQL (enter appropriate database machine name)

Look over the file before you enter the info into the database

Fourth, upload firmware

http://glacier.lbl.gov/releases

2) Upload the appropriate mainboard release to the DOMs (use ldall)

Flasher-board-firmware.html

typically, the version of domcal you need is different from the version in the mainboard release

make sure you are using the correct TEST-DAQ meta project, and correct domhub-app project, and correct PyDOM, etc.

Fifth, run domcal

clear out files in stringproc directory ~/domcal

update domcal-wrapper script to operate on the right number of hubs/ports

"on all", "gotoiceboot", "dtsxall" on hubs, "domcal-wrapper" on stringproc (make appropriate edits to domcal-wrapper first)

domcal can take 90 minutes to run nowadays

make sure domcal ran successfully on all doms (ls *.xml | grep domcal -c)

you need to histogram the domcal HV fits and make sure that they make sense every time you run domcal

put HV values into domtune database using "java icecube.daq.domcal.HV2DB /mnt/data/testdaq/domcal"

you need to do this every time you run domcal

Sixth, generate steering files and update domtune database one more time

make a subdirectory, cd to it, and "autogen-wrapper" (make sure it is pointing to the correct hubs, and domhub-services must be running on all hubs)

enter command updateDB.pl

you can either enter the lcmodes one-at-a-time, or enter "99" and then edit the file later (lcmode 0 is for ref boards, 1 is normal, 2 is if there is no DOM below and 3 is if there is no DOM above on the string). Remember, the "B" or "U" DOMs are above the "A" or "T" DOMs for DOMs on the same wire pair - "on all" prints the pairs in reverse order by the way

Look at updateFAT.sql to make sure it looks reasonable

then update the domtune database with "mysql fat -u tester -h fat-database-machine -p < updateFAT.sql"

The database has now been setup - you need to regenerate the steering files again since the first time they didn't have the correct lcmodes

To update the domprodtest database with info from the doms database, use the command "java icecube.daq.db.app.FAT2DPT"
This command doesn't do as much as it will in the near future.
At pole you need to use the "-d -f" options!

Create a ~/bin/ explanations.txt file using a query like
select thename,thedomid,mbid,explanation from domtune where domhub="fathub1" or domhub="fathub2" or domhub="fathub5";

Lastly, update /usr/local/python/domnames.dat and rerun "monitor". Keep the info in monitor.dat - it is very useful to have these initial readings at the start of a FAT for later reference. Check that monitor.dat doesn't have any missing info in any of the fields (this would indicate database problems).

Other checks prior to starting data-taking

Clear out all old data (all old data should be moved to the official data tree)

hub cold boot data

hub /var/log/messages*

hub STF files

hub monitor files

string processor multimon data

string processor lcchain output files

string processor domcal files (especially domcal.out files)

TestDAQ data

Initialize the fat logbook

move old logbook and change docushare pointer for previous FAT

find appropriate pictures for the current FAT logbook

Initialize new logbook schedule

Reinitialize /usr/local/etc/ready.properties on each hub

ensure that no DOMs have been disabled in the ignoreDOM field; should read "ignoreDOM=000"

Ensure that error reporting is active for all DOMs in checklogs.pl

Check there is plenty of diskspace

Check that the reference PMT is on

make a file called nicknames.txt (for running mmdisplay.jar and fatdisplay.jar)

select mbid,thedomid,thename,dor from domtune and copy and paste all output to nicknames.txt

Check that the laser is connected properly (after calibrations)

update cold reboot list_of_doms_to_test.dat file so that it points to the DOMs you want to test

update domcal-wrapper script to operate on the right number of hubs/ports (already done)

update multimon-wrapper script to operate on the right number of hubs

update lux-wrapper script to operate on the right number of hubs

update reboot-wrapper script on each hub if necessary (this is new for UW-FAT-09)

update ~/bin/automate on the string processor so it points to the correct number of hubs

add machine to stf (if first time) using "java icecube.daq.db.app.AddTestInfo "

update lcchain-wrapper to operate on the right list of DOMs/hubs

make sure that checklogs.pl is going to send email to the right people

if you need to, update ~/bin/testdaq so it points to the right data tree

SWEDEN-FAT-03.meta.xml as an example for what Ciny Mackenzie needs

Checks after you start data-taking (particularly important at the start of a FAT)

check the dataquality logs for the LC MultiPE TestDAQ runs to make sure that all DOMs are producing reasonable numbers of hits - if any are producing too many then the lcmodes are wrong (DOMs are likely triggering on their own reflections if the rate is too high - only DOMs in LCMODE=0 will trigger quickly (like the reference board)

use standard TimeResolution code (fatdisplay.jar) to verify that the laser is firing properly

verify that lux appears to be doing the right thing (use mmdisplay.jar)

if a DOM has an HV problem and produces no hits, disable it in checklogs.pl (at this moment, the only DOM which is disabled in the auto-emailer is "Plaksy", so find it and add more DOMs)

if a DOM has a really serious problem, causing emails to be sent which you cannot control, disable it in /usr/local/etc/ready.properties on the hub

don't forget to look at the domcal histograms every time you run domcal