#!/usr/bin/perl
#
# M Krasberg, April 2004, U of Wisconsin
#
# this is a simple help utility
#

$user=getpwuid($>);

$command = $ARGV[0];


if ($command eq  "domhubapp") {

print <<EOF;

'domhubapp' will launch both rmiregistry and domhubapp

If you run 'domhubapp' from a workspace directory 
	(such as /home/$user/icecube/work) 
 then the jar files found in that workspace tree will be used.

If you run 'domhubapp' from another directory then the jar files found in 
 the standard software repository (/home/icecube/work) will be used.

'domhubapp' requires jar files from the following packages:
  'domhub', 'domhub-common'

EOF
;

}
elsif ($command eq  "domhub-maintenance") {

print <<EOF;

 To update the domhub software using the master installation script you
   should (as 'root') execute /root/icecube_software_install.pl

 User accounts should be added using (as 'root') /root/add_icecube_users.pl

 The preferred way to update software on the standard software repository
  is to (as 'root) 'su - icecube', and then to execute the script 
  'update_all_icecube_software.pl /home/icecube/work'.


EOF
;

}

elsif ($command eq  "icetop-prefill") {

print <<EOF;
                                                                                                                   
IceTop PreFill Commissioning Instructions 2006/2007
                                                                                              
log in to the hub as "testdaq"
                                                                                              
the following command can be entered at any time from a domhub:
"status"
                                                                                              
this shows what DOMs are on, their status, what software is running etc...
                                                                                              
example output:
dor dor-serial comm? loc   dom-name               dom-mbid     dom-id
curr volts  dom-stat
03A R1B0619D05 COMM TS-64  Dr_Jekyll              7b00f53b4cac TP6P1311  36
mA  89V    iceboo
                                                                                              
"03A" above is DOR card 0, wire pair 3, domA
                                                                                              
0) please keep track of when you do things in a log - we have a detailed
speradsheet which we have to fill out for each station. Most software below 
stores its output in ~/Results/

1) All Quads MUST have been tested with the Quad Connectivity Tester (QCT)
     expect one DOM to pass on each wire pair
                                                                                              
2) verify the following software versions are installed using "rpm -qa"

 dor-driver V02-11-00    <-- out-of-date on spts64
    (if you have trouble with the dor-driver, after installing the testdaq rpm,
    go to /mnt/data/testdaq/rpms/ and you can install the non-rpm by hand.)
                                                                                                                  
 domhub-tools 210        <-- out-of-date on spts64
 moat 02.06.0
 domapp-tools 01.03
 jboss-icecube 2.0.8
 pydom 4.1-1             <-- out-of-date on spts64
 testdaq 7.0             <-- out-of-date on spts64 
 java jdk1.5.0_07 (look in /usr/java)

                                                                                              
3) check for anomolous currents in configboot state:
run following program. If it takes more than around 20 seconds to complete,
issue command "status", then "off all". Save screen output as a result from
"status". If "status" takes more than a few seconds, then abort and "off
all", and consult.
                                                                                              
"current-test-configboot"
                                                                                              
look at current and voltage readings.
an anomoly of 5 mAmps is cause for concern.
                                                                                              
In the event of a current anomoly, switch quads, repeat, and determine if
problem is with the DOM or with the DOR card. If the problem is with the 
DOR card (ie if the anomolous current stays in place), continue. If the 
problem is with the DOM, then be prepared to replace the DOM.
                                                                               
                                                                                              
4) check for anomolous currents in iceboot state:
                                                                                              
this program takes a little longer to run (under a minute though)
                                                                                              
"current-test-iceboot"
                                                                                              
look at current and voltage readings.
an anomoly of 5 mAmps is cause for concern.
                                                                                              
In the event of a current anomoly, switch quads, repeat, and determine if
problem is with the DOM or with the DOR card. If the problem is with the DOR
card, continue. If the problem is with the DOM, then be prepared to replace the
DOM.
                                                                                              
make sure the DOMs read out (for a given DOR card) in the order
61->62->63->64 and make sure the expected string is connected to said DOR
card.
                                                                                              
(note - in iceboot, database is preloaded - this will not be the case for
InIce).
                                                                                              
5) check local coincidence connections:
                                                                                              
dtsxinit
lcchain-wrapper-icetop 0
                                                                                              
will test DOR card 0.
                                                                                              
if you don't get all "P"s then something is wrong.

6) check pressure for each DOM
dtsxinit
test.py 0 0 a  <--- will test card 0, pair 0, domA
                                                                                              
7) check communications, part1:
"shortmoat-wrapper" will run "shortmoat" (15 minutes or so?)
output will be put into ~/Results/CommsTests/shortmoat
                                                                                              
when it is finished (use "status") you can continue
                                                                                              
the command "stopcommstests" is an imperfect way to abort communications
testing. use "ps -ef" to clean up what is left.
                                                                                              
8) check communications, part2:
"comms-tests-wrapper card#"
this will run quadtool (20 minutes for one quad) followed by a 14-hour moat
on all DOMs. If there are severe time restrictions, the moat test can be
aborted (the command imperfect "stopcommstests" command will attempt to 
shut communications tests down).
                                                                                              
Results are put into ~/Results/CommsTests/quadtool and
~/Results/CommsTests/moat14
                                                                                              
                                                                                              
EOF
;

}

elsif ($command eq  "misc") {
                                                                                                                   
print <<EOF;

old commands:
 source /home/krasberg/bin/set_classpath
 rmiregistry
 java icecube.daq.domhub.DOMHub /usr/local/etc/dh.properties
 java icecube.testdaq.control.TestDAQControl

coincidence code:
 java -Xmx1024m icecube.testdaq.coincidence.Coincidence zipfile 1000 4

independent data collector:
 java icecube.testdaq.datacollector.DataCollectorFactoryImpl

to run ezplot: java icecube.testdaq.ezplot.Plotter

to run RapCalTest:  java icecube.testdaq.datacollector.RapCalTest <tcalibFile>

to run data quality test: java icecube.testdaq.test.DataQualityTest <ZIPfile>

to run domtest:
nice /usr/java/j2sdk1.4.2_04/bin/java -Xmx2048m icecube.testdaq.domtest.DomtestMain SteeringFile

for domcal: 
off all, on all, gotoiceboot, dtsxall, then: the following will do first 64 doms on card 0:
java icecube.daq.domcal.DOMCal localhost 5000 64 /home/testdaq/domcal/ calibrate dom calibrate hv


java icecube.daq.domcal.HVHistogramGrapher /domcal /domcal/histos .

  java icecube.daq.domcal.HVHistogramGrapher {domcal xml dir} {output dir}
{html root}

to run and put graphs into histos subdirectory:
  java icecube.daq.domcal.HVHistogramGrapher ./ histos ./


to install latest release and domcal on a dom:
 to upload a build to ALL DOMs:
  off all
  insall.sh /home/testdaq/release239/release.hex
 for domcal:
  xymodem -v5 -z /dev/dhc0w0dA domcal.something.gz domcal

 after laoding the software you must do this:
   echo "0" > /proc/driver/domhub/blocking



for TCALs:
tcaltest /proc/driver/domhub/card0/pair0/domA/tcalib 5

stf: (dtsx should NOT be running)
 off all
 source setclasspath /home/testdaq/work-stf/
 java icecube.daq.stf.STF
 or, to run stf interactively:
 off all
 stf.sh /home/testdaq/work-stf temperature

to copy fat database DOM Ids to domprodtest database:
 must run from "/home/testdaq/work" workspace because recent version of this
   program is in the HEAD of the daq-db-common project
 java icecube.daq.db.app.AddHardware
 

after a reboot the following needs to be done:
 as 'root': "nohup domhub-services &"
 execute command: "resetdsb"
 then use toothpick to reset the DSB

to generate steering files:
 from the domhub: autogen-wrapper, this does the following:
  on all
  gotoiceboot
 EITHER on the domhub:  autogen-steering -H databasemachine
 OR on the databasemachine: autogen-steering domhubmachine


to stop jboss:
sudo /usr/local/icecube/bin/jbossMgt.sh stop
                                                                                 
to put doms in iceboot using se.pl
on all
se.pl all r r.+\?\>
                                                                                 


EOF
;
 
}

elsif ($command eq  "domhub-services") {

print <<EOF;

domhub-services should always be running.
after a domhub reboot, launch is as "root" using the command:

nohup domhub-services &

EOF
;

}
elsif ($command eq  "status") {
                                                                                                                   
print <<EOF;
                                                                                                                   

The command status can be entered at any time from a domhub.
                                                                                               
It shows which DOMs are on, their status, what software is running etc...
                                                                                               
example output:

DOR Port Qud DORserial# Stat Pos    NAME                 MBID        DOMID    RawC  CorrC  Volts    State
10B 5009 Q_4 R1B0620D05 COMM C-09   IPA              ad1e249c87f7  UP4P0284  81 mA  82 mA  89V    iceboot
10A 5010 Q_4 R1B0620D05 COMM C-10   Sergeant_Towser  4208dd9ec4ff  TP6P1691  80 mA  81 mA  89V    iceboot
                                                                 
"10A" above is DOR card 1, wire pair 0, domA
dtsx would set it to communicate on port 5010. On an InIce string it would
be found in position #10 on quad #4.


This command will report if the wrong number of DOMs/DOR cards are
responding on the hub. eg:

***spts64-ithub01 : Unexpected # of communicating DOMs: expected 8; found 9.
   if 9 is the correct #, please update /mnt/data/testdaq/hubConfig.dat

+++spts64-ithub01 : Unexpected # of DOR cards: expected 3; found 2.
   if 2 is the correct #, please update /mnt/data/testdaq/hubConfig.dat
                                                                                               

EOF
;

}
elsif ($command eq  "domhub") {
                                                                                                                   
print <<EOF;
                                                                                                                   

The command "domhub" can be entered at any time from the string processor.
        
1st argument is typically the hub list (can be "all", "inice", "icetop", an individual
hub, or a file containing a list of hubs)
2nd argument is the command
                             
'domhub icetop table' will get the status of all of the IceTop DOMs
'domhub all getdomcal' will get all the domcal files and histograms from all hubs and put them into /mnt/data/testdaq/domcal on the string processor.                       
'domhub all gotoiceboot' will execute "iceboot all" on all hubs
'domhub inice ready-iceboot' will execute the command "ready-iceboot" on the inice domhubs                                                                        
'domhub sps-ichub57 checkGPS' which check the GPS on sps-ichub57
'domhub hublist.txt monitorDomhubApp' which monitor the progress of TestDAQ's domhubApp software on all hubs 
						in the file hublist.txt - hubs are listed one per line


NOTE:
 'domhub' can hang if a domhub has crashed.
 you can always to "ps -ef" and kill the hung ssh process.
 this means there is likely to be some kind of problem which requires immediate attention...

EOF
;

}
elsif ($command eq  "distribute") {
                                                                                                                   
print <<EOF;
                                                                                                                   

The command "distribute" can be entered at any time from the string processor.
                                                                                               
"distribute file.txt directory" will put the file "file.txt" into the directory "directory" on all domhubs.

eg:
"distribute status /mnt/data/testdaq/bin/"

EOF
;

}
elsif ($command eq  "TestDAQ") {
                                                                                                                   
print <<EOF;
                                                                                                                   
"TestDAQ" is easy to run.
If you are running on large groups of hubs you should make use of the command "domhub" from the string processor.

The overall procedure is as follows (you can enter the commands directly on a domhub):

The "domhub" commands:
1) ensure that pdaq has stopped
2) enter command 'domhub all stoptestdaqnopowercycle'
4) enter command 'domhub all gotoiceboot'
5) enter command 'domhub all gotoiceboot' again
6) enter command 'domhub all table'  (from a domhub the command is "status")
    Ensure that all DOMs are in iceboot (there is a DOM 39-22 "Liljeholmen" which always comes up BUSY)
7) If all the DOMs are in "iceboot" then issue the command 'domhub all ready-iceboot'
8) All hubs are now ready to take data

The "string processor" tasks:
1) Use either sps-testdaq01 or sps-testdaq02
2) Edit an appropriate "automate.???" script (eg automate.icetop.weekly)
      A) Ensure that the "steering_dir" is pointing at the steering files you want to use
      B) Make sure that you either have a continuous loop, or not, depending on what you are trying to do
      C) Make sure the line after "testdaq" is pointing at the goup of domhubs you want to use (remember a "-d" for each domhub)
3) Enter command "go.????" (eg "go.icetop.weekly") 

TestDAQ will now start up.
1) a directory called /mnt/data/testdaq/output???????/ will be created  ("???????" is the run number)
2) you can "cd" into this directory and see things start up. 

you can monitor the progress of the TestDAQ software on each hub with the
command  'domhub all monitorDomhubApp'   <--- this will determine which hub
has a problem.

3) an email will be sent out saying TestDAQ is starting up
4) you should watch the first run to make sure it succeeds
5) When you see ".hit" file growing, run is likely going fine
6) run will end and "dataquality" software will run
7) If there a missing DOM in the "dataqualitylog" then an email will be sent
8) At run end, run will be moved from "output????" to /mnt/data/testdaq/latest_data
      a tarball and semaphore file of the entire run will be put into /mnt/data/testdaq/spade/
      Hit and TCAL files will be deleted from "latest_data"
9) Tarballs (and semaphores) to go onto satellite need to be moved to /mnt/data/testdaq/dropbox/
     (the tarballs must be moved first, moving * moves tarballs first since they are first alphabetically)
10) Tarballs (and semaphores) which will not go on satellite need to be moved to /mnt/data/testdaq/tape/
11) If not in a continuous loop, when the runs end, an email will be sent

NOTE - any emails indicating an error should be regarded as being important...

 

EOF
;

}
elsif ($command eq  "weekly-runs") {
                                                                                                                   
print <<EOF;

See "TestDAQ" for how to run TestDAQ.
Note that if you want to do only icetop stuff, or inice stuff, then use
"icetop" or "inice" instead of "all" below.

1) Ensure pdaq has stopped running
2) choose a string processor (sps-testdaq01 or sps-testdaq02)
3) enter command 'domhub all stoptestdaqnopowercycle'                                                                                                                    
4) enter command 'domhub all gotoiceboot'
5) enter command 'domhub all gotoiceboot' again
6) enter command 'domhub all table'
    Ensure all DOMs are in ICEBOOT (39-22 "Liljeholmen" always comes up as busy)
7) enter command 'domhub all ready-iceboot'
8) enter command "go.icetop.weekly" from one stringprocessor
9) enter command "go.inice.weekly" from other stringprocessor
10) ensure that data is being taken.
stuck hubs can be pinpointed via:
'domhub all monitorDomhubApp'

11) put appropriate data into ~/dropbox, and other data into ~/tape 

     Typically, put the 2nd DarkNoise IceTop run and also the IceTop cosmic run into ~/dropbox/  
       and put the other IceTop runs into ~/tape/ for Cindy to tape.
     Put all the InIce runs into ~/dropbox/

12) after the runs have completed, put the hubs into a neutral state:
     'domhub all stoptestdaqnopowercycle'
     'domhub all gotoiceboot'
     'domhub all gotoiceboot'
     'domhub all table'
     
     Ensure all DOMs are in iceboot. (remember, 39-22 "Liljeholmen" will not get into IceBoot.
     The hubs are now ready for pdaq...


EOF
;

}
elsif ($command eq  "TestDAQ-problems") {
                                                                                                                   
print <<EOF;

See "TestDAQ" for how to run TestDAQ

==================================================
In the rare instances where you get error emails:

1) read the email to understand what the problem was
2) if a solitary DOM drops out it is not serious, if lots of DOMs drop out it is very serious
3) you can either do a controlled shutdown, or an abort
     aborts kill the current run and also runs which are being post-processed
4) executing a controlled shutdown
     A) issue command "pausetestdaq" on the appropriate string processor
     B) wait until run ends, then reinitialize whatever hub(s) need to be reset
         you can do this on individual hubs, or use the command "domhub"
5) exeecuting an abort
     A) issue command "stoptestdaq" on the appropriate string processor
     B) issue command "stoptestdaqnopowercycle" or "stoptestdaq" on the hubs, as appropriate 
            (it depends on whether or not you want to turn the DOMs off, usually you do not)
6) investigate/fix problem and then restart

==================================================
If TestDAQ does not start up properly:

try this command from the testdaq string processor:

domhub all monitorDomhubApp

this can be used to locate a "stuck" hub.
If you find a stuck hub:

The solution is to "stoptestdaq" on the string processor
Then, on the offending hub:
"stoptestdaq"
"pCycle"           <-- power cycles all DOMs and puts them into iceboot
"ready-iceboot"    <-- starts up domhubApp software
And finally, on the string processor,
enter the appropriate "go.????" command again...




EOF
;

}
elsif ($command eq  "inice-commissioning") {
                                                                                                                   
print <<EOF;

InIce Commissioning is almost finished, but there are DOMs still in water, and high current DOMs!!!

Once a week, issue command "current-test-configboot" on the sps-wczar domhub (as testdaq).

sps-wczar breakdown

card 0:  two high current DOMs from last year (01B=40-51 "Juneberry" and 01A=40-52 "Alfa_Romeo_Spider")
card 1:  Two DOMs in water ( 8 DOMs from strings 58 and 74)
card 2:  top conenctor: four water DOMs from string 56, not communicating - cable not jumpered...
card 3:  SPOTS DOMs  (the South Pole Outdoor Test System 32B="Benzene" and 32A="Dopamine")
card 4:  Four high current DOMs from string 72
card 5:  Two high current DOMs from string 72
card 6:  Six high current DOMs from string 46
card 7:  "PSU DOM" and its partner, from string 48, in bottom connector
         also note, top connector has missing screw
         
The DOMs on card 1 can be considered to be commissioned, and can be integrated into the DAQ when
they are no longer in water

No other DOMs on sps-wczar are commissioned...


To commission a string (this has to be done from the beginning when uncommissioned DOMs are reintegrated into strings),
the basic commands are:

from the domhub:

1) ensure all pdaq processes are terminated and testdaq processes should be terminated with "stoptestdaq"
2) "current-test-configboot" - ensure no corrected DOM current is greater than 65mA
3) "current-test-iceboot"    - ensure all DOMs make it into iceboot and currents look reasonable
   current-test-iceboot creates a file called /tmp/dbinsertion.pl
   copy this file to /mnt/data/testdaq if there are "new" DOMs (they will have no location on the screen).
   then:
      cat dbinsertion.pl | mysql -u tester -p -h sps-testdaq01
      (you should know the password, otherwise look in /usr/local/etc/domcal.properties )
   then:
      execute "make-nicknames-file-wrapper"  (this will update /mnt/data/testdaq/nicknames.txt )
4) then cd to /mnt/data/testdaq/Results/Monitor
   do:
     dtsxinit
     monitor
     
   look at monitor.dat 
    for the current time period, ensure the rates look reasonable and no DOM is in water (if the DOM temp is
      less than -10 C then it is frozen)
5) check LC with command "lcchain-wrapper" (DOMs should start off in iceboot and dtsx needs to be running
   ensure that all pairs pass (if 59 out of 59 pass then all is good!)
6) run domcal, again DOMs need to be in iceboot and dtsx needs to be running
   command is "domcal-wrapper"
   output files go in to /mnt/data/testdaq/domcal/new-files/XXXXXXXXX/ where XXXXXXXX is a directory with the current time.
   Files are then automatically copied to /mnt/data/testdaq/domcal/ and histograms to /mnt/data/testdaq/domcal/histos/  
7) all domcal files from all hubs can be picked up later from the testdaq string processor with the command "domhub-all getdomcal"
8) check rate stability for 2+ hours 
      put DOMs into iceboot and start up dtsx ("dtsxinit")
      "multimon-deadtime-wrapper"
      check rates and temperatures with program "mmdisplay"      
9) run "stf" (start DOMs in iceboot and run dtsx)
    IMPORTANT - never test all DOMs at once - either test even or odd DOMs.      
10) run testdaq for 24 hours
      with the DOMs in iceboot, and domhub-services running, go to the string processor.
      create a directory under /mnt/data/testdaq/steering_files/commissioning/
      execute command:
        autogen-steering-new  -H sps-testdaq01 -S SPS -T 600 sps-ichub79 sps-ichub80    (if you want to test hubs 79 and 80)
         check that the expected number of DOMs are in the xml files with "grep -i domid *.xml -c"
        edit automate.commissioning to point to the directory you created
        ensure automate.commissioning is set to run in a continuous loop
        get the DOMs ready for TestDAQ on the hubs, and execute the command "go.commissioning" on the string processor
    after 24 hours stop TestDAQ - pay attention to all TestDAQ emails in the meantime, and fix problems as they occur!
11) test DOM communications with "dmt-tests-wrapper" (takes several hours)
    test DOM communications with "comms-tests-wrapper all" (takes around 20 hours unless MOAT fails, which it usually does)
        
   


EOF
;

}
elsif ($command eq  "stoptestdaq") {
                                                                                                                   
print <<EOF;

"stoptestdaq" is an abort command.
it will stop all TestingDAQ software and turn off the DOMs (if it is run on a hub)

EOF
;

}
elsif ($command eq  "stoptestdaqnopowercycle") {
                                                                                                                   
print <<EOF;

"stoptestdaqnopowercycle" is an abort command.
it will stop all TestingDAQ software but will not turn off any DOMs.


EOF
;

}
elsif ($command eq  "dtsxinit") {
                                                                                                                   
print <<EOF;

"dtsxinit" power cycles the DOMs and starts up dtsx (port numbers are mapped to DOM device files).
it will stop all TestingDAQ software but will not turn off any DOMs.

port 5001 corresponds to the DOM at the top of an inice string ( 00B )
port 5002 corresponds to the DOM in the 2nd spot (00A)

port 5060 corresponds to the bottom DOM (71A)


EOF
;

}
elsif ($command eq  "dtsxinitnopowercycle") {
                                                                                                                   
print <<EOF;

the same as "dtsxinit" except no DOM power cycling is performed

EOF
;

}
elsif ($command eq  "stf") {
                                                                                                                   
print <<EOF;

STF is the "Simple Test Framework"
With it you test the function of DOM mainboards/HV supply/flasher boards

The most important thing about STF is that it should NEVER EVER be run on
"T" DOMs and "U" DOMs at the same time.

The default is to run on all DOMs - you can never run it in default mode at
the pole!

As long as you remember this fact, no damage will be done.

Also, bear in mind that STF uses the flashers and puts light into the
detector.

To run stf you put the DOMs on, put them in iceboot and turn on dtsx
(ie use the "dtsxinit" command, on sps-wczar there is a command called
"dtsxinit-doms-to-commission", and YOU MUST USE THIS COMMAND INSTEAD
on sps-wczar!)


then, on the hub, you run the command "stf"

If nothing happens (ie if it immediately hangs), then the database has not been set up
properly.

note - graphical tunneling mmust be available - ie "ssh -X" to get to the
hub - otherwise you will get an error that a window could not be opened up.

after you start up "stf" you want to click on "select DOMs from server" and
then, since you are running "stf" from a hub, you select "localhost"

port numbers should be 5001 thru 5064 (ie 64 DOMs)

It is VITAL that you select "only T DOMs", and then the second time you run
it "only U DOMs"

then you select "all-tests" for the set of tests to run - ie
select the directory /mnt/data/testdaq/all-tests/

it is important that you do not double click on the directory "all-tests"

in the end you will have a group of DOMs listed, and a group of tests.
I think there are around 54 tests in the all-tests directory, and in the
case of running "dtsxinit-doms-to-commission" on sps-wczar in mid april of
2007, 16 DOMs should be found on sps-wczar. These numbers should be shown on
the bottom right corner of the screen.

Then you start the tests - three windows will pop up and ask you questions
like:
what is the temperature?  Answer, -30
Do you want to test integrated DOMs? Answer, yes

If you get asked a question to enter a particular DOMID or DOM serial
number, then tell me what the serial number is and abort the test (and enter
command "stoptestdaq"). DO NOT ENTER ANY DOM SERIAL NUMBER INFORMATION!!!

then start the tests.

Then a window will pop up and you will eventually (hopefully) see lots of
gren boxes. red boxes are bad.

tell me which boxes are red. I expect red boxes for the STF ADC test for the
DOMs "Ant_Nebula" and "Spirograph".


Remember, run STF for "T" and "U" DOMs at different times. 
run it from spw-wczar if you want to test DOMs on sps-wczar.
DO NOT RUN IT ON ALL DOMs at once!


Good luck!


EOF
;

}
elsif ($command eq  "status-output") {
                                                                                                                   
print <<EOF;

here is a sample output from the "status" command


-------------------------------------------------------------------------------
SPS-ICHUB56 SUMMARY:

DOR Port Qud DORserial# Stat Pos    NAME                       MBID      DOMID    RawC  CorrC  Volts    State
00B 5001 Q_2 R1B0472D04 COMM 56-01  Hawaii                 03f964a55891 UP5P0934  78 mA  77 mA  89V       busy
00A 5002 Q_2 R1B0472D04 COMM 56-02  Sparv                  171dacb4189d TP5H0131  78 mA  77 mA  89V       busy
01B 5003 Q_2 R1B0472D04 COMM 56-03  Desmoche               2fad6ab546b8 UP6P1380  80 mA  78 mA  88V       busy
01A 5004 Q_2 R1B0472D04 COMM 56-04  Pankow                 276f2e443b2e TP5Y0143  80 mA  78 mA  88V       busy
02B 5005 Q_3 R1B0472D04 COMM 56-05  Whooping_Coughish      39f663d7f943 UP5P0700  81 mA  78 mA  89V       busy
02A 5006 Q_3 R1B0472D04 COMM 56-06  Silent_Hill            9a4c2fc88494 TP6P1275  80 mA  77 mA  89V       busy
03B 5007 Q_3 R1B0472D04 COMM 56-07  Hermannskobb           c8ea2f91956e UP5H0134  82 mA  80 mA  88V       busy
03A 5008 Q_3 R1B0472D04 COMM 56-08  Eiebt                  ed02ce817a2e TP6Y4479  82 mA  80 mA  88V       busy
10B 5009 Q_4 R1B0473D04 COMM 56-09  Houtu                  f158c8761a37 UP6Y4464  79 mA  79 mA  89V       busy
10A 5010 Q_4 R1B0473D04 COMM 56-10  Ma_Mien                32b90a69b8ce TP6Y4383  79 mA  79 mA  89V       busy
11B 5011 Q_4 R1B0473D04 COMM 56-11  Epidemiology           30c2b92f9b90 UP5P0782  80 mA  78 mA  89V       busy
11A 5012 Q_4 R1B0473D04 COMM 56-12  Paraschurama           472d57d94c7c TP6Y4373  80 mA  78 mA  89V       busy
12B 5013 Q_5 R1B0473D04 COMM 56-13  Storskarv              d7f8f7bfdf5c UP5H0240  80 mA  79 mA  89V       busy
12A 5014 Q_5 R1B0473D04 COMM 56-14  Touchmenot             e58f9a082602 TP6P1259  79 mA  78 mA  89V       busy
13B 5015 Q_5 R1B0473D04 COMM 56-15  Spurge                 df3a39ce9f06 UP6P1252  78 mA  78 mA  89V       busy
13A 5016 Q_5 R1B0473D04 COMM 56-16  Onchocerciasis         86a3fd1a9406 TP5P0991  78 mA  78 mA  89V       busy
20B 5017 Q_6 R1B0474D04 COMM 56-17  Ahau_Chamahez          520fd910f398 UP6Y4324  79 mA  79 mA  88V       busy
20A 5018 Q_6 R1B0474D04 COMM 56-18  Sleepy_Hollow          4a3e558dbb75 TP6P1295  79 mA  79 mA  88V       busy
21B 5019 Q_6 R1B0474D04 COMM 56-19  Elephantiasis          1eb997390c7d UP5P1062  79 mA  81 mA  88V       busy
21A 5020 Q_6 R1B0474D04 COMM 56-20  Dchi                   ffce4453a7f5 TP6Y4273  79 mA  81 mA  88V       busy
22B 5021 Q_7 R1B0474D04 COMM 56-21  Autassasinophilia      1d8883c5eb05 UP5P1000  79 mA  78 mA  88V       busy
22A 5022 Q_7 R1B0474D04 COMM 56-22  Entita                 035cd56b0c12 TP5H0139  79 mA  78 mA  88V       busy
23B 5023 Q_7 R1B0474D04 COMM 56-23  Maneki_Neko            c0bb2366b6e7 UP6P1510  83 mA  80 mA  88V       busy
23A 5024 Q_7 R1B0474D04 COMM 56-24  Elm_Street             178fb493ef75 TP6P1309  83 mA  80 mA  88V       busy
30B 5025 Q_8 R1B0475D04 COMM 56-25  Gollum                 a861a18355e1 UP5P0566  81 mA  82 mA  89V       busy
30A 5026 Q_8 R1B0475D04 COMM 56-26  Spades                 87ae7f55ed9b TP6P1387  81 mA  82 mA  89V       busy
31B 5027 Q_8 R1B0475D04 COMM 56-27  Horsten                62481b3949e9 UP5H0142  81 mA  82 mA  89V       busy
31A 5028 Q_8 R1B0475D04 COMM 56-28  See_No_Evil            7c4492e84f06 TP6P1285  81 mA  82 mA  89V       busy
32B 5029 Q_9 R1B0475D04 COMM 56-29  Conchology             fdc5695d1990 UP5P0722  78 mA  81 mA  89V       busy
32A 5030 Q_9 R1B0475D04 COMM 56-30  Psittacosis            acb8521e5103 TP5P1007  79 mA  82 mA  89V       busy
33B 5031 Q_9 R1B0475D04 COMM 56-31  Peony                  5579ab70e064 UP6P1216  81 mA  82 mA  89V       busy
33A 5032 Q_9 R1B0475D04 COMM 56-32  NN0014                 d7de449b3be8 TP4H0014  81 mA  82 mA  89V       busy
40B 5033 Q10 R1B0476D04 COMM 56-33  Etesianish             6a7d013b1f79 UP5P0500  83 mA  83 mA  89V       busy
40A 5034 Q10 R1B0476D04 COMM 56-34  Dragonsmouth           4d7af237c3f5 TP6P1255  83 mA  83 mA  89V       busy
41B 5035 Q10 R1B0476D04 COMM 56-35  Coyote                 5038af08fa94 UP4P0296  84 mA  81 mA  89V       busy
41A 5036 Q10 R1B0476D04 COMM 56-36  Mumps                  4e94b4fc575e TP5P1017  84 mA  81 mA  89V       busy
42B 5037 Q11 R1B0476D04 COMM 56-37  Firethorn              4f369ab46dbf UP6P1208  83 mA  81 mA  89V       busy
42A 5038 Q11 R1B0476D04 COMM 56-38  Horror1                5673b2f9ba6f TP6P1247  83 mA  81 mA  89V       busy
43B 5039 Q11 R1B0476D04 COMM 56-39  Mazda_RX7              4ae40cd8a13b UP5Y0144  81 mA  83 mA  89V       busy
43A 5040 Q11 R1B0476D04 COMM 56-40  Luxing                 ca54112824aa TP6Y4447  81 mA  83 mA  89V       busy
50B 5041 Q12 R1B0477D04 COMM 56-41  Solsidan               68521dbc618b UP5H0248  81 mA  81 mA  89V       busy
50A 5042 Q12 R1B0477D04 COMM 56-42  Ljuskobben             4bac501e2e5f TP5H0107  81 mA  81 mA  89V       busy
51B 5043 Q12 R1B0477D04 COMM 56-43  Camellia               3f32d31b4ea8 UP6P1220  81 mA  83 mA  89V       busy
51A 5044 Q12 R1B0477D04 COMM 56-44  New_Orleans            342a20afb5b0 TP5P0905  81 mA  83 mA  89V       busy
52B 5045 Q13 R1B0477D04 COMM 56-45  The_Others             a908a67793b0 UP6P1276  82 mA  83 mA  89V       busy
52A 5046 Q13 R1B0477D04 COMM 56-46  Pygmy_Shrew            29c29d5c727e TP4P0263  83 mA  84 mA  89V       busy
53B 5047 Q13 R1B0477D04 COMM 56-47  Cholera                bc02916c64b8 UP5P1038  82 mA  83 mA  89V       busy
53A 5048 Q13 R1B0477D04 COMM 56-48  Tepeyollotli           f1f54d2818c8 TP6Y4363  82 mA  83 mA  89V       busy
60B 5049 Q14 R1B0478D04 COMM 56-49  Orre                   bb4684878fd7 UP5H0154  81 mA  83 mA  88V       busy
60A 5050 Q14 R1B0478D04 COMM 56-50  Sake                   a14806c131c6 AP4P0071  81 mA  83 mA  88V       busy
61B 5051 Q14 R1B0478D04 COMM 56-51  Bride_Of_Chucky        61a0d760953f UP6P1308  80 mA  83 mA  89V       busy
61A 5052 Q14 R1B0478D04 COMM 56-52  Sweet_Pea              910e941307eb TP6P1265  80 mA  83 mA  89V       busy
62B 5053 Q15 R1B0478D04 COMM 56-53  Hodophobia             52cae978ebb0 UP5P0690  83 mA  83 mA  88V       busy
62A 5054 Q15 R1B0478D04 COMM 56-54  Benu                   7cf34d883b4d TP6Y4257  83 mA  83 mA  88V       busy
63B 5055 Q15 R1B0478D04 COMM 56-55  Blennophobia           533893a39a29 UP5P0674  84 mA  83 mA  88V       busy
63A 5056 Q15 R1B0478D04 COMM 56-56  Rabies                 4e6a97fcef81 TP5P0987  83 mA  82 mA  88V       busy
70B 5057 Q16 R1B0479D04 COMM 56-57  Hackspett              d43401450483 UP5H0102  80 mA  81 mA  89V       busy
70A 5058 Q16 R1B0479D04 COMM 56-58  Tengu                  0491472ce359 TP5P0615  81 mA  82 mA  89V       busy
71B 5059 Q16 R1B0479D04 COMM 56-59  Dopping                9792e6e3b123 UP5H0176  80 mA  81 mA  89V       busy
71A 5060 Q16 R1B0479D04 COMM 56-60  Mirugai                e9bd3e571f1d AP4P0067  80 mA  81 mA  89V       busy

communicating 60 DOMs; busy 60 DOMs;
software: 64 dtsxs; 1 multimons;

-------------------------------------------------------------------------------


EOF
;

}
elsif ($command eq  "current") {
                                                                                                                   
print <<EOF;

the command "current" shows the DOR current for all wire pairs.
This is useful is DOMs come up as non-communicating.

Usually, if all the DOMs draws current but come up as non-communicating, then
the GPS is not working.

EOF
;

}
elsif ($command eq  "pDAQ-interaction") {
                                                                                                         
print <<EOF;

After pDAQ has been stopped issue the following domhub commands
(remember, you can type domhub-icetop or domhub-inice or domhub-all)

domhub all stoptestdaqnopowercycle (ensures that all testdaq processes are
stopped)
domhub all gotoiceboot
domhub all gotoiceboot
domhub all table

and ensure that all DOMs you want to be in IceBoot are in IceBoot.

watch for messages saying the number of DOMs on a string is incorrect... 

Do exactly the same thing before you give the DOMs back to pDAQ


EOF
;

}

elsif ($command eq  "iceboot") {
                                                                                                                   
print <<EOF;

the command "iceboot all"  will put all the DOMs into iceboot after you turn them on,
assuming no other process has locked them.

It is important not to leave a DOM in configboot (the state they are in after you turn them
on) because a DOM can get its FLASH erased easily when it is in configboot.

It is currently often necessary to run "iceboot all" twice, because
of a DOM softboot problem. I have never had to run the command three times
in order to get all DOMs into iceboot. Note that 39-22 "Liljeholmen" will
never get into iceboot (it is always "BUSY").

After you put the DOMs into iceboot you should run "status" from the hub
(or "domhub table" from the string processor) to ensure that all
DOMs are indeed in iceboot.

EOF
;

}
elsif ($command eq  "pDown") {
                                                                                                                   
print <<EOF;

the command "pDown"  will power down groups of DOMs. Appropriate power information
is recorded for all DOMs just prior to power down.

'pDown'        all DOMs are powered Down
'pDown 3'      all DOMs on DOR card 3 are powered Down
'pDown 3 1'    all DOMs on DOR card 3 wire pair 1 are powered Down
'pDown Q06'    all DOMs on quad 6 are powered Down
'pDown Q6'     all DOMs on quad 6 are powered Down
'pDown Q_6'    all DOMs on quad 6 are powered Down


EOF
;

}
elsif ($command eq  "pUp") {
                                                                                                                   
print <<EOF;

the command "pUp"  will power Up groups of DOMs and put them into iceboot. Appropriate power information
is recorded during power Up.

'pUp'        all DOMs are powered Up and put into iceboot
'pUp 3'      all DOMs on DOR card 3 are powered Up and put into iceboot
'pUp 3 1'    all DOMs on DOR card 3 wire pair 1 are powered Up and put into iceboot
'pUp Q06'    all DOMs on quad 6 are powered Up and put into iceboot
'pUp Q6'     all DOMs on quad 6 are powered Up and put into iceboot
'pUp Q_6'    all DOMs on quad 6 are powered Up and put into iceboot


EOF
;

}
elsif ($command eq  "pCycle") {

print <<EOF;

the command "pCycle"  will power Down and then power Up groups of DOMs and put them into iceboot.
Appropriate power information is recorded at power DOwn and then again during power Up.

'pCycle'        all DOMs are power Cycled and put into iceboot
'pCycle 3'      all DOMs on DOR card 3 are power Cycled and put into iceboot
'pCycle 3 1'    all DOMs on DOR card 3 wire pair 1 are power Cycled and put into iceboot
'pCycle Q06'    all DOMs on quad 6 are power Cycled and put into iceboot
'pCycle Q6'     all DOMs on quad 6 are power Cycled and put into iceboot
'pCycle Q_6'    all DOMs on quad 6 are power Cycled and put into iceboot


EOF
;

}
elsif ($command eq  "checkGPS") {

print <<EOF;

the command "checkGPS"  will check the status of the GPS for all DOR cards
on the hub. It will report if the wrong number of DOR cards are found, or if 
a DOR card clock is set to internal mode, or if something is wrong with the
GPS, or if the wrong number of communicating DOMs are found.

reading the GPS signal can interfere with data-taking, so the user is
prompted before the check is actually made...

example output looks like:



                Check GPS
Checking GPS may conflict with processes
currently running on the DOMs.
Do you with to continue? (yes/no) yes
Check GPS: confirmed.


Key:
        .       - External & GPS, all looks good
        BAD     - External & No GPS
        INT     - Internal & GPS
        DOR     - Internal & No GPS
        " "     - No DOR card detected
        *?*     - Program Error: type from clksel unkown.
        #?      - Program Error: unexpected dir structure.

-----------
Hub:    01
crd0    .
crd1    .
crd2
crd3
crd4
crd5
crd6
crd7
-----------

+++spts64-ithub01 : Unexpected # of DOR cards: expected 3; found 2.


EOF
;

}
elsif ($command eq  "gotoiceboot") {

print <<EOF;

the command "gotoiceboot"  will make an attempt to move all of the DOMs into
iceboot. Note that DOMs can be locked out by other users/processes, so the command may
not always work...

EOF
;

}
elsif ($command eq  "ready-iceboot") {

print <<EOF;

This command starts up TestDAQ domhubApp software.
It is assumed that the DOMs are in iceboot when you enter this command.

EOF
;

}
elsif ($command eq  "monitorDomhubApp") {

print <<EOF;

This command monitors the progress of TestDAQ domhubApp software on a hub.
This has proven necessary because hubs can hang occasionally after the
handoff from pDAQ to TestDAQ. The reason is not currently known, but this
command can help to pinpoint which DOMHub has the problem. 

The solution is to "stoptestdaq" on the string processor
Then, on the offending hub: 
"stoptestdaq"
"pCycle" 
"ready-iceboot"
And finally, on the string processor,
enter the appropriate "go" command again...

EOF
;

}

else {
print <<EOF;
                                                                                                                   
    For help on a particular topic type 'icehelp <topic>' or 'icehelp <topic> | less '
                                                                                                                   
    Topics include:
    
    Standard Hub Commands:
      pDown                   (powers Down groups of DOMs, can do by DOR/pair/Quad)
      pUp                     (powers Up groups of DOMs and puts them into Iceboot, can do by DOR/pair/Quad)
      pCycle                  (power Cycles groups of DOMs and puts them into Iceboot, can do by DOR/pair/Quad)
      status                  (shows status of all DOMs and software on the hub)
      checkGPS                (checks GPS on the hub)
      gotoiceboot             ("iceboot all" puts DOMs into iceboot, currently necessary to run this command twice)
      domhub-services         (should be running on all domhubs at all times, needs to be restarted after a reboot)
      stoptestdaq             (aborts Testing DAQ software and turns off DOMs)
      stoptestdaqnopowercycle (aborts Testing DAQ software but does not turn off DOMs) 
      dtsxinit                (power cycles DOMs, starts up dtsx) 
      dtsxinitnopowercycle    (softboots DOMs, starts up dtsx, no power cycle)
      stf                     (runs STF, the "Simple Test Framework", on a hub)  
      ready-iceboot           (starts up domhubApp software on the hub - DOMs should be in iceboot mode already)
      monitorDomhubApp        (monitors the state of TestDAQ's domhubApp software)

    Standard String Processor Commands:
      domhub             (for issuing commands on groups of domhubs simultaneously)
        eg: "domhub all checkGPS" or "domhub inice table" or "domhub icetop monitorDomhubApp" )
      distribute         (for moving a particular file onto groups of domhubs)


    General Instructions:                                                                                                                   
      TestDAQ              (how to run TestDAQ)
      TestDAQ-problems     (in the rare instance TestDAQ encounters a problem)
      icetop-prefill       (IceTop PreFill Commissioning Instructions)
      inice-commissioning  (InIce Commissioning Instructions)
      icetop-commissioning (IceTop Commissioning is finished)
      weekly-runs          (how to run the IceTop and InIce weekly runs - no InIce runs yet...)
      pDAQ-interaction     (how to get DOMs from and give back to pDAQ)

    Miscellaneous Commands:
      current            (shows DOR current for all DOMs)
      status-output      (shows sample output of status)
      misc               (assorted commands)


 REMEMBER TO KEEP THE DOMs ON ( ie after a power cycle "on all" and "iceboot all" )

    
EOF
;

}
exit(1);


