updated 8/11/09

current test page is 
<a href=http://icecube.wisc.edu/~testdaq/domhub-testing.html> 
http://icecube.wisc.edu/~testdaq/domhub-testing/2009/domhub-testing.html
</a>


When swapping in a hub for testing it is important (vital) to check each of
the DOR power connectors to ensure they have been wired correctly.

also, when removing a hub from active testing the procedure is:

Justin-start at step 1:
----------------------------------------------------------------
0) copy the data files we want to save from the hub...
ensure that 
   A) current data is retained
         from testdaq@spts64-testdaq01:
             domhub sps-ichub60 getcurrent   (to save all current info from sps-ichub60)
also copy the dor-card-offsets.txt file from the hub and rename it as
dor-card-offsets.txt.hub-name-and-##
Do not overwrite the file in the home directory!

   B) shortmoat passed
   C) testdaq data looks good
--------------------------------------------------------------------

1) issue command 'stoptestdaq' on the domhub as testdaq
you should see the DOMs power down

2) turn off DOM power switch at top back middle of hub

3) to shutdown the hub log into it as yourself and issue the command
"sudo /sbin/shutdown -h now"

4) turn off both DOMHub power switches at top back left of hub

5) unplug all cables and pull hub out of rack
HUBs should be returned to the spot they came from 


==============================================================

justin: dont do this part:
---------------------------------------------------
(if possible, log in to testdaq@pub and edit
~/public_html/domhub-testing/2008/domhub-testing.html as appropriate)
----------------------------------------------------

TO SET UP THE HUB FOR TESTING:

to log in to a hub on the build system (this is how we test the hubs when
they are hooked up to the mainboards on scube on SPTS)
log in as yourself to spts64-access from pub
log in as "testdaq" to the hub in question eg "ssh testdaq@spts64-ichub37"

note that if you are using the KVM on the test system then you must log in
to pub.icecube.wisc.edu before you log in to sps-access...



in $HOME:
 hubConfig.dat needs to be edited to 
    correspond to the hardware setup.



BE SURE TO LOG OUT AND LOG BACK IN AFTER ALL CHANGES HAVE BEEN MADE!

if you don't get a color-coded prompt at login, note the problem and for now
use 'source .bashrc'

=====================================================
0) check software

rpm -qa testdaq   (check version is 10.0)
   if not install the rpm and then issue the command "post-kickstart"
fvers.pl          (check version is 02-11-05 and that all DOR cards have
                   PCI=9, FW=104q)

now for the tests:

1) 'checkGPS'
 check all is good in GPS land

2) 'current-test-configboot'
  look for DOR card pairs which are registering abnormal current
  ALL 64 mainboards need to respond

3) 'current-test-iceboot'
  (same as above)
  make sure all DOMs are in order on the final list... 

4) 'pDown' to turn DOMs off

5) 'pUp' with all DOMs on
take a look at the currents -
run the script offsetDORs.pl

If all DOMs are in iceboot and currents look good, then, from $HOME:
quickstatus > quickstatus.iceboot.txt

Now is a good time to edit $HOME/hubConfig.dat

Do quickstatus > quickstatus.txt

5) check all communicating properly
with doms off:
'shortmoat-wrapper'  <-- takes around 20 minutes, check for SUCCESS in "latest_moat"
  in ~/Results/CommsTests/shortmoat/...

6) then, run TestDAQ

'pCycle'
'ready-iceboot'

in 2 or 3 seconds you will see: " Waiting for RMI method calls"
hit cntrl-c
then:
'go.scube.quick'   <-- should take 5 minutes, sends out email...
  
after it completes successfully:
'go.scube'    <--- runs forever



this will take data continuously until you 'stoptestdaq' or 'pausetestdaq' ...
(pausetestdaq allows the ongoing 30-minute run to finish normally)


==============================================================

It is important that the current information got saved.
cat this file:

~/Results/Current/00.dat




if any DOMs turn off and firmware fuses blow:

as yourself: 
pDown
sudo /etc/init.d/dhrc restart    <-- will reload dor-driver
pUp