2005/07/15 9:00 9:00;;2005/07/15 9:00;;+25;;LCChain
2005-7-18 10:30;;mtk ;; Mark reran with the full chain - not split across DOMHubs
2005-7-18 11:00;; All passed - 143 links.;;
2005/07/15 9:30 9:30;;2005/07/15 9:30;;+25;;STF A/B
2005-07-16 15:30;; kdh;; strange event - ran manually and got many fails. then ran using mark's stfA/stfB scripts and things look OK
2005-07-16 18:30;;4 FAILs
1. UP5P0652 - atwd_pulser_spe
2. UP5P0920 - flasher brightness - this is "Daikaku_Inumura", one of the Golden PMT Doms
3. TP4P0093 "Apple" - ADC
4. AP5P0439 "Pharmacology" - fadc_fe_pulser;;DO NOT USE STF results from 12:00 to 15:30 on this date - they are BAD!
2005/07/15 11:30 11:30;;2005/06/15 11:30;;+25;;DOMCal
2005-7-15 10:19;; Mark;;
2005-7-15 11:19;; TP5P0753 88776d7e42bf "Technology" is a high gain DOM which failed to calibrate. I set the nominal opertaing voltage to 1172 volts in the database.
see the domcal plots at
inital domcal plots
;;
2005/06/15 12:30 12:30;;2005/06/15 12:30;;+25;;Lux
2005-7-17 13:00;;kdh;;Runs are in 2005/07/17.lux.
The sync board shows 345 kHz - not 500 kHz but this is same as FAT7
2005-7-17 18:00;;;;
2005/07/15 14:30 14:30;;2005/06/15 14:30;;+25;;TestDAQ
2005-7-19 2:11;; kael and mark;; runs 7385 thru 7411 looked succesful, then ran out of disk space
i (mark) stopped TestDAQ, deleted all of the hit files for the recent runs, and started up TestDAQ again with run 7430
2005-7-19 2:11;; Genealogy fathub2.52B went dead during run 7446;;
2005/06/15 20:30 20:30;;2005/06/15 20:30;;+25;;Soaking / stability
2005-7-17 17:00;;kdh;;75000 sec run
2005-7-18 10:30;; ;;
2005/06/18 08:30 08:30;;2005/06/18 08:30;;+25;;Cooldown
2005-07-19 08:34;;kdh;;Did a manual defrost at 10:05 am - 2 liters H2O expelled from DFL. Defrost time of about 30 min should be sufficient
2005-07-19 16:40;; ;;
2005/06/18 16:00 16:00;;2005/06/18 16:00;;-20;;Defrost
2005-7-20 11:30;;kael;;did short defrost - no water expelled from DFL
2005-7-20 12:30;; ;;
9; 2005/06/18; 17:00 ; -20; cooldown
2005/06/19 08:00 08:00;;2005/06/19 08:00;;-45;;STF
2005-7-20 12:30;;kdh;; several fails
2005-7-20 14:00;; ;;
2005/06/19 10:00 10:00;;2005/06/19 10:00;;-45;;DOMCal
2005-7-20 14:40;;kael;;
2005-7-20 15:30;; all doms finished;;
2005/06/19 11:00 11:00;;2005/06/19 11:00;;-45;;TestDAQ
2005-7-20 16:54;;kael;; No dead DOMs.
2005-7-21 00:50;; RUNS 7465 - 7511 are GOOD.;;
2005/06/19 18:00 18:00;;2005/06/19 18:00;;-45;;LUX
2005-7-21 6:30;; kael;;
2005-7-21 8:45;; OK;;
2005/06/19 20:00 20:00;;2005/06/19 20:00;;-45;;Soaking / stability
2005-7-21 8:50;; kael;;
2005-7-22 12:30;; ;; Note that Demonology has bad HV setting so the rate is bad. Musicology has slow monitor readout.
2005/06/19 20:00 20:00;;2005/06/19 20:00;;-45;;Cooldown
2005-7-22 12:35;;kael;;simple turn down of switch to -53 C. Did short (30 min) defrost prior to turn-down.
2005-7-22 12:54;; ;;
2005/06/20 08:00 08:00;;2005/06/20 08:00;;-55;;STF
2005-7-26 11:30;;kael;;
2005-7-26 14:30;;Barat serious HV failure;;
2005/06/20 10:00 10:00;;2005/06/20 10:00;;-55;;DOMCal
2005-07-23 03:30;;kael;;150 DOMs in run
2005-7-23 4:45;;2 very high gain DOMs (TP5P0733 - Vexillilogy and TP5P0753 - Technology) failed to calibrate. SPE fits OK but rejected by DOMCal fitter.;;
2005/07/21 11:00 11:00;;2005/06/20 11:00;;-55;;TestDAQ
2005-7-25 9:21;;kael;;Dead DOM
- runs 7521-7534 and 7543 - 7575 OK.
MK: "Barat", a FAT05 Wind DOM, appeared to lose its HV/PMT during run 7538.
STF will likely disagnose this problem further.
2005-7-25 23:45;;;;Seismology and
Cryptology went DEAD during run 7517 to 7520 fathub2.03 A and B
run 7517 was PSL-FATDAQ1_run0007517_DarkNoise-MultiPE-FullReadout-ATWD1
2005/06/20 18:00 18:00;;2005/06/20 18:00;;-55;;LUX
2005-07-26 9:45;; kael;;
2005-7-26 11:00;; ;;
2005/06/20 20:00 20:00;;2005/06/20 20:00;;-55;;Soaking / stability
2005-07-23 5:10;;kael;;Remembered to insert new voltages into DB.
2005-7-25 9:10;; May need more soaking - fathub1 crashed on Sunday. Data files are in 2005/07/23 directory. Additional soak started on Monday 25 July at 11:47 PM. Data into 2005/07/25;;
2005/06/21 20:00 20:00;;2005/06/21 20:00;;-55;;Warmup
2005-7-26 15:00;;kael;; started multimon - going into 2005/07/26
2005-7-26 22:00;; ;;
2005/07/25 08:00 08:00;;2005/07/25 08:00;;-20;;STF
2005-7-27 12:30;;kael;;T DOMs run on 7/27.
U DOMs run on 7/28 between 21:00 and 23:00
2005-7-28 23:00;; ;;
2005/07/25 10:00 10:00;;2005/07/25 10:00;;-20;;DOMCal
2005-7-26 22:00;;kael;;
2005-7-27 7:23;;UP5P0502 "Barat" failed to produce an XML. All others OK;;
2005/07/25 11:00 11:00;;2005/07/25 11:00;;-20;;TestDAQ
2005-7-27 21:21;;kael and mark and hagar 
;;run 7596 "Apple" goes dead
Hagar zips up successful steering files. Runs restarted 7600 by david - no fpgall -
runs restarted again by mark starting with 7603. we got a run which did not terminate in run 7606
mark restarted runs again with 7607
2005-7-27 21:21;;kael Note that GPS was bad for this period - all runs which rely on RAP sync of DOMs are BAD.
For LC/Timeres runs with good GPS look to 7638 - 7660.;;
2005/07/25 18:00 18:00;;2005/07/25 18:00;;-20;;LUX
2005-07-27 07:20;; kael;;
2005-7-27 7:24;; ;;
2005/07/25 20:00 20:00;;2005/07/25 20:00;;-20;;Soaking / stability
2005-7-28 04:16;;kael;;Paused at 10:30 to reboot hubs because of GPS problem
2005-7-28 11:16;; ;;
27; 2005/06/25; 08:00; -20; Defrost
2005/07/26;;2005/07/26;;-14;;Warmup1
2005-7-29 08:00;;kael;;put data in 2005/07/29.warmup
2005-7-29 22:00;; ;;
2005/07/26;;2005/07/26;;+15;;Warmup2
2005-7-29 22:0;;kael / hagar;;stopped warm-up at +25 C (DOM MB temp)
2005-7-29 22:0;; ;;
2005/07/27 08:00 08:00;;2005/07/27 08:00;;+25;;LCChain
2005-7-29 22:2;;kael;;skipped
2005-7-29 22:2;; ;;
2005/07/27 08:30 08:30;;2005/07/27 08:30;;+25;;STF
2005-7-30 14:45;;kael;;Zymology FAILS to complete tests - crashed in atwd_baseline and flasher_width.
2005-7-30 16:57;; ;;
2005/07/27 10:30 10:30;;2005/07/27 10:30;;+25;;domcal
2005-7-29 22:00;;kael;;147 DOMs calibrate
- UP5P0502 "Barat" fails - HV bad - hangs.
- UP5P0520 "Antarctic_Wind" produces XML but fits fail
- TP5P0593 "Photophobia" Ibid.
2005-7-29 22:55;; ;;
2005/07/27 11:30 11:30;;2005/07/27 11:30;;+25;;TestDAQ
2005-7-30 19:3;; mark and kael;; stopped run 7670 since TP5P0759 Zymology was failing.
Kael had previously started TestDAQ twice and Zymology was always failing.
Mark removed Zymology from the error checker, and restarted TestDAQ
beginning with run 7671.
stopped TestDAQ in run 7673
removed Zymology from ready.properties and restarting TestDAQ
forgot to do fpgall
restarting with run 7676
kael stopped the runs and completed the pedestal pattern and time res the next day - runs 7716 thru 7721 - he installed the auto reboot switch before this...
2005-7-30 19:3;; ;;
34; 2005/06/26; 18:30; +25; Lux
2005/07/27 20:30 20:30;;2005/07/27 20:30;;+25;;Soaking / stability
2005-7-29 23:18;;kael;;data into 2005/07/29.
2005-7-29 23:18;; fathub1 crashed morning of July 31
around 05:00
Jul 31 05:21:30 fathub1 ntpd[1624]: synchronisation lost
;;
2005/07/29 08:30 08:30;;2005/07/29 08:30;;+25;;Cooldown
2005-07-31 22:00;;kael;;Begin cooldown at end of LUX.
2005-08-01 08:00;; ;;
2005/07/29;;2005/07/29;;-20;;Defrost
2005-8-1;;Claire Pettersen;;
2005-8-2 15:4;; ;;
2005/07/29;;2005/07/29;;-20;;Cooldown
2005-8-1 14:00;;Claire ;;
2005-8-2 15:5;; ;;
2005/08/01 08:00 08:00;;2005/08/01 08:00;;-45;;STF
2005-8-2 15:31;;kael;;stfA run at 15:31
TODO stfB.
2005-8-2 16:33;; ;;
2005/08/01 10:00 10:00;;2005/08/01 10:00;;-45;;DOMCal
2005-8-2 08:00;;kael;;
2005-8-2 15:27;;AP5P0438 fails /w/ low PMT gain (3.8E7)UP5P0502 and TP5P0503 and TP5P0733 fail completely.
;;
2005/08/01 11:00 11:00;;2005/08/01 11:00;;-45;;TestDAQ
2005-8-2 18:38;;kael, 
mark;;kael starts testdaq with run 7722
"Craniology" , fathub1.73B, goes dead near start of PSL-FATDAQ1_run0007731_LC-MultiPE-ATWD1
Here is the error in the domhubapp log:
5293674 [TimeCalibrationServiceThread] DEBUG (RealDOMDriver.java:724) - For DOM : 643f1bf2fdb9 - updated and validated syncgps rec for
5293676 [DataMsgServiceThread] ERROR (DataMsgService.java:170) - 565ea2e2ee72: Data Request Timeout:
millisecs since oldest data request: 2001
requests in flight: 3
5293679 [DataMsgServiceThread] DEBUG (DataMsgService.java:174) - Driver Comm Stats for 565ea2e2ee72:
/dev/dhc7w3dB
RX: 217652337B, MSGS=460176 NINQ=0 PKTS=1135152 ACKS=362997
BADPKT=0 BADHDR=0 BADSEQ=140 NCTRL=0 NCI=338 NIC=6
TX: 3485409B, MSGS=362999 NOUTQ=1 RESENT=14 PKTS=1134823 ACKS=771811
NACKQ=0 NRETXB=2 RETXB_BYTES=16 NRETXQ=0 NCTRL=0 NCI=407 NIC=340
NCONNECTS=345 NHDWRTIMEOUTS=0 OPEN=true CONNECTED=true
RXFIFO=notempty TXFIFO=almost empty,empty DOM_RXFIFO=notfull
5293681 [DataMsgServiceThread] DEBUG (DataMsgService.java:176) - Card 7 FPGA:
FPGA registers:
CTRL 0x4000000f
GSTAT 0x00000fc2
DSTAT 0x000070ff
TTSIC 0x0000fffd
RTSIC 0x0740803f
INTEN 0x00000200
DOMS 0x000000ff
MRAR 0x00000000
MRTC 0x00000000
MWAR 0x00000000
MWTC 0x00000000
CURL 0xe19e8c0a
DCUR 0x0c7130ab
FLASH 0xff000000
DOMC 0x00000000
DCREV 0x00000018
FREV 0x07010171
and then a little later:
5295303 [DataMsgServiceThread] ERROR (DataMsgService.java:231) - 565ea2e2ee72 Monitor Request Timeout:
millisecs since last moni request: 2002
requests in flight: 1
5295307 [DataMsgServiceThread] DEBUG (DataMsgService.java:235) - Driver Comm Stats for 565ea2e2ee72:
/dev/dhc7w3dB
RX: 217652337B, MSGS=460176 NINQ=0 PKTS=1135152 ACKS=362997
BADPKT=0 BADHDR=0 BADSEQ=140 NCTRL=0 NCI=338 NIC=6
TX: 3485409B, MSGS=362999 NOUTQ=1 RESENT=28 PKTS=1134837 ACKS=771811
NACKQ=0 NRETXB=2 RETXB_BYTES=16 NRETXQ=0 NCTRL=0 NCI=407 NIC=340
NCONNECTS=345 NHDWRTIMEOUTS=0 OPEN=true CONNECTED=true
RXFIFO=notempty TXFIFO=almost empty,empty DOM_RXFIFO=notfull
5295309 [DataMsgServiceThread] DEBUG (DataMsgService.java:237) - Card 7 FPGA:
FPGA registers:
CTRL 0x4000200f
GSTAT 0x00000dc2
DSTAT 0x000070ff
TTSIC 0x0000ffff
RTSIC 0xff00807f
INTEN 0x00010200
DOMS 0x000000ff
MRAR 0x00000000
MRTC 0x00000000
MWAR 0x00000000
MWTC 0x00000000
CURL 0xe19e8c0a
DCUR 0x0c7130ab
FLASH 0xff000000
DOMC 0x00000000
DCREV 0x00000018
FREV 0x07010171
John Jacobsen investigating...
kael restarts TestDAQ a couple of times - good runs start with run 7735 (thru 7759)
run PSL-FATDAQ1_run0007760_DarkNoise-980Disc-5E7Gain-FullReadout-ATWD0 is a special dark noise run where the HVs were set for 5E7 gain and the discriminator was set to 980. There was a long multimon run with the same settings...
run 7761 is a special run where the disc is 1023 and the operating voltage is 0.7 of the way from hv3 to hv2. This run appeared to have a GPS problem, and it was repeated with run 7762.
PSL-FATDAQ1_run0007762_DarkNoise-1023disc-PMT-voltage-under-1E6-FullReadout-ATWD0
PSL-FATDAQ1_run0007763_DarkNoise-100VoltPMT-FullReadout-ATWD0 is a special 100 volt run to look for "blips".
2005-8-2 18:38;; ;;
2005/08/01 18:00 18:00;;2005/08/01 18:00;;-45;;LUX
2005-8-3 12:00;;kael 
;;1st run fathub2 and fathub5 crashed! move this data to 08/03.lux-A. Good run in 08/03.lux
2005-8-3 22:00;; ;;
2005/08/01 20:00 20:00;;2005/08/01 20:00;;-45;;Soaking / stability
2005-8-3 22:02;;kael;;fathub1 crashed again on 8/8 at 05:00 - rebooted - this data is in 08/04 directory
2005-8-10 10:00;; ;;
2005/08/04 08:00 08:00;;2005/08/04 08:00;;-45;;Soaking / stability at 2000 kV
2005-8-12 22:44;; mark;;during testing, Hematology came up as non-communicating
FATHUB1 crashed at around 2:30pm on saturday aug 13
2005-8-12 22:44;; ;;
43; 2005/06/02; 08:00; -45; Soaking / stability
2005/08/09 10:30 10:30;;2005/08/09 10:30;;-45;;Warmup
2005-8-14 12:15;; 
Claire;;
2005-8-14 12:15;; ;;
45; 2005/07/09; 08:30; -45; UnLoad