2005/08/17 17:00 17:00;;2005/08/17 17:00;;+25;;Initialization
2005-8-17 23:7;;Mark;;
2005-8-17 23:7;;No output from 84de... "Glucoseish". possible bad PMT (the DOM "Glucose" had the same problem and its mainboard ended up being replaced)
flasher not uploaded on 2.23A and 2.43A
08/18 afternoon
"Glucoseish" and "Thalassaphobia" are removed from the DFL and replaced with "Discophilia" and "Europhilia".
Also, "Potophobia" and "Leprechaun", two spikey DOMs, are added to the special calibration stations, bringing the total number of mainboards to 150;;
2005/08/19 8:00 8:00;;2005/08/19 8:00;;+25;;LCChain
2005-8-17 23:2;;Mark, hagar;;
2005-8-18 16:2;;On Aug-18, lcchain succesful ! (one failure in the 4 extra wire testing doms hub5:21A to 22B)
__________________
DOM thalassophilia, fathub2.30A thalassophilia - failed on both sides.
* its neighbour were only tested on one side: above it is Francophilia, and below it Germanophilia.
After changing cabling, the same DOM failed in the same place. The other passed. Thalassophilia will be removed and replaced with "Europhilia". And also Glucoseish will be removed.;;
2005/08/19 8:30 8:30;;2005/08/19 8:30;;+25;;STF A/B
2005-8-18 20:46;;Mark;;
fathub1 stfA
fathub1 stfB
fathub2 stfA
fathub2 stfB
fathub5 stfA
fathub5 stfB
2005-8-18 20:46;;had to enter TP5P0827 "Europhilia" and UP5P0930 "Discophilia" when I ran STF. Don't know why I never got asked about AP5P0450 "Palaeophilia";;
2005/08/19 10:30 10:30;;2005/08/19 10:30;;+25;;DOMCal
2005-8-17 23:8;;Mark, hagar,Erik;;
2005-8-18 20:06;; look here
* rheophilia - 1026.... xml was not odwnloaded correctly. downloaded manually and seems fine.
320..waterspout - "too few data point to do the regression"
redone 8/18
look here
Pali and Discophilia seem to have not produced any histograms.
Mark's bad - "Discophilia", one of the two new DOMs, did not have the right version of domcal on it - i will rerun domcal
histograms updated. see above. Pali still sucks (as we knew it would), but Disco looks pretty now.;;
2005/08/19 11:30 11:30;;2005/08/19 11:30;;+25;;TestDAQ
2005-8-17 23:36;;Mark, Hagar;;Start at run 7804 - no need to save 7804, extra run. Start from 7805.
Glucoseish,Thalassophilia where commented from the automatic mailing warnings.
run 7820, fathub1 crashes. runs restarted with run
7854.
run 7854 failed to start - domhub-app appears to have gotten stuck on both fathubs 2 and 5. Reason unknown - problem occurs during transition to iceboot.
on fathub2:
112175 [RMI TCP Connection(2)-172.16.34.160] DEBUG (DOMHubComImpl.java:456) - Sending 'r' to 62A
on fathub5:
130385 [RMI TCP Connection(2)-172.16.34.160] DEBUG (DOMHubComImpl.java:456) - Sending 'r' to 00B
run 7884 - Islandophilia dead DOM
OFFICIAL START OF UW FAT 09 beings with run 7887
2005-8-18 20:54;;good runs:
7809-7819,
7855- 7882;;
2005/08/19 18:30 18:30;;2005/08/19 18:30;;+25;;Lux
2005-8-19 12:20;;Erik;;
2005-8-19 14:26;;LUX successful. Everthing looks good.;;
2005/08/19 20:30 20:30;;2005/08/19 20:30;;+25;;Soaking / stability
2005-8-19 14:28;;Erik, hagar;;fathub1 crashed twice:
- 19/8 -
- 7 pm 20/8 - around 6 am
2005-8-21 16:00;;;;
2005/08/22 08:30 08:30;;2005/08/22 08:30;;+25;;Cooldown
2005-8-21 12:00;;Claire 
;;
2005-8-22 14:58;;;;
2005/08/22 16:00 16:00;;2005/08/22 16:00;;-20;;Defrost
2005-8-21 23:00;;Claire;; The defrost took a very long time because there was a lot of ice on the coils. After 2 normal defrosts and 2 short defrosts, there was no water coming out of the DFL
2005-8-22 02:30;; ;;
2005/08/22 17:00 17:00;;2005/08/22 17:00;;-20;;cooldown
2005-8-22 03:00;;Claire;;Claustrophilia fathub2.03B (port 5007) dtsx died at 03:20am this morning,
and, as a result, this DOM stopped being monitored
(we need to always check for CHUNK in nohup.out, and i intend to change
my script to solve the permanently dead DTSX problem (i have been
meaing to have it restart dtsx every time).
Apodysophilia had a solitary comm failure at around -32.7 degrees
(this DOM is on fathub1)
Claustrophilia fathub2.03B failed again (CHUNKSIZE) when the reboot script was restarted at 2pm on fathub2. Temperature was around -33 degrees.
fathub1.12A (port 5012) and fathub1.12B (port 5013) had CHUNKSIZE failures at 1:09.
2005-8-22 15:45;;;;
2005/08/23 08:00 08:00;;2005/08/23 08:00;;-45;;STF
2005-8-22 20:00;;Erik;;
fathub1 stfA
fathub1 stfB
fathub2 stfA
fathub2 stfB
fathub5 stfA
fathub5 stfB
2005-8-22 23:30;;ran once, screwed something up.
ran again. see links for failures.;;
2005/08/23 10:00 10:00;;2005/08/23 10:00;;-45;;DOMCal
2005-8-22 16:00;;Erik;;
2005-8-22 17:00;; look here
Walrusish - 0318.... "couldn't find an acceptable LED brightness! Aborting calibration."
Pali - 49ef... "All-zero HV ID! No HV base attached; aborting gain calibration."
"Claustrophilia" didn't show up. tried to reload mainboard software. failed twice. DOM looks to be dead.;;
2005/08/23 11:00 11:00;;2005/08/23 11:00;;-45;;TestDAQ
2005-8-23 8:4;;Hagar+Erik;;start at run 7942
Somethin funny with the dom after hipogrif on aad2afa683f5 on 70B hub1 .
then with
.
Continuiing with multimon now. go back to testdaq later.
Removed clausiphilia 03B, fathub2 from ready.dh
start at run 7954 -7976 OK
dead dom in run 7978
Malophilia.
starting in 7980
/data/psl full
cleared some, continue at 7989
on run 8006- dead dom-apodysophilia
5 last runs are missing: TimeRes and LC runs left - will continue tomorrow - running multimon for the night
2005-8-24 10:0;;good:
7954-7976
7980-7982
7989-8004
8008-8012
;;
2005/08/23 18:00 18:00;;2005/08/23 18:00;;-45;;LUX
2005-8-24 9:45;;Hagar+Erik;;
2005-8-24 9:45;;24.lux
Not on Claustrophilia;;
2005/08/23 20:00 20:00;;2005/08/23 20:00;;-45;;Soaking / stability
2005-8-22 23:37;;Erik;;soaking overnight. then stopping to run testdaq. then more soaking.
2005-8-22 23:37;;RanDOM as higher rates in the last 10 hours. about 2khz.
Does not effect other doms.
at;;funny looking:
Demonology 810 - rates increase+high rates at end
sosuke_inu 913 - rates increase+high rates at end
Leprechaun, Potophobia - high rates
walrusish-313-0318..dead
xenophilia-739-3e1e.. start to increase in the end.
Pali-535 - increase at the end
Butadienish - spiky
Suestado - increase at teh end
2005/08/25 08:00 08:00;;2005/08/25 08:00;;-45;;Cooldown
2005-8-24 14:38;;Claire;;
2005-8-24 21:00;; ;;
2005/08/26 08:00 08:00;;2005/08/26 08:00;;-55;;STF A/B
2005-8-25 11:50;;Erik+Kotoyo;;
fathub1 stfA
fathub1 stfB
fathub2 stfA
fathub2 stfB
fathub5 stfA
fathub5 stfB
2005-8-25 12:42;;See links for failures.
forgot to save screen shots for stfB. see stf_fail... for failures.;;
2005/08/26 10:00 10:00;;2005/08/26 10:00;;-55;;DOMCal
2005-8-25 10:35;;Kotoyo+Hagar;;
2005-8-25 11:47;; look here
Pali - 49ef... "All-zero HV ID! No HV base attached; aborting gain calibration."
Walrusish - 0318.... "couldn't find an acceptable LED brightness! Aborting calibration."
totally dead
Demonology 4755.... "couldn't find an acceptable LED brightness! Aborting calibration."
seems to have calibrated anyway. looks ok. see histograms.;;
2005/08/26 11:00 11:00;;2005/08/26 11:00;;-55;;TestDAQ
2005-8-25 1:20;;Erik+Kotoyo;;start with run 8013
"Sinophilia" 2595... dead in run 8037 and 8038. Throw out 8036 too.
8046 ran into storage problems. Hagar cleared space. starting again at 8051
2005-8-25 23:50;;8013-8035 good
8039-8044 good
8051-8069 good;;
2005/08/26 18:00 18:00;;2005/08/26 18:00;;-55;;LUX
2005-8-25 23:50;;Erik+Hagar;;
2005-8-26 3:00;;fathub1 crashed at 12:50. Hagar rebooted.
rest went well.;;
2005/08/26 20:00 20:00;;2005/08/26 20:00;;-55;;Soaking / stability
2005-8-26 3:3;;hagar+Erik;;fathub1 crashed at 25-Aug around 7:00 am. rebooted at 8:00
restarted 8/26 3 AM.
crashed around 5. restarted 8:39.
2005-8-27 12:00;;;;Oops, I did not run domcal before. do not use - will be deleted
marked above run as bad and started new.
2005/08/29 09:00 09:00;;2005/08/29 09:00;;-55;;Warmup
2005-8-28 12:00;;Claire;;
2005-8-28 15:30;; ;;
2005/08/30 08:00 08:00;;2005/08/30 08:00;;-20;;STF
2005-8-29 10:25;;Kotoyo+Sean;;fathub1 stfA
fathub1 stfB
fathub2 stfA
fathub2 stfB
fathub5 stfA
fathub5 stfB
2005-8-29 13:21;;See links for failurs.
"Claustrophilia" didn't show up at stfB. Removed it from the script (fathub2.03B).
;;
2005/08/30 10:00 10:00;;2005/08/30 10:00;;-20;;DOMCal
2005-8-29 9:25;;Kotoyo+Sean;;
2005-8-29 10:20;;look here
Pali - 49ef... "All-zero HV ID! No HV base attached; aborting gain calibration."
Demonology 4755.... "couldn't find an acceptable LED brightness! Aborting calibration."
seems to have calibrated anyway. looks ok.
Nyct-d8c7 only 3 accepted datapoints and linefit failed;;
2005/08/30 11:00 11:00;;2005/08/30 11:00;;-20;;TestDAQ
2005-8-29 13:31;;Kotoyo+Sean;;start from run 8070.
8073 freezed. restart from 8074.
8096
"ERROR: exception caught in RapCalImplBS constructor (id=Gale b2fb40bee8ac fathub1.10B) (tcalfit 1): java.lang.ArrayIndexOutOfBoundsException: -1".
DAQ didn't stop automatically. Manual restart.
2005-8-29 23:30;;8074-8094 good
8098-8124 good;;
2005/08/30 18:00 18:00;;2005/08/30 18:00;;-20;;LUX
2005-8-30 0:10;;Kotoyo+Sean;;29.lux :
light sources are not turning on. restart.
Use 30.lux.
2005-8-30 2:32;;potophobia,Demonology, Sosuke_inukawa,Leprechaun:
Big errorbars
;;
2005/08/30 20:00 20:00;;2005/08/30 20:00;;-20;;Soaking / stability
2005-8-30 2:44;;Kotoyo+Sean;;fathub1 died at 17:16.
Restarted 17:45.
;;;;
2005/09/01 08:00 08:00;;2005/09/01 08:00;;-20;;Defrost
2005-8-31 08:45;;Claire;;
2005-8-31 11:45;; ;;
2005/09/01;;2005/09/01;;-14;;Warmup1
2005-8-31 11:45;;Claire;;
2005-8-31 16:00;; ;;
2005/09/01;;2005/09/01;;+15;;Warmup2
2005-8-31 16:00;;Claire;;
2005-8-31 18:00;; ;;
2005/09/02 08:00 08:00;;2005/09/02 08:00;;+25;;LCChain
2005-9-1 16:30;;hagar;;
2005-9-1 17:16;;Success !;;
2005/09/02 08:30 08:30;;2005/09/02 08:30;;+25;;STF
2005-9-1 13:58;;Kotoyo+Sean;;+25 fathub1 STFA
+25 fathub1 STFB
+25 fathub2 STFA
+25 fathub2 STFB
+25 fathub5 STFA
+25 fathub5 STFB
2005-9-1 13:58;;see plots for results. fathub1 STFA had an initial error that said "error communicating with OM." Reran STFA, worked okay. STF was finished late because STFB on fathub1 froze when almost finished.;;
2005/09/02 10:30 10:30;;2005/09/02 10:30;;+25;;domcal
2005-9-1 15:10;;Kotoyo+Sean;;
2005-9-1 16:54;; look results
Pali - 49ef... "All-zero HV ID! No HV base attached; aborting gain calibration."
Dwarf - b288... and Gale - b2fb... failed to calibrate. Retry.;;
2005/09/02 11:30 11:30;;2005/09/02 11:30;;+25;;TestDAQ
2005-9-1 17:23;;Kotoyo+Sean;;start from run 8125
at Run 8142, an error message mailed without stopping daq. The output of checkQuality.sh test was brank, but the dataqualitylog lacks DOM names. For example, first line of .dataqualitylog is "Discovered new DOM in mon stream: 8cc775dded6b".
This is the sign of disk overflow.
Clean up disk and continue remaining runs tomorrow.
Restarted 9:45AM 9/2 from Run 8152, but Sues-b53a died. Restart again 10:30 AM from Run 8157
Run 8164 Apple-91c4 died. Restart from Run 8167.
2005-9-2 15:52;;Run 8125 - 8140 good
Run 8157 - 8162 good
Run 8167- 8192 good;;
2005/09/02 18:30 18:30;;2005/09/02 18:30;;+25;;Lux
2005-9-1 22:0;;hagar;;Starting lux, until enough free space will clear
2005-9-2 0:00;;last 4 doms (FATHUB5.22A, 5.22B, 5.23A, 5.23B) has big errorbars;;
2005/09/02 20:30 20:30;;2005/09/02 20:30;;+25;;Soaking / stability
2005-9-1 0:05;;Hager, Kotoyo, Sean;;Stopped at 9:30 AM to run TestDAQ
Restarted 16:07 9/2
23:51, fathub1 died. Restarted 0:05 9/3.
;;DOM Autassasinophilia extremely spikey.
DOM Hoplophilia has a large abrupt baseline shift.;;
2005/09/05 08:30 08:30;;2005/09/05 08:30;;+25;;Cooldown
2005-9-3 13:00;;Claire;;
2005-9-3 23:00;; ;;
2005/09/05;;2005/09/05;;-20;;Defrost
2005-9-4 12:30;;Claire;;
2005-9-4 13:30;; ;;
2005/09/05;;2005/09/05;;-20;;Cooldown
2005-9-4 13:30;;Claire;;
2005-9-4 23:30;; ;;
39; 2005/09/06; 08:00; -45; STF
;;Sean;;
;;;;
2005/09/06 10:00 10:00;;2005/09/06 10:00;;-45;;DOMCal
2005-9-5 2:50pm;;Sean;;
2005-9-5 16:56;; look here
no results for Walrusish, Pali. Results for other DOMs look fine.;;
41; 2005/09/06; 11:00; -45; TestDAQ
;;Sean;;
;;;;
42; 2005/09/06; 18:00; -45; LUX
;;Sean;;
;;;;
43; 2005/09/06; 20:00; -45; Soaking / stability
;;Sean;;
;;;;
46; 2005/09/13; 10:30; -45; Warmup
;;Claire
;;
2005/09/14 08:30 08:30;;2005/09/14 08:30;;-45;;UnLoad
;;Not Me;;
;;;;
