1;;2005/04/15 15:00 ;;2005/04/15 15:00;;+25;;LCChain
2005-4-14 ;; mark ;; no problems
2005-4-14 ;; 4 doms need to be repeated. Also, Pumpkin and Walrus are the DOMs at the far ends and LC was only tested in one direction ;; Santa_Ana, Cape_Doctor, Ostria, Etesian (ccd...e3e, b35..233, 45a..f62, 818..d0e)
2;;2005/04/15 15:20 ;;2005/04/15 15:20 ;;+25;;STF
2005-4-15 16:30;;hagar;; ran smoothly
2005-4-15 17:00;; OK (Many doms failed flashe_clock_flasher_clock test
all other tests successful. ;; updating software to version 12
Finally all passed OK
3;;2005/04/15 16:20 ;; 2005/04/15 16:20 ;;+25;;DOMCal
2005-4-17 18:11:0;; mark;; ok
2005-4-14 17:00;; ok ;;
4;;2005/04/15 17:20 ;;2005/04/15 17:20 ;;+25;;Lux
2005-4-15 ;;kael;;
2005-4-15 ;;fine .;;
5;;2005/04/15 19:20 ;;2005/04/15 19:20 ;;+25;;TestDAQ
2005-4-15;;all;; machine rebooted at least twice.
some runs are missing some dom's data stream.
2005-4-16 11:00;; ;;
6;;2005/04/16 08:20;;2005/04/16 08:20 ;;+25;;Soaking/stability
2005-4-15 1:00;;;;
2005-4-15 8:20;; ok;;
7;;2005/04/16 09:20;;2005/04/16 09:20;;+25;;Cooldown
later;;
later;;
8;;2005/04/16 17:20;;2005/04/16 17:20;;-20;;Defrost
later;;
later;;
9;;2005/04/16 18:05;;2005/04/16 18:05;;-20;;Cooldown
later;;
later;;
10;;2005/04/17 08:05;;2005/04/17;;-45;;STF
2005-4-17 18:21:47;;hagar;;no problem
2005-4-17 18:21:47;;
Output here
;;many transit time STF failured
2 Flasher Clock failures (old doms ika & cucumber)
1 pmt hv-ramp (bali_wind )
1 atwd pulser spe (waterspout)
11 pedestal noise chip B, !!!!!!!!channel 0
2 pedestal noise chip A, channel 0 !!
many pmt led max light faliures - a Mark's est
11;;2005/04/17 09:05;;2005/04/17;;-45;;DOMCal
2005-4-17 11:50;;hagar;;no problem
2005-4-17 12:30;;hv.gt.2000 for gain 5e7 for AP5P0400 Wind_Shear.
xml files in:
/mnt/data/testdaq/domcal/fat5-45;;
12;;2005/04/17 10:05 ;;2005/04/17 10:05 ;;-45;;Lux
2005-4-17 12:40;;hagar;; no problem
2005-4-17 14:50;;mon files in: /mnt/data/testdaq/monitoring/2005/04/17/
plots in:
;;
13;;2005/04/17 12:05 ;;2005/04/17 12:05 ;;-45;;TestDAQ
2005-4-17 14:31;;hagar;;runs 5077-5131 with missing tcal streams.
fathub1 was rebooted just before the start of run 5046.
2005-4-18 12:55;;use runs 5046-5077, 5131-5161;;Missing data streams for most of the runs.
14;;2005/04/17 18:05;;2005/04/17;;-45;;Soaking/stability
2005-4-18 12:53;;hagar;;switched to fathub2, on April 20, noon
Sometimes some doms experiencing lower readout rate. probaly due to DMA problem.
between 22-24/4 testing was paused.
2005-4-26 15:1;;output in:
/mnt/data/testdaq/monitoring/2005/04/18
/mnt/data/testdaq/monitoring/2005/04/20
/mnt/data/testdaq/monitoring/2005/04/25
/mnt/data/testdaq/monitoring/2005/04/26
/mnt/data/testdaq/monitoring/2005/04/28
/mnt/data/testdaq/monitoring/2005/04/29
1 DOM pressure leak b641d54f973a-TP5P0509-Sirocco.
2 DOMS high rate>3000: 087c5871c6a5-UP5P0542-Haru_I
3e75e434c9e0-TP5P0541-Warm_B;;on run, times: 1114435443.5 - 1114446254.9
laset was left on from hub testing. rate is higher !
15;;2005/04/26 08:05;;2005/04/28;;-45;;Moat
2005/04/28 11:20;;hagar;;
2005/04/28 19:20;;OK.
output at:
fathub2::/home/testdaq/MOAT-FAT5-45/;;5 doms failed.
This is due to DMA bug in REV1. when running on less doms all passed.
16;;2005/04/26 08:05;;2005/05/01;;-45;;Cooldown
2005-5-1 10:00;;kael;;Rebooted fathub2 because of comm problems with dor channels 03B and 10B - looks OK now. Run laseron.py during cooldown period - monitor is running.
2005-5-1 19:00;;Cooldown monitoring run put in $HOME/monitoring/2005/05/01/cooldown;;
17;;2005/04/26 16:05;;2005/05/01;;-55;;STF
2005-5-2 12:49;;hagar;;db was moved to /mnt/scratch
flasher-width did not run. JohnK ran it and sent results.
I was not able to run in doing the same proceudre. probably related to :
May 2 15:43:15 fathub2 kernel: dh: HARDWARE TIMEOUT on minor 57, did comms. reset
found in /var/log/messages
2005-5-2 15:51;;output here
26 ATWD failures
5 flasher failures
1 pulser
1 disc.
total 30 failures
;;john:
cucumber - all flasher tests
ika - flasher clock
wind shear - flasher brightness
gale - flasher width
bali-wind - flasher bright
walrus - flasher width
18;;2005/04/26 17:05;;2005/05/01;;-55;;DOMCal
2005-5-2 9:49;;hagar;;Disk on sastrugi was full.
2005-5-2 12:52;;output in:
domdaq1::~testdaq/domcal/fat5-55/
2 doms with HV>2000 for
AP5P0402 ( 39368d8f1d7c Pampero)
AP5P0400 ( 80b0adb4915b Wind_Shear);;
19;;2005/04/26 18:05;;2005/05/01;;-55;;Lux
2005-5-2 2:42;;hagar;;output in: /mnt/data/testdaq/monitoring/2005/05/02
2005-5-2 9:45;;;;
20;;2005/04/27 08:05;;2005/05/02;;-55;;TestDAQ
2005-5-2 16:46;;hagar;;none :-)
2005-5-2 22:12;;runs 5708-5758;;
21;;2005/04/27 14:05;;2005/05/02;;-55;;Soaking/stability
2005-5-3 0:38;;hagar;;
2005-5-3 9:50;;monitoring/2005/05/03;;
22;;2005/04/27 15:05;;2005/05/02;;-55;;Warmup
2005-5-3 13:7;;hagar;;started cold_boot
2005-5-3 13:7;;;;
23;;2005/04/27 18:35;;2005/05/02;;-20;;STF
2005-5-3 18:35;;hagar;;ran smoothly
2005-5-3 20:00;;
Failure list here
Full list here
6 awd pedestals,
2 flasher clock
1 hv ramp;;
24;;2005/04/27 19:35;;2005/05/02;;-20;;DOMCal
2005-5-3 20:00;;hagar;;
2005-5-3 20:49;;db +
/mnt/data/testdaq/domcal/fat5-20/;;
25;;2005/04/28 08:35;;2005/05/03;;-20;;Lux
2005-5-4 0:48;;hagar;;0f945673f4ef-UP5P0540-Coromell is reading out 0.
rebooted hub.
Pressure is a bit low (29). c
ard = 7 pair = 1 domAB = 1 port is 5059
2005-5-04 8:31;;in
monitoring/2005/05/04/;;
26;;2005/04/28 10:35;;2005/05/03;;-20;;Test DAQ
2005-5-3 20:49;;hagar;;run 5781... :
DOM Pampero 39368d8f1d7c
fathub2.70A failed configuration change
stopped run, restatred domhub-app.
Restarted....run 5785:
6: Coromell 0f945673f4ef fathub2.71B -- Requested DOM not present in hit stream
run 5787 report no errors- actually, there was only one hit record for Coromell in this run - and then same message, same DOM
run 5802-5809 missing hit streams: Coromell AND OSTRIA. restarted domhubapp
2005-5-4 16:15;; ;;Mark upgraded to
domhub-app V02-05-16
27;;2005/04/28 16:35;;2005/05/03;;-20;;Soaking/stability
2005-5-4 16:15;;hagar;;
2005-5-4 17:15;; ;;
28;;2005/04/28 17:35;;2005/05/03;;-20;;Defrost
2005-5-4 17:15;;claire;;Started coldReboot
2005-5-4 21:00;; ;;
29;;2005/04/28 18:20;;2005/05/03 18:20 ;; -14;;Warmup1
later;;
later;;
30;;2005/04/29 08:20;;2005/05/04 08:20;; +15;;Warmup2
later;;
later;;
31;;2005/04/29 10:20;;2005/05/04;;+25;;LCChain
2005-5-5 10:14;;hagar;;
2005-5-5 10:25;;here
;;as before: failure between:
santa-ana =cape_doctor
ostria=etesian
32;;2005/04/29 10:40;;2005/05/04;;+25;;STF
2005-5-5 10:36;;hagar;;
2005-5-5 11:41;;Here
And here
+ Bali wind faile hv ramp test.
+ MANY doms failed led max light. this is on room temp, and shouldn't happen. need tob check with Azriel.;;
33;;2005/04/29 11:40;;2005/05/04;;+25;;DOMCal
2005-5-5 11:48;;hagar;;0f945673f4ef (Coromell) does not finish.
killed.
2005-5-5 13:00;;;;
34;;2005/04/29 12:40;;2005/05/04;;+25;;Lux
2005-5-5 21:14;;hagar;;
2005-5-6 9:00;;;;
35;;2005/04/29 14:40;;2005/05/04;;+25;;TestDAQ
2005-5-5 13:10;;hagar;;
2005-5-5 21:14;;;;
36;;2005/04/30 08:40;;2005/05/05;;+25;;Soaking/stability
2005-5-5 23:15;;;;see previous lux entry.
2005-5-6 9:15;;;;
37;;2005/04/30 09:40;;2005/05/05 9:40 ;;+25;;Cooldown
later;;
later;;
38;;2005/04/30 17:40;;2005/05/05;;-20;;Defrost
2005-5-6 17:16;;;;
2005-5-6 17:16;;;;Mark updated data-collector to
cvs co -r V07-01-02 testdaq-collector
39;;2005/04/30 18:25;;2005/05/05 18:25 ;; -20;;Cooldown
later;;
later;;
40;;2005/05/01 08:25;;2005/05/06;;-45;;STF
2005-5-7 10:18;;hagar;;ran smoothly
2005-5-7 11:18;;- atwd pedestal noise
18 failures
atwd_pulser_spe - 1 failure
- flasher bright 1
- flasher clock 2
- hv ramp 1
;;
41;;2005/05/01 09:25;;2005/05/06;;-45;;DOMCal
2005-5-7 11:32;;hagar;;
2005-5-7 13:05;;in
domcal/fat5-45b-05.00.00
and
domcal/fat5-45b-05.04.00
;;this was domcal 05.00.00
Later domcal 05.04.00 was tested - 3 doms did nt calibrate well, and I went back to 05.00.00
42;;2005/05/01 10:25;;2005/05/06;;-45;;Lux
2005-5-7 13:10;;hagar;;none
2005-5-7 15:38;;~testdaq/monitoring/2005/05/07.lux/;;
43;;2005/05/01 12:25;;2005/05/06;;-45;;TestDAQ
2005-5-7 18:01;;hagar;;Domhubapp restarted at ~20:30:
runs 5933-5937 OK
run 5938: Jet-stream dom data stream missing in mon entry
run 5939..5949:Jet_Stream 73659395df5e fathub2.40B No reply to slow control message
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247)
icecube.daq.domhub.common.DHDriverException: DOM: Jet_Stream 73659395df5e fathub2.40B No reply to slow control messaggzip -d e
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247)
6: Jet_Stream 73659395df5e fathub2.40B -- Requested DOM not present in hit stream
run 5962-64 : Ostria is not present in hit stream. restarting domhubapp
2005-5-7 1:28;;5933-5937,
5951-5961
5966-;;
44;;2005/05/01 18:25;;2005/05/06;;-45;;Soaking/stability
2005-5-8 1:45;;hagar;;
later;;;;
45;;2005/05/02 18:25;;2005/05/07 18:25 ;; -45;;Cooldown
later;;
later;;
46;;2005/05/03 08:25;;2005/05/08 08:25 ;; -20;;STF
later;;
later;;
47;;2005/05/03 09:25;;2005/05/08 09:25 ;; -20;;DOMCal
later;;
later;;
48;;2005/05/03 10:25;;2005/05/08 10:25 ;; -20;;Lux
later;;
later;;
49;;2005/05/03 12:25;;2005/05/08 12:25 ;; -20;;TestDAQ
later;;
later;;
50;;2005/05/03 18:25;;2005/05/08 18:25 ;; -20;;Soaking/stability
later;;
later;;
51;;2005/05/03 19:25;;2005/05/08 19:25 ;; -20;;Defrost
later;;
later;;
52;;2005/05/04 08:10;;2005/05/09 08:10 ;; -14;;Warmup1
later;;
later;;
53;;2005/05/04 13:10;;2005/05/09 13:10 ;; +15;;Warmup2
later;;
later;;
54;;2005/05/04 13:10;;2005/05/09 13:10 ;; +15;;Warmup2
later;;
later;;