500U a809b636d4c7 22.94 96
This became the Commonwealth of "Virginia"
comm problems on both sides:
domhub2 73B f8d252d28d84 UP4P0256
This DOM was removed from the DFL
DOMs 968 Alaska and 970 Mississippi may be mixed up.
Claire identified them as being in the correct spot in the DFL. Andy says their mainboard ids are switched - is the label wrong?
also, Minotaurish TP5P0545 will not load the flasher firmware properly
also, Meteorology AP5P0446 seems to like to crash frequently (the mainboard actually crashes). I will try reuploading the mainboard software once...
readings at the start of fat 10
uploaded release 322, domcal 5-11-01 and the flasherboard firmware (version 12) - Minotaurish failed to upload the firmware, and Meterology gives flash upload errors
DOM positions at start of FAT 10
2005-9-16 18:12;;as part of the initialization in the future it will be necessary to do this on each domhub:
echo 1 >
/proc/driver/domhub/verbose
I think this will have to be done after each reboot;; 2005/08/19 8:00 8:00;;2005/08/19 8:00;;+25;;LCChain
2005-9-17 20:51;;
;; 2005/08/19 8:30 8:30;;2005/08/19 8:30;;+25;;STF A/B 2005-9-16 17:18;;
STF results, U DOMs, P25A ;;many U doms fail pulser_1pe_chipA_ch0
149 doms tested. Meterology was not. 2005/08/19 10:30 10:30;;2005/08/19 10:30;;+25;;DOMCal
2005-9-16 0:39;;
on all DOMs
Indiana hung in the middle - one DOM has no .out file or .xml file - which one??? I have no idea...
starting up domcal-wrapper again to see if i can find the missing dom - rebooted all doms first
| a19e1b2d911d | AP5P0446 | Meteorology | fathub1.12A |
appears to be the DOM which is not producing any king of domcal.out or domcal.xml file...
2005-9-16 0:39;; domcal histos at start of fat 10
;; 2005/08/19 11:30 11:30;;2005/08/19 11:30;;+25;;TestDAQ
2005-9-16 3:53;;
run 8251 was a total failure
ERROR (DOM.java:1093) - DataMsgService WF data stream is not ready
run 8255 is the first run which worked...
testdaq test runs taken successfully after reverting back to domhub-app 02-05-V18, dor-driver V02-06-01h.
note the new dor-driver-pci was left as is (version 9) - this is a change from fat 09
in the test runs the laser pulse looks good - claire has since turned on the calibration PMT
fathub1 crashed in run 8288 - runs restarted with run 8294
errors reported in run 8306 - a strange set of DOMs to fail. will repeat previous run...
2005-9-16 3:53;; run 8258 is offical start of FAT 10 TestDAQ runs
8288 to 8293 are bad runs
8305 thru 8307 are bad runs
last of the P25A good runs is 8314
semaphore files moved to data goes to madison...
;; fathub1 crashed during run 8288 2005/08/19 18:30 18:30;;2005/08/19 18:30;;+25;;Lux
2005-9-16 19:36;;
claire turned on ref pmt earlier in the day 2005-9-16 19:36;; ;; 2005/08/19 20:30 20:30;;2005/08/19 20:30;;+25;;Soaking / stability 2005-9-16 14:55;;
multimon started around 4am
multimon restarted at 1pm because the hv set point for Cheyenne was changed from 3500 volts (3500 volts actually corresponds to arround 1600 volts to 1185 volts, per jim brauns rec.
16.lux is the P25A lux run...
in the overnight running of domcal (Sep 17 morning) the DOM Richmond did not produce data - reason unknown - i think it was because i had killed multimon and restarted it without rebooting the DOMs - was too tired to do this
started multimon again at 10:20pm on sep 17
restarted multimon-wrapper at 6:45pm on Sep 18
at 7pm i changed my mind and started up the cold reboot scripts. 2005-9-16 14:55;;
Plots here
Temperature changing drastically over time;;both fathub2 and fathub5 crashed during morning of sep 18 - at around 7am and 7:15am. the fathub5 crash is surprising. Multimon restarted at 11am.
fathub1 crashed at 7pm on Sep 18 2005/08/22 08:30 08:30;;2005/08/22 08:30;;+25;;Cooldown 2005-9-17 22:35;;
I started up the reboot scripts early to look for failures at warm temperatures (the new fat9 reboot script found a lot of problems) 2005-9-17 22:35;;;; 9; 2005/08/22; 16:00 ; -20; Defrost 2005/09/19 17:00 17:00;;2005/09/19 17:00;;-20;;cooldown
2005-9-19 15:4;; ;;at 10:30m I (Mark) did "dtsxall" on fathub5. Two of the wire pairs had generated CHUNKSIZE errors
(one of them was for "Richmond" at around +2 C, and its partner "Texas" went down at the same time.
"Charleston" and "Tennessee" had failed a few hours later. 2005-9-19 15:4;; ;; 2005/09/20 08:00 08:00;;2005/09/20 08:00;;-45;;STF 2005-9-21 10:13;;Jake, hagar;;at least one T Dom failed running status
& sytanx error or access violation on SQL
at least one U Dom failed runstate change
2005-9-21 12:4;;
Here For T's
and here for U's ;;148 doms tested.
Meterology and ?? were not tested.
2005/09/20 10:00 10:00;;2005/09/20 10:00;;-45;;DOMCal 2005-9-21 12:53;;Jake + hagar;; 2005-9-21 13:53;;14fc92c73f8c
08f71471129a - have strange fits in HV's
Histos here ;; 2005/09/20 11:00 11:00;;2005/09/20 11:00;;-45;;TestDAQ 2005-9-21 16:11;;Jake, hagar;;Took us a while to find that claustrophilia was giving testdaq a hard time. Not willing to go into softboot.
It was removed from dh.propertiess (2.50B) and the LC modes for its neighbours were changed:
43A, ba088ea10a64 to "2"
and:
50A, e8933be1bf50 to "3"
2005-9-22 23:28;;run started at 8327 .
with hit format error:
good runs:
8374-8379
8383- 8424
Checked time res plots and data quality is good.
;;runs 8327-8350 - ok accept for "hit record format errors". some runs have many entries, and other only a few. stopped testdaq until further investigation.
removing meterology from testdaq (fathub1.12A) 2005/09/20 18:00 18:00;;2005/09/20 18:00;;-45;;LUX
2005-9-21 22:59;;
;; 2005-9-22 0:45;;21.lux;; 2005/09/20 20:00 20:00;;2005/09/20 20:00;;-45;;Soaking / stability
2005-9-22 0:45;;
;;Rocish had a file size that was less than half of the others, perhaps because it is only sampling every 3 secs instead of in less than 2, appears fine on restart may just be a software problem
file 2005-9-25 6:7;; ;; 16; 2005/08/25; 08:00; -45; Cooldown 2005/09/23 08:00 08:00;;2005/09/23 08:00;;-55;;STF A/B 2005-9-26 12:25;;JB;;none 2005-9-26 1:32;;Continued failure of pulser test
A
B
Special runs:
A
B
;; 2005/09/23 10:00 10:00;;2005/09/23 10:00;;-55;;DOMCal
2005-9-25 6:14;;
;; 2005-9-25 10:20;; no calibration for Claustrophilia.
here results ;; 2005/09/23 11:00 11:00;;2005/09/23 11:00;;-55;;TestDAQ
2005-9-26 6:34;;
;;hub1 crashe in gainvshv-1700 run 2005-9-26 13:57;;good runs:8426-8441
8446-8452
8461-8465
8469-8492
Stops due to
1. hub1 crash
2. dead dom -
3. dead dom - hartford;; 2005/09/23 18:00 18:00;;2005/09/23 18:00;;-55;;LUX
2005-9-25 10:19;;
;; 2005-9-25 12:19;; ;; 2005/09/23 20:00 20:00;;2005/09/23 20:00;;-55;;Soaking / stability
2005-9-25 6:35;;
;;size of Arkansas is really small 17 compaired to 86 2005-9-27 11:18;;about 10 hours in the above lux run;; 2005/09/26 09:00 09:00;;2005/09/26 09:00;;-55;;Warmup
2005-9-27 11:16;;jake;;ff245ac... up5p0948, Tenessee and 9984c001b14c-tp5p0901-Charleston(5.03A and 5.03B) lost multimon at 27//9, 14:48 in -28 degrees.
/var/log/messages. current ok, units turned on OK
error in nohup.out: time out error
2005-9-27 11:16;; ;; 2005/09/27 08:00 08:00;;2005/09/27 08:00;;-20;;STF 2005-9-28 12:01;;jake;;included meteorology in even, will not include claustrophilia in odd 2005-9-28 14:26;;redid tests pulser_1pe_chipA_ch0 and pulser_1pe_chipB_cho. all passed this time;;screen shots for st only,sorry 2005/09/27 10:00 10:00;;2005/09/27 10:00;;-20;;DOMCal
2005-9-28 5:28;;
;; 2005-9-28 6:53;; no calibration for:
Meterology,
and Claustrophilia;; 2005/09/27 11:00 11:00;;2005/09/27 11:00;;-20;;TestDAQ
2005-9-29 4:44;;
;;start at run 8493
8500 - fathub1 crashed on run 8500, GainVsHV-1200
8507-Trenton dead. restart 8509 2005-9-29 13:37;;8493-8498
8509-8550;; 2005/09/27 18:00 18:00;;2005/09/27 18:00;;-20;;LUX
2005-9-28 7:10;;
;; 2005-9-28 10:10;; ;; 2005/09/27 20:00 20:00;;2005/09/27 20:00;;-20;;Soaking / stability 2005-9-28 14:31;;jake;;Fathub1 crashed at 7:28pm on the 29th 2005-9-30 9:22;;;; 28; 2005/09/01; 08:00; -20; Defrost 2005/09/29;;2005/09/29;;-14;;Warmup1 2005-9-30 12:00;;jake, edited by Claire;;rates of wisdom seem a little high at start ranging from 141 to 202
Restarted multimon at Noon. 2005-9-30 15:33;; ;; 30; 2005/09/01; ; +15; Warmup2 2005/09/30 08:00 08:00;;2005/09/30 08:00;;+25;;LCChain
2005-10-5 2:41;;
;; 2005-10-5 3:29;; OK;; 2005/09/30 08:30 08:30;;2005/09/30 08:30;;+25;;STF 2005-10-3 16:45;;Erik;; 2005-10-4 9:15;;no picture. the screen capture isn't working. not too many errors though.
U doms failed the pulser_1pe_chipA_ch0 test in large quantity. ran a special run on just this test, and they all passed. hmmm. make of it what you will.;; 2005/09/30 10:30 10:30;;2005/09/30 10:30;;+25;;domcal 2005-10-2 18:26;;Erik;;didn't seem to work when I ran it sunday night. so I ran it again monday morning. 2005-10-3 9:11;;no Meteorology
results ;; 2005/09/30 11:30 11:30;;2005/09/30 11:30;;+25;;TestDAQ 2005-10-3 10:30;;Erik;;start at 8554 2005-10-3 16:59;;8554-8564 good
8568- 8604
only crashed once, for Baton_Rouge. Not bad.;; 2005/09/30 18:30 18:30;;2005/09/30 18:30;;+25;;Lux 2005-10-4 9:24;;Erik;; 2005-10-4 12:30;;;; 2005/09/30 20:30 20:30;;2005/09/30 20:30;;+25;;Soaking / stability 2005-10-3 18:50;;;;soak overnight. lux in the morning.
WisDOM rates seem high. ~150 on average 2005-10-6 13:51;;stop at 2005-10-4 8:36 to run STFB and lux. then start up again till claire cools.
start again at 12:30;; 2005/10/03 08:30 08:30;;2005/10/03 08:30;;+25;;Cooldown
2005-10-5 10:00;;
Claire;; 2005-10-5 17:00;;fathub1: 031,131,101,731 have comm errors.
fathub2:630,600,601 have comm errors
fathub5: 110,031 have many comm errors. 200,120,011 have some.
see .dat files for details.
;; 2005/10/03;;2005/10/03;;-20;;Defrost
2005-10-8 7:48;;
Claire;; 2005-10-8 7:48;; ;; 2005/10/03;;2005/10/03;;-20;;Cooldown
2005-10-8 7:48;;
Claire;; 2005-10-8 7:48;; ;; 2005/10/04 08:00 08:00;;2005/10/04 08:00;;-45;;STF 2005-10-7 09;00;;
Claire;;One database Error. Probably due to non-communicating DOM 2005-10-7 10:55;;Rerun on the pulser_1pe_chipA and chipB on all doms using the new STF. no fails !
;; 2005/10/04 10:00 10:00;;2005/10/04 10:00;;-45;;DOMCal
2005-10-6 13:19;;Erik,
Claire;;hung with 6 DOMs to go first time. Had some troubles with the Histo generator, but it worked in the end. 2005-10-6 17:19;;no Meteorology or Claustraphilia
results
Demonology cal at high gain looks funky.;; 2005/10/04 11:00 11:00;;2005/10/04 11:00;;-45;;TestDAQ 2005-10-7 13:33;;Erik,
Claire;;start good runs at 8612
stopped running background_it.pl on runs after a while.
started over. no dice. going back to the soak for a while.
started up again 12:50 monday. 2005-10-10 18:39;;8612-8615 good.
8637-8680.
no trouble once it got going.
;;screwed up start, so 8605-8611 are no good.
/data/psl was full. moving semaphore files. Will redo testDAQ when space is available. 2005/10/04 18:00 18:00;;2005/10/04 18:00;;-45;;LUX
2005-10-6 17:32;;Erik,
Claire;; 2005-10-6 19:32;;ok;; 2005/10/04 20:00 20:00;;2005/10/04 20:00;;-45;;Soaking / stability 2005-10-6 20:15;;Erik,
Claire,
Mark;;10-6 20:15 to 10-7 9:00
10-7 11:00 to 10-7 12:30
10-7 15:34 to 10-10 12:50
10-10 18:50 to
2005-10-14 14:42;;fathub1 crashed on oct 7 at around 6:30pm
mark restarted everything at around 5am on oct 8
fathub1 crashed at 13:45 on oct 8. restarted at 14:30.
fathub1 crashed on oct 14 at around 2:30pm
;;Oct 8, 6pm, mark started up multimon using a new version of dtsx. we may learn something about the CHUNKSIZE errors with this new version... 2005/10/11 10:30 10:30;;2005/10/11 10:30;;-45;;Warmup
2005-10-8 7:45;;
Claire;; 2005-10-8 7:45;; ;; 46; 2005/09/14; 08:30; -45; UnLoad