2005/06/10  16:30    16:30;;2005/06/10;;+25;;LCChain
2005-6-10  18:00;;Hagar;;
2005-6-10 18:30;;Success. <br>Connection between hub was not tested.<br>hub2-hub1-hub5 <br>The hubs connection was tested succesfuly <br><br>setup here:<br><a href=http://icecube.wisc.edu/~hagar/fat/fat7/monitor.hub1.start.FAT7.dat> fathub1 </a><a href=http://icecube.wisc.edu/~hagar/fat/fat7/monitor.fathub2.start.fat7.dat> fathub2 </a><br><a href=http://icecube.wisc.edu/~hagar/fat/fat7/monitor.fathub5.start.fat7.dat> fathub5 </a><br>;;one doms (700) was not communicating. It was replaced by fat 6 dom Leprechaun (620)
2005/06/10  16:50    16:50;;2005/06/10;;+25;;STF
2005-6-11 18:49;;hagar;;- fathub2 : ok<br>- 3 doms failed flasher clock. after fb update passed.  fathub1, fathub5: stoppetd before flasher-width. many flasher clock failures.all passed after fb update.<br>---------<br>run u and t speratly for flasher width.<br>all doms passed.
2005-6-15 11:00;;problem running flasher-width on hub1 and hub2 <br><br>2 failures: flasher bright on 13215a559856 Hadephobia, 666<br> <br><br>flasher width on Ecclesiophobia 664;;
2005/06/10  18:20    18:20;;2005/06/10;;+25;;DOMCal
2005-6-10 22:40;;hagar;;Updated domcal on all doms. <br> 7b0de8658562 LeukoPhobia Failed to calibrate. It is on fathub1 111U. updated domcal (again) but still no HV info while updating HV2DB
2005-6-10 23:40;;~/domcal/domcal+25/ <br>1 dom failed <br><a href=http://icecube.wisc.edu/~hagar/fat/fat7/histo25/hv.html> histos here.</a><br>;;
2005/06/10  19:20    19:20;;2005/06/10;;+25;;Lux
2005-6-11 10:40;;hagar/kael;;error: connection refused from lightsource.py.<br>Rates very high 20k-30k<br>one dom readouts=0, the same one who did not calibrate <br>Chris killed a python light control script, and now ok.
2005-6-11 18:47;;11.lux<br>;;
2005/06/11  08:00    08:00;;2005/06/11;;+25;;TestDAQ
2005-6-14 21:16;;hagar and later mark;;Beginning of run was very bumpy. <br><a href=http://www.icecube.wisc.edu/~hagar/fat/fat7/fat7crisis.txt><br>more details here </a> testdaq is now mworking after a domhubaoo update using Kael's fix. for more details contact Mark.<br> <br>more gps problems on hub5, card2, although hub was rebooted earlier.<br>Another problem is that the output fiels are not written to the same directory.<br>rebooting hub.... <br>....now run 6874 - fathub2 card3 gps is gone, did fpga 3 f 3 and restrted the run.<br> <br>again for gps hub5,2 fpga 2 f 3....boring <br><br>unbeleivablem after disk space problem, for runs 6884, 6885 messages: "requested dom not present in hit stream" for all hubs and "<br>icecube.daq.domhub.common.DHDriverException: Reply does not match a known DOM App message format<br>        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247)<br>4635 [Thread-11] ERROR TestDAQ Control  - DOM Hodophobia 52cae978ebb0 fathub2.32B failed configuration change<br>icecube.daq.domhub.common.DHDriverException: Reply does not match a known DOM App message format<br>        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247)<br>"<br><br>rebooted all three hubs, started testdaq.<br> <br>Runs (which Hagar started at run number 6887) were successful during the night until the DOM Ephebiphobia went dead during the first time resolution run (#6930)<br><br>consequently, i am repeating the previous run (the last of the ped pattern runs) and also the timeres runs.<br><br>these runs start with run number 6943<br><br>I deleted runs 6930 to 6942 from the data warehouse staging area.<br><br>
2005-6-15 9:00;;runs:<br>6887-6928, 6944-6947;;
2005/06/11  16:30   16:30;;2005/06/11;;+25;;Soaking / stability
2005-6-12 18:52;;kael;;see lux run above
2005-6-12 18:52;;;;
2005/06/11  17:30   17:30;;2005/06/11;;+25;;Cooldown
2005-6-15 13:11;;Claire / Hagar;;cold_rebbot started
2005-6-17 9:35;;failures:<br>fathub2:<br>630 - 17 failures  - Obesophobia, 3270c...<br>530 - Murophobia, 39f663d...1 failure-<br>610 - Radiophobia, ea6f08e...3 failures<br>730 - Mythophobia, 12d916...- 1 failure<br><br>fathub2:<br>730 - fe4fff5cfe19 -Photophobia- two failures<br><br>;;
8; 2005/06/12 08:30 ;  -20; Defrost


9; 2005/06/12 09:15 ;  -20; Cooldown


2005/06/12  22:00  22:00;;2005/06/12;;-45;;STF
2005-6-17 9:45;;hagar;;
2005-6-17 11:49;;fails:<br>725, novercaphobia - flasher bright <br>4e41e17f6b7e atwd pulser, chipB 1<br>e4194d677ad0 - disc scan<br>6458149340b pmt hv ramp<br>0912cde99f01 pmt hv ramp<br>01c99<br>01c997d687a6 atwd pulser 40pe, chip A1<br>69a843a5e181 pmt hv ramp<br>34bb7d414187 pmt hv ramp<br>721, textophobia - flasher bright<br>597, xyrophobia, flasher width<br>679, symmetrophobia flasher bright<br>669, sciaphobia, flasher bright<br>peccatophobia, 689 flasher bright<br>nomatophobia 647 - flasher bright<br>scoleciphobia 693 - flasher bright, flasher width<br> <br>limnophobia, 676 flasher bright<br>666, hadephobia flasher bright<br>648, hobophobia flasher bright<br>702, globophobia, fash bright<br>lavophobia flasher bright<br>618, ephebiphobia flasher width<br>ideophobia 708 - flasher bright<br>714, graphobia flasher width<br>638, cancerphobia - flasher width;;
2005/06/13  09:00  09:00;;2005/06/12;;-45;;DOMCal
2005-6-17 11:51;;hagar;;upgrdae to domcal 5.8
2005-6-17 13:5;;all doms calibrated but<br>7b0de8658562 <br>Leukophobia;;
2005/06/13  10:00  10:00;;2005/06/13;;-45;;TestDAQ
2005-6-17 13:6;;hagar;;run 6950- error in card 2 fathub5<br>and card 3 and 1 on fathub2<br>Reloaded pages 3 and 1. without rebooting,<br>----<br>run 6951<br>I forgot to "ready" on hub2. restart.<br>-----<br>run 6952-6954 - gps on hub5<br>rebooted hubs<br> <br>----<br>run 6955-6958 ok<br>-----<br>run 6959 hung <br><br><br>run 6960 encountered GPS errors, fathubs 2 and 5 were rebooted<br><br>started sequence of testdaq runs where we left off with run 6961<br> <br><br>assorted GPS problems occured until we rebooted the fathub5 again and started up run 6967 <br><br>debugged problems with new script until data taking for real began with run 6970<br>---<br><br>run 6970 -gps. <br>reloaded page 3,1 on fathub5<br>------<br> run 6975 seems ok <br>-------<br>run 6979 : gainvsHv 1300 - Leukophobia does not respond to slow control <br>----<br>fine til 6995 where Sitiophobia 3a5... is lost(first linearty rn )...restarting tetsdaq and reloading all cards on hub2 and hub5<br>---<br>6997.-7010ok<br> <br>run 7012 - Papaphobia dropped out from slow control. <br>restarted testdaq and fpga hub2 and hub5<br>------<br>run 7013 did not start well<br>-----<br>run 7014, 7015 ok<br>---<br>run 7016 Trichopathophobia failed comm...<br>----- <br>run 7018... 7027
2005-6-18 2:06;;good runs:<br>6955-6958,<br>6975-6993 (Leukophobia not responding, but this guy is dead already) <br>6997.-7010 <br>7014 , 7018-7027;;
2005/06/13  17:00  17:00;;2005/06/17 21:00;;-45;;LUX
2005-6-19 18:40;;hagar;;
2005-6-19 21:38;;06/19.lux<br>;;
2005/06/13  19:00  19:00;;2005/06/17 23:00;;-45;;Soaking / stability
2005-6-18 2:7;;hagar;;
2005-6-20 9:56;;06/18 = 40.2 hours<br>06/19 = 12 hours <br>;;
2005/06/14  19:00  19:00;;2005/06/19  21:00;;-45;;Cooldown
2005-6-20 9:57;;Claire;;started cold reboot at 10:00
2005-6-20 20:26;;;;
2005/06/15  08:30  08:30;;2005/06/20  8:00;;-55;;STF
2005-6-20 21:15;;hagar;;using new version of stf, with the new commands called "stfA" and "stfB"<br> <br>running on all tests but flasher using old stf
2005-6-20 23:39;;<a href=http://icecube.wisc.edu/~hagar/fat/fat7/stf_fail.txt><br>here </a>;;
2005/06/15  10:00  10:00;;2005/06/20  10:00;;-55;;DOMCal
2005-6-20 19:59;;mark;;upload domcal v5.9 and started domcal
2005-6-20 21:00;;all doms calibrated by 7b0de8658...;;
2005/06/15  11:00  11:00;;2005/06/20  12:00;;-55;;Lux
2005-6-20 23:41;;hagar;;3592798f3eb0-TP5P0597-Xyrophobia is not communicating restarting. <br>The light source were left on, so I stopped it again when the light source are off.<br>restarted. now there are 146 doms with good starting rate.<br>happy I am !
2005-6-21 10:59;;first in 20.lux, but only the nohup file is in.<br>then in 21.lux<br> <br>9 hours of monitoring;;
2005/06/15  13:00  13:00;;2005/06/20  14:00;;-55;;TestDAQ
2005-6-21 10:59;;hagar, mark;;start at 0007028 - card 2 hub 5 gps.<br>I put a script called fpgall in bin to reload pages 1,3 on all cards. <br>---<br>run 7029-7036 used the wrong steering files. I had a typo in automate<br>---<br><br>7037...7059 ok<br>on 7061  Panophobia 6e41ab3703aa failed response to slow control <br>7069-7084 <br>fathub1 crashed on Tuesday evening.<br> <br><br>restart of testdaq runs failed because of GPS<br><br>2nd restart beginning with run 7104 - look good so far<br><br><br>7104,7105,7106 ok
2005-6-22 18:30;;;;
2005/06/15  21:00  21:00;;2005/06/15  21:00;;-55;;Soaking / stability
2005-6-22 18:30;;hagar;;
2005-6-23 23:43;;contuing the soaling from lux.<br>22/<br> <br><br>started warm up around noon.<br>fathub2 crashed;;
2005/06/17  08:00  08:00;;2005/06/17  08:00;;-55;;Warmup
2005-6-23 23:44;;claire;;
2005-6-24 10:1;;;;
2005/06/17  11:30  11:30;;2005/06/17  11:30;;-20;;STF
2005-6-24 11:30;;hagar;;
2005-6-24 13:49;;6 failures.<br>Ranidaphobia: bright<br>ephebiphobia: 618 : width<br>hadephobia: 666 : bright<br>ideophobia: 708 : bright<br>oneriphobia: 416 : pulseatwd<br>rhytiphobia: 665 :pulse atwd;;
2005/06/17  13:00  13:00;;2005/06/17  13:00;;-20;;DOMCal
2005-6-24 10:1;;hagar;;After rebooting fathub1 who was not responding:<br><a href=http://icecube.wisc.edu/~hagar/fat/fat7/crashes.txt>Logs of all the crashes here.<br></a><br>
2005-6-24 10:55;;;;
2005/06/17  14:00  14:00;;2005/06/17  14:00;;-20;;Lux
2005-6-25 0:23;;hagar;;fathub1 crashed on 25/6 at 10:55<br>machine rebooted.<br><a href=http://icecube.wisc.edu/~hag7/crashes.txt> see<br>messages </a><br>
2005-6-25 0:24;;06/25.lux;;
2005/06/17  16:00  16:00;;2005/06/17  16:00;;-20;;TestDAQ
2005-6-24 13:53;;hagar;;run 7123...7134 ok<br>-----------------<br>run 7136 : Doraphobia 21023f61af76 fathub2.03B No reply to slow control message. restarted<br>--------------<br>7138-ok<br>--------------------<br>7140 - DOM: Doraphobia 21023f61af76<br>fathub2.03B No reply to slow control message<br>rebooted hubs. note: this is the same dom as before, and on hub2 card0<br>------------<br>7143-7153: Doraphobia  fathub2.03b and Pagophobiba 2.03A failed config change<br>------<br>7154-7156 ok<br><br>----------<br>7158-7161 doraphobia, pagophobia failed config change<br>from 7160 also textophobi fathub2.20Aa998c...<br>rebooting fathub2<br>-------<br>7162...7163<br> 7165-7178 pagophobia and doraphobia,<br>-----------<br>7180-7187<br><br>=============<br>run 7189: Doraphobia, pagophobia and telephonophobia (b352) fathub2.03B failed configuration.<br>I will start multimon and continue tomorrow <br>------------<br>7192 - 7212, 7218, except 7203.  <br>run 7203: java error <br><br>in RapCalImplBS:<br>exception caught in RapCalImplBS constructor (id=Pagophobia 58681d58fc77 fathub2.03A) ...ArrayIndexOutOfBoundsException=-1
2005-6-25 19:42;;;;
2005/06/18  08:30  08:30;;2005/06/18  08:30;;-20;;Soaking / stability
2005-6-25 19:50;;hagar;;
2005-6-29 11:38;;in /25<br>and /25.lux<br>;;
2005/06/19  08:30  08:30;;2005/06/19  08:30;;-20;;Defrost
2005-6-26 16:30;;Claire Pettersen;; 
2005-6-26 17:15;; ;; 
2005/06/19  09:15  09:15;;2005/06/19  09:15;;-14;;Warmup1
2005-6-26 17:15;;Claire Pettersen;;/mnt/data/testdaq/monitoring/2005/06/26
2005-6-26 21:15;;;;
2005/06/19  14:15  14:15;;2005/06/19  14:15;;+15;;Warmup2
2005-6-27 07:00;;Claire Pettersen;;
2005-6-27 08:00;;multimon results in <br>/mnt/data/testdaq/monitoring/2005/06/26;;
30; 2005/06/20 09:15;   +25; LCChain


2005/06/20  10:00  10:00;;2005/06/20  10:00;;+25;;STF
2005-6-27 8:39;;hagar;;
2005-6-27 15:00;;;;
2005/06/20  11:00  11:00;;2005/06/20  11:00;;+25;;domcal
2005-6-27 175:26;;hagar;;
2005-6-27 27:26;;;;
2005/06/20  12:05  12:05;;2005/06/20  12:05;;+25;;TestDAQ
2005-6-28 16:0;;hagar;;7227-7246 OK<br>--------<br>7248 - Papyrophobia - no reply to slow control<br>--------<br>7250-ok<br><br>---------------<br>7252 - Ideophobia - no reply to slow control.<br>-----------<br>rebooted hubs,<br>7254-7279<br>
2005-6-29 5:00;;;;
2005/06/20  19:05  19:05;;2005/06/20  19:05;;+25;;Lux
2005-6-28 17:25;;;;the pulser power supply was drenched in water and no longer working.  It has been replaced <br>Chris had to open the light sources box and thread cables in and out;  <br>It's unlikely, but conceivable, that the laser pointing was  <br>disturbed (which would spoil the station-to-station calibration).<br><br>
2005-6-28 19:40;;28.lux;;
2005/06/20  21:05  21:05;;2005/06/20  21:05;;+25;;Soaking / stability
2005-6-28 22:41;;hagar;;
2005-6-28 16:30;;/mnt/data/testdaq/monitoring/2005/06/27<br>/mnt/data/testdaq/monitoring/2005/06/28;;
2005/06/21  16:30  16:30;;2005/06/21  16:30;;+25;;Cooldown
2005-6-29 10:30;;Claire Pettersen;;start cold reboot
2005-6-29 17:30;;;;
2005/06/22  08:30  08:30;;2005/06/22  08:30;;-20;;Defrost
2005-6-30 5:00;;Claire Pettersen;; 
2005-6-30 5:30;; ;; 
2005/06/22  09:15  09:15;;2005/06/22  09:15;;-20;;Cooldown
2005-6-30 5:30;;Claire Pettersen;; 
2005-6-30 13:00;; ;; 
2005/06/22  19:15  19:15;;2005/06/22  19:15;;-45;;STF
2005-6-30 22:30;;hagar;;
2005-7-1 11:32;;;;
2005/06/23  08:00  08:00;;2005/06/23  08:00;;-45;;DOMCal
2005-6-30 22:00;;hagar;;
2005-6-30 22:45;;;;
2005/06/23  09:00  09:00;;2005/06/23  09:00;;-45;;Lux
2005-7-1 00:44;;hagar;;
2005-7-1 09:30;;/mnt/data/testdaq/monitoring/2005/07/01.lux/;;
2005/06/23  11:00  11:00;;2005/06/23  11:00;;-45;;TestDAQ
2005-7-1 11:34;;hagar;;run 7318
later;;;;
2005/06/23  19:30  19:30;;2005/06/23  19:30;;-45;;Soaking / stability
2005-7-1 11:36;;hagar;;hub2 crashed on 7:00 am, 3/7 <br>3/7, 9:40: set Leprechaun hv1=1200 (was 1396). restart multimon.<br>cyberphobia hv1=1200<br>(was 1420) 5898d6a825cf <br> <br>hv1=1100 5/7, 11:00 <br>hub1 crashed on 4/7, <br>----<br>5/7, 11:15, reloaded dor drivers on hub1 and hub2.<br>for hub5 got "dh: Device or resource busy".no proccesses were found. did not reboot fathub5<br> reoad dh again on hub1 and hub2, 6/7/05 at 22:20<br> <br>hv of Leprechaun and cyberphobia back to nominal value <br>----------<br>fathub1 crashed on  Jul7, evening.
later;;06/18 = 40.2 hours<br>06/19 = 12 hours <br>07/01.lux = 7 hours <br>07/01 - 38.25 hours /46.4 hours<br>07/03 - 23.2 hours/43.4 hours <br> <br>07/05 - 35.5 hours<br>07/06 - 20 hours<br>;;