HomePage | RecentChanges LogFebruary February 2005 electronic log book ______________________________________________________________________________________________________________________________________________________________________________________________________________ Fri Feb 11 12:33:36 GMT 2005 Marc Hellwig Reboot of sps-ichub-dat01 and sps-ichub-dat02. This is a test whether there is a hardware related problem in communication that disappears after rebooting. Removed the following services rhnsd, rhn_hardware_profile_update, canna, cups, apmd from both machines. They're not necessary for normal operation. Wed Feb 9 22:02:36 GMT 2005 Ran DOMCAL on ic01 and ic02. Only created 44 cal files. Wed Feb 9 21:30:12 GMT 2005 After off/on and dtsx restart all DOMs passed lcchain.py Wed Feb 9 21:27:17 GMT 2005 steffen Testing localhost 11A to localhost 20B Traceback (most recent call last): File "/home/testdaq/bin/lcchain.py", line 427, in ? if (lcpair(hubs[hub_hi], port_hi, hubs[hub_lo], port_lo)): File "/home/testdaq/bin/lcchain.py", line 138, in lcpair q_lo = daq.ibx(host_lo, port_lo) File "/usr/lib/python2.2/site-packages/icecube/domtest/ibidaq.py", line 257, in __init__ txt = self.send('domid') File "/usr/lib/python2.2/site-packages/icecube/domtest/ibidaq.py", line 286, in send raise IBEX, 'Timeout Error' icecube.domtest.ibidaq.IBEXTimeout Error Wed Feb 9 21:22:25 GMT 2005 steffen run lcchain.py on 02. Failed with: Testing localhost 11B to localhost 11A Traceback (most recent call last): File "/home/testdaq/bin/lcchain.py", line 427, in ? if (lcpair(hubs[hub_hi], port_hi, hubs[hub_lo], port_lo)): File "/home/testdaq/bin/lcchain.py", line 138, in lcpair q_lo = daq.ibx(host_lo, port_lo) File "/usr/lib/python2.2/site-packages/icecube/domtest/ibidaq.py", line 257, in __init__ txt = self.send('domid') File "/usr/lib/python2.2/site-packages/icecube/domtest/ibidaq.py", line 286, in send raise IBEX, 'Timeout Error' icecube.domtest.ibidaq.IBEXTimeout Error Mon Feb 7 11:31:05 GMT 2005 krasberg testdaq operational documentation is now at http://amanda.spole.gov/icecube/private/daq/icecube/documents/The_IceCube_TestDAQ.html started taking combined inice and icetop data yesterday _____________________________________________________________________________________________________________________________________________________________________________ Sat Feb 5 13:42:14 GMT 2005 krasberg there are still 36 doms hooked up to domhubs 1 and 2 a log of what happened today: 1) set up 2nd dom at tch - both inside a large box - noise rates are normal 2) extensively modified autogen to create steering files automatically for flasher runs 3) took dark noise and LC runs and flasher runs 4) hub 1 card 3 pair 1 is suspect since the cable was disconnected and reconnected 5) today'd dark noise runs do not exhibit the timing problem - however, flasher runs do... 6) uploaded release-pole-fb-01 to all doms on domhub 1 (4 doms had the old version !!) 7) ran stf on domhub2 (ran stf on domhub 1 yesterday, all doms passed) _____________________________________________________________________________________________________________________________________________________________________________ Thu Feb 3 05:04:04 CST 2005 krasberg we seem to have a major problem some runs have bad gps corrected times it looks like it depends on the data rate. ie a localcoincidence inice run works fine there is no error in domhub-app or testdaq control logs. sample runs which look good: SPS-DAQ-IT01_run0000067_DarkNoise?-ATWD0 SPS-DAQ-01_run0000360_LocalCoincidence?-ATWD0 sample runs which are bad: SPS-DAQ-01_run0000356_DarkNoise?-ATWD0 every flasher board run I have lookd at so far SPS-DAQ-IT01_run0000065_LC-station39HGonly-5 i dont have too many ideas at the moment! Mark ______________________________________________________________________________________________________________________________________________________________________ Thu Feb 3 10:51:52 GMT 2005 krasberg DOM rates are now falling below 1kHz as the DOMs freeze in: http://amanda.spole.gov/icecube/private/daq/icecube/strange-rates/strange-rates-6.html _____________________________________________________________________________________________________________________________________________________________________________ Wed Feb 2 14:04:52 GMT 2005 krasberg the latest leak-testing spreadsheet, courtesy of Jim Baccus http://amanda.spole.gov/icecube/private/daq/icecube/leak-testing/leak-testing.html _____________________________________________________________________________________________________________________________________________________________________________ Wed Feb 2 13:47:28 GMT 2005 krasberg The icetop string processor SPS-ICETOP01 has been set up to take data with the IceTOP? DOMs. The prefix for data files taken with this machine is SPS-DAQ-IT01 Additionally, some of the scripts and jar files were modified, and these changes will be propogated back to SPS-STRIGNPROC01 ________________________________________________________________________________________________________________________________________ Wed Feb 2 07:09:43 CST 2005 krasberg the rates are going down! see http://amanda.spole.gov/icecube/private/daq/icecube/strange-rates/strange-rates-4.html ________________________________________________________________________________________________________________________________________ Tue Feb 1 16:04:55 UTC 2005 Jacobsen Ran moat on all 4 DOMs on Quad 3 plugged into domhub2 -- all passed a short moat test versus configboot, iceboot firmware. See first draft of commissioning checklist for this quad (Gary has it). ________________________________________________________________________________________________________________________________________ Tue Feb 1 15:25:34 UTC 2005 Jacobsen Managed to upload release-pole-fb-01.hex to all four DOMs on domhub2: Card 1 Pair 0 DOM A ID is fe92d7ff4480 Card 1 Pair 0 DOM B ID is abcfd5e5a352 Card 1 Pair 1 DOM A ID is fb0944d283fd Card 1 Pair 1 DOM B ID is 32bb0201e5a7 There were lots of retxs for 11B. ________________________________________________________________________________________________________________________________________ Tue Feb 1 10:55:24 UTC 2005 Jacobsen Checked comms on the quad plugged into DOMHub 2 -- had signs of water leak before, Jim told Mark water leak signs went away. But comms test failed with hardware timeout on 11b (flash is hosed, presumably because Mark tried to upload new software). Tried to upload new software myself with ldall. 11b is happy at first, then retxs occasionally, then retxs constantly the same packet (stuck). I powered off the quad. Also checked DOMHub 1 wire pair (2, 1). 21A and 21B were on domhub3 before, moved to domhub1. These two DOMs are also very sick, with retxs and hardware timeouts. ________________________________________________________________________________________________________________________________________ Tue Feb 1 06:29:54 GMT 2005 krasberg problems problems problems 1) when the flasher board rate is set to 20 Hz, it triggers at 600 Hz. Will try setting it to one and see what happens (hopefully the rate is 2 to the power of x, and it maxes out at 600 Hz or so). Addendum - this did not turn out to be the case - 600Hz is fixed. 2) testdaq darknoise runs with the inice doms wipe out the data-collector. Intend to try with Arthur's tinydaq disabled, to see if that will make a difference, since I have never seen this happen before... 3) one of the Swedish DOMs (pos 11, "Erik_Segersaell", is having a problem communicating. http://amanda.spole.gov/icecube/private/daq/icecube/data-taking/swedish_dom_comm_problem.dat Both DOMs on this wire pair seem to be losing comms. I have stopped using this wire pair. _______________________________________________________________________________________________________________________________________________ Tue Feb 1 04:42:47 GMT 2005 krasberg Quad 3 (domhub1 card 1) has been disconnected and plugged into domhub2 card 1 (in case it is decided that it is safe to do tests) We are now ready to try to do flasherboard tests... ________________________________________________________________________________________________________________________________________ Tue Feb 1 03:49:22 GMT 2005 krasberg Quad 3 hooked up to domhub1 dor card 1 is having Comm problems flash download failed for 11B - it may have failed for 11A as well, not too sure. lots and lots of these in /var/log/messages: Jan 31 21:45:58 sps-ichub-cont01 kernel: RETXB(1526, minor=11, dt=25, tmax=20): SEQN(1972) -> retxq. Jan 31 21:45:58 sps-ichub-cont01 kernel: RETXB(1551, minor=11, dt=25, tmax=20): SEQN(1972) -> retxq. i briefly turned on the DOMs after the problems - dor card 1 doms were all communicating. we have sadly decided that we should keep this quad off for now, until an expert can look at /var/log/messages ________________________________________________________________________________________________________________________________________ ________________________________________________________________________________________________________________________________________ Tue Feb 1 03:03:32 GMT 2005 krasberg rewiring of DOMs is taking place. DOMs in middle of string are being moved to domhub1. DOMs at end of string are being moved to domhub3. One quad (Q3 - doms 5 to 8) is going to be plugged in for the first time. Mapping will be: Q2 --> domhub3 dor card 0 Q3 --> domhub1 dor card 1 (the new never turned on DOMs) Q4 --> domhub1 dor card 2 Q5 --> domhub1 dor card 3 Q6 --> domhub1 dor card 4 Q7 --> domhub1 dor card 5 Q11 --> domhub3 dor card 7 (this is a strange mapping for this quad, in case you were wondering) Tue Feb 1 03:03:32 GMT 2005 krasberg Sample flasherboard waveform and current pulse can be found at: http://amanda.spole.gov/icecube/private/daq/icecube/flasherboard-tests/flasherboard-tests.html Tue Feb 1 01:45:29 GMT 2005 Krasberg Strange rate changes which seem correlated between some DOMs... http://amanda.spole.gov/icecube/private/daq/icecube/strange-rates/strange-rates-3.html Mon Jan 31 18:23:11 CST 2005 Krasberg The rates at the top are on the rise again! http://amanda.spole.gov/icecube/private/daq/icecube/strange-rates/strange-rates-2.html Tue Feb 1 00:01:06 GMT 2005 Krasberg Added Jim Baccus's xcel leak testing spreadsheet at http://amanda.spole.gov/icecube/private/daq/icecube/leak-testing/leak-testing.html Mon Jan 31 17:49:40 CST 2005 Krasberg I believe we have made a mistake. Each plugged in quad for data-taking is at least two quads away from another quad. ie we are analyzing doms 1-4, 13-16, 21-24 and 37-40 (see http://amanda.spole.gov/icecube/private/daq/icecube/normal-data-taking-doms.dat ) Meanwhile, DOMs 9 thru 12 and 17 thru 20 are being used to monitor the temp and pressure (see http://amanda.spole.gov/icecube/private/daq/icecube/cold-boot-doms.dat ) For flasherboard runs we should have a "mini-string" I propose we switch doms 17-20 with doms 37-40 this way we can analyze a string from DOM 13 all the way to DOM 24 with the flasherboard. ---------------------------------------------------------------------------------------------------- Mon Jan 31 22:18:00 GMT 2005 Krasberg All 24 DOMs continue to communicate with the surface. Dom lists are at: http://amanda.spole.gov/icecube/private/daq/icecube/normal-data-taking-doms.dat and http://amanda.spole.gov/icecube/private/daq/icecube/cold-boot-doms.dat The rates are still strange - here are the rates for the top two DOMs (the strangest rates): http://amanda.spole.gov/icecube/private/daq/icecube/strange-rates/strange-rates.html In data taking I found two potential problems just now: 1) on icehub3, the cold boot DOMs, one DOM called "Sverker" (20T) did not communicate properly at first and had to be rebooted in /var/log/messages I see: Jan 31 16:09:12 sps-ichub-cont03 kernel: RETXB(59551, minor=18, dt=23, tmax=20): SEQN(0) -> retxq. Jan 31 16:09:12 sps-ichub-cont03 kernel: RETXB(59576, minor=18, dt=25, tmax=20): SEQN(0) -> retxq. 2) in multimon monitoring I see an icetop "event" (major noise spike) which spans two DOMs called "Lithium" and "Black_bean". These DOMs are both in station 30. The strange thing is that the DOMs are in different tanks, which makes me slightly suspicious of the wiring! see http://amanda.spole.gov/icecube/private/daq/icetop/Akagai-Black_bean-event/Akagai-Black_bean-event.html Perhaps a flasher run would determine whether or not a miswiring has occured... ---------------------------------------------------------------------------------------------------- Mon Jan 31 16:24:01 UTC 2005 Jacobsen At the request of Dan Wahl at PSL, I measured the wire pair current while a DOM was flashing both on the surface (domhub #7) and in-ice (domhub 1, 00a). Results: surface DOM off: 0 mA configboot: 28 mA iceboot: 28 mA domapp: 35 mA domapp flashing at 610 Hz, brightness=1: 48mA brightness 2: " brightness 4: " brightness 32: " brightness 64: 50mA brightness 127: 56mA DOMMB temperature is -14 C. DOM 00a in the hole: off: 0 mA configboot: 57 mA iceboot: 64 mA domapp: 73 mA domapp flashing at 610 Hz, brightness=1: 89mA brightness 2: " brightness 4: " brightness 32: " brightness 64: 90mA brightness 127: 95mA DOMMB temperature is -1.56 C. I sent the results to Dan and a few other people this AM. ____________________________________________________________________ Tue Feb 1 16:14 NZ / South Pole time 2005 Baccus After testing the leakage current for Quad 3 and finding that it is well below the 10uA limit for safe connection, I have now connected this to DOM Hub 1 / DOR 1. Also, connections of quads to DOR cards were changed and are now connected in the following manner: Q2 => Hub 3 / DOR 0 Q3 => Hub 1 / DOR 1 Q4 => Hub 1 / DOR 2 Q6 => Hub 1 / DOR 4 Q11 => Hub 3 / DOR 7 All other quad to DOM Hub connections remained the same. One additional note, a large release in pressure was noted on Paro 2, but not Paro 1 at approximately 15:35 NZ time on Feb. 1. HomePage | RecentChanges Edit text of this page | View other revisions Last edited 2005-02-11 12:42 UTC by Marc Hellwig (diff) Search: HomePage | RecentChanges LogJanuary January 2005 electronic log book -------------------------------- ______________________________________________________________________________ Mon Jan 31 10:16:32 UTC 2005 Jacobsen Called Jerry P. this AM on Iridium regarding DOR wire pair current on flashing DOMs. We observed currents in excess of 100 mA and were concerned enough to abort the run and called Jerry. Jerry felt that up to 120 mA was probably ok and is conferring by e-mail with the other experts. We went ahead and ran the flashers and measured that the current was about 104 mA except for a brief spike at startup < 1 sec at about 125 mA. I have let Jerry and the others know by e-mail what we are up to and asked them to get back to us ASAP if there is cause for concern. [Also see Feb. 1 entry] _____________________________________________________________________________ Mon Jan 31 05:45:21 GMT 2005 Karle Tested connectivity of all quads that are not currently connected to a Hub. All these quads tested positive on correct connection from patch panel to DOM. I used the DOM tester in the TCH. A green LED indicates correct connection for each pair. That means that these connectors may be penetrated by water but the electrical connectivity to DOMs is OK. ____________________________________________________________________ Mon Jan 31 02:28:20 UTC 2005 Blaufuss First entry in the [Wiki] electronic log book _______________________________________________________________________ Mon Jan 31 01:09:25 GMT 2005 [Krasberg?] flasherboard runs for "Lemur" finishing - too tired to continue flasherboard runs: ____________________________________________________________________ ____________________________________________________________________ Old log text (from amanda.spole.gov:~krasbema/log.txt): Sun Jan 30 23:32:55 GMT 2005 [Krasberg?] flasherboard runs starting... ____________________________________________________________________ Sun Jan 30 20:41:41 GMT 2005 [Krasberg?] tested flasherboard release in Madison - same failure as before - ie flashing DOM went into LED DC mode. new release to fix this problem from John J and this release is now being downloaded to the 16 DOM string and also to domhub 7. started flasher runs - confirmed with Jerry P that the current (105 mA) was OK for one wire pair running domcal - successful - strange that the DOM UP4Y0014 got a calibration voltage... it is at least partly working! have a sequence of flashboard runs preprepared. ____________________________________________________________________ Sun Jan 30 16:25:31 GMT 2005 [Krasberg?] one of the DOMs >UP4Y0014 | Wickueler appears to have a busted HV - rate is only 20Hz _____________________________________________________________________ Sun Jan 30 15:29:36 GMT 2005 [Krasberg?] release 308 has been downloaded to the icetop DOMs lcchain was run - all DOMs pass. ____________________________________________________________________ Sun Jan 30 14:32:15 UTC 2005 Jacobsen While the moat runs look good, there are still the GPS hiccoughs we saw in icetop and on jerry's doms... this will have to get addressed by Kalle. ____________________________________________________________________ Sun Jan 30 14:19:23 UTC 2005 Jacobsen This time 5 minutes on configboot and 5 minutes on iceboot firmware: moat -d 10 -t 300 -c 300 -s -r 0 -g /home/jacobsen/moat-runs/MOAT__2005-01-30__08:19:18 Test passed! ____________________________________________________________________ Sun Jan 30 14:14:17 UTC 2005 Jacobsen This time ran 60 seconds of moat versus configboot firmware. moat -d 10 -t 0 -c 60 -s -r 0 -g /home/jacobsen/moat-runs/MOAT__2005-01-30__08:14:12 Only 00B was flaky as before: 10 ReTXes?, test PASSED. ____________________________________________________________________ Sun Jan 30 14:12:25 UTC 2005 Jacobsen Ran 60 second moat test on all connected DOMs versus release firmware: moat -d 10 -t 60 -c 0 -s -r 0 -g /home/jacobsen/moat-runs/MOAT__2005-01-30__08:12:19 Passed! ____________________________________________________________________ Sun Jan 30 14:07:53 UTC 2005 JJ: While trying to test configboot comms I accidentally started a flash reprogram of each of the 16 DOMs attached. Not good. Fortunately ldall of release 308 succeeded with no problems. All 16 DOMs now have release 308 on them [session text removed]. _______________________________________________________________________________ 13:40 turning on the 4 ichub-cont01 quads above to chcek temps and rates... _______________________________________________________________________________ 13:35 downloading release 308 to all icetop doms [Krasberg?] _______________________________________________________________________________ 13:30 tests successful with icetop hub _______________________________________________________________________________ 11:20 UTC [Krasberg?] installed arthur's tiny daq am going to take an icetop run to test the code _______________________________________________________________________________ Jan 30 11:00 UTC [Krasberg?] more quads plugged in after leak testing (no change from yesterday) Q2 --> ichub01 dor 0 (same as before) Q4 --> ichub03 dor 2 Q5 --> ichub01 dor 3 Q6 --> ichub03 dor 4 Q7 --> ichub01 dor 5 (same as before) Q11 --> ichub01 dor 7 (this is not in the right spot, since it should be plugged into ichub02) ____________________________________________________________________ HomePage | RecentChanges Edit text of this page | View other revisions Last edited 2005-01-31 22:19 UTC by Gary Hill (diff)