2007-08 South Pole DOMHub Testing at SPTS/SPBS

last updated 8/29 16:00 CDT by mark

Instructions

SPBS Hub Testing Schedule/Results

HUB

Start of Tests

Results

Retest Desired?

Current Status

ichub60 8/24 mainboards 57 and 58 rebooted back to configboot (power glitch?) no This hub is not going to the pole (old revision hub)
ichub63 8/24 IP address wrong in /etc/hosts no tested OK
ichub70 8/27 no problems no tested OK
ichub77 8/28 mainboards 57 and 58 rebooted back to configboot (power glitch?)
later, DOR card #1 dropped out of data-taking
yes, need to check DOR card #1 fail
ichub76 8/28 IP address wrong in /etc/hosts
mainboard flash erased accidentally!
computer crashed at start of test cycle!
yes fail
ichub75 8/29 yellow GPS cable not plugged in to DOR card #7 no tested OK
ichub62 8/29 no problems no tested OK
ichub69 8/30 shortmoat repeatedly fails. After talking to John, moat was found to be failing to capture what the error was, so all we know is that shortmoat failed twice, but we don't know why. yes, and if shortmoat fails again John needs to debug the software fail
ichub68 8/30 no problems no tested OK
ichub61 8/31 green and yellow LEDs on front of display channel are reversed. we need to cut the tie wraps to make some slack to move the LEDs into the correct positions. no tested
ichub52 9/1 no problems no tested
ichub53 9/1 no problems no tested
ichub44 9/2 plastic case on DOM power suply was cracked and DOMHub was making an awful noise - Sebastian adjusted the plastic and noise went away no, but plastic power case for Acopian power supply needs to be checked; fan cover for left supply, bottom fan, was badly cracked at the bottom. Decision is to send supply to pole as is. tested
ichub54 9/2 no problems no tested
ichub45 9/3 no problems no tested
ithub05 9/4 sudo misconfiguration no tested OK
ithub06 9/4 no problems no tested OK
ichub41 9/5 initially, checkGPS claimed cards 4,5 and 7 were BAD
yellow cables were loose
no tested OK
ichub42 9/5 has new DSB fuse; plan is for all hubs from now on to be tested with DSBs which have the new fuse no tested OK
ichub31 9/6 when hub was moved for memory testing, the black plate covering the push button switches (reset buttons etc) on the front panel had somehow lodged over the buttons. The hub would not power up, and would send no video to a monitor which was hooked up to it. Eventually, the problem was discovered and rectified; also, sudo misconfiguration no tested
ichub32 9/6 after kickstart, 7 of 8 DOR cards had incorrect 101 version of firmware instead of 104q and dor-driver had to be reinstalled no tested
ichub33 9/6 DOR card 7 had loose yellow ethernet cable no tested
ichub34 9/7   no tested
ichub35 9/7 DSB (#109) acted like it had a blown fuse - replaced with DSB (#113)   tested OK
ichub76 retest 9/7 hub repeatedly crashes - all i had to do was run "domstate 01B" after powering up the DOMs and the hub would crash. When we hook up the 8 "DOMs in boxes", the same thing happens. Nothing appears on the console. Bug in dor-driver coupled with strange hub/DOR card? On investigation, John J suggests replacing DOR card 0, so we are trying this and then will retest the hub... yes fail
ichub69 retest 9/7 unable to reproduce shortmoat failure no tested OK
ichub77 retest 9/8 DOR card #1 problem not present no tested OK
ichub36 9/9   no tested OK
ichub37 9/10   no tested OK
ichub22 9/10   no tested
ichub76 retest 9/10 after DOR card 0 replacement, yellow cable was not plugged in all the way no tested
ichub60 (the real one) 9/11   no tested OK
new ichub51 9/11 old ichub51 was abandoned (wouldn't power up properly); DOR power connector was wired up backwards - caught during assembly by Matt and Will; Hub power module (left one?) failed a couple of times during power up in the rack - we aren't sure what happened; no DOM power until DOM power switch was turned off and on no tested OK
new ichub43 9/11 old ichub43 was abandoned (wouldn't power up properly)   tests ongoing
ichub23 9/12   no tested OK
ichub24 9/12   no tests ongoing

Other events of note


ichub43 does not power up - Paul Wisniewski (from The IceCube Project at the University of Wisconsin-Madison) thinks it could be hard drive problem; hub was replaced with a completely new hub
ichub51 does not power up - Paul Wisniewski (from The IceCube Project at the University of Wisconsin-Madison) thinks it could be hard drive problem; hub was replaced with a completely new hub

Brief instructions