My work day after a night of non-sleep went pretty well; I actually didn’t feel particularly awful. But man, I sure fell asleep fast and easy when I got back to the hotel.
So why have I been up since 5AM? Because the Zappos primary DB went down! Whee! That hasn’t happened in a very long time. But at around 1AM PDT the filesystem got corrupt — duplicate inodes. Damn you ReiserFS! So I assisted our excellent sysadmin Kris as best I could while he wrangled things back into order. Props to Richard for answering the call, and Eric for identifying the source of the problem.
In all fairness to ReiserFS, it’s never done this before and we’ve been hammering on it for years now. Everyone is entitled to a little mistake every few billion transactions I suppose.
Alright, see if I can get another hour of sleep…
crashy crashy
it’s funny – on of the worst crash we had at LE was due to a system problem too. someone had mistakenly unmounted a remote drive, we had chron jobs copying log files to that drive, there was no error handling so the log files just built up, disk ran out of space, db hung becaue it couldn’t create a new – um – forget what the exact file name was, but instead of crapping out and giving a useful error oracle just wouldn’t accept new transactions. Because someone unmounted a remote drive.
Mount it! Mount it now dammit!
Maybe I missed it but did you ever get to try the Zulu beer?