My work day after a night of non-sleep went pretty well; I actually didn’t feel particularly awful. But man, I sure fell asleep fast and easy when I got back to the hotel.
So why have I been up since 5AM? Because the Zappos primary DB went down! Whee! That hasn’t happened in a very long time. But at around 1AM PDT the filesystem got corrupt — duplicate inodes. Damn you ReiserFS! So I assisted our excellent sysadmin Kris as best I could while he wrangled things back into order. Props to Richard for answering the call, and Eric for identifying the source of the problem.
In all fairness to ReiserFS, it’s never done this before and we’ve been hammering on it for years now. Everyone is entitled to a little mistake every few billion transactions I suppose.
Alright, see if I can get another hour of sleep…