prince.org staff blog

OK, so the BET awards last night caused a pretty significant influx of additional traffic to prince.org. Once we got 3.5x the usual traffic, things started to go downhill quickly. We ran out of memory, and then out of CPU. It’s not supposed to happen; we should be able to handle reasonable spikes in usage, but then again, we ain’t runnin’ Google here

Here’s what it looked like from a “load” perspective on our Linux server:

During normal days, we stay under 1.0, which means essentially, nothing the server is asked to do, waits… everything is serviced as fast as it is requested. Even a load of 2 or 3 can be manageable, but when our web traffic drives us this hard this quickly, we exhaust resources and go into a ‘death spiral’. It doesn’t happen often, but it happens when Prince is the center of major media attention (the previous time was the Super Bowl). So, even though it can be inconvenient, and I do apologize… I kind of wish there was a reason for Prince to make the site crash more

Heads-up, we’ll have a (hopefully!) short outage in the next 12-72 hours for database maintenance. I’m hoping to do it late tonight Pacific Time (i.e., around 11pm tonight) but it’s possible I might not get around to it until over the weekend. Don’t be alarmed, don’t be afraid… we’ll be back shortly when it happens!

And no, I won’t be putting the “crazy chatroom that broke in 10 minutes” thing up while the DB is down, like last time

Sorry for the downtime today — had to try and repair a piece of the database. While I was at it, I upgraded some other bits and pieces of the system software, which required a reboot… but we seem to be all back and happy now. Let me know if you see something amiss!

The org decided it needed a little nap this morning. Ok, well, actually, I upgraded some parts of the server software to keep up on the security and maintenance, and it borked the webserver such that it wouldn’t restart. I apologize, and all has been rectified now.

I’m going to take another shot at moving the .org to the faster server over the holiday, probably starting Sunday night. Expect some downtime! You’ve been warned

Thanks for your understanding!

Ben

Sorry, as usual, I forgot to take the necessary steps to ensure the chat would keep working on January 1st… I think this is the 5th year in a row that I’ve managed to neglect that! Sorry, it’s all working again now, and thanks very much to luv4u for reminding me!So, aside from that minor glitch, I certainly hope everyone is having a joyous New Year! Personally, 2008 wasn’t a bad year for me — but certainly was tough for a lot of folks. I’m definitely hoping it improves for everyone in ’09; fingers crossed!Ben 

Sorry for the unavailability this morning — turns out I (d’oh!) didn’t update the credit card on file at the hosting provider and they turned off the traffic. All fixed, paid in full, happiness again  

Whew. That was fun (not!)

My apologies for our outage for the last 5 days. Not that it was my fault, but I still feel bad about it. I know many of you have missed your “org fix” as one person wrote to me in email. In case you didn’t get the mail I sent out (some people seem not to have — reminder, check the email address the org has on file for you!), we were down starting last Saturday night, due to an electrical fire and subsuquent explosion (!) where our servers are hosted, in Texas, at theplanet.com. This was a catastrophic failure for their power systems, and while things like a normal power cut, would not have caused any problem, an explosion destroying much of their underground wiring, certainly did Over 9,000 servers were affected, approximately 7,000 customers, including, of course, our beloved org. Fortunately, no one was hurt in the fire and explosion!

The folks at theplanet.com worked round the clock for days to restore power, and finally restored it to all the servers about a day ago. The org wouldn’t come right back up, however, and after some manual intervention on their part, we were able to return to service about 6 hours ago. My thanks to the good folks at The Planet, and I really appreciate everyone’s patience while we waited for them to slog through the repairs. It’s great to be back!!

Two months ago when I started working on moving us to a new server, I had hoped to have everything done within a week. I spent about a week’s worth of effort on getting everything set up and installed, etc., and still wasn’t ready for the crucial last set of steps to actually move the entire site over. I’m going to be working on that some more this week, hopefully even tonight, and try to get us onto the faster equipment by the end of May.

Please understand if the site is down for a while, and it may also take time for your ISP to update their DNS cache (i.e., to point to the new site’s address). Best case, that takes only a few minutes, but worst case, a couple of days.

Anyhow, I’ll try to have meaningful messages posted up on the old site as I make progress.

Ben

All:
Sometime Sunday night (I’m not exactly sure when I’ll get around to this, but probably after 9PM Pacific time), I will be taking the org down for a little while for database maintenance, and one of the critical steps towards moving the site to new (faster!) server.  I expect the downtime to only be about an hour. I will try to have a “maintenance” page up during the process, as well.

If all goes well, within a week or so we should be all migrated to the new machine, however, which I think will be a noticeable speed improvement, so it’s all for a good cause!

Thanks for your understanding,
Ben