I wanted to give everyone a status update on where we stand and the issues we've had the last couple of days.
If you haven't noticed, the server has been up and down for about 3 days now. There were a few corrupted tables in our MySQL database, which have been repaired (this was Wednesday night).
Yesterday the site was down from about 3:30pm CST until about 9pm. I was running some update scripts to update Apache and PHP. Then I had to tweak both again to get the load down. I haven't updated MySQL yet, but I might be doing that when time permits.
Since last night, the site seems to be more stable, but we'll be able to tell more as people get on today.
Page loads might be a little slower than you're used to. I haven't gotten xCache installed with the new version of PHP yet. I wanted to see how things ran without it before I tried to configure that. If things seem to level out, I will be installing this again.
Since last night, the server load has rarely gotten above 6, which is a lot better than its been. Search disables at 10 and you'll get the "Server too busy" message at 15.
Over the last couple of days we were running out of RAM and had high CPU usage, causing us to use the swap space (like 770mb in swap). When I logged in this morning it looks like we have used about 7mb in swap overnight, so that's a huge difference.
Basically, we're starting to outgrow this server (which is at a hosting company). Its a Quad Core Xeon processor with 4gb of RAM.
We have a couple options we will be looking at from this point on:
1) Keep this server and get a 2nd server for either Apache or MySQL.
2) Upgrade this server to Dual Xeon Quad Core processors and increase the RAM (8, 16, 32...whatever we can get).
Eventually I still want to get the FTP off of this server and put it on the cheaper one Mike still has.
But while we're looking at the other options, hopefully the changes I did last night will be enough to hold us over for another week or so.
Thanks again for all your patience while we go through our growing pains!
Update 10/17/09 1pm CST: See my post
here regarding the outage this morning.