So we launched www.aminormalornot.com to the outside world today. We did a limited friends-and-family style launch 3 days ago in order to get some feedback and make sure there were no glaring bugs and in general, we got a lot of positive feedback. Given this, we decided to push it on the social news sites reddit and digg.
What we weren’t expecting was how quickly our relatively cheap and wimpy server got crushed under the load. I think getting swamped by a massive surge of traffic is every little startup’s dream but it was a bit pathetic when it happened with only a few hits a second.
After looking into the matter, it turns out that the 256MB of RAM on our VPS was much too small for serving a Rails app with the configuration we had. What was happening was that each new request was spawning a new apache and ruby process which would push an old request out to disk. When the old request started up again, it had to pull it back into memory and then push the other request to disk. We spent roughly 0.3% of our CPU time actually serving files and 99.7% shuttling files back and forth.
After upgrading to a larger slice, we’re back up and serving requests again at a tolerable pace. We’re about 10 minutes into getting the server back up and we’re already at 340MB of RAM so I’m going to keep an eye on this and see what we can do about reducing memory usage.
What are the lessons learnt from this? I don’t quite know. I’m sure there’s a huge amount of inefficiency we could wring out of the code that would decrease memory usage dramatically. But I still think to prematurely optimize this would have been foolish. The chances that we we’re actually going to be a big hit with this site is still low. We managed to get this site from concept to first alpha in 10 hours and from alpha to ready-to-release in 4 days. The only reason why we decided to play around with this was because we knew the cost in time for doing it was so low. Now that we are starting to get traffic, we’ll look into optimizing some stuff.
Also, much props to our host, Slicehost for their prompt support and super easy admin tools which helped us get up and running as soon as possible.
More stuff to come as I get to grips with this…