In fact it was a design flaw, though not in the server.
The code for checking whether a username was taken was inefficient, so it was taking a huge amount of time to create a new account. The culprit was this line:
The reason the TechCrunch post slowed the site down so much was not the traffic per se but the number of those visitors who were creating new accounts. I pasted a fix into the repl, so the site should be faster now.
Thanks to Rtm for narrowing the problem down to account creation.
The reason [the site was slow] was not the traffic...
but the number of visitors who were creating new accounts.
The only better startup problem than that is "the site was slow because so many people were trying to give us money that our credit card processor crashed".
My friend Mike once told me: "When you can't tell the difference between public opinion and a DDoS, you're either doing something right, or doing something horribly wrong."
Any chance you could fix a bug in the header handling as well? It's not accepting "Content-length" headers (lower case l) which is perfectly acceptable HTTP and happens to be what w3m produces. Means I can't post comments from w3m.
What type of tests do you guys maintain for news.yc other than the live deployment? Did this hiccup result in something like an account creation stress test being added to an automated test suite?
Django has a shell that automatically loads all model objects. If you've got a decent shared-nothing architecture and point your config to the live DB (bad idea, but for the sake of argument...) that's effectively a REPL for the webserver.
For that matter, you can get the same effect with the MySQL prompt and a shared-nothing PHP architecture.
The code for checking whether a username was taken was inefficient, so it was taking a huge amount of time to create a new account. The culprit was this line:
The reason the TechCrunch post slowed the site down so much was not the traffic per se but the number of those visitors who were creating new accounts. I pasted a fix into the repl, so the site should be faster now.Thanks to Rtm for narrowing the problem down to account creation.