Brokekid.net

Did you know? According to Google, here’s what happened yesterday:

[Monday] morning (Pacific Time) we took a small fraction of Gmail’s servers offline to perform routine upgrades…However, as we now know, we had slightly underestimated the load which some recent changes placed on the request routers – servers which direct web queries to the appropriate Gmail server for response.

At about 12:30 pm Pacific a few of the request routers became overloaded and in effect told the rest of the system "stop sending us traffic, we’re too slow!" This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

As a result, people couldn’t access Gmail via the web interface because their requests couldn’t be routed to a Gmail server. IMAP/POP access and mail processing continued to work normally because these requests don’t use the same routers.

The engineers were quickly alerted and moved to bring additional request routers online. Google is working to make sure this type of incident doesn’t happen again, for example, by making sure there are more than enough request routers to handle demand and that when they start to become overloaded, they don’t just stop handling traffic be degrade gracefully.