How one employee ruined millions of people’s day

Author: Shariq Syed

So this past Tuesday when you were browsing the internet, did you notice that some sites went down? Or maybe you were trying to access a PDF of a file which was hosted some website such as Piazza and weren’t able to? Well that error that caused drastic inconveniences for hundreds of thousands of users has been investigated and the reasoning has been announced.

Most of us know Amazon to be an electronic commerece store where we can order items and if we have prime membership we can get those items within a day if we would like. We also know they’re known for the futuristic ideas such as drone delivery services. But do we know that they also provide Cloud Infrasturure services? In fact they are the largest provider of IaaS. According to Amazon, “One of the key benefits of cloud computing is the opportunity to replace up-front capital infrastructure expenses with low variable costs that scale with your business.“ The saying with great power comes great responsibility is no small saying.

On Tuesday, February 28 2017, the internet hit pretty hard. The Amazon S3 team, Simple Storage Service, was working on a bug that increased the time to access the billing team. According to Amazon, “at 9:37 a.m. Pacific, starting the outage: “an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.” Before is a chart showing the most commonly affected areas that was impacted by the outage.

AWS Most Reported Issues
S3 EC2 Price Log-in
43% 37% 18%

In summary, one tiny little error caused hundreds and thousands of internet users to be without specific documents and websites. Not only did this have effect on people’s businesses it also had effect on day to day people. Imagine a business owner not being able to do what he does for a living because of a simple error. This accident translated over poorly in Amazon stock prices as the company’s stock dipped from $851 per share to $845.

Feburary 28, 2017 AMZN Stock Price
Opening Price Closing Price High Low Volume
$851.45 $845.45 $854.09 $842.05 2.79M

Yet in reality, these sort of errors happen all the time and aren’t something that people will obsess over. Do you remember the latest hack on a major company of the top of your head. Do you remember that millions of passwords were stolen and hackers were able to deplete some people bank accounts using that information? Probably not because we will forget it ever happened. Some may not forgive and others may try to make a statement by leaving certain memberships but ultimetly people will come back and won’t move back and forth between companies. In this case, Amazon is the largest IIaS provider which a pretty solid track record. Is it the lesser of two evils if people were to switch to the other competition? Probably not because Amazon is still the leading and most stable provider out there.

So next time something goes down on the internet, it may be because that someone, somewhere ended up making a tiny mistake that cost the company hundreds of thousands of dollars.


Author: Shariq Syed