> 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches within the affected Availability Zone. Connectivity and power to other data centers within the affected Availability Zone, or other Availability Zones within the US-EAST-1 Region are not affected by this issue, but we would recommend failing away from the affected Availability Zone (USE1-AZ4) if you are able to do so. We continue to work to address the issue and restore power within the affected data center.

So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain functions", so I guess it's not enough to handle full load while generators spin up if the 2 primaries go out. Or perhaps some failure in the source switching equipment (typically called a "static transfer switch").

Some detail on different approaches: https://www.donwil.com/wp-content/uploads/white-papers/Using...

If they claim tier4, then they basically have everything in n+n configuration.

I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated which one was hot). A major power outage hit, and there was a thunderous roar as all of the generators fired up. The channel never went off the air.

Our main computer lab had a serial UPS that was online 100% of the time, though the inverters where under a very light load. If the mains even acted 'weird' (dips, bad power factor, spikes) the UPS jumped full on, and didn't revert to main power until the main was stable for some duration of time. The UPS was able to carry the full lab (which was quite large) for about two hours, allowing plenty of time for the generator to fire up.

The UPS ran a lot, and because the main was 'weird', the outages were often short, the generator wouldn't even start during the first ten minutes of UPS coverage. Of course, the rest of the building would be dark, other then emergency lighting.

I was a embedded firmware engineer, and our development lab was directly on the wall behind the UPS. When it fired into 100% mode, it roared, mostly from cooling. It was sort of a heads up that the power was likely to fail soon.

Why in the world would the state need to be involved in this level of decision?

A few minutes seems correct for one place I worked.

This was back in the 90's, before UPS technology got really interesting. Our system was two large rooms with racks and racks and racks of car batteries wired together. When the power went out, the batteries took over until the diesel generator could come online.

I saw it work during several hurricanes and other flood events.

I always found the idea of running an entire building off of car batteries amusing. The engineers didn't share my mirth.

The "when" shouldn't really matter- Diesel engines aren't a new thing. Warming them up isn't really a thing either- they'll have electric warmers hooked up to the building power to keep them ready to go.

Edit to add: I was at a place that took over a company that had one of these. With all of the dead batteries, it was just a really really large inverter taking the 3-phase AC to DC back to AC with a really nice and clean sine wave.

I wonder if a lot of AWS dc design in this area predates the battery grid storage revolution with (what my impression is) a far faster adaptation/switchover time than a generator spin up, and possibly software systems that work to detect and switch over quickly?

AWS can claim it will be best of breed, but they aren't going to throw out a DC power redundancy investment (or threaten downtime) that they can't wring more ROI on.

Tesla apparently did some early pilot stuff: https://www.datacenterdynamics.com/en/analysis/teslas-powerp...

Maybe it could affect people buying services as well.

We're still waiting on the RCA for last week's us-west outage...

"almost unusable" is maybe exaggerating, but there were definitely issues affecting more than just the single AZ.

Yes, some of these we should be better at handling ourselves, but... it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands.

edit: just to short circuit any "well, why aren't you running redundant regions" - we run redundant regions at all times. But for reasons of latency, many customers will bind to their closest region, and the nature of our technology is highly location-bound It is not possible for us to move active sessions to an alternate region. So something like this is... unpleasant.

> it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands

"Expect to lose an AZ" includes not being able to make any changes to existing instances in the affected AZ.

If you had instances across multiple AZs behind an ELB with health checks, then the ELB should automatically remove the affected instances.

If you have a different architecture, you would want to: * Have another mechanism that automatically stops sending traffic to impaired instances (ideal), or * Have a means to manually remove the instances from service without being able to interact with or modify those instances in any way

Does that help, or have I misunderstood your problem?

Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails?

IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between?

Deploying to multiple places is more expensive, it's not wrong to choose not to, it's trading off reliability for cost.

It's also unclear to me how often things fail in a way that actually only affect one AZ, but I haven't seen any good statistics either way on that one.

It's turtles all the way down, and underneath all the turtles is us-east-1.

Off the top of my head, US-EAST-1 is:

(1) topologically closer to certain customers than other regions (this applies to all regions for different customers),

(2) consistently in the first set of regions to get new features,

(3) usually in the lowest price tier for features whose pricing varies by region,

(4) where certain global (notionally region agnostic) services are effectively hosted and certain interactions with them in region-specific services need to be done.

#4 is a unique feature of US-East-1, #2-#3 are factors in region selection that can also favor other regions, e.g., for users in the West US, US-West-2 beats US-West-1 on them, and is why some users topologically closer to US-West-1 favor US-West-2.

For example, in railway signaling, drivers are trained to interpret a signal with no light as the most restrictive aspect (e.g. "danger"). That way, any failure of a bulb in a colored light signal, or a failure of the signal as a whole, results in a safe outcome (albeit that the train might be delayed while the driver calls up the signaler).

Or, in another example from the railways, the air brake system on a train is configured such that a loss of air pressure causes emergency brake activation.

Fail-safe doesn't mean "able to continue operation in the presence of failures"; it means "systematically safe in the presence of failure".

Systems which require "liveness" (e.g. fly-by-wire for a relaxed stability aircraft) need different safety mechanisms because failure of the control law is never safe.

And even then, you still need to define "safe". Imagine a lock powered by an electromagnet. What happens if you lose power?

The safety-first approach is almost always for the unpowered lock to default to the open state — allow people to escape in case of emergency.

Conversely, the security-first approach is to keep the door locked — nothing goes in or out until the situation is under control.

A more complex solution is to design the lock to be bistable. During operating hours when the door is unlocked, failure keeps it unlocked. Outside operating hours, when the door is set to locked, it stays locked.

The common factor with all these scenarios is that you have a failure mode (power outage), and a design for how the system ensures a reasonable outcome in the face of said failure.

* Negative temperature coefficient of reactivity: as temperature increases, the neutron flux is reduced, which both makes it more controllable, and tends to prevent runaway reactions.

* Negative void coefficient of reactivity: as voids (steam pockets) increase, the neutron is reduced.

* Control rods constructed solely of neutron adsorbent. The RBMK reactor (Chernobyl) in particular used graphite followers (tips), which _increased_ reactivity initially when being lowered.

It's also worth noting that nuclear reactors are designed to be operated within certain limits. The RBMK reactor would have been fine had it been operated as designed.

Source: was a nuclear reactor operator on a submarine.

e.g. consider a railway track circuit - this is the way that a signaling system knows whether a particular block of a track is occupied by a train or not. The wheels and axle are conductive so you can measure this electrically by determining whether there's a circuit between the rails or not.

The naive way to do this would be to say something like "OK, we'll apply a voltage to one rail, and if we see a current flowing between the rails we'll say the block is occupied." This is not fail-safe. Say the rail has a small break, or if power is interrupted: no current will flow, so the track always looks unoccupied even if there's a train.

The better way is to say "We'll apply a voltage to one rail, but we'll have the rails connected together in a circuit during normal operation. That will energize a relay which will cause the track to indicate clear. If a train is on the track, then we'll get a short circuit, which will cause the relay to de-energize, indicating the track is occupied."

If the power fails, it shows the track occupied because the relay opens. If the rail develops a crack, the circuit opens, again causing the relay to open and indicate the track is occupied. If the relay fails, then as long as it fails open (which is the predominant failure mode of relays) the track is also indicated as occupied.

"fail-safe" doesn't mean "doesn't fail", it means that the failure mode chooses false negatives or false positives (depending on the context) to be on the safe side.

Or you ask if it's a lesson about how real systems operate? Because yes, it's a very serious lesson about how real systems operate.

Anyway, you seem out of grasp on system engineering. Your reply downthread isn't applicable (of course fail-safes can fail, anything can fail). If you want to learn more on this area (not everybody wants, and its ok), following that link of system theory books on the wiki may be a good idea. Or maybe start at the root:

https://en.wikipedia.org/wiki/Systems_theory

Notice that there is a huge amount of handwaving in system engineering. I don't think this is good, but I don't think it's avoidable either.

In my experience, you can be specific, but then you get the problem that people think that if they just 'what if' a narrow solution to the particular problem you're presenting they've invalidated the example, when the point was 1. that this is a representative problem, not this specific problem and 2. in real life you don't get a big arrow pointing at the exact problem 3. in real life you don't have one of these problems, your entire system is made out of these problems, because you can't help but have them, and 4. availability bias: the fact that I'm pointing an arrow at this problem for demonstration purposes makes it very easy to see, but in real life, you wouldn't have a guarantee that the problem you see is the most important one.

There's a certain mindset that can only be acquired through experience. Then you can talk systems engineering to other systems engineers and it makes sense. But prior to that it just sounds like people making excuses or telling silly stories or something.

"(of course fail-safes can fail, anything can fail)"

Another way to think of it is the correlation between failure. In principle, you want all your failures to be uncorrelated, so you can do analysis assuming they're all independent events, which means you can use high school statistics on them. Unfortunately, in real life there's a long tail (but a completely real tail) of correlation you can't get rid of. If nothing else, things are physically correlated by virtue of existing in the same physical location... if a server catches fire, you're going to experience all sorts of highly correlated failures in that location. And "just don't let things catch fire" isn't terribly practical, unfortunately.

Which reiterates the theme that in real life, you generally have very incomplete data to be operating on. I don't have a machine that I can take into my data center and point at my servers and get a "fire will start in this server in 89 hours" readout. I don't get a heads up that the world's largest DDOS is about to be fired at my system in ten minutes. I don't get a heads up that a catastrophic security vulnerability is about to come out in the largest logging library for the largest language and I'm going to have a never-before-seen random rolling restart on half the services in my company with who knows what consequences. All the little sample problems I can give in order to demonstrate systems engineering problems imply a degree of visibility you don't get in real life.

It conveys reality, that "fail-safe" isn't literal, as if anyone believed that.

Fail safes do fail. Often due to severe user error.

I'm not sure why this is a big deal though, this is why Amazon has multiple AZ's. If your in one AZ, you take your chances.

IMO it was handled rather well and fast by AWS... not saying we shouldn't beat them up (for a discount) but being honest this wasn't that bad.

I once had to prepare for a total blackout scenario in a datacenter because there was a fault in the power supply system that required bypassing major systems to fix. Had some mistake or fault happened during those critical moments, all power would've been lost.

Well-designed redundancy makes high-impact incidents less likely, but you're not immune to Murphy's law.

It's turtles all the way down.

For example, too often people will set up clustered databases and whatnot because "they need HA" without much thought about all the other potential effects of using a cluster, such as much more complicated recovery scenarios.

In the vast majority of cases, an active-passive replicated database with manual failover is likely to have fewer pitfalls and gives you the same operational HA a clustered database would, even though in the case of a (rare) real failure it would not automatically recover like a cluster might.

Interesting quote:

“This is exactly the sort of design that lets me sleep like a baby,” said DeSantis. “And indeed, this new design is getting even better availability” – better than “seven nines” or 99.99999 percent uptime, DeSantis said.

1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power in them.

I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the wrong one and took both power circuits down at the same time.

However, a power cut (in the traditional sense where the supplier has a failure so nothing comes in over the wire) should have literally zero effect!

What am I missing?

I've never worked anywhere with Amazon's budget so why are they not handling this? Is it more than just the imcoming supply being down?

Nothing happens if you remember that your new capacity limit per DC supply is 50% of the actual limit, and you're 100% confident that either of your supplies can seamlessly handle their load suddenly increasing by 100%.

I've seen more than one failure in a DC where they wired it up as you described, had a whole power side fail, followed by the other side promptly also failing because it couldn't handle the sudden new load placed on it.

Normally this is factored into the Rack you buy from a hardware provider, they will tell you that you have 10A or 16A on each feed, if you exceed that: it will work, but you are overloading their feed and they might complain about it.

Obviously people can operate things however they want, but you wont get a tier 3 classification with that setup.

However, with their comment DC == Data Center, not Direct Current.

DC = Datacenter? makes no sense, so my head replaced it with "Power Supply" instead of "DC Supply", second sentence does make sense as being datacenter though.

This is all local scale. Your setup would not survive a data center scale power outage. At scale power outages are datacenter scale.

Data centers lose supply lines. They lose transformers. Sometimes they lose primary feed and secondary feed at the same time. Automatic transfer switches cannot be tested periodically i.e. they are typically tested once. Testing them is not "fire up a generator and see if we can draw from it"

It is cheaper to design a system that must be up which accounts for a data center being totally down and a portion of the system being totally unavailable than to add more datacenter mitigations.

And to top it off each rack had its own smaller UPS at the bottom and top, fed off both rails, and each server was fed from both.

We never had a power issue there; in fact SDGE would ask them to throw to the generators during potential brown-out conditions.

Of course this was a datacenter that was a former General Atomics setup iirc ...

Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in?

Again, this is all pure speculation: I have absolutely no idea of the exact failure, nor how their infrastructure is held together - this is all just speculation for the hell of it :)

That's ATS. It is not really advisable to test their under load performance because the failure of an ATS would be catastrophic. ATS typically would be tested at the installation and after that their parameters would be monitored.

Replacing a functional in line ATS would be a 9-12 months long project.

> Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in?

At high energy you are pretty much always going to use an ATS.

Because that would mean no power at all to the DC and no way to get it back? (I am completely ignorant on this topic)

While most of smarts in the ATS are in the electronics, the really nasty failures come from the mechanical part.

At the end of the day a high energy ATS looks just like a switch behind a meter in your house. There's a lip that goes from one position to another, except in a high energy ATS the lip is big and when the transfer occurs it slams from one source to another.

There are only so many of those physical slams that it can withstand to being with so you want to minimize that number.

The second failure mode is that after transfer to non-main source, the lip can get stuck there, making it impossible to switch back on the main. [Once I have seem the lip melt into the secondary position. While I thought it was weird, the guys from the power company said it is not that uncommon.] This creates a massive problem as the non-main source is typically not designed for long term 24x7 operation. So now you are stuck on a secondary feeding system and you cant just transfer to main without de-energizing the system i.e. taking the power out of the entire data center.

I've had bad power supplies fry out taking the whole power circuit with it, and thus half (or whatever fraction) of the rack's power. I've also had bad power supplies bring down the whole machine as they shunted everything internal too.

When things go bad, anything can happen. You can provide the best effort, and it'll usually work as expected, but there will always be something that can overcome your best efforts.

Citation needed - the same issue with testing, data races and expensive bandwidth come up.

Software is much easier than hardware. If you are to start a project today in this kind of hardware, you will be operating it in 2029, without changes.

I don't think this makes sense, you are using the three statements "Software is cheaper", "Software takes less time" and "software is easier" as if they all mean the same thing, and proving one means proving all of them.

Hardware takes a long time, okay, that does not mean it's expensive. Building a hydroelectric dam takes 20 years, but it provides the cheapest source of electricity that ever existed. Ships can take a decade from order to delivery, they are the cheapest mode of transport.

For big DC workloads, it is usually, though not always, better to take the higher failure rate than add redundancy.

Actually, now that I type that it makes sense. Scaling a few tens of dollars to a bajillion servers on the off-chance that you get an inbound power failure (quite rare I'd reckon) might cost more than what they'd lose if it does actually fail.

So yeah, they're potentially just balancing the risk here and minimising cost on the hardware.

Edit: changed grammar a bit.

More modular and a lot less copper at 10x the voltage. Still a lot of copper.

Perhaps we are going to discover how AWS produces such lofty margins by way of their next RCA publication.

My guess is that they cheaped out in having redundant PSUs to get you to use multiple availability zones. (More zones = more revenue)

Even a single PSU shouldn’t be an issue if they plugged in an ATS switch though.

Somewhat surprising to see how many things are failing though, which implies, either that a lot of services aren't able to fail-over to a different availability zone, or there is something else going wrong.

[0] https://news.ycombinator.com/item?id=29648992

Specifically large parts of the management API, and IAM service are seemingly centrally hosted in us-east-1. So called Global endpoints are also dependent on us-east-1 and parts of AWS' internal event queues (eg. event bridge triggers)

If your infrastructure is static you'll largely avoid the fallout, but if you rely on API calls or dynamically created resources you can get caught in the blast regardless of region

The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Related to that, I have been critical of lambda/FaaS/serverless infrastructure patterns for similar reasons. Always felt like a brittle house of cards to me (N.B. I do still use aws lambda, but keep it constrained to non-critical workloads).

Agreed; however, this is somewhat difficult to do correctly. There are all sorts of systems that might have hidden dependencies on managed services. e.g. AWS IAM roles will almost always be checked at some point if your services need to interact with AWS managed services.

I think cloud providers could meet developers half way here, by providing ways to reduce API usage; but I'm not sure if it aligns with their incentives.

Sometimes this _can_ be costly. For example with something like autoscaling, thats an active system I've seen fail when seemingly unrelated systems are failing. The result is scaling out systems intentionally ahead of time to deal with oversubscription or burst traffic which can leave you with (costly) idle compute.

I don't mind this tradeoff personally, but can understand that budget constraints are going to be different org to org.

It's like the duality of modular code. If you want to manage one change in a lot of places, it's easiest to change it in the one module that everything else sources. But that means that one change to that module can take down everything. The alternative where you copy+paste the same change everywhere is the most resilient to failure, but also the most difficult and expensive.

AWS provides a lot of modular, dynamic things because that's what their customers want to use. But using each of those things increases the probability of failure. It's up to the customer to decide how they want to design their system using the components available.... and the customers always chose the easy path rather than the resilient path.

The great thing is that with AWS, at least you have the option to design a super freaking reliable system. But ultimately there's no way to make it easy, short of a sort of "Heroku for super reliable systems". (I know there are a few, but I don't know anything about them)

I've managed to stick to EC2/ELB and S3 as passive systems for the vast majority of what we build at my org (~90% of our stack). And for the most part, AWS failures are hitless for us as a result.

One incident I recall was involving our GCP regional storage buckets, which we were using to achieve mutli-region redundancy. One day, both regions went down simultaneously. Google told us that the data was safe but the control plane and API for the service is global. Now I always wonder when I read about MR what that actually means...

Your point here deserves highlighting. A failure such as a zone failing is nowadays a relatively simple problem to have. But cloud services do have bugs, internal limits or partial failures that are much more complex. They often require support assistance, which is where the expertise of their staff comes into play. Having a single provider that you know well and trust is better than having multiple providers where you need to keep track of disparate issues.

You really need multi-region and also not be relying on any AWS service that’s located only in us-east-1 (including everything from creating new S3 buckets to IAM’s STS).

edit: The question isn't necessarily AWS specific, just any data on amount of downtime per cloud provider on a timeline would be nice.

There indeed has been an uptick in AWS outages recently. You can see a bit of the history here: https://statusgator.com/services/amazon-web-services

If you can’t even admit you’re having an issue how can you keep an accurate record?

This is what happens when we've been affected by outages (even without involving support).

Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZs, typically by just dropping nodes in the down AZ.

People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down.

Isn't that because Elasticache will distribute the cluster across AZs automatically?

https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...

Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or cross cloud which is quite more complex to handle and require more resources to do well.

And I hope that nobody is listening your blaming and pointing fingers advice, that's the worst way to solve anything.

It's AWS job to ensure that things are reliable, that there is redundancy and that multi-AZ infra should be safe enough. The amount of issues in US-EAST-1 lately is really worrying.

I do agree that the end of this year has been a very bad period for AWS. I wonder whether there’s a connection to the pandemic conditions and the current job market – it feels like a lot of teams are running without much slack capacity, which could lead to both mistakes and longer recovery times.

On our side we saw some EC2 VM totally disconnected from the network in 3 AZs.

In the past I've seen both of those systems seamlessly handle an AZ failure. Today was different.

Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. These entire datacenter failures should be anticipated, but to expect them is a bit too easy I think. There are plenty of hosters who don't have this stuff even once for the last decade in their only datacenter. I do not find it strange to expect or even demand that level but to protect yourself if it happens in any case if that fits your specific project and budget.

Edit; OK meant that raid-5 remark in the same context as the hosting; it can and does happen but it shouldn't; you should plan for a contingency but expect it goes far. We never had it (1000s of hard-drive, decades of hosting, millions of sites) and so we plan for it with backups; if it happens it will take some downtime but it costs next to nothing over time to do that. If we expected it, we would need to take far different measures. And we had less downtime in a decade than aws AZ had in the past months. I have a problem with the word 'expect'.

There are plenty of situations where this might happen if they’re in your room: a lightning strike can cause a surge that causes the disks to fry, a thief might break in and steal your system, your house might burn down, an earthquake could cause your disks to crash, a flood could destroy the machines, and a sinkhole could open up and swallow your house. You may laugh at some of these as being improbable, but I have seen _all_ of these take out systems between my times in Florida (lightning, thief, sinkhole, and flood) and California (earthquake and house fire).

The fix for this is the same fix as being proposed by the parent post - putting physical space between the two systems so if one place become unavailable you still have a backup.

Here are some examples where that happened:

1. Drive manufacturer had a hardware issue affecting a certain production batch, causing failures pretty reliably after a certain number of power-on hours. A friend learned the hard way that his request to have mixed drives in his RAID array wasn’t followed.

2. AC issues showed a problem with airflow, causing one row to get enough warmer that faults were happening faster than the RAID rebuild time.

3. UPS took out a couple racks by cycling power off and on repeatedly until the hardware failed.

No, these aren’t common but they were very hard to recover from because even if some of the drives were usable you couldn’t trust them. One interesting dynamic of the public clouds are that you tend to have better bounds on the maximum outage duration, which is an interesting trade off compared to several incidents I’ve seen where the downtime stretched into weeks due to replacement delays or manual rebuild processes.

Same manufacturer, same disk space, same location, same operator, same maintenance schedule, same legal jurisdiction, same planet, you name it, and there's a common failure to match

HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner.

All drives were purchased at the same time, and happened to all come from a single batch from the manufacture. The manufacture confirmed this via serial numbers, and admitted they had an issue during production. All drives were replaced and at a larger volume size.

TL;DR: Drives will fail, and manufacturing issues happend. Don't buy all of your drives in an array from the same batch! It will happen. To say it won't is just pure inexeperience.

The context of the parent seems to be that they intermittently couldn't get to the console. That seems fair to me. If we're blaming developers and finding gaps in HA design, then AWS should also figure out how to make the console url resilient. If it's not, then AWS does appear to be down.

I imagine it's pretty hard to design around these failures, because it's not always clear what to do. You would think, for example, that load balancers would work properly during this outage. They aren't. Or that you could deploy an Elasticache cluster to the remaining AZs. You can't. And I imagine the problems vary based on the AWS outage type.

Similarly, with the earlier broad us-east-1 outage, you couldn't update Route53 records. I don't think that was known beforehand by everyone that uses AWS. You can imagine changing DNS records might be useful during an outage.

As others have said, they are not being forthright about the severity of the issue, as is standard.

edit: of course, AWS does have this: AWS Fault Injection Simulator

Be in multiple AZs, and even multiple regions but if you're going to be in only one AZ or one region, make it us-east-2.

I have a server at OVH (not affiliated to them) which, at this point, I keep only for fun. It has 3162 days of uptime as I type this.

3 162 days. That's 8 years+ of uptime.

Does it have the traffic of Amazon? No.

Is it secure? Very likely not: it's running an old Debian version (Debian 7, which came out in, well, 2013).

It only has one port opened though, SSH. And with quite a hardened SSH setup at that.

I installed all the security patches I could install without rebooting it (so, yes, I know, this means I didn't install all the security patches for some required rebooting).

This server is, by now, a statement. It's about how stable Linux can be. It's about how amazingly stable Debian is. It's also about OVH: at times they had part of their datacenter burn (yup), at times they had full racks that had to be moved/disconnected. But somehow my server never got affected. It may have happened that at one point OVH had connectivity issues but my server went down.

I "gave back" many of my servers I didn't need anymore. But this one I keep just because...

I still use it, but only as an additional online/off-site backup where I send encrypted backups. It's not as if it gets zero use: I typically push backups to it daily.

They're only backups, they're encrypted. Even if my server is "owned" by some bad guys, the damage he could do is limited. Never seen anything suspicious on it though.

I like to do "silly" stuff like that. Like that one time I solve LCS35 by computing for about four years on commodity hardware at home.

I think it's about time I start to do some archeology on that server, to see what I can find. Apparently I installed Debian 7 on it in mid-october 2013.

I've created a temporary user account on it, which at times I've handle the password (before resetting it) to people just so they could SSH in and type: "uptime".

It is a thing of beauty.

Eight. Years. Of. Uptime.

Awesome! Are you Bernard Fabrot [0]?

[0] https://www.csail.mit.edu/news/programmers-solve-mits-20-yea...

At its current use, it's likely not a major issue but imagine if someone saw this uptime and thought to take it as a statement of reliability and built a service on it. I for one, would want that disclosed because this is a disaster waiting to happen. I'd much rather someone disclose that they had a few servers each with no longer than 7 days of uptime because they'd been fully imaged and cycled in that time...

In reality, most people don't need to scale. An occasional spike in traffic is a nuisance, but not the end of the world, and security is not terribly hard, if you keep your servers patched (which is trivial to automate).

I really don't understand why there's so much FUD around running your own stuff.

When your server requirements get into needing 5-6 servers (not at all atypical for a startup in their first year of being launched), running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount.

What challenges are you thinking of? You buy a full-rack in colocation and then just buy servers/hardware when required.

If a company has the budget for AWS or some other cloud provider then they would have a budget for colocation; which in long term is cheaper. I see no additional challenge other than maintaining X amount of hardware than just one.

Buying upfront hardware is not feasible even if I had the cash(which most don't), I don't know if the company would last that long or would be doing things that require x servers .

What you are saying is similar to saying may be it is cheaper to buy the building /floor instead of renting space for office. - most small biz cannot afford do that, or expect their business to change (fail/take off) in the time frame ROI would come to take that commitment.

This is all assuming that a the startup has skill in setting up and managing physical servers and there is no opportunity costs( delayed features) on doing so, both are not a given.

small companies ( and poor people) typically don't buy low quality stuff or buy into rent seeking business models because they are dumb it is usually because they cannot afford to do long term thinking.

You only need one server, slap on a hypervisor and your rocking. Heck you can buy entry servers from dell for budget; upgrade later.

A 10u rack, which is adequate for any small business comes to around $500 a month in LA. 4u would be more than enough for a startup and that's around $200/month.

Is the start-up not going to purchase computer hardware, monitors, television screens for their clients when they sit in the waiting room? Email accounts with Office365, a website, a domain name? If they can fit that in to their budget I am pretty sure they could afford a server and colocation space.

> What you are saying is similar to saying may be it is cheaper to buy the building /floor instead of renting space for office. - most small biz cannot afford do that, or expect their business to change (fail/take off) in the time frame ROI would come to take that commitment.

But colocation is dynamic. Contracts can be negotiated.

> buy into rent seeking business models

And AWS isn't a rent seeking business model?

Looking at EC2 instances, for 120GBHD, 32 cores "Dedicated" instance, your looking at 679.54 USD for a month. 120GB isn't much especially when the developers start doing their thing.

For $500 you can have so much more, and hardware you actually own and that if the company does not lift off it can then be sold. Is that not the better investment?

No remarks to lack of skills.

This comment just screams of engineer-only focus. Running servers yourself brings almost no value to the customer and is a specialized skill that you have to pay for. All to solve... a couple days of service-specific downtime a year? People need to chill out with their non-mission critical software. God forbid someone can't access their HR portal for an hour.

No. You rack the server, connect the cable to the switch press the on-switch, let it boot and then operate as you would with any other computer. Need a new server? Get the DCOps remote hands to Rack the server, cable it to the switch and press the on-switch. If you can install Windows 10, you can install Linux.

> This comment just screams of engineer-only focus. Running servers yourself brings almost no value to the customer and is a specialized skill that you have to pay for.

Hmm, the data is yours; You own the data and that's value to me if I am the customer. And there's no value to the customer if you were to host it in the cloud either, much of a loss retrospectively.

There is no specialized skill you have to pay for, sure if you were going to run a high-density compute mainframe running some specialized OS like AIX then yeah sure. To buy a server, install a OS and plug it in to a switch and navigate it with SSH requires no overqualified anything.

To your second point: We still own the data for hosted solutions. We have our backups, we have direct access. Barring a catastrophic failure and wiping of AWS, our data is there and ours. The value to our customers for using cloud providers is the time saved on building infrastructure is instead spent on delivering value to the customer in the form of features/bug fixes. And yes I know the argument of "you end up spending more time debugging AWS", but I think you don't need to reach that point if you keep things pretty simple, especially early stage.

I think you're vastly oversimplifying the task. And if I had to guess, it's something you're pretty familiar with so it makes sense that it's easy to you! I'm sure an early stage company would be happy to have you to save a ton of money in their early days on infrastructure costs

All which you have to do on a cloud provider. Fire/Power are normally handled by the DC. You have to have someone with knowledge in the first place to operate that in the cloud and taking an application such as HAProxy is on the same skill level. Especially with vast fields of blogs you can find on the topic.

The cloud brings the instant "power-up" methodology and I would compromise, you could be right. If you want a perfective optimum cloud platform you would need a dedicated "cloud" engineer if you want to ensure security, connectivity etc. etc. But if your going to do that then again you might as well move in-house and hire a system admin, you've still got to operate your companies infrastructure. It's a moot point.

> The value to our customers for using cloud providers is the time saved on building infrastructure is instead spent on delivering value to the customer in the form of features/bug fixes.

I suppose this is a mixed area and what customers value varies. For me I value a service that uses it's own hardware rather than cloud. On the basis that they are willing to put the skill in to operate their own infrastructure rather than.

> I think you're vastly oversimplifying the task.

I don't think so. People seem to think that setting up what you can in the cloud is impossible on bare metal when really it isn't. What did devOp's do before the cloud providers? Amazon,Azure,X have only lassoed FOSS software, constructed a webGUI-admin panel and throw it as a service.

This is not to say Cloud doesn't has a purpose, otherwise it wouldn't exist today.

> it's something you're pretty familiar with so it makes sense that it's easy to you

While true, I won't disagree, I've been working in the SysAdmin field since 2009. But disagree as I started with very little knowledge and gained it through setting up such infrastructures. I've educated a few and those with very little knowledge of servers could setup an infrastructure that companies run in AWS.

Of course it doesn't. Why are you asking antagonistic questions?

I don't think people care that AWS has other customers, they want their workload to work, if it doesn't: then that's a today issue.

Is this not antagonistic? It's pointless to make these statements, so your parent comment pointed it out. Go downvote the first one instead.

Put another way: even if your home ISP has had 100% uptime, are you comfortable saying that was true for all of their customers?

Better uptime than paying for EC2 on AWS US-East-1.

Obviously this approach isn't scalable but it serves me well.

It's perfectly scalable. Just give everybody their own home server.

“ditch your on-prem infrastructure and migrate to a major cloud provider”

And its starting to seem like it could be something like:

“ditch your on-prem infrastructure and spin up your own managed cloud”

This is probably untenable for larger orgs where convenience gets the blank check treatment, but for smaller operations that can’t realize that value at scale and are spooked by these outages, what are the alternatives?

A much faster and more effective solution that doesn't have you trading cloud problems with on-prem problems (the power outage still happens, except now it's your team that has to handle it) would be to update your services to run in multiple AZs and multiple regions.

Get out of AWS is you want, but don't get out of AWS because of outages. You should be able to mitigate this relatively easily.

Everything fails, we can argue the rate. But I would argue that understanding your constraints is better.

if you know that your secret storage system can't survive if a machine goes away: well, you wire redundant paths to the hardware and do memory mirroring and RAID the hell out of the disks. And if it fails you have a standby in place.

But if you use AWS Cognito.

And it goes down.

You're fucked mate.

I remember we had a power outage in 2006, it actually took one of my services off air. Since then of course that has been rectified, and the loss of a building wouldn't impact on any of the critical, essential or important services I provide.

Source? Has there ever been an industry wide survey that compares availability from "insert average colo/data center operations" with the cloud ones?

And I'm not talking about "we have 12 SREs who are based in Cupertino and are all paid top dollar to support a colo"...I'm talking average.

I worked through the ranks at a large enterprise that ran a “big” datacenter for a decade. The facilities team was about 6 people, average salary around $90k. I can only remember one power interruption affecting more than a rack, caused by a failure during a maintenance event that requires a shutdown for safety reasons. The rest is like any other industrial facility - you have service contracts for the equipment, etc and maintain things.

There’s a cost/capability curve that you need to plan around for these matters. You need to make business and engineering decisions based on your actual circumstances. If the answer is automatically “AWS <whatever>“, you’re making a decision to burn dollars for convenience.

Ok so $540k salaries + benefits, so ~$700k. Then you have transaction costs:

- Annual salary increases

- Any cost associated with people leaving (severance, hiring, recruiters, HR, HR systems)

- Systems that run in the data center (logging, monitoring, etc.)

- Procurement costs with changing costs in hardware (silicon shortages, etc.)

- Security compliance overhead and associated risks

- Finance resources required to capitalize and manage asset allocation

- etc. etc.

Versus

- Click a button and voila it works.

- Hire way less engineers to manage the system administrative portion

> If the answer is automatically “AWS <whatever>“, you’re making a decision to burn dollars for convenience.

100% AGREE. The answer is always "it depends", but just like people are saying "just put in the cloud", the opposite of "well it worked for us using a data center" isn't that simple.

Obviously, there’s a huge capital investment component too that has to be incorporated. Those costs may be really high if you’re in a growth phase as you need to overbuy capacity.

Just to be clear, I’m not arguing that on-prem is magically cheap. :) But it has its place too!

I’ve been deploying to AWS for years and can’t remember and outage on their side in my region. But this is anecdotal and doesn’t necessarily reflect the statistics.

It is as if the software industry has collectively forgotten how to run basic data center operations. Something that used to be a blue collar skill is now treated like arcane magic.

It's not arcane magic. It's undifferentiated toil that requires hiring for a different skill set than tech companies generally want to hire for. Of course when you get to a certain size it may make sense to take on this cost.

I want us to stop pretending individual actors lack the agency to make their own decisions, and they're all blind to how AWS is charging them a fortune for such simple things they can do themselves. You get value from AWS or you stop using AWS.

And what rate is this? It gets attention because it impacts more people, but AWS / GCP / Azure uptime is still better than what I've seen for small / mid size businesses trying to manage their own infrastructure.

So again, we're talking about cloud providers because of their scope and size, and they're still doing better than MidSizedBank managing their own infrastructure.

Commenters will show up like clockwork and say shit like:

“What man, it’s not like cars didn’t crash before? Haha”

Don’t be dense dude. And definitely don’t pursue a leadership position anytime in the future.

The problem with both this example, and the AWS one (it needs to have better availability than your personal home-spun solution, and it does), is that people are amazing at deluding themselves.

"Yes, cars are dangerous, because other people can't drive. But I'm a better than average driver"

"Yes, other people will build unreliable systems. But I know how to architect for my use case and ensure that for my needs the availability will be higher than AWS's"

Both are true* in the micro sense and false in the macro sense.

* Not really. 88% of americans think they are "above average" drivers.

If your app depends on a few 3rd party services -- SendGrid, Twilio, Okta and they're all hosted on different infra then congrats! You're gonna have issues when any one of them are down, yayyy.

Also the marketing benefit can't be downplayed. If your postmortem is "AWS was having issues" then your execs and customers just accept that as the cost of doing business because there's a built-in assumption that AWS, Azure, GCP are world class and any in-house team couldn't do it better.

In my experience, execs and customers don't treat an outage differently because AWS is at fault. Though the developers do often have the attitude that it's "someone else's problem", which can actually can make execs more worried than if the problem was well known and under the company's control.

Edit: Restarting Slack does update the edited messages.

Edit 15:24 CET: Slack is back up.

- edits failing or working with big lag;

- "Threads" view slow;

- can't emoji-react;

- can't upload images;

- people also say they can't join new channels.