Hacker News new | past | comments | ask | show | jobs | submit login
AWS: IPv4 addresses cost too much, so you’re going to pay (theregister.com)
155 points by penda on July 31, 2023 | hide | past | favorite | 185 comments



I know at least one company that, after complaining bitterly over the weekend, freed up more than 50% of their IPv4 addresses today after a quick audit and change.

Seeing something like that makes me think that AWS is completely justified in bumping the price on IPv4 addresses. People used IPv4 indiscriminately and didn't care because AWS ensured that their customers would always have enough addresses available.


Not exactly. Most of the AWS services you can't release the IPv4 addresses. You automagically get 3 IPv4 addresses assigned to you when you create a load-balancer, even if you want that load-balancer to be IPv6 only.

And their native support for IPv6 within their services are hit-and-miss at best.


Yeah, this is my only real complaint with this. Internally their IPv6 support is pretty garbage, which is kind of fine since your internals shouldn't be exposed externally anyways, so it's in theory possible to have an internal IPv4 network that exposes traffic over IPv6 to the public using something like a load balancer.

Load balancers are already somewhat expensive- the base cost for each load balancer is already $16.43 a month before bandwidth. Three IP addresses, 12 cents per day each, over 30 days is another $10.80 a month. In other words load balancers just had their base price increase by 65%.


Sounds like they will end up with some pressure to get IPv6 working correctly if any of their customers are capable of pressure.. If they don't get it working now then it is definitely time to run away from their platform while you can still afford the fees to get out.


I have started working for a startup recently. My main responsibility is to develop networking features for our cloud on bare metal. We started ipv6 by default but soon we discovered that the biggest issue is "not" the setup side. Ipv6 setup is actually quite straightforward, if you're starting from scratch. The biggest problem of ipv6 is that the ecosystem is not ready for it, at all. You cannot even use github without a proxy! Hence, we had to start implementing ipv4 support immediately, because VMs for developers that only has ipv6 is almost useless.


Github is one of the most idiotic IPv4 exclusive services. Microsoft and Azure has all the knowledge and equipment to make IPv6 available to practically any site, but Github seems afraid to ask. They had IPv6 for a short while and turned it off later.

https://github.com/orgs/community/discussions/10539 is full of people voicing their grievances but I don't think Github is paying this issue any attention anymore.

Luckily almost all providers or IPv6-only networks also offer NAT64 or similar NAT mechanisms to make IPv4 addresses reachable.


ZScaler is worse on a practical level. I have to disable IPv6 system-wide else I can't access internal services (it only routes IPv4). The crazy thing is that they call out VPNs for being archaic, but force users to use an even more archaic technology.


Bless you for mentioning that ZScaler doesn’t play nice with IPv6 - this gives me another possible cause for why a certain MSFT distributed app sometimes, but not always, installs on our users’ PCs when ZScaler is enabled, but works with it off.

Thanks so much!


Microsoft used to “get” IPv6 and they added support to Windows back in the very early 2000s.

All those engineers left or retired and have been replaced by outsourcers and H1Bs.

Nobody at Azure can even spell IPv6.

Turning it on is difficult and then it breaks everything, including unrelated services in other peered vnets.

It’s just shameful how much the engineering skill has degraded in Redmond.


As long as the stock goes up, no-fin-body cares.


Yeah, the situation is pretty awful for something so big and not lacking of technical talent as GitHub :(


> You cannot even use github without a proxy!

Luckily that does not seem to be an issue here. You only have to pay for a public IPv4 address, you still have a full IPv4 stack and are able to make outbound connections via NAT.


AWS won't give you a NAT without you paying for it (NAT Gateway) which requires a public IPv4 address (which you will soon have to pay for).

There ain't no such thing as a free lunch.


I don't think you need that based on https://aws.amazon.com/blogs/aws/let-your-ipv6-only-workload...

It's possible that the business model has changed, but at least on this blog they say it's available free of cost.


> Pricing and Availability > These two new capabilities to the VPC NAT gateway and Route 53 are available today in all AWS Regions at no additional costs. Regular NAT gateway charges may apply.

Specifically:

> Regular NAT gateway charges may apply.

Which should say: "Regular NAT gateway charges apply"

Since you still pay for the traffic that is processed by the NAT gateway (per GiB).


That article makes it clear that the NAT Gateway needs IPv4

> The NAT gateway recognizes the IPv6 address prefix, extracts the IPv4 address from it, and initiates an IPv4 connection to the destination. As usual, the source IPv4 address is the IPv4 address of the NAT gateway itself.

If the NAT Gateway doesn't have an IPv4 address it won't work.


Not free of cost, free of additional cost if you were already using a NAT Gateway before they added NAT64 capability.


I recently tried to deploy GitLab from scratch on an IPv6-only network, and the initial experience was anything but smooth. I was met with an exception right in the console during the initial setup. GitLab attempted to obtain a Let's Encrypt certificate and immediately failed, as it doesn't listen to IPv6 addresses by default. A year ago, we (at work) faced similar issues when trying to deploy GlusterFS on an IPv6-only network, and it also failed. (I pushed for V6 only, my manager was not happy) It's evident that while IPv6 may be the future, the present ecosystem doesn't seem fully prepared to support it. For years, I have wanted to use Docker with IPv6 only, and I am really thinking about learning Go so I can write my own IPv6-only driver.


I’m getting python 2 v 3 flashbacks


Yeah, it's a real shit show when you get down to actually trying to utilize IPv6 in any scenario that needs legacy IPv4 access in a straight-forward way.

I'm somewhat happy in that I've moved away from being way down at the low-level ISP/network side of things, so I may be missing something, but I don't see how we are ever going to elegantly transition away from IPv4 addresses. Everything just seems hacky and fragile in terms of trying to run a "pure" IPv6 environment, and be connected to the rest of the Internet.


I think that ISP-wide single-stack IPv6 deployments are the key. They throw the legacy IPv4 internet behind a huge NAT, while letting IPv6 internet to function natively. There is an IPv6 address range that represents the totality of the legacy IPv4 addresses, and the IPv4 addresses are translated into that, so from the IPv6 side of the NAT, every IPv6 service looks like it has an IPv6 address. From the IPv4 side, it looks like the your standard carrier-grade NAT, with huge numbers of users sharing IPv4 addresses.

That should simplify the network over dual-stack deploys, plus it makes providing services in "native" IPv6 the more attractive choice over NATted-to-death IPv4.

There are already some ISPs doing this. In Japan, where I live, one of the big three, NTT Docomo transitioned into a deployment like this just last year.


The protocol is called MAP and it is nearly stateless, cause the IPv4 address is in the IPv6 packet, and more efficient than carrier-grade NAT. The downside is that requires special gateway on customer side. The advantage is that ISP can have IPv6 internal network with IPv4 on the edges.


Oh, I was talking about NAT64/DNS64 which is what NTT Docomo is doing. Now I got to read about MAP... But how can the mapping be almost stateless? When packets are sent from IPv6 to IPv4, if there is significant exhaustion, multiple IPv6 address are going to map into a single IPv4 address, unless the ISP has enough for everybody, in which case I don't see the motivation to hasten moving to IPv6?

Edit: Ah, uses port numbers as extra bytes to encode the original senders address information. Smart!


Exactly this. As more ISPs go through this process, service providers are incentivised to provide v6 services or get poor service. As things continue in that pattern, v6 becomes dominant and v4 becomes the afterthought to be neglected on the sideline (like v6 is currently for many ISPs with lots of v4 resources)


I would be surprised if gitlab didn't support ipv6.

Some of the ecosystem must be ready for it, and ipv6 support can be just another requirement to choose among solutions.

Also, you can have a reverse proxy and a cloud behind NAT64 to run servers on ipv4, but access them with ipv6.


GitLab supports ipv6, but just not out of the box. My private gitlab instance is v6 only. I had problems with the updates in the beginning because because the official repo was not on V6 but I think they fixed it some years ago. (I am not gitlab expert, but i think my type of install is called Omnibus)


Do "dig -t AAAA github.com": you'll see they have no AAAA records. No github for you. Why then should anyone else bother unless forced by cost?


They were talking about GitLab in comparison to GitHub, not talking about GitHub. GitLab has AAAA records.


Is/was DNS64+NAT64 a viable way option for your use case?


AWS NAT Gateway comes with native NAT64 support so GitHub just works over ipv6. No issues whatsoever.


Cloudflare has supported a free IPv4 to IPv6 gateway for IPv6-only web servers since 2011: https://blog.cloudflare.com/introducing-cloudflares-automati...

If you need more than web traffic, you can use our Tunnel service.


Back when I had Comcast (and thus native IPv6 at home), this was a great way to expose a web server at home without resorting to either weird port forwarding or setting up a proxy + SNI. Both of those work, but this is super clean.

(Now I only have IPv4, so I just use Tunnel).


Even using something like Hurricane Electric, this is still a nice solution to get access to services hosted on a residential connection. Feels a lot cleaner to me than weird reverse tunnelling solutions.


Are there any plans for SSH tunneling without using cloudflared at the client side? Also: supporting both a SSH and an HTTP tunnel on the same A record would be nice


We do actually, on our paid plans: https://blog.cloudflare.com/cloudflare-for-ssh-rdp-and-minec...

You get A and AAAA records by default.


I'm using Zero Trust Tunnel for some web apps I host in my home, but I'm trying to think if the older service (IPv4 to IPv6) you describe would be useful for anything, like ssh'ing into my home from an external VPS.

Would the earlier product be used for something like a router, which can't run the Tunnel service?


Does this end up being similar to say an haproxy doing domain based load balancing to an ipv6 endpoint(s)? I assume you have loads of customers on any single ipv4 IP ingress right?


It's unnecessary indeed to not use Ipv6 addresses, 2^128 addresses and the many many features it offers like unicast etc. Ipv6 makes a server as a middlemen for some applications (Ipv4 only) completely obsolete.

But a big problem is that there is still no Ipv6 auto configuration at all on a lot of devices (e.g. no default gateway or no global address configured). Especially android devices and from experience also on Windows. Linux depends on the distro. Changing routing settings on android devices from Ipv4 to Ipv6 does often not work or is not offered by the ISP strangely.

And there are other problems like routers having enabled incoming and outgoing Ipv6 connections by default, which is good, but having router advertisements blocked by default, which is bad. Since there is no way for the OS to get the prefix to construct global addresses automatically. Most users today have little to no knowledge about networking and computers in general. So auto configuration is a must.

That leads to Ipv6 only servers being not reachable and thus the buying of Ipv4 addresses makes a lot of sense at this point.


Building IPv6 autoconf into the protocol was a mistake. DHCPv6 is better.

The problem is that when you autoconf on a local network you usually want more than just a route and basic DNS. Trying to do it in the IP protocol is a bad idea since the IP protocol is intended to almost never change. It belongs in a protocol that's less tightly bound to the IP stack that can be more easily extended like DHCP.

DHCP can also integrate with things like local DNS, while this is much harder to do with IPv6 RA and SLAAC.

SLACC is something that sounded good on paper but doesn't adequately capture the entire problem domain.

IPv6 in general needs to just deprecate all the parts of the protocol that are anything but just making IP addresses larger. Everything else is "second system effect" cruft that tends to impede adoption by adding complexity or adding features in the wrong place in the stack.


For clients, IMHO, SLAAC is fine and means I don't have to maintain and run a DHCP service anymore. One less thing that can fail, while SLAAC only fails me if the routers IPv6 link inside the given network goes down.

Servers on the other hand, I will provision with a static IP subnet on deploy, as part of the PXE install or configuration management process, depending on the environment. They will have an ephemeral address during the install, but then query for and persist their allotted address before rebooting into the installed environment as part of their post-install.

I guess we agree that we need a single source of truth, what physical device has what IP (range) in their possession at any time. DNS is a classic way to do that, but there are other solutions, from ITIL-style CMDBs to simple config management git repos. And of course the latter doesn't mean that we don't also update DNS based on IP-assignement, DHCP is not the only tool that can be made to interface with a DNS service.


> SLACC is something that sounded good on paper but doesn't adequately capture the entire problem domain.

Good news! Nothing in SLACC (sic) prevents you from using DHCPv6.

But now, since we have SLAAC as well, you get auto-magic working with simple link-layer connectivity without having to bother with extra infrastructure. If you need extra functionality, you have the option (not necessity).


Standard SLAAC only uses 64-bit prefixes. This is a problem if you want subnets and have an uncooperative ISP handing out 64-bit prefixes. Or maybe you have a /48 or /56 but want a nice readable hierarchy of subnets.

You don't even need 64 bits. IIRC with SLAAC you send NDP messages to the entire broadcast domain. So you need broadcast-domain-sized (i.e., small) device counts on every data-link-layer subnet. So why assign /64? But because some devices won't give up if SLAAC routers aren't there and and some mysteriously default to SLAAC when they shouldn't and in fact some devices only use it, now every bottom level subnet has to be /64. Blech.

This is just one of many IPv6 annoyances that don't even need to exist.


/64 is about a lot more than SLAAC, that's just one easily seen place it shows up. Even without SLAAC the /64iveness of IPv6 would still be there. End client subnets being /64 (with the exception of host routes and p2p networks) allow hardware route scaling to be roughly twice as high, allow network operators to never worry about what the client subnet size is (it's /64 and that's big enough for anything, if it's bigger it's an aggregate route, the address is always split in the half for routed part and client part), and allow various tricks like SLAAC assignment methods to work reliably. A /56 gives 256 subnets (and a /48 65,536), if a house needs more than that for hierarchy then something besides the base /64 is wonky.

Uncooperative ISPs will always exist. If the minimum size was /96 they'd hand that out instead of a /64. If there was no minimum size then they'd hand out a /124 or something equally as stupid. For all of the corporatize ISPs and well planned uses, starting at a /64 makes the rest of life easier.

It's the same logic, though often more accepted in this case, as the minimum advertiseable prefix on the internet being a /48. That also has nothing to do with how many theoretical end clients could fit in that and everything to do with what that actually means to equipment, scaling, and implementation planning.


The practical issues are unrelated to whether or not the two are mutually exclusive. In practice, the lack of RDNSS support from the get-go was quite the nuisance. Especially when combined with clients that refused to support the optional DHCPv6 method (e.g. Android/ChromeOS). Even with RDNSS, the feature set is quite limited compared to DHCPv6 but you may have to end up implementing both since there weren't enough incentives for all clients to go the extra mile and support it even if some of your other clients do.

Overall these days, beyond Google products, I think the ecosystem is in a healthy spot now and I think IPv6 chose the right path. It can still be a bit more hassle to manage in the enterprise space though and that's the one that needs to most convincing to migrate at this point.


Can you guess which of the popular OSes doesn't support stateful DHCPv6?

Answer: Android.

This makes DHCPv6 pretty much useless in practice, when 30% of the clients simply can't use it.


It's been a while since I set up an IPv6 network, what's wrong with RDNSS?


As long as ISPs are unwilling to actually work on the problem on letting their customers use ipv6, applications/services will continue to be uninterested in exposing ipv6 for usage.

Some countries are doing better than others (https://www.google.com/intl/en/ipv6/statistics.html#tab=per-...), but still, ISPs are really dragging their feet...


The worst foot-draggers are major sites like Github and cloud infrastructure. Google only got IPv6 in GKE this year in most regions.

The other big foot-draggers are corporate networks. Even if the ISP supports V6 many corporate networks do not because two generations of IT professionals learned how to do networking entirely through the lens of NAT as a requirement and don't understand how to do things without it. I've seen many IT peoples' brains just melt at the idea of things just having one address. In reality it simplifies things dramatically but sometimes getting people to grasp a simpler solution is actually harder than getting them to grasp a complex one.

I live in the USA and have had IPv6 at home for over a decade (and have used three different ISPs in that time). Many mobile networks are IPv6-first.


NAT is a hack and it going away will be good.

That being said, you can do NAT on IPv6 if you really want to, and maybe it will be needed to help soothe those with those emotional attachment to certain numbers. [fc00::192:168:1:0]/120 or [fc00::10:44:0:0]/96, for example.


Even with NAT IPv6 is better because the NAT can do 1:1 mapping for every IP inside and does not need to remap ports. There is no port exhaustion and P2P always works.

V6 NAT is unnecessary and dumb but it works better than V4 NAT.


In my opinion, that should be a legal issue.

Nowadays, you shouldn't be allowed to advertise "internet access" if ipv6 isn't supported.

Ipv6 is the current protocol. And some sites don't have ipv4. (Amazon charging an extra for ipv4 is another sign that ipv4 should be a protocol for particular use cases, not for "the internet")

And it should be the same for software and connected hardware. No ipv6 ? That's not a product that works over the internet.

On a personal side, what I host is only working on ipv6, as my ISP has stable ipv6 but not ipv4, and for the convenience of configuration.

And even cheapo internet plans on mobile and landline support ipv6 by default nowadays. (The government pushed for it)


> On a personal side, what I host is only working on ipv6, as my ISP has stable ipv6 but not ipv4, and for the convenience of configuration.

For me it's the other way around - I disable IPv6 on all my servers and only host anything on IPv4. I know it's frowned upon in networking circles, but IPv4 "just works" for me, and I want to reduce attack scope and maintenance burden (I had some problems with IPv6 messing things up, or my ipv6 firewall misconfigurations).


Exacly that! I do the same. Accessing of internal resources hosted on LAN? No problem, just make overlay VPN network.

Need to host something via HTTP? mod_proxy to the rescue.

IPv6 is junk protocol, overengineered. I hope IPv6 will be used for all those internet consumers and IPv4 will stay where its place to be, interesting R&D projects :)


> Ipv6 is the current protocol

It's A protocol

> And some sites don't have ipv4

Yet there are far more sites that don't have ipv6 access.

What's the ipv6 address for hacker news?


The sad part about all this is that Ipv6 was already standardized in the 90s and supported by most network interfaces in the early 2000s.


All major ISPs have had native ipv6 for customers in the US for at least 5 years. Not some funky bastardized implementation but native full ipv6.

Ipv6 is overly complicated and has been riddled with bugs for 30 years now. As long as ipv4 is an option many are going to choose to completely disable it. Some of the security concerns cannot be effectively filtered at all. There are numerous examples of these vulnerabilities from even just the last few years.

It’s hard for teams of engineers to secure properly much less a home user.

I completely disable ipv6 even with a deep understanding of it.


> All major ISPs have had native ipv6 for customers in the US for at least 5 years. Not some funky bastardized implementation but native full ipv6.

CenturyLink (or Lumen or whatever they want to call themselves today) only has 6rd, at least in my neck of the PNW. And it's best if you don't use it, as their CPE tends to do bad things if you do; my initial CPE would reboot if a fragmented 6rd packet came in over the WAN interface. The current CPE doesn't reboot, but v6 packets sometimes take about 1 second to transit the CPE, so I gave in and run the CPE as a bridge and do PPPoE on my own equipment.


Here in Boise I have two choices for internet, Sparklight or CenturyLink. CenturyLink will only deliver 80Mbps to my house and Sparklight does not support IPv6.


>Ipv6 makes a server as a middlemen for some applications (Ipv4 only) completely obsolete.

Not really, proxying also provides user privacy, and enables DDoS protection (this is especially an issue in the video game world).


Absolutely, a very good point indeed. But I meant more specific applications like end-to-end messaging. Surely 'obsolete' is a bit of an overstatement. At the end it depends how one looks at and want things to be done.


IPv4 addresses have always had a cost (sort of, though they've gone from pennies per IP to $60+ per). I get the feeling Amazon was happy to eat the cost to reduce friction in deploying EC2 instances but now they've hit maximum saturation and now they can just add another charge to the pile that 99.99% of users will never notice.


> I get the feeling Amazon was happy to eat the cost to reduce friction in deploying EC2 instances [...] and now they can just add another charge to the pile that 99.99% of users will never notice.

This always leaves me puzzled about the concept of "free markets." How can smaller entities compete when these massive conglomerates can perpetually introduce loss leaders or subsidize pricing in new sectors using profits from their existing businesses? This strategy effectively shields them and reduces competition.

My initial thought is that it should be illegal for companies to invest in sectors unrelated to where they generated their profits. However, I recognize this could lead to numerous unintended consequences.

So, what could be an alternative solution?


I'm not sure I agree with the premise of the question. Not charging for IPs was less likely to be some massive subsidization plan to create a loss leader and capture market and was more likely just what everyone else was doing - ignoring charging for the few pennies because the juice wasn't worth the squeeze and they've got better ways to spend time trying to make real money. Now prices are getting very high and that's no longer true.


It’s a case by case basis. You can’t set a rule and say that every time an established leader eats a cost that it’s right or wrong.

In this case, AWS has had plenty of competition via other cloud services like Azure and Google Cloud as well as other hosting options. The fact that they ate this cost was immaterial and I don’t see any issue with it.

Even with all the competition, the alternatives still kind of pale in comparison so it’s definitely not a competition problem.


Predatory pricing is already illegal. Or at least was, but apparently SCOTUS weakened it substantially.


When was the last time antitrust laws were actually used in the US "against" these huge companies?


It's been a while. Wikipedia [1] is relatively useful here:

1890 - Start of conventional anti-trust enforcement

1930 - Ramp up of law's usage (under FDR)

1966 - First dissent against anti-trust (Brown Shoe Co. v. United States)

1974 - First decision against anti-trust (United States v. General Dynamics Corp.)

1982 - United States v. AT&T allows break up of Ma Bell. Weakened enforcement allows re-merger.

1999 - Microsoft successfully fights off anti-trust enforcement prevent company from ever being split.

If you want anti-trust enforcement, do not elect Reagan and his descendants.

[1]: https://en.wikipedia.org/wiki/United_States_antitrust_law


It happens all the time.

But usually it’s blocking mergers and acquisitions. Here’s an example: https://www.nytimes.com/2022/11/21/books/penguin-random-hous...

Waiting to break up a company for anti-trust is like waiting for your house to completely flood instead of fixing the pipe before it gets to that point.


Not just huge companies! The entire venture capital model relies on providing a service substantially below cost and burning money to acquire users and eliminate competition so you can raise prices and profit later. In a sane world, Uber undercutting taxis with VC money should have been illegal.


Is it how it works in the EU? Has it worked well for them?

If the cost of initial operation were prohibited to be eaten by investment money, why the cost of development would not be, by the same logic?

I do think that there are cases of competition stifling through dumping, and that's illegal for a reason. Unfortunately, things are not as clearly delineated as with e.g. burning down your competitor's factory.


Well since aws has been hugely profitable, you can easily argue it was simply priced in. I guess they did buy a lot of their ips when they were much cheaper, but it's similar to buying and holding land in many ways.

I had heard more than anything it was due to behind the scene implementations anyways, they likely finally resolved those.


> My initial thought is that it should be illegal for companies to invest in sectors unrelated to where they generated their profits. However, I recognize this could lead to numerous unintended consequences.

Main consequence would be forcing companies to go bankrupt, instead of pivoting to new areas, when their current market becomes obsolete/commoditized.


This would be impossible to ban, and even small businesses do it. Imagine if the corner store charged you to park in the car park, charged you to stand in the store, charged you for every staff interaction, if you open the fridge they charge you a small amount for the power you consumed, etc.

IPv4 addressees were once as insignificant as any of these costs, now they aren't, so they are charging for it.


Speaking of IPv4 addresses, it's far more worse than "free markets". It's a rent seeking by internet early adopters (specifically the US). New indian ISP doesn't have much choice. IMO it's good thing because AWS users will waste less IPv4 addresses.


I run quite a few small production aws accounts for clients and this is a big increase to their bill. If you use a lot of t4g.nano instances, the IPs are more than the machines. I think the large customers that are 99% of the revenue won't care, but the bottom 50% of customers will notice.


Most customers will have very few public IPs. If your architecture is based around dozens (or even more) of public IPv4 addresses then you need to rethink your design because cost isn’t the only risk you’re exposing.


I am pretty sure that the default VPC assigns public IPs (and a default route to the IGW) so if you grew organically without some kind of plan I can see many startups having tons of ephemeral addresses or EIPs.

Also I just did a cleanup recently myself of this and migrated an account to a hub/spoke security VPC with transit gateway. AWS makes it real pain for running instances to migrate to private IP only. You have to play games with adding a second NIC/EIP to workaround this.


AWS really need to deprecate the default VPC. Experienced engineers know to avoid it and novices are encouraged to design things badly because of it.

There should be a “wizard” type walkthrough to guide novices through creating their first VPC (they’d generally be managing AWS via “clickops” so a wizard type UI would work here).


You should try the latest VPC creation wizard. It collects all the required information and then creates what you need.

The biggest complaint I have with it is that it doesn’t write ID values to well known SSM parameters within the region.


I disagree. AWS should make it easier to just use routed IPv6, which makes the default VPC much more usable.

And then just use one of the zero-trust overlays to secure communications between nodes.


Hahahaha, you are giving the average AWS customer way too much credit.


Now that you mention this, it's interesting for me to consider in retrospect. At least one prior employer had me architect solutions that were designed around scaling up using very small instances in large numbers during peak load.

In theory those are deployed in the private address space behind a load balancer, but getting any actual information on the production deployment was like pulling teeth.


I've seen a similar setup ~12 years ago: a very elastic setup on AWS following load changes.

At peak times, the clusters had substantially more than 1k API instances running together; none of the the API instances ever had a public address, and not for cost reasons.

Everything was on EC2, behind a bastion; you had to jump through two jump hosts to SSH to particular boxes (containers were not a thing yet). Everything ran on private networks. Everything was exposed through load balancers (HAProxy) via 2 or 3 public IPv4 addresses; I think one of them was only to interact with an off-AWS third-party service over the public internet.

Yes, it was a startup, just very technically-minded.


At AWS or in general? To my knowledge, existing assignments aren't incurring any annual fees (or if so, not more than IPv6).

There's just a secondary market for v4 these days, but that's also a one-time cost, as far as I know.

In other words, either AWS is charging a recurring fee for an asset they purchase at a one-time flat fee (which is great if you use a service for less than the year or so it takes to amortize, and not so much afterwards), or I missed a development in the IPv4 exhaustion saga.


I assume it’s a one time fee for AWS and pay forever for all of us that can’t afford to buy a block.

I thought it was closer to $40 per IP last time I looked. AWS charging $3.60 per month looks pretty lucrative either way since the payback is only 1-1.5 years.


We recently bought a /24 at $65 per IP. Larger blocks are somewhat cheaper, but not by much.


You mean like my landlord?


If you're renting property at a rate that would amortize after 12 months if you'd purchase instead, I suggest you do that – it shouldn't be hard to find a one-year mortgage ;)


It's pretty galling for AWS to ask their customers "to be a bit more frugal with your use of public IPv4 addresses and to think about accelerating your adoption of IPv6 as a modernization" when they themselves have been dragging their feet in IPv6 adoption, and in many cases are still blocking or at least making it unnecessarily difficult to use IPv6.


Well, ask yourself: if you're Amazon and you have the choice to spend money getting ipv6 working properly, or you can make money selling v4 addresses without any risk of customers jumping ship, what would you do?


> without any risk of customers jumping ship

actually, changing of cloud services is in the sweet spot of being both manageable, and pretty complex so you won't do it every other day.

So if companies are fed up for long enough (or if engineers find it so complicated or costly that they learn to do cloud with something else), they will change. And when they will change, they will never come back. Amazon would be just like AOL or Yahoo (or Jenkins or SVN) or any forgotten giant.

Additionally, many companies don't rely on any cloud service so far. One day, when they will eventually implement it, they won't take what has annoyed others.

AWS is already more expensive than the services of smaller companies (OVH and the like). So Beware.

This change on its own won't make much difference, but they shouldn't be too pushy.


> actually, changing of cloud services is in the sweet spot of being both manageable, and pretty complex so you won't do it every other day.

Exactly. Once you're tied to Amazon's cloud, and especially their managed services, moving away is enormously costly and complex. Early in the cloud game people used to pretend to build abstractions to try to avoid being locked down to a single vendor, but these days that's all but done.

The result is AWS is beyond sticky.

> So if companies are fed up for long enough (or if engineers find it so complicated or costly that they learn to do cloud with something else), they will change.

Which is why Amazon will keep their fees for IPv4 high enough to be profitable but low enough that investing in changing cloud providers isn't worth the cost to their customers.

> Additionally, many companies don't rely on any cloud service so far. One day, when they will eventually implement it, they won't take what has annoyed others.

No one ever got fired for picking AWS.


A significant number of AWS Customers are "internal", and—believe it or not—the cost of resources does come up in design meetings at Amazon. This change might actually light a fire those teams to actually start supporting IPv6 properly.


Amazon really needs to put a ton of work into making v6 work for everyone on the server side or this is a very big price increase on the low end.

If they had a compelling case to do the devops work and then everythings fine, I wouldn't mind this at all. The reality is a ton of stuff is ipv4 only (cloudfront origins, albs require ipv4, etc etc).

They realistically need free NAT or free 6to4 as a transition plan.


This has been driving me crazy for years now. AWS still doesn’t have complete IPv6 support, in 2023. They are front and center to IPv4 exhaustion yet seem unconcerned.


Even Amazon can't make every single ISP in the world provide IPv6 connectivity, which would be required to actually deprecate IPv4 on the server side (or at least at the load balancer or other type of HTTP reverse proxy).


They should offer NAT64/DNS64 gateway. It would work like other gateways and give IPv4 access to IPv6 hosts. Would probably be expensive like the NAT Gateway.


Is the gateway described here: https://aws.amazon.com/blogs/aws/let-your-ipv6-only-workload... no longer available?


That's the wrong direction (IPv6 clients calling IPv4 services). That doesn't help you for running a service yourself.


You'll need something it you want to download files from Github, because for some stupid reason they still don't provide IPv6 services for cloning and such.


you can run an NLB that terminates v4 and forwards to v6 internally: https://aws.amazon.com/about-aws/whats-new/2020/11/network-l...


ELBs (Elastic Load Balancer) supported IPv6 termination since 2011.


It looks to be available, but (from the bottom of the article) "Regular NAT gateway charges may apply."


Yes the problem is NAT gateways are expensive, not free. AWS could easily run a "central" 6 to 4 in each region (not in your VPC), which the incremental cost of using would be near 0. Then you could use pure v6 (no IP charges). And actually be able to talk to things like github, stripe, apple, etc.


I didn't know about that.


Do you mean NAT46? NAT64 is for IPv6 clients to access IPv4 servers, no? And how would you assign, e.g., port 80 to more than one IPv6 IPs in the backend?

The other option are application-level proxies/load balancers etc., but that's not easy if you want to support all possible protocols.


I'm talking about IPv6-only instances needing to access IPv4. Github is the big example.

Network load balancers (NLB) are for other protocols.


Sure, but NAT64 is much easier than NAT46 or application-level load balancing. It’s probably been how most people in the world access the internet for a while, to say nothing about NAT44.


The "official" solution is SIIT-DC but it looks like AWS would prefer that you use ALBs.


TIL about SIIT-DC, thank you.

This still requires mapping tables similar to explicit port forwarding, and also does not help with e.g. a scenario where all hosts need a specific port (e.g. 443) to be v4 reachable, right?


The last time I tried to set up IPv6 with my VPC, it was an absolute nightmare. Maybe I'm not devops-y enough, who knows. But all three of my earnest efforts to use IPv6 have gone pretty badly.

Has anyone successfully used AWS's IPv6 offerings to stand up a VPC/ECS/ALB/RDS using secure best practices without friction? What tutorials did you follow? I'm all ears.


Not every service supports IPv6. Some big ones are APIGW and Lambda.

For RDS, you have to set up your instance as dual stack explicitly even if you’re deploying it into an IPv6 subnet.


whoa! really APIGW and lambda are not ipv6 compatible ?

i was planning to deploy an internal developer platform (think local PAAS) using lambdas behind an api gateway. no ipv6 there ?


TBH, not supporting IPv6 in this day and age is just unconscionable.


not when comparatively nobody uses it


All US mobile ISPs (which are 100% IPv6 and use NAT64 to access v4-only servers!) are “nobody” to you?

Many fixed-line ISPs also only provide v4 over DS-lite or similar, these days.


There's no incentive to use it when IPv4 is free. There are two main ways for network providers to move the needle, assuming that they actually offer IPv6 as an alternative:

(1) charge for IPv4

(2) move IPv4 behind CGNAT


Lambda does support private IPv4 networks though.


This explains a lot. I wanted to be a good citizen and use IPv6 exclusively internally and keep IPv4 at the edge, then I found I couldn’t create a database without a bunch of IPv4 settings I hadn’t configured.


The tools and tutorials are not optimized for ipv6 only workflow.

Guess they will improve soon as amazon start charging


I think so as well. When M1 Macs started pushing ARM chips out there were growing pains, but most of the ecosystem adapted fairly quickly.


Incentive are the opposite, surely. The harder it is to use v6, the more people will pay to stick to v4.


I think this is probably the main issue, too complex to set up on the server. But I agree with what AWS is doing.

For example, when I do an ifconfig, I get 3 ip6 addresses but 1 ip4 address.

'?' indicates a unique value, 'x' means values match between the IP addresses. That alone indicates the complexity of ip6 on setting up the server.

inet6 ????::????:????:????:???? prefixlen 64 scopeid 0x20<link>

inet6 xxxx:xxx:xxxx:xxxx::???? prefixlen 128 scopeid 0x0<global>

inet6 xxxx:xxx:xxxx:xxxx:????:????:????:???? prefixlen 64 scopeid 0x0<global>


My IPv4 server has 127.0.0.1/8, 10.64.78.37/32, 172.17.2.1/16, and a public IP hidden somewhere. The 172/12 networks I see are usually Docker doing Docker things but I'm still left dealing with three different IP addresses.

Not that it matters much, because they all just appeared on the right interfaces and started working.

You may need to know some basic things about IPv6 for your firewall ("fe* means local link") but the same is true for IPv4 ("10.* means local network"). I think they're equally difficult to manage, but I can understand how daunting it may look to someone whose been taught networking by outdated textbooks lacking IPv6 like so many other people.


Why is this even relevant in AWS?


I am showing why I think setting up an IP6 server is harder than IP4. With IP4 I get 1 IP address, IP6 I get 3


When do you deal with ip addresses at all in AWS?



With all that investment in addresses I'm surprised AWS is still the first cloud provider to charge for them. (As far as I know.) It will be interesting to see if other cloud providers will follow, and if the cloud providers compete over the price or just match AWS. It kind of feels like AWS charging for V4s will "give permission" to other providers to charge.

I'm also curious if the price will come down over time as addresses are yielded back. I guess it depends on if their goal is to recoup all the money they spent on addresses, or just to avoid running out.


AWS is the last major cloud to start charging, not the first.


Ah, should have googled. I stand corrected.


GCP and Azure have always charged for IPv4 addresses. If anything, AWS was the last of the public cloud providers to provide them for free.


I've even seen small VPS hosts have discount plans if you turn off the v4 address.


Good. Should honestly charge more. The slow adoption of IPv6 is an embarrassment for everybody in tech. Tech talks of inclusivity yet looking behind the curtains that is not always the case.

Developing countries often do not have the money to buy/lease IPv4 public addresses so therefore they force their subscribers to IPv6. With most of the internet still on ipv4, this makes them inaccessible (ie, github) unless you are technically inclined.


Is any ISP actually providing users with no v4 access? I've never heard of this. It's always that they provide v6 and then some kind of CGNAT or bridging service so that v4 still works.



I looked up the cost of buying my own /24 block (which would be 256 addresses). From the auction houses I looked at, it appears that floats around $9,000-$10,000.

What the...

Good thing YouTubers aren't selling IP Addresses as investments. Yet.


The day they started forcing users to use VPCs on new EC2 instances it was pretty clear to me they’d really screwed up.

That was the point when they should have gone pure v6 for everyone except enterprise customers. And if you wanted to expose services over v4, they should have made you pay for a load balancer. That would have made it pretty easy to pass through the real cost, and would have gotten devs used to using v6 for management and internal connectivity.

I saw a comment saying ISPs should drop v4. I consult for one and they absolutely cannot. Too many customers still rely on stuff that doesn’t support v6. It’s become a major headache. Not only are they expensive, but you also have to be careful not to buy blocks that are on legacy blacklists.


For people running t3.nano instances, this is a 96% increase (from $3.75/month to $7.35/month).

For t4g.nano it's a 119% increase (from $3.04/month to $6.62/month)


IPv6 is still an afterthought for nearly every level of networked software development, and that will probably never change as long as IPv4 is a viable alternative. V6 is just... Annoying, compared to V4. Without a concerted industry wide effort to abandon V4, V6 won't take its place.


$0.005/hr is $44/year; not nothing!

If this is what IPv4s cost, are small VPSes now uneconomical? For example, the current listed pricing for the smallest Lightsail instance, which includes an IPv4, is $42/year. [1]

[1] https://aws.amazon.com/lightsail/pricing/


There is a lot of variance in what an IP costs in practical terms. If you need to go out and get a new IP block then that pricing should roughly break even in 1 year. In that sense, it might significantly increase the time to break even and long term margins for that type of service.

For ultra cheap VPSes IPv6 only options are becoming available for just this reason.


I wonder if we could get ultra cheap VPSs on CGNAT. You might want to run something which only makes outgoing API connections. Maybe it's just a chatbot or something that doesn't need it's own IP.


I don't know of anyone offering that service (a bit too niche and low margin). An option for low volume personal use might be to just piggyback off a public NAT64 provider though. For something a bit more enterprise, AWS will do NAT64 by default on any NAT gateways you define


More companies should have IPv6 as standard offering and offer IPv4 as extra.

But it also requires tools to make sure they properly prefer IPv6 over IPv4, otherwise you're paying extra because your customers run bad software.

For example WireGuard on iOS. To this day it prefers an A record over AAAA when resolving a domain. That costs money. And breaks on 464xlat.


I wish we would get an "IPv6 as it was meant to be" - v4 just with more octets. That's it. That's all we wanted. That's all we needed. If that was the IPv6 spec, we'd have already been using it for the past decade. We'd have avoided the #1 problem of v4, address scarcity, while retaining its superior user experience, and articles about expensive v4's wouldn't exist.

IPv6 is a travesty because its creators failed to consider the users. Nobody wants to deal with it. And that's why its adoption will eternally be "just around the corner!"


What exactly are the problems that you think would have been avoided? As far as I can tell, the stumbling block to v6 adoption is just the fact that v6-only and v4-only hosts can't communicate without help, and you would have that problem in any form of "v4 with more octets".

Sure, v6 made a few other changes, but why do you think those are the problem?


There were plenty of stumbling blocks early in the process.

1. SLAAC had been half-baked for the first 10 years. You were supposed to run SLAAC _along_ with stateless DHCPv6 just to get the DNS settings.

2. The initial rollout was embarrassing. The initial idea was to avoid the PI (Provider Independent) prefixes.

3. There is a plethora of non-intuitive features, like on-link addresses instead of the simple "subnet mask" in IPv4.

4. DHCPv6 is strictly worse than V4 for management. Whoever thought that DUIDs are a good idea...

5. IPv6 managed to break fragmentation and PMTU _even_ _more_ than IPv4.

6. HappyEyeballs that mitigate the V4/V6 instabilities became a standard only around 2012.

Etc.

Honestly, there's pretty much nothing that went right in the IPv6 development.


v4 with more octets is still backwards incompatible with v4. It would have had all the same problems and headaches in rollout, not only would "nobody want to deal with it', it would have even fewer reasons to switch. IPv6 at least has some forward-facing features to give people reasons to switch (even if people still argue if they "are worth it").


From what I can tell from the announcement the change will apply to a NAT gateway, so the cost of that will rise from 0.045 to 0.05, the same as having 10 IP addresses. If they are really seeing high costs associated with IPs then I would like to see them reduce the price of the NAT gateway to the same as having 3 or 4 IP addresses or less, ideally free then they would probably see a lot bigger uptake.


The NAT gateways should be free.


At very least some free tier for NAT64 gateways, or shared DNS64/NAT64 alike https://nat64.net/public-providers for Amazon users would be nice. I'm fine with ipv6-only connectivity for a lot of hosts, but once in a while they need to download some software / updates from github.com which is still ipv4-only. Paying fixed extra cost for every subnet just for 100MB/mo download per host is meh.


Yeah, Jeff Bezos needs to squeeze few more dollars out for another rocket to space.

The question is whether IPv6 is really the way to go. Not speaking technically of course. End consumer does not really care about it and if you want to create (and sell) a product, you just cannot say "it will not work, get a better internet". So right now it means I have to work with IPv4 anyway and then it is a question of why even bother with dual-stack at all?

That means there is essentially no push from the masses and therefore no real reason for ISPs to even bother with pricing it and including into their processes.

IPv6 just failed to crack the chicken-and-egg problem. It did not offer anything substantial (or good enough) to the end user. Most of them will never care about not being able to get SIP calls working peer-to-peer as they will use WhatsApp or something like that (that works right now already).


3.50/m for an ip? You can get a lowend vps for that amount.

Pretty exorbitant pricing especially when coming for free, but that's really par for the course for aws. I guess most users really don't need all that many, so maybe it's just a way to force some users have been going wild to setup a proper vpc.


> Pretty exorbitant pricing

good, clearly no one is going to start implementing IPv6 until using IPv4 begins to hurt


This has been coming for a long time, and I expect other services to follow suit.

After you hit IPv4 exhaustion, the price crunch only gets worse. I guess it finally hit a tipping point where it’s going to be taken seriously.


For normal use, you can just use a NAT gateway and keep a couple of IP addresses: Gateway and Bastion host. Trivial to implement. Shouldn't affect most people.


DNS64+NAT64 seems to be an elegant solution, but it's so damn expensive on AWS... as if their $0.09/GB egress wasn't already bad, there is an additional $0.045/GB charge for transfer through NAT64 gateway.. and you still have to pay $0.045/hour as well.


I wish I understood IPv6 as well as IPv4, even with NAT. I have an OVH server I've moved most of my personal stuff to, running ProxMox, but it was easier for me to setup a virtual nic with a natted ipv4 block for internal services than it was to figure out how to use my massive ipv6 block.

I mean, I get AAAA records, but does anyone have a good primer on ipv6? Another on ipv6 with docker would be nice too.


Ho? So this is why they bought our /16... We had downgraded to a /18 following that, but still have the same old v6 space.

There is no reason to not use v6, and excuses don't count. RFC2460 has been out since 1999 and most everything even defaults to v6 for localhost. Set up AAAA records and never think about it again...


How can I buy a small number of IP addresses?


Join RIPE and you may get a /24 allocation though the wait time varies

https://www.ripe.net/participate/member-support/become-a-mem...


"wait time varies"

Although reasonably consistently >400 days.


And for the rest of the folks, here are the rest of the RIRs (as RIPE is Europe/West+Central Asia specific):

- AFRINIC - Africa

- ARIN - North America + misc

- APNIC - East+South Asia + Pacific

- LACNIC - South America + surrounding


Do they also issue allocations as part of membership? I didn't mention them because I'm not aware if the same "free" allocation is made in those RIRs.


That depends on what your definition of "a small number" is and what you plan on doing with them. You can only buy them in powers of 2 with a minimum block of 256 and there are many brokers happy to facilitate a sale. You can also get them assigned (in a limited and often delayed fashion these days) but only if your intent is to actually use them as a new entity (or if your intent is to try to game the system for money then that's its own path with its own pitfalls). Once you have them you'll also need to pay regular registration dues while you hold them. If you plan on using them you'll need a BGP ASN and a peering relationship or a provider/carrier willing to advertise it under their ASN and then pipe it to you. AWS supports this and calls it "BYOIP". Advertisements to the internet are made in chunks the same sizes as you can buy.

The above is slightly simplified but it should give you the gist.


I believe AWS can advertise your networks via BGP these days, so you only need an IP allocation and an AS number.


Yes, that's the AWS "BYOIP" feature noted towards the end.


They used to support BYOIP, but without advertising the clients' prefixes themselves.

So you could use your own IPs, but you had to use something like DirectConnect or VPNs to forward traffic to your network.


The wait time for IPv4 address allocations from the RIRs is years, and probably getting longer at least as fast as time is passing.

The most common way to get addresses is now on the resale market; they are currently trading at $40 or $50 per address (depending on the size of the allocation) tho prices have been as high as $60. For example, see https://ipv4.global/reports/

The minimum allocation is a /24 so you can expect to pay more than $10,000


Let's just go full native IPv6 already. Clinging to the past like this doesn't make sense in the long run.



Good. Charge more, and more cloud providers should do this. Kill it already.


Can IPv4 reasonably be killed yet? The ISPs are still dragging their feet on IPv6. I can't browse over IPv6 at home because my ISP (Cox) hasn't been able to consistently deploy it to residential. I'll have support for a few weeks, then go 3 months without, and then repeat. It's been like this for like 4 years.

When I helped setup our new office earlier this year, I requested IPv6 support and was told that Cox just doesn't support it at all for business lines right now.

How can anyone drop support for v4 when the ISPs still don't support v6? What good is a service that can't be accessed due to ISPs being awful?


It's a chicken-and-egg problem. ISPs have very little incentive to deploy IPv6 when not enough of their customers are demanding it. Meanwhile, people don't demand IPv6 when all of their services work just fine over IPv4.

A big drive for IPv6 adoption seems to be that some ISPs are genuinely running out of addresses. They are forced to re-engineer their network stack in order to switch to CGNAT anyways, so deploying IPv6 along with it isn't too much extra work. As long as the ISP has plenty of IPv4 space remaining, their engineers are going to have a hard time selling spending time on IPv6 to management.


considering the Lindy effect, ipv4 is likely going to be around in some form forever.

https://en.m.wikipedia.org/wiki/Lindy_effect


Remind why everything "needs" a public ip address? NAT and SRV records solved all of these problems decades ago. Not sure why we're still living out an XKCD comic.


I'm confused, how can NATs and SRV DNS labels replace a public IP address?


They can help people put up to 65535 services on a single IP address rather than taking up a /16 of routable space through port numbers alone. Double that if you add UDP, and triple that if you can find that many services that can use SCTP.

You still need at least one IP address, but for services using hundreds of individual hosts the cost shouldn't rise too much this way.


What common applications commonly support SRV? I can't load a HTTPS in a modern web browser on a non-443 port without explicitly specifying it for example.


XMPP and Matrix do, at least. Office365 uses it for Lync I believe. Minecraft uses it too. Wikipedia has a short list: https://en.wikipedia.org/wiki/SRV_record

For HTTPS and such protocols, where the port number is hardcoded, you'll be out of luck with plain SRV.

SVCB and HTTPS records should allow for other ports to be used with modern HTTPS, though I haven't personally played around with those.


The theory is that very few companies actually need more than one ipv4 address. In fact, the overwhelming majority of internet consumers would be better served without a public address.


That's not really true. Most people, companies and consumers alike, have networks of more than one machine and they actually really want those networks to be part of the Internet rather than being separate and using a proxy. You can't do that without public address space.

You can try to fake it with NAT, but that introduces a whole bunch of problems of its own which then need to be worked around. It doesn't make sense to use it unless you can't get public address space in the first place.


NAT needs a public IP address. SRV needs a public IP address. In any case, web browsers doesn’t suppport SRV. HTTP can serve multiple hosts from one IP address.

Company requires at least 2 IP, one for outgoing and one incoming.


> Company requires at least 2 IP, one for outgoing and one incoming.

Where did you learn that?




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: