One rather insidious one I'd never considered prior was that lowercase n is one bit from '.'. So you can also bitsquat on things like, say, wwwngoogle.com or mailngoogle.com.
A researcher brought this to my attention years ago with a set of domains in particular I won't name. What was most interesting to me is just how frequent bit errors must happen. According to the research, they'd basically received thousands of emails destined for the correct domain. Really makes you think.
Reminds me of a former colleague of mine who was one of the principal developers behind an Android app, where they could really go to town on making it crash free, since Android offers a phone home function when a crash occurs.
At some point, he made the observation that 1-2% of crashes occur not because of a programmer error or anything, but "chance"; bit flips and the like, in either the app's memory or the phone's services.
So while on an individual basis it's rare, and in practice nobody will really notice an issue, statistically bit flips are significant enough to need attention.
I had a project where we were reaching for the 'cosmic ray' idea. But it turns out it was a supercap that did not hold enough charge on NAND during power off and the way NAND ages. It worked when 'new' but as the device aged and was used it the NAND would slowly take extra time to write a page. Eventually the time was just enough that it would mostly write everything. It would also sometimes randomly flip bits here and there depending on where it was in the write cycle.
I don't think bit errors happen nearly as often as people seem to think on HN. Sure they happen but they're relatively rare, especially on good quality servers. We have ECC/hashing all over the place looking for it at various layers. I bet it's 1 in a trillion emails that get a bit error in the addressing if not more. I have written embedded code (FPGA) that ran for YEARs that has hashing and I could probably count on one hand the number of times we've had to address any issue (always failed hardware) and it was caught every time unless SHA512 has failed us.
> relatively rare, especially on good quality servers
Good quality servers are the top of the reliability standings. The bottom... the bottom is very far down. Very far.
I'm a hardware guy now. I'm still amazed anything ever works at all. The worst are things like DDR SDRAM that work well enough that people think they can get away without error detection. At least modern MLC NAND flash is so bad that it obviously doesn't work at all without ECC. Better that than silent failures.
The first Guild Wars game actually added a Prime95-style iteration before sending a bug report/crash dump back. An astonishing number of clients failed this test; if I'm remembering the conversation I had correctly, something like half of all oddball crashes got filtered out by that test. Half! (To be fair, overclocked cheap gaming machines may actually be the bottom tier of reliability, but the point stands.)
> I don't think bit errors happen nearly as often as people seem to think on HN.
Back in the days of spinning disks, I had a coworker who had previous worked on the SQL Server test team.
One thing they tracked was bit errors. He said they were surprisingly common to encounter (and of course then account for)
Or, to put it another way, there is a reason modern file systems are paranoid.
Also way back when I managed a huge distributed test cluster, we ran ~3 million tests a week. A non-trivial # of test failures (few hundred) were spurious network failures where the test package didn't get copied to the test client correctly, we detected it "something went wrong" and marked the test as failed. I know in theory Ethernet promises the data gets there without issue, but in reality, at scale, it wasn't. Flaky hardware, bad cable, failing switch, a couple machines with unreliable RAM? Who knows, but it happens.
> I could probably count on one hand the number of times we've had to address any issue (always failed hardware)
Another product I worked on, the bus timings for the DRAM were slightly off. And I do mean slightly, I wasn't directly involved but I think it was something stupid like being 1 cycle short on waiting for something.
We had amazing logging. Soon as we had a few hundred thousand units out, we get ~10 stack traces come in from crash dumps, always in the same place. My team spent a month going over every line of code related to that place in memory.
New build goes out, more failures, again tiny tiny %, you'd only notice this at scale. We track it down, pointer being corrupted, it was stored in the same memory address we had been looking at last time.
Bit was being flipped. We only noticed it because it had hit a pointer. Most of the DRAM, the vast vast majority, was capable of operating 1 cycle out of spec, but some chips weren't.
We only detected and fixed this bit flip issue because the principle engineer on that product insisted that we investigate and fix EVERY single product crash. He insisted on 100% reliability.
Do you think every firmware team for every single component of every single motherboard has that same dedication to quality? Because if not, errors are going to slip through, and they won't even be noticed, not to mention investigated.
Cool writeup. I think some groups or orgs suffer the opposite of survivorship bias, that is, even if an occurrence is rare, they get to see it a lot of times.
> Do you think every firmware team for every single component of every single motherboard has that same dedication to quality?
The answer is, of course, no. Build quality and work conditions vary wildly.
But what about consumer hardware the might be failing? So your typical 7-10 year old PC still trucking along serving as a Facebook and word machine. Genuinely curious as to how common bit flips are in the wild.
For this bit flip to work it would have to happen before it was handed down to the actual resolver, since on the wire this flip would just cause a invalid parse of the DNS label
This "attack" is against the ordinary user, so yeah is before or at the time of the dns query. I'd expect(probably incorrectly) for the server side stuff like recursive resolvers to have ECC.
So imagine you are sending a mail to jim@mail.example.com. You see it in the address field as correct. You click send, and your client resolves mailnexample.com, gets an address, and delivers it, while you are none the wiser.
This is particularly bad for third level domains, which are more common than you'd expect.
Since you have experience, maybe there's something you can help explain with what was mentioned:
> In fact, out of the 32 valid domain names that are 1-bitflip away from windows.com
"windows" is 7 characters long so how is there 32 1-bitflip combinations?
Also one of the ones mentioned was "windo7s", but "w" is 01111000 and "7" is 0000111, which isn't a single bit flip away unless I'm doing something wrong.
In theory, this should have lower efficacy than bit flips elsewhere in the domain. Periods aren’t actually sent in DNS queries, so you’re decreasing the number of places in memory where the necessary bit flip could occur. The equation changes slightly when you’re trying to intercept HTTPS traffic, though, since a bit flip in a DNS query wouldn’t help you anyway; it’d have to happen elsewhere.
DoD address range is not so mysterious: it’s 11.0.0.0/8, and it’s not been seen on the internet. So it is extremely tempting to take one’s 10.0.0.0/8 and turn it into 10.0.0.0/7.
I would bet this is what is going on there - a large network who decided to take a shortcut with addressing, no evilz haxxorz.
The buried lede here is what's going on with Baidu. Presumably the initial erroneous request was typed into a browser, does Baidu operate a browser in China? Do they operate a portion of the backbone network infrastructure, and are inspecting traffic?
If not, how could they possibly have identified this as a target for crawling, unless perhaps they are being fed traffic by the Great Firewall?
That's easy enough. There are WWW pages that contain "time.wiodows.com". I turned up a couple with Bing, in examples of how to use the ntpdate program, and there are no doubt more, including ones where this is hyperlinked by some "smart" program. No need for a nefarious explanation when a WWW spider found this on a WWW page and decided to crawl it will do just as well. (-:
They could both have been the WWW spider. Search engines have been known to not announce themselves, to check what they are seeing against what everyone else sees.
Even if they're not hyperlinked, at this point, I wouldn't be surprised to learn of a crawler looking for FQDNs and things that just generally look like URLs in text to crawl into.
I don't think they operate a browser anymore. My guess at how this happend was that some user tried to open the site, it failed and the copied and pasted the url into Baidu search which then crawled it, and probably also corrected the misspelling.
Since you have experience, maybe there's something you can help explain with what was mentioned:
> In fact, out of the 32 valid domain names that are 1-bitflip away from windows.com
"windows" is 7 characters long so how is there 32 1-bitflip combinations?
Also one of the ones mentioned was "windo7s", but "w" is 01111000 and "7" is 0000111, which isn't a single bit flip away unless I'm doing something wrong.
It partially is, the biggest issue with parallel busses like that of DRAM is skew between the bit lanes. You need the electrical pulses to physically arrive at the same time. It's more difficult to layout 71 bits of exactly equal length than 64.
I did this years ago with the DNS name for a large CDN (think akadns.net but it wasn't Akamai) and also saw lots of interesting stuff behind it.
Its hard (maybe impossible) to identify identify scale, so I don't know if it was 1%, .1% or .0001% of traffic, but I was seeing hundreds of requests per second.
Since you have experience, maybe there's something you can help explain with what was mentioned:
> In fact, out of the 32 valid domain names that are 1-bitflip away from windows.com
"windows" is 7 characters long so how is there 32 1-bitflip combinations?
Also one of the ones mentioned was "windo7s", but "w" is 01111000 and "7" is 0000111, which isn't a single bit flip away unless I'm doing something wrong.
You can't consider all bit-flip possibilities, you have to consider only those which result in valid domain names. Flipping a bit which results in a w becoming a / will never result in a valid domain name, for example.
Your bit math here is wrong (not sure why).
> "windo7s", but "w" is 01111000 and "7" is 0000111
I had McAfee flag a restaurant's website because they mentioned alcohol (i.e included their wine menu). I went to the hassle of submitting a request to change the category ... and it was rejected!
What I'd love to see is a split by which bit is affected and are any versions more popular in the truly benign cases (like ntp).
Although it was confirmed with idle devices that we do see bitflips, I keep wondering if in real world we get more changes due to memory overflows than cosmic rays / failed ram refresh.
The “US DoD” IP could simply be a mobile network operator’s internal IP address leaking via the XFF header. Mobile network operators are known to use public but unadvertised IP ranges such as 25.0.0.0/8 (UK MoD: https://blog.wireshark.org/2010/04/t-mobile-clever-or-insane...) for their internal networks in order to avoid clashes with actual private IPs. For example, by using a “public” IP address, any private IP address requests from a client will route through e.g. WiFi instead of through the cell network. It would not surprise me if a Chinese mobile network operator was using chunks of unadvertised US DoD IP address for the same purpose.
Semi-related: I remember a story from HN a few years ago where someone started experiencing crashes and tracked it down to a single flipped bit in memory which went away when he purged his page cache so the binary was reloaded from disk. Does anyone happen to have a link to it?
I don't remember it on HN, but this is prodding my memory about an article where the specific executable that was corrupted was `bc' on Linux, if that helps any other readers track it down. I think it specifically broke subtraction or something?
> 199,180 NTP Client connections from 626 unique IP addresses
This seems like an abnormally high connections/IP, even if we assume some of the IPs represent multiple clients. Perhaps it's because of retrying as the author does not seem to have sent NTP replies back?
My guess is that for some of these clients they made a typo in their ntp.conf file/NTP config but never noticed it was wrong. I have manually configured hosts/devices in the past for a variety of public NTP servers including time.windows.com.
That's over 14 days, so it averages to just under 1 request per IP per hour. That seems reasonable for an NTP server.
What seems stranger to me is the incorrect clock settings on 30% of those computers. Why do they all think it's the end of September (even though the years are drastically different)?
I suspect this is just because the author is sorting the text, and the days at the end of September are lexicographically last given that date formatting.
But to have your computer bitflip 300 times in a couple weeks is impossible. Either the address is cached incorrectly for a long time or it's just a typo in a config file.
Somewhat related: I run an http server on the local network for various services.
Due to the large number of bogus requests I set up a honeypot vhost which responds only to requests sent without a Host header, i.e. received directly on the IP address. It also logs these requests separately.
It's fun to check the logs once in a while and see all sorts of exploit attempts. Wordpress and PHP seem to attract a lot of attention.
As a fun exercise sometime, spin up any cheap EC2 instance in AWS, set up an HTTP server listening on port 80 and 443, and watch your SSH and HTTP logs.
The last time I did this I had random SSH and HTTP requests rolling in within minutes.
I recently interned in cloud security for a mid-sized firm.
In my first week, I spun up a free-tier EC2 for some testing (Only port 22 was open) and soon enough, the logs were full of port scans and unauthorized ssh attempts.
It is only then I realized the scale at which malicious parties operate.
Suffice it to say, I was very careful with my filters after that incident.
In 2004 a study was done that showed an unprotected PC connected to the internet was attacked within 20 minutes, half the time of a previous study just a few years prior.
If we suppose that it continues to half, say ever five years, it went to within 10 minutes in 2010, within 5 minutes in 2015, and is now somewhere around within 3 minutes for an unprotected PC to be attacked once connected the the internet.
The behavior seen here seems to indicate typos more than bit flipping - ex, look at the NTP requests: it works out to roughly 1 request/hr from the ~600 IPs that are hitting them, that's not an in-memory bit-flip.
Certain in memory or on disk bit flips would make it use the wrong domain repeatedly until rebooted, or even until the system was reinstalled.
Having said that, I suspect that if these were investigated in detail you would find software bugs or typos to be the majority. Simple things like a race condition when suspending to RAM could flip a few bits occasionally if the RAM isn't correctly refreshed.
Oh goodness! After domain squatting, one more squatting to think about! Do I have to worry about these things? How likely is it that a solar flare modifies a computer memory right when typing a domain?
The likelihood of your computer experiencing a bit flip at the wrong time is microscopic. The likelihood that some computer, somewhere on the planet at any time? Much higher.
aren't Ethernet frames CRC'd? Are there other checksums down the call stack, or up the stack at the other end, that would reject such bit flips in the raw data?
Yes, Ethernet frames are CRC'd but some (most?) memory isn't. The web address is bit-flipped in memory, and then the network stack generates a valid Ethernet frame using the already corrupted address.
RAM (the temporary memory where your operating system, programs, etc are stored) stores bits as charge in a capacitor. Now this is where my understanding falls apart, so I'm going to take an educated guess and hope someone smarter corrects me: Random energy from thermal heat or stray cosmic particles can cause these bits to leak from the capacitor or for an extra charge be thrown into the wire as the data gets transmitted.
the use of the department of defense IP there is most likely because it's a ISP using one of the DoD's ipv4 /8 blocks internally. There's more than a few (and not just all in China) that have done that, in an attempt to shovel back the tide of needing to fully migrate to ipv6, or due to lack of other ipv4 resources for unique customer numbering.
A researcher brought this to my attention years ago with a set of domains in particular I won't name. What was most interesting to me is just how frequent bit errors must happen. According to the research, they'd basically received thousands of emails destined for the correct domain. Really makes you think.