Probably a misconfiguration involving X-Forwarded-For. The frontend sticks the IP in there, you set your backend to trust X-Forwarded-For headers from, say, 10.0.0.0/8, but somehow your frontend and backend end up on the machine and the connection comes from 127.0.0.1 and fails the check for a connection from 10.0.0.0/8. So you distrust X-Forwarded-For and just log the IP address that the TCP connection came from, which was 127.0.0.1.
I have never been able to make sense of all the rules around X-Forwarded-For and neither have the various library implementers. I recently wrote an authentication plugin for Envoy that just extracts what Envoy thinks the remote address is, and puts it in the authentication header that goes to the backend. Then the backends can't get it wrong; if the signature on the message is right, you're getting the IP address that the frontend Envoy got. If something is misconfigured, the header probably won't have a valid signature, and so the request will be rejected outright. Less failsafe than what Wikipedia did... but easier to detect.
That’s important if you don’t control all the systems. Back to there being no rules some systems prepend addresses at each layer and some append them. And if you don’t know or don’t control the behavior at each layer it’s useless IP soup. I’ve not dealt with that in a long while but your comment brought back memories.
To solve the append/prepend dilemma there's also X-Real-IP too. (At least the Nginx module does this.) So basically just log the x-f-f and use the other one as the real client IP.
Of course, if you don't control the layers, then probably you should consider those headers invalid for an incoming request.
(Though for email there's ARC to sign the added headers, maybe if someone really wants to provide at least marginally accountable HTTP proxies, they can use something like that.)
Wikipedia appears to have a two-layer varnish cache system, and if the frontend and backend cache is the same host, the edit was attributed to localhost.
They have been for years. The issue is that they're tables, and often nested tables (there's nothing to prevent how nested afaik) which don't render that well on the mobile UI. Rather than figure out how to render them or come up with an alternative layout, they opted to hide them.
It says "All edits attributed to this IP (besides those resulting from a server misconfiguration in 2013 and another one in 2015) have been made by one of Wikimedia's system administrators" with links to discussions of the two misconfiguration incidents. What more were you looking for?
Just be glad you didn't have to explain an in joke about ftp sites, the local loopback address, and a troll, in a deposition, under oath, to Scientology lawyers, like Keith Henson did.
Readers of alt.religion.scientology were astonished to notice
a large collection of alleged secret, copyrighted and trade
secret protected documents of the church of scientology posted
anonymously over the weekend of May 5. An expert source known to
Biased Journalism verified the documents as authentic.
[snip--to transcript from a deposition of Keith Henson by the "Church"
of Scientology. Lieberman is their lawyer.]
Lieberman: do you know who Patrick J. Volk is?
Henson: to the best of my knowledge I've never heard of this
person.
Lieberman explains that Volk is apparently communicating from
some educational institution in Pittsburgh. Henson still doesn't
recognize the name. Lieberman hands Henson a document.
From: hkhenson@shell.portal.com (H Keith Henson)
Newsgroups: alt.religion.scientology
Subject: Re: OT Materials...
Date: 6 Apr 1995 19:35:38 GMT
Parick J Volk (pjvst+@pitt.edu) wrote:
: Screw the courts....
: I have an ftp site for all the OT materials...
: ftp:127.0.0.1 /pub/texts/news/alt/religion/scientology
: I don't know how long I'll have it up.
: P J Volk
: (alt.2600 lives! All hail the clams and trolls!)
Great stuff! But don't you expect the 'ho to blow a gasket?
Henson: (cracks up) this is a great troll.
Lieberman: (acidly) you find this amusing?
Henson: yes. It's an in joke.
Lieberman quotes from the Volk post: "screw the courts" and
also says that he has an ftp site for all the OT materials. "Mr. Henson is laughing hysterically about this posting for reasons that I suppose he understands--" Henson offers to explain.
Lieberman: What's an ftp site?
Henson explains that ftp means file transfer protocol. You
can use almost any machine on the Internet to access a file on almost
any other machine, that has been placed in an ftp directory, he says
with relish. [He goes on at length about how this is done.]
Lieberman: Okay. "So when he said 'I have an ftp site for
all the OT materials,' he is saying he has all the OT materials on
a site which people can access." Was Henson aware of Patrick
Volk's ftp site? Does this refresh your recollection? he demands.
Henson: well, you see right after the colon, it says
ftp:127.0.0.1?
Lieberman: yes.
Henson: that's a loopback address.
Lieberman wants to pursue the question of the site with the
OT materials. Was Henson aware of Patrick Volk's ftp site?
Henson: (patiently) It's at 127.0.0.1. This is a loop back
address. This is a troll.
Lieberman: what's a troll?
Henson: it comes from the fishing where you troll a
bait along in the water and a fish will jump and bite the thing,
and the idea of it is that the internet is a very humorous place
and it's especially good to troll people who don't have any sense
of humor at all, and this is a troll because an ftp site of 127.0.0.1
doesn't go anywhere. It loops right back around into your own
machine.
Lieberman [not getting it]: So the idea here was to make the
church think that this person had an ftp site and to take action
against him and, in fact, he didn't have it; is that your point?
Henson: Oh, it's really humorous, and I picked up on it
and instantly added something to extend the troll. Extending the
trolls like this is an art form of the highest order.
Lieberman (acidly): I see. So this is part of your art
form where you say, "don't you expect the 'ho to blow a gasket?"
Henson: yes.
Lieberman (starting to lose his temper): so you do remember
this posting apparently?
Henson (helpfully): I can't remember for certain that I
did this one, and certainly I could not swear to any of the material
on here being letter perfect on it (but he goes on to say that it
is such a good one that he would be happy to take credit for it).
Lieberman: You find this whole thing kind of amusing, don't
you?
Henson: Oh, this is screamingly funny.
Lieberman (no more Mr. Nice Guy): You find it amusing to
make Helena Kobrin and the church go after you or other people for
this sort of thing, whether you have the materials or not; is that
right?
Henson: It's a great game.
Lieberman: It is a great game. You really find it amusing,
don't you?
Henson: It's an extremely amusing thing.
Lieberman: All right. You find it amusing when you receive these letters from Ms. Kobrin, the cease and desist letters? It's part of the game; isn't it? [This goes on for awhile as Lieberman hammers at the point. Henson reiterates that he is amused, and wants to talk about the SP levels.]
Lieberman: You find it an amusing part of the game when you receive these cease and desist letters, right?
Henson: No, no. It's not amusing, it's a major increment in status.
Lieberman: I see. You feel this increases your status, right? On the internet, on a.r.s.
Henson: Yes, absolutely.
Lieberman: All right. And it's all part of this game, right?
Henson: Absolutely.
Lieberman: It's all part of the troll, right?
Henson (waving exhibit): This is a great troll. I mean, anybody in the computer business instantly would have spotted this, ftp:127. In fact, it even says trolls in here (indicating). In fact, this was cross-posted from --
Lieberman has heard more than enough about trolls: "There is no question pending. You can hold your comments."
Lieberman (with an air of getting into the bizarre nature of the situation): why did you think this would cause Ms. Kobrin to blow a gasket?
Henson: this wasn't addressed to Helena. He goes on to explain that the message is a loop back. If it worked at all it would be a loopback to your own machine. If you tried it you'd discover it's a troll. The 127 is the loopback address! It's a joke, but the lawyer isn't getting it.
[The observer notices that the RTC lawyer has connected "the 'ho" with Ms. Kobrin. Evidently the nickname has made transit to the solid world. Ms. Kobrin is stuck with it for life.]
It's not "sort of" hacked by an unauthorized outsider! The explanation is pretty clear cut and perfectly believable, no hackers or malice involved at all..
I have never been able to make sense of all the rules around X-Forwarded-For and neither have the various library implementers. I recently wrote an authentication plugin for Envoy that just extracts what Envoy thinks the remote address is, and puts it in the authentication header that goes to the backend. Then the backends can't get it wrong; if the signature on the message is right, you're getting the IP address that the frontend Envoy got. If something is misconfigured, the header probably won't have a valid signature, and so the request will be rejected outright. Less failsafe than what Wikipedia did... but easier to detect.