ISP DNS servers really ought to be banned, they are always so bad. I've seen traffic days later on a record with 1 hour TTL. In general I see like 50% traffic move after the initial 1-2x TTL interval, another 40-45% over next several hours up to one day, and then the last 5-1% can take forever.
For round-robin, I've actually had it work reasonably well for API usage. Of course it's not ideal, but when I wanted to roll out new things slowly over several days and could not use a load balancer or reverse proxy, it kind of worked. I think most API users are just running with a reasonable resolver and not residential ISP ones.
When I moved reddit from one datacenter to another, about 70% of the traffic shifted within the TTL. Another 20% moved within a week. Took till the end of a month after the change to get to about 98%
But after two months, about 1% was still going to the old server (I had set it up as a proxy for the cutover). Most of that traffic looked like crawlers that were written in things like Python or Ruby and had probably hard coded the IP or done something where it just didn't know what a TTL was.
So at that point I just shut down the old server.
You're probably right about API clients using better resolvers though. I was talking about consumer facing things where a lot of people would be on ISP DNS.
> Some users may knowingly install this software on their devices, lured by the promise of “monetizing” their spare bandwidth.
Sounds like they’re targeting networks even if the users are ok participating in, precisely what you’re saying is ok.
As for malware enrolling people into the network, it depends if the operator is doing it or if the malware is 3rd parties trying to get a portion of the cash flow. In the latter case the network would be the victim that’s double victimized by Google also attacking them.
Users are OK with acting as proxies because they don't understand all the shady stuff their proxy is being used for. Also consumer ISPs generally ban this.
But then would you make the same arguments for running a tor node (presumably, you don't know what shady stuff is there, but you know there's shady stuff)?
Personally I consider Tor less shady than these residential proxy networks because Tor has some normal users but yes, the considerations are similar. (I ran one of the earliest Tor exit nodes.)
> These SDKs, which are offered to developers across multiple mobile and desktop platforms, surreptitiously enroll user devices into the IPIDEA network.
> These SDKs, which are offered to developers across multiple mobile and desktop platforms.
> other actors then surreptitiously enroll user devices into the IPIDEA network using these frameworks.
I’m not saying Google did the wrong thing, but it is one private entity essentially handing out a death sentence on its own. The only mitigating thing is that a) technical disruptions were either on their own infra b) legal judgements they then enforced with cooperation from others like Cloudflare. But it’s not clear what the legal proceedings were actually like
Am I the only one cynically thinking that "Russia, Iran, DPRK, PRC, etc" is the "But think of the chiiildren!!!" excuse for doing this?
And when Google say
"IPIDEA’s proxy infrastructure is a little-known component of the digital ecosystem leveraged by a wide array of bad actors."
What they really mean is " ... leveraged by actors indiscriminately scraping the web and ignoring copyright - that are not us."
I can't help but feel this is just Google trying to pull the ladder up behind then and make it more difficult for other companies to collect training data.
>I can't help but feel this is just Google trying to pull the ladder up behind then and make it more difficult for other companies to collect training data.
I can very easily see this as being Google's reasoning for these actions, but let's not pretend that clandestine residential proxies aren't used for nefarious things. The vast majority of social media networks will ban - or more generally and insiously - shadow ban accounts/IPs that use known proxy IPs. This means that they are gating access to their platforms behind residential IPs (on top of their other various blackboxes and heuristics like fingerprinting). Operators of bot networks thus rely on residential proxy services to engage in their work, which ranges from mundane things like engagement farming to outright dangerous things like political astroturfing, sentiment manipulation, and propaganda dissemination.
LLMs and generative image and video models have made the creation of biased and convincing content trivial and cheap, if not free. The days of "troll farms" is over, and now the greatest expense for a bad actor wishing to influence the world with fake engagement and biased opinions is their access to platforms, which means accounts and internet connections that aren't blacklisted or shadow banned. Account maturity and reputation farming is also feeling a massive boon due to these tools, but as an independent market it also similarly requires internet connections that aren't blacklisted or shadow banned. Residential proxies are the bottleneck for the vast majority of bad actors.
> The vast majority of social media networks will ban - or more generally and insiously - shadow ban accounts/IPs that use known proxy IPs. This means that they are gating access to their platforms behind residential IPs (on top of their other various blackboxes and heuristics like fingerprinting)
Social media will ban proxy IPs, yet gleefully force you to provide your ID if you happen to connect from the wrong patch of land. I find it difficult not to support any and all attempts to bypass such measures.
The fact is that there's now a perfectly legitimate use for residential proxies, and the demand is just going to keep growing as more websites decide to "protect their content", and more governments decide to pass tyrannical laws that force people to mask their IPs. And with demand, comes supply, so don't expect them to go away any time soon.
This really just sounds like a rehash of the argument against encryption. "Bad people use it, so it should go away" - never mind that there are completely legitimate uses for it. Never mind that using a residential proxy might be the only way to get any privacy at all in a future where everyone blocks VPNs and Tor, a future where you may not even be able to post online without an ID depending you where you live, a future which we're swiftly approaching.
It's already here, in fact. Imgur blocks UK users, but it also blocks VPNs and Tor. The only way somebody living in the UK can access Imgur is through a residential proxy.
> The only way somebody living in the UK can access Imgur is through a residential proxy.
And very little of value was lost.
> This really just sounds like a rehash of the argument against encryption. "Bad people use it, so it should go away" - never mind that there are completely legitimate uses for it.
Except that almost everything that uses encryption has some legitimate use. There are pretty much no legitimate uses for residential proxies, and their use in flooding the Internet with crap greatly outweighs that.
If I plumbed a 30cm sewage line straight into your living room would you be happy with it? Okay, well, tell you what, let's make it totally legit - I'll drop a tasty ripe strawberry into the stream of effluent every so often, how about that?
No, what they're saying is what they said, what you're implying reveals a strange bias. Web scraping through residential proxies? Please think through your thoughts more. There's much more effective and efficient ways to do so. Multiple bad actors, like ransomware affiliates, have been caught using residential proxy networks. But by all means, don't let facts and cyber threat intelligence get in the way.
> Am I the only one cynically thinking that "Russia, Iran, DPRK, PRC, etc" is the "But think of the chiiildren!!!" excuse for doing this?
Maybe. But until I dropped all traffic from pretty much every mobile network provider in Russia and Israel, I'd get up every morning to a couple of thousand new users of whom a couple of hundred had consistently within a few hundred milliseconds created an account, clicked on the activation link, and then posted a bunch of messages in every forum category spreading hate speech.
If they said "could" then I would agree but they said it did happen. those actors DID do it, not could. So it's not a think of the children excuse. Unless they are outright lying but I doubt the security team came up with a business type excuse
Getting rid of malware is good. A private for-profit company exercising its power over the Internet, not so much. We should have appropriate organizations for this.
The proxies is the reason why you get spam in your Google search result, spam in your Play store (by means of fake good reviews), basically spam in anything user generated.
It directly affects Google and you, I don’t see why they should not do this.
Spam in Google search results is due to Google happily taking money from the spammers in exchange for promoting their spam, or that the spam sites benefit Google indirectly by embedding Google Ads/Analytics.
I don't see any spam in Kagi, so clearly there is a way to detect and filter it out. Google is simply not doing so because it would cut into their profits.
"SEO spammers being more advanced than multi-billion-dollar search conglomerate" is a myth. Spam sites have an obvious objective: display ads, shill affiliate links or sell products. All these have to be visible, since an ad or product you can't see/buy is worthless. It is trivial to train a classifier to detect these.
But let's play devil's advocate and say you are right and spammers are successfully outsmarting Google - well, Kagi does use Google results via SerpAPI by their own admission, meaning they too should have those spam results. Yet they somehow manage to filter them out with a fraction of the resources available to Google itself with no negative impact on search quality.
Many are "compensated" (in the way of software they didn't pay for), so the real question is that of disclosure (in which case many software vendors check the box in the most minimal way possible by including it as fine print during the install)
No, the question is not just disclosure. People have their bandwidth stolen, and sometimes internet access revoked due to this kind of fraud and misuse - disclosure wouldn’t solve that
Also, as a website owner, these residential proxies are a real pain. Tons and tons of abusive traffic, including people trying to exploit vulnerabilities and patently broken crawlers that send insane numbers of requests, and no real way to block it.
It's just nasty stuff. Intent matters, and if you're selling a service that's used only by the bad guys, you're a bad guy too. This is not some dual-use, maybe-we-should-accept-the-risks deal that you have with Tor.
I run a really small forum and I've been absolutely inundated with a bunch of junk traffic. I had to tighten my Cloudflare WAF rules a whole bunch, and start issuing browser challenges way more aggressively.
Excluding known "good" crawlers, well over 99% of the traffic trying to hit the site has been attempting to maliciously scrape. Most of this traffic looks genuine, but has random genuine-looking user agents and comes from random residential proxies in various countries, usually the US.
For the traffic that does make it all the way to a browser challenge, the success rate is a measly 0.48%. Put another way, over 50% of traffic is already blocked by that point, and of the under 50% that makes it to a browser challenge, more than 99.5% fails that challenge.
It's been virtually no disruption to users either, since I configured successful challenges to be remembered for a long period of time. The legitimate traffic is a gentle trickle, while the WAF is holding back garbage traffic that's orders of magnitude above and beyond normal levels. The scale of it is truly insane.
> Ones which you pay for and which are running legitimately, with the knowledge (and compensation) of those who run them.
The problem is, it is by default unethical to have residential users be exit nodes for VPNs - unless these users are lawyers or technical experts.
No matter what you do as a "residential proxy" company - you cannot prevent your service being used by CSAM peddlers, and thus you cannot prevent that your exit nodes aren't the ones whose IP addresses show up when the FBI comes knocking.
I learn: proxy networks run by large corps are good. True internet is bad. While I understand that often we are talking about Malware/Worms etc that enable this. However, i find it often disturbing to here often a lot of libertarian speech from the tech scene, while on the other hand are feeling themselves very comfortable to take over state power like policing efforts to save the world.
People say the reason nigerian prince scammers use such ridiculous story, or bank phishing has so many typos, is to pre-filter dumb and gullible people so the scammers don't waste time on targets that won't get scammed in the end.
All these AI "hacks" seem to be based on the same principle.
To your point, from the article: "To me, giving a Claude skill all your credentials, and access to everything important to you, and then managing it all via Telegram seems ludicrous, but who am I to judge."
To be fair, dang's position has consistently been (and he posted as such yesterday[1]) that the flag brigading comes from two distinct groups: 1. Users who don't want to see "politics" and 2. Users who are partisan and want to hide politics they don't like.
My view is that these are essentially the same group. "Not wanting to see politics" is itself just a partisan view in favor of whatever the Status Quo currently is. So if you're going through and flagging articles describing wrongdoing or calling for change, because they are "political," then you're just operating in the service of whoever is currently in power.
> My view is that these are essentially the same group. "Not wanting to see politics" is itself just a partisan view in favor of whatever the Status Quo currently is. So if you're going through and flagging articles describing wrongdoing or calling for change, because they are "political," then you're just operating in the service of whoever is currently in power.
But some people do that regardless who is currently in power, what do you call that? I'd call that "not wanting to see politics" and not being partisan.
> Given the consistency and speed of flags though, I don't think #2 is just "organic" flagging.
Thousands of people view these very quickly once they hit the frontpage, its enough 0.1% of those are willing to flag to see consistent near instant flagging.
Followed by the usual sycophantic support: "You're doing a great job dang, we all love you so don't worry about these cranks", and "Well I'VE never noticed any censorship so it obviously doesn't exist" (Yeah dude that's how it works lol).
The CO2 cycle is problematic because of timelines. We are releasing millions of years of CO2 accumulation.
Rain is more of a location problem. The evaporated water returns as rain quickly, but maybe somewhere else, such as over ocean. And the aquifer compresses and loses water retention ability.
Exactly. I'm watching this from the Netherlands. Until last year I always ignored political posts here but now it's become an existential necessity to be involved.
They most likely use GPON so the optic is going to see return traffic for your neighbors. So they make it hard (but not impossible) to bring your own optic or media converter.
AFAIK GPON uses encryption, so you actually get the traffic intended for all your neighbors but can't do anything with it. If you bring your own converter, you wouldn't be able to handle your own traffic either.
Also the authentication might rely on weak secrets. I know my ISP provided FTTH router has a six letter password and a guessable username (derived from my last name), and I can't change either.
Though the research is quite old now. Couldn't find anything recent specifically for DT.
At least for Germany, you can buy the Digitalisierungsbox Glasfasermodem or any other modem. You just have to register it with the DTAG via their hotline.
For round-robin, I've actually had it work reasonably well for API usage. Of course it's not ideal, but when I wanted to roll out new things slowly over several days and could not use a load balancer or reverse proxy, it kind of worked. I think most API users are just running with a reasonable resolver and not residential ISP ones.
reply