> In general, bots are pretty easy to exclude. They usually advertise themselves by a user agent string. Yes, that can be faked – but it seems highly unlikely that bots using faked user agents create such a large number of impressions that Google has to use this route against them.
There's legits ones that the site owners will generally find to provide a positive tradeoff. These bots identify themselves by the user-agent, the requests come from a predictable set of IPs, and the they obey the robots.txt. Think most crawlers for search engines (though not Brave's), bots that handle link previews for apps like WhatsApp, even RSS readers!
Then there's the abusive ones. These are usually hitting resources that are expensive and contain valuable information. The will not obey robots.txt. They'll run residential IP botnets to avoid IP blocks. They'll make their bot as similar to legit traffic as possible, the user-agent is literally the first thing they'd look at changing. They'll hire mechanical turks to create fake accounts to look like signed in users.
Now, it's pretty obvious why the author's methodology for supporting the statement is so silly. First, it was circular. They identified bots by user-agent, and then declared that since there were bots that had a distinguishing user-agent, the other traffic can't have been bots. The other is that they looked at the logs of a server that doesn't contain any data that somebody would be scraping maliciously. Ocean's 11 will do a heist of a casino, not a corner store. Likewise the professional bot operations are scraping valuable information from people trying to actively defend against it, not your blog.
> Now, it's pretty obvious why the author's methodology for supporting the statement is so silly.
i also found this odd. the target matters.
when i did more sophisticated dual tracking (js interaction and http logs), the js interaction detected 2x-3x the bots vs pure logs from just UA strings and ip ranges.
Speaking as someone who works at Google, but not on anything at all related to ads, browsers, or ad spam detection, I only wish that the attackers who (try to) make a living out of scamming Google and its advertisers out of money were as incompetent as the author of this article appears to be.
I mentioned this in the other WEI thread and I’ll do it here again:
Instead of simply flailing our collective arms around complaining about an evil corporation, has anyone written to the respective competition authorities (such as the FTC in the US or CCI in India) about the potential anticompetitive effects of this proposal?
My thoughts exactly. These GitHub protests, while emotionally satisfying, do not work. Google does not care and they are already drunk on monopolist power.
I would like to bring your attention to Google’s recent proposal to add a feature to its Chrome (Chromium family) of browsers called Web Environment Integrity. This provides a mechanism to reinforce Google’s already dominant browser market position by creating a technological control that can be used to nullify a user’s choice of browser, device and operating system. This technology also has the potential for abuse by preventing users from using browser extensions that can enhance security by blocking unwanted and potentially malicious content, as well as browser extensions that help vulnerable users with enhanced accessibility needs, such as color blindness and visual impairment.
Google’s dominant, near-monopoly position in the browser market already harms me as a consumer by reducing browser choices and preventing a competitive market for developing new browsers. Allowing Google to include this feature will reduce my browser choices and consolidate the browser market even further, and it is incumbent on [INSERT AUTHORITY HERE] to take action against this abusive behavior.
Subject: Potential Threat to Digital Freedom from Google's Web Environment Integrity
Dear [Recipient],
I'm writing to highlight an important matter regarding the digital freedom and competitiveness on the internet. Google is rapidly advancing a policy named "Web Environment Integrity" (WEI) in their Chromium browser.
WEI allows developers to regulate browser configurations, which could lead to limiting the usage of free browsers or operating systems. This creates a potential for a web environment that discriminates based on browser usage. Further, this scenario could pave the way for governments and corporations to enforce specific browser usage and could also allow Google to restrict access to their services based on browser compliance.
This practice contradicts the fundamental principles of an open and competitive digital marketplace. I strongly encourage your agency to investigate the potential impacts of Google's WEI and consider taking necessary actions.
Your proactive engagement is vital in preserving the principles that ensure a free and open web.
If you are in the UK you can also contact the competition and markets authority.
I've also created a parliament petition, which has gotten the 5 min supporters it needs before they review and publish it. I will share it on HN once its published.
Edit: removed the link to the petition for now (it'll come back after its published)
Trying to reach official authorities is a good idea.
I will quote and extend part of my call to action I did in an other thread
- ban Google all together in your personal life. No chrome and no excuses. Stop the bullshit or leave this profession. Use startpage, duckduck or whatever for searching.
- develop with and for firefox and friends only, introduce usability problems for chrome
- employ the same tactics as google.
-> Bundle firefox with the software you are distributing.
-> Like google did, remove the competition altogether from the users device.
-> make your npm-module or your website slower in chrome
-> let your customers know that your service for non-chrome users is cheaper. Money motivates.
-> show a popup urging users to download firefox, provide a link to download or page with more explanation.
Tell that you detected that their current chrome has security and privacy risks and that you recommend to take action immediately. Average user is easily scared into action.
-> use as many tricks as you can think of to spoil the well for google.
Destroy search results, fill their storage with /dev/random, whatever your imagination leads you too. You keep telling us you are so smart. Show it.
- remember, Google's capital is data. Hit that and the beast will die.
> show a popup urging users to download firefox, provide a link to download or page with more explanation. Tell that you detected that their current chrome has security and privacy risks and that you recommend to take action immediately. Average user is easily scared into action.
If I recall correctly, this was Google's approach with Chrome.
This is incredibly hypocritical. I would never want to work with or for, or employ, or be friends with or associate myself with someone who blatantly displayed this level of hypocrisy and lack of integrity.
This comment is also clearly violating the HN guidelines - it's not intellectually interesting, it's naked political activism.
I'm not well versed on scrapers. How do you fill their storage with /dev/random? Did they employ this tactic on competitors at one point as with the rest of your psot?
gmail, google storage, contacts, content farms, Google slurps everything it can find. /dev/random would be something for storage. For other stuff you would fake data. No, not all the tips listed have been take from their playbook.
> has anyone written to the respective competition authorities
Just a reminder that several states have already filed an antitrust suit (in part) over a previous Google plan to turn the web into their own walled garden.
> Project NERA was Google’s original plan to create a closed ecosystem out of the open internet. Google documents reveal that Google’s motive was to “successfully mimic a walled garden across the open web [so] we can protect our margins.”
According to Google’s internal documents, the strategy would allow Google to extract even higher intermediation fees. A Google employee aptly described Google’s ambition for Project NERA to “capture the benefits of tightly ‘operating’ a property … without ‘owning’ the property and facing the challenges of building new consumer products.”
Google main strategy to do this was to leverage its popular browser, Chrome, to track users, by forcing them to stay logged into the browser. Google did this by logging users into the browser when they logged into any Google property such as Gmail or YouTube, and logging them out of services when they logged out of the browser.
> "Instead of simply flailing our collective arms around complaining about an evil corporation, has anyone written to the respective competition authorities (such as the FTC in the US or CCI in India) about the potential anticompetitive effects of this proposal?"
Yes, I have. A couple times now.
Google has been strongly signaling this since last year. No one wanted to believe it last year though, before the tech bubble burst. Now that people see Google isn't so awesome right now, perhaps more people will write and contact their representatives.
Has anyone sent such a message to their authority? Please share, as more authorities (Norwegian anti-competition authority will surely want to hear about taht) need to be contacted with well-researched text.
So you're upset about that decision. Fair enough! I can't think of any regulatory agency that hasn't made a decision I thought was terrible at some point or another.
But that doesn't mean that what they do, big picture, is pointless.
Fact of the matter is acquisitions are a corner stone of the tech economic model. If they actually cared about consumers and competition they would go after MS for bundling Teams. Or MS for round-tripping cash with OpenAI.
I'd rather they bring cases and lose some instead of never bringing cases and letting the corporations do whatever they want with no fear of consequences.
If they bring cases and lose them corporations will continue doing whatever they want since there will be no consequences. It's important for the FTC to know what fights are worth fighting, and they will be taken seriously.
So a technical note I’m posting here mostly because I dug into this only for a comment I was responding to to be deleted:
What they lost in the case they filed in June was an enjoinment to prevent the merger and acquisition of Activision/Blizzard until their own FTC judge (read: an administrative law judge that exists outside of Article III and is within the chain of command of the Executive branch) could hear the case on August 2nd. The merger had a termination date of July 18th, so they needed that to continue their administrative review. Discovery was finished, it was just the trial, but without being able to enjoin the trial because in the opinion of Judge Jacqueline Scott Corley they were unlikely to prove the merits of their assertions, the trial before the FTC judge would have been moot by the time it occurred. It’s been formally cancelled by the FTC by the way.
I’m not disagreeing with you by the way, I just wanted a place to park this information in the discussion. They started this action in December and failed to win even an enjoinment against Microsoft and Activision temporarily stopping the merger until their own guy could hear the case.
FTC is actually the most active it's been in decades. Blocking mergers is one of its core functions. Not to mention the stance against NDAs, etc. Where have they been lacking?
I actively despise google and avoid any product they make (except youtube, got no choice there) for this reason.
It goes back all the way 2010, I remember opening up chrome to try it, right clicking on a youtube channel background to attempt to download like I could on firefox just fine, and it not having the option, why would you go out of your way to restrict a user easily being able to right click and download? well, because you believe you own the web.
Never used Chrome and never will, if you use Chrome you are actively making it worse for yourself in the future once they implement enough bad policies that it becomes near impossible for almost anyone or anyone to bypass their restrictions.
...you may feel that your insights and experience can be valuable to help steer the platform from making what you're sure is a huge mistake. That's great!! Getting involved in web platform discussions is essential to ensure it's built for and by everyone.
...
In cases where controversial browser proposals (or lack of adoption for features folks want, which is a related, but different, subject), it's not uncommon to see issues with dozens or even hundreds of comments from presumably well-intentioned folks, trying to influence the team working on the feature to change their minds.
In the many years I've been working on the web platform, I've yet to see this work. Not even once.
--- end quote ---
"We do so love for everyone to join the discussion. It also never influences our decisions, not once"
1. Often the feedback goes completely to the wrong address. You won't stop Google from doing google things.
2. Most often the depth level at which the discussions on web standard are made will alienate most people, so instead of participating in "standards making" they turn somewhere else (1.).
The web is awesome and it got awesome because for the first 15 years of its existence it was actually very straight forward to run a web entity. But success brought ever growing companies and ever more complex interests. The discussions also vary a lot nowadays. There are still things being done to make the web more approachable but at the same time we see stuff like "Web Environment Integrity", DRM etc.
The problem is that a process that requires the public to be vigilant will eventually fail if the public cannot appoint people to be vigilant full time for them.
> Most often the depth level at which the discussions on web standard are made will alienate most people, so instead of participating in "standards making" they turn somewhere else
It also takes a lot of time. You have to read quite a few proposals, and there are literally hundreds of them, you have to participate in discussions in the GitHub issues, on the w3c mailing list, and in multiple face-to-face discussions.
Even the most technical people find this daunting because they are not paid for this (unlike the people making and promoting the specs). So even the technical people often come into an issue, voice their concerns briefly (or not-so-briefly) and are summarily dismissed.
I've seen Google engineers misrepresent and ignore any input from engineers working on Firefox and Safari, and just push their specs forward. So what chance does an outsider have?
It's a mess.
Granted, it's a better mess because so many discussions are happening in the open unlike 10-15 years ago, but it's still a mess.
Clearly the implication is that rushing to join a professional discussion just to yell about some or another controversial proposal you read about on HN is not going to work to sway the stakeholders. If you want influence, you need to cultivate it over time by building trust in the community you want to influence. That's hardly controversial.
In particular, taking a fairly dry proposal like WEI, which is intended as a anti-bot/anti-cheat framework for web content, and spinning it with a shitpost title like "Google vs. the Open Web" is really not going to ingratiate you with the people who think hard about very difficult problems every day.
Is it a good proposal? Honestly I don't know. But the problems it's trying to address are real, so I'm inclined to give the benefit of the doubt to the people trying to solve them in good faith over the shitposters.
Characterizing a fairly reasonable position like "we should have a way prevent bots at the client level" as an attack on "The Open Web" is pretty much the definition of a shitpost, no? It's a terrible strawman and it pollutes the discourse.
Do you really not agree that people might want the former and not the latter? You genuinely think that the standards folks are being driven by a conspiracy and not what they say they want?
There are ways to argue against WEI that don't involve the existence of enemies you have to fight. Maybe you could try them?
1. (noun) any content on the internet whose humor derives from its surreal nature and/or its lack of clear context. Differs from a meme: whereas a meme's humor comes from its repeatability, a shitpost is funny simply because it isn't a predictable repetition of an existing form. Shitposts can become memes, but memes cannot become shitposts.
2. (verb) to create such a post
No idea where you're citing. Oxford gives me "a deliberately provocative or off-topic comment posted on social media, typically in order to upset others or distract from the main conversation", which fits my usage perfectly. You'd agree this framing is "deliberately provcative", no?
Wikipedia explains it similarly: In Internet culture, shitposting or trashposting is the act of using an online forum or social media page to post content that is satirical and of "aggressively, ironically, and trollishly poor quality"; it may be considered an online analog of trash talk.
Even Urban Dictionary is on board: A post of little to no sincere insightful substance. Especially a "shit"(low)-effort/quality-post with the sole purpose to confuse, provoke, entertain or otherwise evoke an unproductive reaction.
Frankly I have to assume you went out of your way (like, off the front page of a Google search even) to find a definition that you could cite just to prolong an online argument. I wonder if there's a word for that.
None of those definitions fit the original blog post. It's not satirical or ironic, it's not aggressive, it's not trolling, it's not off-topic, it's not even a "comment". It's not poor quality (imo). It's an opinion piece.
My definition was from urban dictionary btw, the first entry, maybe it sorts differently for different people.
"Google is Attacking The Open Web!" is 100% aggressive. Whether it's trolling or not depends on how people react to it and not its content per se. And here we are in this ridiculous subthread. So, yeah, it was trolling too.
Come on. I repeat: it's a complicated subject and a real problem, and a sincere but potentially flawed proposed solution. It deserves serious discussion and not a bunch of yahoo's throwing bombs about the evil corporate overlord of the week.
- The WEI check will be designed with a level of simplicity that tech-savvy individuals or hackers can easily bypass. Criticisms or objections will be quieted with comments like, "You just need to initiate the browser using these 50 different settings and you're good."
- On the other hand, the WEI check will be intricate enough that an average user won't be able to circumvent it, resulting in them being obligated to view ads.
In this way, it's a win-win situation: the hackers maintain their access to an "open" web, while the vast majority (99%) of the population will navigate through a "Google" web.
This is already one of the use-cases listed for WEI. The intended implementation of WEI will be Play Protect which lives in ARM TrustZone and thus runs above the kernel[0]. So you'll have something even more invasive than kernel-level anticheat.
[0] In ARM speak, kernel mode is EL1, hypervisor mode is EL2, and TrustZone mode is EL3. Each exception level is a higher level of privilege.
It might be more common than you think. Some major SAST tools complain if you aren't checking if the device is rooted, and it wouldn't surprise me if some naive shops blindly followed the recommendation without need.
In the 2000s it was funny how corporates just failed to understand how client server models worked. Nowadays it is just sad and a reason to move more and more towards crypto for the day to day banking stuff...
>Oh right! I said crypto on hacker news! what was I thinking?! :-D
Post that comment again when crypto accounts are FDIC[0] (or whatever scheme, if any, is used where you live) insured. I'm sure you'll get a different response.
SecuROM is DRM for PC games that installs a rootkit. I first learned about it when it was used with Spore 15 years ago and it bricked my Windows install.
No, Valve is specifically noted for not having a kernel-driver anticheat in a landscape where most competitive games do use them. Notably, Easy Anti-Cheat, BattlEye, and Valorant's Vanguard all use kernel drivers, but no Valve Anti-Cheat has, because they've focused on server-side heuristics and crowed-sourced detection instead of trying to force the client to rat itself out.
This is a clear sign of Google's weakness. They are losing their monopoly and are desperately trying to hold on to the net. In the last few weeks, they have announced that they will try to block navigation if you have an ad blocker installed (for example, when watching a video on Youtube).
Take a look at Fuchsia for another example ... they are losing the control on Android, so they started this new project ... it is another sign.
My recipe: AdGuard Home, Brave browser (phone, tablet, desktop), Bromite (phone), Firefox (desktop) + uBlock origin plugin ... and FreeTube on desktop.
Just using Brave on the phone is enough to kill all ads and trackers.
In the open source community, there will always be someone smarter than they think who will find a way around their gates...
Few days ago Kevin Mitnick passed away, sadly, but there will be always another Kevin Mitnick ...
Google will lose all respect from the community and will collapse sooner or later.
> Take a look at Fuchsia for another example ... they are losing the control on Android, so they started this new project
I work on fuchsia and can honestly say I have no idea what you're talking about. Fuchsia and android are more complimentary than they are competitive. I've noticed that when there is a lack of information, people tend to invent things that fit their narrative, but that's a really dangerous habit.
Taken from Wikipedia :
Fuchsia is an open-source capability-based operating system developed by Google. In contrast to Google's Linux-based operating systems such as ChromeOS and Android, Fuchsia is based on a custom kernel named Zircon. It publicly debuted as a self-hosted git repository in August 2016 without any official corporate announcement. After years of development, its official product launch was on the first-generation Google Nest Hub, replacing its original Linux-based Cast OS.
And from 9to5google.com
Work on this Fuchsia project within Android — dubbed “device/google/fuchsia” — stalled in February 2021, with no public indication of how things were progressing. This week, all of the code for “device/google/fuchsia” was removed from Android, formally signaling the end of this particular avenue.
In its place, we have a lone “TODO” message, suggesting that Google may be building up something new in its place. The developer responsible for the change primarily works on Fuchsia’s “Starnix” project.
First shared in early 2021 as a proposal, Starnix is designed to make it possible for Fuchsia to “natively” run apps and libraries that were built for Linux or Android. To do this, Starnix would act to translate the low-level kernel instructions from what Linux expects to what Fuchsia’s Zircon kernel expects.
So ... custom kernel and a custom OS that will support Android applications as far as I understand ...
Hiroshi Lockheimer once confirmed that Fuchsia at this stage (to be exact, 4 years ago) is more of testbed for OS technologies that cannot be readily integrated into Android. It is quite absurd to say that Fuchsia is a competitor against Android. It is more close to Midori with a slightly clearer path to productization.
Yeah, I think that's the ambitious moonshot at least for Fuchsia team, and Google might hope it to be realized. But it probably also acknowledges that it's a very unrealistic goal. More likely scenario is to gradually replace some important core systems (including its kernel?) with Fuchsia while keeping the overall Android ecosystem.
I agree. I don't think they can abandon the Android ecosystem at all, I mean all the apps and the store, that's the real value. They could design a migration path for the apps and make them work seamlessly on Fuchsia in the meantime, gradually replacing the ecosystem with the promise of new "shiny" features for developers. Building an OS from scratch is very expensive in terms of resources and money. I cannot see a valid technical reason for this move. My view is that many projects are now using AOSP to build their own operating system and trying to get rid of Google services, which is a threat to Google and its business based on bombarding the user with ads. The biggest obstacle at the moment is getting the applications that rely on Google services to work. e/OS/ uses MicroG and in my personal experience everything works seamlessly, including banking applications. The other crucial aspect is the availability of stores for the apps. Aurora is just an alternative front-end client to Google PlayStore, but it is a huge step forward in removing direct dependency. e/OS/ has AppLounge which does the same thing.
I'd certainly prefer to download and install my bank's app directly from a protected area of the bank's website rather than from a generic store. Implementing a custom authentication mechanism (e.g. signed with GPG) and an auto-update feature is certainly doable.
>Google will lose all respect from the community and will collapse sooner or later.
I love this little bubble all of HN (or at least a vocal majority) seems to live in. Google is most definitely not collapsing anytime soon, and their products are loved by millions, if not billions, of users all over the world.
>They are losing their monopoly
No, they most definitely aren't. Brave Browser runs on top of Google's Chromium. Firefox runs on top of Google's money. Their lead in search does not seem to be going away anytime soon - there is a reason literally everyone on earth uses Google as a search engine. There is a reason literally everyone on earth uses YouTube to watch any video they want. There is a reason 70% of all phone users use Google's operating system. There is a reason Gmail is by far and away the clear leader in the personal email space.
>They have announced that they will try to block navigation if you have an ad blocker installed (for example when watching a video on YouTube).
As they rightly can. You are under no obligation to use YouTube - and if you do use it, you must pay for it, either by watching ads, or by paying for YouTube Premium.
HN can keep complaining about Google all they want, but Google is one of the few companies that has truly made the Internet the Internet. Their impact on humanity has a whole has so far most definitely been net positive, and you are under no obligation whatsoever to use their products. There is a reason they are the clear leader in the products they offer, and that is because they offer, say, a free tier (as in Gmail), or openness (as in Android).
> you are under no obligation whatsoever to use their products
Well ... with this new proposal they are trying to change this, don't you think ?
Yes, it is not mandatory to watch Youtube, but it should be also mandatory that Google don't collect and sell the personal data without the owner permission or scan all the emails in every Gmail account (free o paid) ... The history of Google is full of these practices and, after discovered, every time they respond "will never do it again" ...
"the rotten tree-trunk, until the very moment when the storm-blast breaks it in two, has all the appearance of might it ever had." - Isaac Asimov, Foundation
While that's a very nice saying, and I appreciate you applying in this context, what you're basically saying is we can never ever assess any organization as strong whatsoever, since every organization that breaks up seems strong at some point.
That's not what was said at all in context, and I do not appreciate you putting words in my valid comment. You dismissed the original argument with your own personal truths.
You didn't really make a comment -- you just dropped in an Asimov quote. And xNeil's interpretation of the relevance of that comment matched my intuition. If you had some other intent with that comment, maybe you should clarify?
I guess I can see why you feel that way - you intended to say a company that seems strong may be at risk of failing just like any other company (in this case, just because Google seems large does not mean they are not failing) - which is something I (sort of) agree with!
But doesn't it logically follow that the same truth holds for any other 'strong' company, thereby rendering our perception of it (or any other company) worthless? I'm sorry you're disappointed, but I just made a logical continuation, that's all.
> Their impact on humanity has a whole has so far most definitely been net positive, and you are under no obligation whatsoever to use their products.
The strategy over the years has always been the same:
1. create a necessary product and give it away "for free"
2. wait until people are used to it and consider it essential and difficult to migrate
3. close the gate and make it no longer free.
For example : Gmail for organisations (at launch free up to 100 users, then 50, then 10, then 0), Maps for websites (lower free tier now), Google Drive (lower free tier now), Youtube is next ...
That these are the "best" products in the world is a subjective affirmation. They are pre-installed on devices and difficult to remove ...
They can do whatever they want with their products, of course, but trying to control the openess of the web as we know now, it is a different thing ...
If you are this paranoid about someone showing ads or collecting your information, maybe Brave isn't the best choice, with their history of getting caught with both hands in the cookie jar. Especially since you already use Firefox elsewhere. Mozilla also collects information btw.
I almost never see meaningful detail regarding what is collected and what it’s used for. Is Google’s collection equivalent to Brave’s, or to Firefox’s? I’d be very surprised if there were not significant differences between what is collected here. Comments like this draw a false equivalence between the three.
That's a bit of a strawman since I don't compare the three. I only compare Firefox and Brave. Of course Google is collecting the most data out of those and the tracking is worse, especially since they have other data points to compare it with.
However, according to my logic using Brave + Firefox simply must cause more data to be collected than using only one of the two, no?
I use different browsers for different websites / web applications.
For bank or accounting, for example I use Firefox in a container.
I configured Firefox to clear everything on closing : cache, cookies, history.
I don't save the passwords in the browser. I keep them in Vaultwarden, installed locally on a miniserver.
I'm currently trying out LibreWolf. It is based on Firefox, but with extra privacy and security features. I also installed Vivaldi, also if I am not a big fan of all these extra features integrated : mail, calendar and notes.
It wasn't meant as criticism. Doing something is better than nothing. Of course the irony is that the more you do the easier it gets to fingerprint and track you with cookieless technology. I've been there and given up. The best solution in my opinion is to blend in and hide in the masses while blocking the worst offenders. I use Firefox with uBlock and DNS blocking in my firewall (OPNsense).
I participated in an experiment that tried to fingerprint without cookies over time. All browsers failed but Firefox did best (for me). So that's what I use.
I mean, there are now many other open source projects based on Android (LineageOS and e/OS/ for example) that are free from Google. If they can't control the operating system on your phone because it's free from their services, they can't control your device, track you and send you their ads...
I've been using e/OS/ for 3 years now on a phone made in Germany (Gigaset). There is always an alternative...
LineageOS isn’t entirely free from Google. It relies on AOSP, which is maintained by Google, and it suffers from the decisions that Google makes. For example, Google made a change in AOSP to require location services to be installed as a system package instead of a user package, something few users know how to do. The result is that users are less likely to use something privacy-protecting like Mozilla’s location services. Moreover, Google has reimplemented a lot of AOSP functionality in its Play Services and the industry now uses those Google APIs instead of the old AOSP ones, so loads of apps won’t even run on LineageOS.
LineageOS is a great illustration of how Google is winning. Years ago I could use LineageOS or Cyanogenmod as my primary phone just fine. Now it's very hard to do that if I want to be able to use an increasing number of apps (banking comes to mind). And now I won't be able to bank with Firefox, either.
A huge number of people only use the internet with their phones, and Google is doing their best to tie the entire system to Google services and the Chrome browser.
Exactly. And why the phones are sold with preinstalled with all the Google software and services and it is difficult to remove them if not via adb cli ?
If they are the best in the World as someone else has stated, the people will flock to install them on a "vanilla" phone, right ?
All your SMS, contacts, emails, location positions, photos ... One time my mother was at a funeral at a cemetry, nearby is a cafè. 20 minutes later she got a notification from Google : "How was your experience at the cafè ?" What have this to do with their own products, their free tier levels, the freedom to not watch Youtube or use Gmail ?
Its sad for me to see Google being herded in this direction over the last few years. Google was one of the main push behind an open web thorough 2000s-2010s as they wanted data for search and when everything was open they had access to everything. But as new web 2.0 companies came about like facebook that started siloing the internet it started changing things. I have been anti facebook for this reason alone not its data mining etc but because it was the reason the web started to change where instead of websites a lot of companies started building facebook pages with the data not being available unless you are logged in to facebook.
Yes. Wasn't Google also one the main forces behind PWAs? Google also introduced so many new web APIs (even WebUSB!) in an attempt to make web apps competitive with native apps and their native APIs.
To resolve this conundrum, Google as a whole cannot be said to be "for" or "against" the open web. Instead, Google's infamous internal infighting means that you can only say some parts of Google are for the open web, others are against, and sometimes one has an upper hand.
I don't want to play devil's advocate, but when I consider the history of personal computing it would almost surprise me if this didn't get through. From locked-down smartphones having become the norm to windows having turned into a carrier for advertisements, this just seems like the next step in some inevitable evolution of IT.
Maybe news agencies aren't much interested because this is still only an early proposal, but considering its implications I find it striking how little it seems to be talked about on the web (outside of hackernews). Rossmann seems to be the only one with a video on the topic up on Youtube. There's only a handful of Twitter results for "web environment integrity api" with next to no replies to them. When I look the keystring up on reddit the only result with a noteworthy amount of debate is not related to WEI at all. Social media is probably just on its last legs, but it doesn't seem like too many people that care are left to fight for what the web, or even general computing, used to be.
How does WEI work with non-browsers, like curl or python requests? I was wondering if there is some motive here to monopolize web scraping (especially with respect to harvesting AI training data)?
Companies like Google love kicking down the ladder. You can bet that the Google crawler will have its own "attestation token" but if you want to crawl the Web with your own code you'll be SOL.
All these billion-dollar tech companies got their start thanks to open, accessible, hackable systems. Now it's all being locked down so only the big guys can play, and the rest of us have to pay a fee just to put our "apps" into their walled gardens, and if we do anything they don't like (or are just unlucky) then we get banned forever.
"You can bet that the Google crawler will have its own "attestation token" but if you want to crawl the Web with your own code you'll be SOL."
Let's be real here and note that while most web properties welcome Google crawlers, there are many, many other scrapers/crawlers that offer zero value to web operators while costing resources.
This is just silly, there exist frameworks like selenium that allow you to run any browser of choice and emulate actual user behavior(clicks, keystrokes). If they go further the emulation layer will have to be moved higher, above the virtual machine running the browser for example. The truth is, this has nothing to do with scraping, scrapers will find a way. This is to stop the majority of people from using ad block.
Hi, Selenium & Appium creator here. I've always been on the test automation side of things. The fact that these tools were also useful for scraping was an interesting coincidence to me. These days I make physical robots that are the "real world" equivalent of Selenium or Appium with a stylus that actually taps the screen and presses buttons. To websites and apps, taps and clicks are real, not emulated. Primary use is still test automation, especially when it also involves a real-world component like a credit card transaction with a credit card reader. The number of people contacting me who are interested in getting a physical robot as a way to circumvent software bot detection is increasing. Yes, scrapers will find a way.
Thanks, although I'm not active day-to-day on the Selenium and Appium projects these days. All my love to the current maintainers keeping the projects going!
Wow, thank you for the great software :-) And the physical robot approach is very interesting. Of course it introduces physical world limits (you can't run 1k tests in parallel to load test the site unless you have 1k robots), but still it is very cool.
If I understand this proposal correctly, this is exactly to prevent such things. Yes, of course, it’s to prevent people from using ad block. But a nice side-effect is to block crawlers, or frameworks like selenium as well, so they can „serve ads only to real people“. Of course, people will always find a way to crawl. We already have bot farms that are just remote controlled smartphones lined up somewhere. But it makes it harder for everyone who isn’t Google to compete with Google.
>If they go further the emulation layer will have to be moved higher, above the virtual machine running the browser for example.
Your hypothetical change of emulation tactics won't work. You're analyzing at the wrong abstraction level.
The "attestation tokens" to validate the integrity of the web browser environment would come from a 3rd-party (e.g. Google Play services).
For example... Today, hacks like youtube-dl work because implementing client-side code to "solve javascript puzzle challenges" is still inside the "world" that Google-server-to-browser-client present to each other. Same for client-side solvers for Cloudflare captchas. The "3rd-party attestation token" breaks those types of hacks.
I'm still really irritated by all the people at Google who claimed "Team Web" like they were the good guys, while working on things like AMP designed to proprietize the web.
What we really need is to get the W3C and IETF to straight up throw out vendors who repeatedly push user hostile proposals.
Do not let anyone with employed by Google contribute a web standard. Period. And reevaluate the ones already accepted while we're at it.
The bright side is, when Google really pushes through with this WEI nonsense, it will not only break the Web, it will also create some kind of premium Web run by Google, analogous to gated communities. And then the worldwide Internet ad market bubble will finally burst.
How can a bot create fake impressions? When a bot (or just a simple program) makes a http request he fetches the raw html code only. AFAIK if you don't actually render the html code in a browser or requesting all the contents afterwards again with http requests (like GET ad.jpg, GET logo.png etc.), no google ad server should be hit. Now you could argue that bots could inflate the popularity of a website and therefore the cost to run ads on it. But I guess websites that show ads have most likely google analytics running, one of the only ways Google can actually calculate the popularity (besides Google Search and maybe Chrome history). So it should be no problem for Google to exclude bots from the popularity calculation by analyzing traffic. Maybe I am just missing something, I am also no ad expert at all.
It's not about bots creating fake ad impressions by accident. It's people writing bots whose purpose is to fake ad impressions and clicks. They'll then run it on their own website that's running ads, with the goal of being paid by the ad network for this fake traffic.
But isn't this a win situation for Google to a certain extent? Since it uses up the budget of the advertiser much faster. And the accuracy of filtering new revenue coming from ads as a company is already fairly limited in general. But maybe there are multiple reasons that Google really only wants to serve real humans to the ads of its clients.
It's not a win. The fake clicks will not convert to sales, and the advertisers are seeing a lower ROI on their ads and will go and spend their budget elsewhere in the future. All ad networks will try to filter out as many fraudulent clicks as possible, because they are not optimizing for the maximum revenue today but for the revenue in the long run.
But yes, of course this is not just about filtering out fake clicks. The draft proposal lists a bunch of use cases, most of which have nothing to do with ads.
Interesting explanation, I totally agree on the click-per-pay part. But how would you track the benefit of ads with paying-per-impression? I know its less expensive, but according to the article paying per view seems to be a quite big part of the ad business.
The article is just straight out wrong about "Google’s ad network charges per impression". The author clearly doesn't know anything about the area, made up some shit on how things could work, and just wrote it into their article with no fact checking.
You're right that attribution and measuring ROI is way harder and less precise for ads sold by impression than by click. That's why they're not the common form of advertising, especially on these kinds of ad networks. But for cases where the ads are per impression, the concerns about fraud would be exactly the same. It's not about a crawler accidentally generating impressions, it's about bots deliberately doing so.
Don't nail me down on this but I think since nowadays' websites are often dynamic, you most likely have to employ headless browsers in order to do whatever it is you want to do. This should then result in fake impressions.
Makes me sad to think how locked-down modern computing is becoming. Between app stores, DRM, TPM, and proposals like WEI future generations of hackers will have a very different experience of what you can and can't do with a computer than I did.
The comment about it killing scraping makes me sad. Figuring out a website's api and collecting your own dataset using python+scrapy for personal ML projects is a wonderful learning exercise that I recommend to everyone. A world of only approved datasets from Kaggle etc. is not the same.
I'm sorry, but using the term "generative system" and complaining that this undermines the internet founded on "generative systems" is perhaps the least impressive way to get anyone to care about the open web. Using buzzwords from some random paper just overwhelms people and doesn't convince them to care.
Tell my uncle, or my aunt, that "Google wants to undermine the internet of generative systems!" Whatever. Tell them "Google wants websites to be able to block any devices you might have modified, in any way, that the website owner doesn't like" and you'll get a much stronger reaction.
>but it seems highly unlikely that bots using faked user agents create such a large number of impressions that Google has to use this route against them
Clickfaud makes the frauders money. There is a financial incentive in fooling Google's bot detection. There is a lot of bets using a faked user agent.
>Websites can refuse service unless you install their proprietary data collection agent. Websites can refuse service if you use the wrong browser
Another year closer to a Shadowrun future, minus the magic, where the most powerful corporations run everything, are more powerful than most nation states, and where your only allowed role in life is a consumer/corpse-servant for life (unless you want to risk the harsh penalties of being illegal and running the shadows).
I think we will get to a PostCapitalist future. The decisions we make in the next 7 years will likely determine whether the probable future is dystopian like Shadowrun, or utopian like Paul Mason (see his book "Postcapitalism: a Guide to Our Future").
Personally, I prefer Mason's, with his goals of:
- Rapidly reduce carbon emissions to stay below 2 °C warming by 2050 (edit: We've lost this battle, see the current 6-sigma sea ice event and recent AMOC reports - maybe we can hold it to 3 °C).
- Stabilise and socialise the global finance system.
- Prioritise information-rich technologies to deliver material prosperity and solve social challenges such as ill health and welfare dependency.
- Gear technology towards minimising necessary work, until work becomes voluntary and economic management can focus on energy and resources rather than capital and labour.
That will not be if we do not bring to heel the FAANG companies now, and prevent things like Apple's Private Access Token, Google's WEI, etc. from taking root (yanking them out of the ground where already present).
The sort of Shadowrun/Snow Crash/ancap future you're talking about is what Cory Doctorow is calling technofeudalism[0]: one in which the primary driver of economic activity reverts to passive income scams[1] instead of active economic activity.
It just splits the web. You will have web properties that don't necessitate its usage. These will be available to everyone. Then you will have the Googlesphere which will have all google sites and all sites integrating google services that will only be available from "verified environments"
It isn’t perfect but we are ahead of most others (Mastodon, Matrix). We have spent TWELVE YEARS building the free, permissionless open source platform for anyone to assemble and host their own community software with all the features of Facebook/Twitter/TikTok for their own community:
We are about to roll out version 2.0 — I have never done this before but I would like to invite whoever wants to learn about it or build on it, to a Zoom webinar where I will demo anything and answer any questions. Starting in Q3 this year all the webinars will take place on our own platform — no Calendly, no Zoom, no Google, just the free open Web.
Anyway, sign up here if you want. Will do it every Sunday throughout August:
Whether you’re a developer, a businessperson, or just want to learn about the latest technologies moving the Free Open Source Web forward, this platform can help empower you to build and engage a community around yourself and your projects.
> It isn’t perfect but we are ahead of most others (Mastodon, Matrix). We have spent TWELVE YEARS building the free, permissionless open source platform for anyone to assemble and host their own community software with all the features of Facebook/Twitter/TikTok for their own community
Why are we using Hacker News? Go post on https://community.qbix.com/ that is powered by Discourse, a fellow open source company doing good work, and that we are integrated with
Why using calendly, zoom, and google? Well, as I said, we haven't launched Qbix Platform 2.0 to everyone worldwide yet. This is if you want to get involved pre-launch.
We dogfood our own stuff, but we also interoperate with everything else out there, such as Discourse (https://qbix.com/ecosystem for example incorporates it), Zoom, Google, Facebook, etc.
Interesting, if it isn't US based and if it isn't using US politics as a guideline (like banning russian patches).
I have lost a big part of my former trust and want for writing OSS this last few months and one thing I have learned is that if those two can't be answered with a resounding no it is a project I won't ever contemplate even though I'm neither American nor Russian.
As a side note, many people on our development team that has worked with us since 2013 have spoken Russian, coming from Ukraine, Armenia, Russia, etc. Many of them continue to work together despite the war their governments are conducting.
We are for empowering people uniting communities around the world, and are pretty critical of government overreach:
Well, I can't give you a rational answer since it is irrational in my opinion, but denying patches from Russians and Russian companies happen, including on the Linux Kernel mailing list:
The link above is an introduction to the Qbix Platform, which you can download and try out, if you want to rather easily build your own community, that has more advanced features than Mastodon or Matrix.
Can't the browser just fake itself to look like chrome/safari without extensions to get the WEI server token?
* CON: The problem is that the WEI server could change it's tracking faster than the browser app updates it's fakeness though. There's more money in bypassing adblockers than there is in blocking them.
* CON: If it does fake itself, when you return to the original website it can assume there's no adblocker and fail to load with the adblocker unlike now where it's usually ignored.
1. Someone could set up a server that proxies WEI required requests to regular clients. The client initiates the process, the request goes to the middleman, the middleman makes the proper WEI authorized request, gets the response, passes the response back to the client.
2. The private key could leak somehow, and so, software can forge the required signature.
I'm not holding my breath for either one. Some kind of regulation has to step in, otherwise Google puts the internet in a chokehold.
My guess is that on desktop, the endgame will involve implementing Easy Anti-Cheat levels of anti-tampering into the browser to prevent anyone from proxying through an automated Chrome instance or whatever. On Android, Google already has SafetyNet or Play Integrity, they can already refuse if the app or operating system has been modified
“They usually advertise themselves by a user agent string” my understanding is this is very much not true, but all that info comes from cloudflare which obviously has an incentive.
Nearly ever single product, service, and idea that Google brought into this world beyond - perhaps - their early days search engine, was created in response to the rise of some problem of Google's own single-handed making.
Inadvertently at first, they were amongst the first to ever even run into some of the complexities of providing internet services at scale after all. But then the rot took hold and Google Chrome was created.
Frankly, it's long past time we had "anti-cheat" available for the web. People vastly underestimate how bad "web cheaters" are for society. They drown search engines and social networks with spam that advances interests for a tiny minority. Places like Reddit are flooded with astroturfed misinformation. They are not as obvious as cheaters in a video game, but far more consequential.
There are obviously a lot of details to work out, but pretending that this is just a power grab by Apple, Google et al. instead of an attempt to address an extremely serious problem (that HN indirectly complains about all the time, e.g. "web 2.0 sucks", "why are search results so bad now"), is just naïve.
It’s a signed attestation. A user agent can be spoofed, this attestation needs to be signed cryptographically with a trusted key, for example a hardware key shipped in your device by an approved vendor. Think Apples Secure Enclave.
The goal is a verified stack - the hardware key proves you have approved hardware. The approved hardware proves you don’t have a tampered OS. The untampered OS proves you have approved binaries. The approved binaries disallow certain actions that users want such as blocking ads or downloading YouTube videos.
What part of attestation don't you understand? If linked with a OS level signing with keys stored on TPM, it's game over for private browsing. The only thing worse than companies proposing such measures are the useful idiots downplaying the impact. If someone disagrees, pray tell us muddle brains how to bypass this on a proprietary OS with locked boot and tpm stored keys.
And if the "attester" decides that IceWeasel on Ubuntu (or Firefox with uBlock/uMatrix/NoScript) isn't "trustworthy," but (unmodified) Chrome is "trustworthy," you've just created vendor lock-in.
The same proposal suggests that users who fail the attestation still access the content. Which is apparently how the Apple version of this same protocol already works.
I'm of the opinion that Google is no longer "Organizing the world's information" but "Stealing the world's information". Their last keynote proved that with all of the AI products. Regardless of this proposal (which is awful) they are a data vampire using all your info for themselves without permission or compensation.
Y'all ought to read Rainbow's End, where a Google-alike is trying to OCR all written information to make it accessible online… while shredding the originals in the process. That book was prescient on so many levels.
In-universe reason for that made no sense whatsoever and was nonsense. It was not even an evil organization (this would make sense) but OCR method as described was absurd and would not work.
Is this serious?