I always feel that a good deal of data-driven anything is basically modern divination. The median data-driven decision might as well have been based on haruspicy.
You can construct experiments to demonstrate damn near anything is true, and without (or even with) a rigorous background in statistics and scientific methodology, it's very hard to examine the supposed evidence.
I know I say this a lot, but constructing solid experiments is time consuming and difficult. It's hard even if you you're trying to be fair. Professors routinely get this wrong despite a literal decade of formal training in the scientific method and half a lifetime of practical experience with this exact craft; despite how fucking up puts their career, reputation and livelihood in jeopardy. Not only does this happen a lot, bad science routinely slips through peer review as well.
That's the state of affairs in the spaces that in general aren't extremely adversarial, where the people involved in constructing the experiments in general don't have a a vested interest in trying to mislead you. (Scientific fraud still happens of course, but Jan Hendrik Schön is the exception and not the rule)
I've worked at several startups now where the people in charge of A/B testing couldn't design a single A/B test that returned statistically significant results out of dozens of attempts each. Not. A. Single. One. Even the so called experts. And that's using their own calculations, which were already poorly designed attempts at p hacking.
The worst part is that some of the tests were still used to justify decisions to and by management. SMH
Is that the fault of the experiment? Or was it a weak manipulation and no-result was correct (e.g., changing the purchase button to a slightly different shade of green and expecting a higher conversion rate)?
If you are unable to provide your manager reasons for new development, he's going to find someone else to do your job, someone who will give him a report with charts and numbers that he can use to justify and expand his teams operational and headcount budget.
Giving a manager a report that says "there are little to no modifications that we can make at this time to improve UX" is a CLM for him.
IMO, this is still a failure to understand split testing. It is not to discover if changing a button color matters, but to explore the universe of all possible treatments and how they impact your most important business metrics.
It’s a global optimization problem, not a scientific way to understand how a specific change impacts users. Testers that have this mindset tend to be locally constrained and less likely to have bigger wins.
A no difference result in a well designed study is very valuable and can be used to justify decisions. It doesn't matter if we A or B so choose based on cost or the CEOs favorite color or a coin flip.
> ...don't have a a vested interest in trying to mislead you.
I witnessed this.
I briefly worked on the recommendations team for a fashion retailer, doing yeoman plumbing and maintenance. Basically an in-house equiv of RichRelevance.com.
So, so much data and logging.
Alas, after cutting out all the obfuscation layers and steps, the "lift" attributed to the team's exquisite algos proved illusory. Some >80% of the conversions came from the "recently viewed" rule. Whereas the conversation attributed to the algos consuming most of our dev and compute resources was basically noise.
I wouldn't say our (very smart) data scientists were fooling themselves, necessarily. More like the feedback loops were never fully validated. (Familiar story of non-programmer domain experts hoisting a POC, which then was promoted to production. Maintenance hell.)
Coincidentally, FWIW, my bro has worked in ad tech (Avenue A/Microsoft, RichRelevance, others?). He's got a math skillset. Was a true believer. It took a long time, but he eventually lost his "faith" that digital advertising (as they were doing it) was effective, on the balance.
Also FWIW, during my brief stint, I advocated "personalization". What I imagine StitchFix is doing. I'd love to know, for real, how they're doing. Like the split between algos and human "style consultants".
> You can construct experiments to demonstrate damn near anything is true, and without (or even with) a rigorous background in statistics and scientific methodology, it's very hard to examine the supposed evidence.
One time I was at this silly daylong corporate team building event where they assigned us to groups and had us do "experiments." As it happened one of my group members has a PhD in Particle Physics and the look on his face was something else. Needless to say he didn't bother telling the take charge type that her experiment design was entirely nonsensical, because why bother, but I was amused.
One doesn't even need to observe all this to recognize the simple fact that craigslist is much nicer to use than twitter, reddit, tiktok and craigslist hasn't changed in decades I dont think.
If you want to take a lot of a thousand products from aliexpress and sell them globally on the internet, ebay is your platform. If you want to sell something of yours locally or rent a local apartment, craigslist is your platform. These platforms reflect these usages when you browse them. Local results from people who just want to move a single item in their home vs some global drop shipping arbitrage opportunity will always be of higher quality.
> Local results from people who just want to move a single item in their home vs some global drop shipping arbitrage opportunity will always be of higher quality.
Sometimes you want one of those Chinese paper cutters, or reasonably priced leather tools. Ebay is mostly good, despite itself. They've also barely changed the UI in my memory, and I've got a 20 year old account.
I will agree on that. If I want a random thing, there will be the same product dropped shipped on both sites but usually ebay comes out a few dollars cheaper.
eBay is my main shopping site. If a vendor doesn't maintain their own web store I usual can find it on eBay. The eBay vendor may have their own web store and I use that.
Which explains why I've been getting ads for brides' maid dresses.
However, it also seems as if a tuned LLM could look at the corpus of my on-line profile (mail, FB, HN) and produce a model of my attention and commentary (positive/negative). Then that model (or several models) could be tested against various ads to determine which are more likely to produce positive responses. Right now I think that would be too expensive except for very high value ads/targets, but perhaps pricing will come down and non-keyword based advertising might appear.
Absolutely not.
Advertising is applied sociology, not psychology.
Ads don't care about your personal wants or desires, they only care about Bayesian probabilities across broad target audience groups.
(E.g. "dog lovers are X more likely to Y than cat lovers".)
That makes sense if we're looking at broad-scope advertising.
I always figured there are some verticals where the value per acquisition was high enough, and the product narrowly-scoped enough, where it might make sense to microtarget individual customers-- profiling them individually and building custom messages for each of them.
The obvious target would be the narrow-scope prescription drugs-- all those ads that say "If you have XYZ-positive, PDQ-negative Exploding Sphincter Syndrome Type XVI with Sprinkles and Cheese Sauce". Presumably there are only a few hundred people with that condition in the country, but you're spending a lot of money to reach everyone else to try to catch them in your net.
Super-premium luxury goods might also fit there, but I wonder if the brand risk would be too high. Rolls Royce probably knows the 200 customers most likely to buy their new model, but it might not fit their brand to actually put together a campaign that seemed like they were actually TRYING to sell cars.
If the cost is reasonable enough, it doesn't even have to be crafted for the individual, simply target them with one of multiple ads, which are modeled to have the highest (positive) effect... If you're not going to interact with it showing it to you has minimal effect. If one is modeled to have 5x higher interactivity, that's the one you want to show.
I think there are high value customers for whom this probably makes sense, but only if those doing the targeting make money from it. Right now you're the product, but the supplier is the one doing the targeting (or not) and the customer doesn't know enough about their potential customers to target. The suppliers don't really make money by effective targeting.
Hmm, I feel like this analysis comes from trying to rationalise what’s happening. From my experience the truth is simpler, and much more disappointing: Advertising is about statistics, and if some wonky new data model creates better conversions the advertisers don’t care how it’s underlying rule sets work.
The cost of pushing ads to people who already have mattresses is insignificant to capturing those that are about to buy them, statistically speaking.
The disappointing thing about this trend is that even if the rule sets are obviously wrong, yet still provide positive results means that their operators have little reason to change them. Statistics don’t work for individuals.
Case in point is credit rating and insurance underwriting
> The cost of pushing ads to people who already have mattresses is insignificant to capturing those that are about to buy them, statistically speaking.
This is called conversion rate and for some businesses it can be surprisingly tiny while still making that business completely viable.
I would hazard that rates in the mattress industry are almost preposterously low (<5%) but margins are so high that it doesn't matter in the slightest
5% is low? You mean that having 1/20 people buy your product after seeing an ad is a bad outcome? If I sold just about anything, I bet that having a 1% conversion rate would be a money printing machine
Click through rate (CTR) is `people clicked ad / people shown ad`. Conversion rate is `people bought thing / people clicked ad`.
For Google search ads pretty good CTR would be about 8%, and a pretty good conversion rate would be about 5%. So it’s more like 1/250 buying if you’re doing pretty well. At average costs, that’ll be one conversion for something like $40 in ad spend. More average would be like 2% and 3% so more like 1/1700 and like $70 per conversion.
(Display ads are more like 0.3% and 0.6%. So more like 1/50,000. Costs are cheaper.)
Those prices are _average_. There’s no flat rate for these—they’re an auction. So generally your higher margin items, especially in industries with higher average click/conversion rates, are going to end up with higher bids and higher costs.
A bank will pay a lot more to convert someone to a mortgage customer than a restaurant will pay to convert them. You may end up paying $30 per click for someone searching for mortgage refinancing. Assuming the same “good” search conversion rates, you’re paying $600 for the conversion at a $30 CPC. (I suspect the conversion rate on something like mortgage refinancing is probably lower.)
It’s been a few years since I was forced to live in the advertising world but (1) no it’s not a money printing machine (except for the ad networks) and in fact (2) I’ve literally never seen a directly attributable positive ROI. Mostly the gap is explained away as "well, it's building _brand awareness_". Most of the positive results I’ve seen were… questionable. Usually “you didn’t acquire new customers, you took existing ones or existing leads and funneled them into your advertising pipeline and attributed them there”. Think advertising on your own company’s name—people searching for your company by name were almost definitely going to end up there anyway.
The whole point of targeted advertisement is to avoid this. The old-school blanket approach had this issue as well, but did not promise to know if the if the potential customer had a mattress already.
Yep, in the past they would advertise in newspapers or magazines or TV or radio where the vast majority of the audience had zero interest in buying a mattress anytime soon. Narrowing that down to "people who have recently searched for a mattress" is a win, even if that includes some people who already bought one. Despite what people think about Google being all-knowing, it's less likely that they know you bought a mattress than they know you searched for one.
There are a whole class of fallacious modes of thinking that revolve around assuming the world is smarter than it actually is and that people are more competent than they actually are.
The most common these days is “conspiracy theory,” which is really an optimistic view that starts with the assumption that someone understands what is happening and can coherently control it. If these conspiracies were true it would mean that all we’d have to do to fix major problems is get “them” out of power or convince “them” to become the good guys. Very, very optimistic.
I'd say the conspiracy stuff is more that either the individuals are ignorant of Hanlons Razor or simply refuse to accept it's applicability.
Combined with supremacist beliefs (They are superior which gives them insight others can't possess) and we get pretty much all the conspiracy theories.
Sadly, there are plenty of very real conspiracies in this world, doing immense harms, yet those are seemingly actively ignored by the theorists... Which I often presume to be a result of the inability for the theorist to hold the "special knowledge" card as any who looks at the evidence would come to the same position and thus the promulgation of such legitimate theories are "too mainstream" for those seeking to further thier self delusion of supremacy.
Some theories of mass propaganda rely on the fact that people who are complicit in it are merely acting on the nth higher level of incentives for themselves, and the end results we see reflect the pileups of these incentives. Its much harder to deincentivise an industry from certain behaviors they’ve organically grown into, than it is to even turn over the entire executive staff. It almost doesn’t matter who or what the company is if the incentives are still in tact, they will fill the untapped niche like a new species in an environment.
This is a popular meta-conspiracy theory. “It takes too many too keep a secret”. Yes, but not everyone has to stay silent. If you have a lot of resources you can buy silence, it is literally a product for purchase (not perfect, but close enough). How do you think Harvey Weinstein got his victims to shut up? How did numerous large companies suppress their own research that their products were harmful, for decades? How did Jeffrey Epstein build a jetset trafficking empire without getting caught (and when he did get caught – the first time – he got a slap on the wrist and no media coverage)?
Worst case, if someone refuses to stay silent, you can just label them “disgruntled”, question their character, their mental well-being, claim that they’re in cahoots by some known evil agent, etc etc.
Point is.. many conspiracies don’t require perfectly competent agents with perfect knowledge.
> If these conspiracies were true
Well, quite obviously all theories are not true. The bulk of random conspiracy theories are most certainly bogus, just like most low-effort theorizing by laymen in general. But aggregates don’t make conspiracies impossible.
I think I've outlined this before on HN, but I've worked with a proper sleazy DSP, who I found out were basically targeting vulnerable folks with ads. These guys have zero ethics and less morals and have an attitude of "if it works it works, and to hell with the consequences".
No, it's not. It never claims advertising doesn't work. It claims that data-driven targeting is a scam on multiple levels. That isn't to say advertising doesn't work, just that "data-driven" doesn't improve its results.
I mean there were a few people correct that mortgage backed securities were much higher risk than they were classified as. Just because a bunch of smart people agree on something doesn’t necessarily mean it’s correct. But it can take a long time for people with a lot of money to go broke.
Except a big part of downvoting has nothing to do with the truth of a statement - yes factual inaccuracies and misinformation will be downvoted but that’s just scratching the surface.
If people are rude or posting something completely unhelpful that’s a downvote even if the statement is overall true.
That’s a rather unhelpful way to use a voting system, considering no two people have the same sensibilities toward rudeness or the same purpose for reading a thread. At least voting based on truth has universal utility. In fact, downvoting true information could be actively harmful in many (most?) situations as many (most?) platforms boost or suppress content based on those votes
I also find it a bit.. idk what the word is, egotistical? Self-centered? Judgemental? To vote based on subjective qualities like perceived rudeness or helpfulness (toward one’s own goals or one’s own understanding of a stated goal)
Then talk to dang and attempt to convince him that the voting system needs to be purely based on the truth (or lack thereof) contained within a comment. As it stands, the voting system is primarily intended to surface comments and posts that foster the discussion of interesting things.
Considering that things like attacking the messenger and bad-faith discourse (via insults, etc) are counter to the stated HN ethos, I don't feel it's unhelpful to downvote assholish comments for the sake of being assholish, even if the comment has accurate factual information that is relevant.
Hallelujah cry the internet trolls! Who now are allowed to post rude and vile comments provided they might be true! Downvoting their perceived rudeness would be wrong.
No wait. Nope, that’s not how it works. Downvoting for rudeness, incivility or low effort comments makes for a much better environment.
"Hey boss, I spent $40 milion on a targeted ad campaign with this fancy new agency. They sent us a report showing 2.5% uplift, but I ran it by our in-house stat nerds and they say it's bogus."
vs.
"Hey boss, I spent $40 million on a targeted ad campaign. The agency proves we got a 2.5% uplift. That's $300 million! Ain't I terrific!? Where's my bonus?"
Edit: and...
CEO to board: "We tried a new targetted ad campaign but it didn't make any difference."
vs.
CEO to board: "Last quarter the industry suffered increased competition, and our uptake through traditional advertising channels suffered. Fortunately I had the forsight to direct Marketting to open up to new, inovative methods. I'm pleased to report that it paid off, to the tune of $300 MM, completely offseting the drop from traditional campaigns."
So much of the time of the ad boom has been the period of 0 interest rates.
So many of the adtech dollars have come from venture capitalists pouring money into startups that will be profitable any year now, they swear.
So much of the advertising business has always been about convincing the entire rest of the economy that they desperately need more advertising in order to compete, regardless of any actual stats on ROI.
Is online advertising—and in particular, the hyper-targeted advertising that fuels the industry's insatiable hunger for every last drop of personal data—worthwhile?
We don't know.
You don't know. I don't know. The adtech companies don't know (but they assiduously avoid acknowledging that fact, often even to themselves). Their customers don't know.
Me, I'm comfortable with my educated guess that it's nowhere near as worthwhile as they want to make us believe it is.
> So much of the advertising business has always been about convincing the entire rest of the economy that they desperately need more advertising in order to compete, regardless of any actual stats on ROI.
I've never worked with anyone that doesn't measure ROI or ROAS. I'm not arguing that some companies don't need to spend so much on advertising or that it's not needed for some but there's a pretty clear measure.
> Is online advertising—and in particular, the hyper-targeted advertising that fuels the industry's insatiable hunger for every last drop of personal data—worthwhile?
In my experience in eCommerce marketing is that both Meta and Google are steering advertisers towards massive audience campaigns that aren't hyper-targeted. They might be doing hyper-targeted advertising but it's no longer as recommended as it once was.
All that said, I use ad blockers and I think data should not be hoovered up without consent or awareness and sold without consent. On my personal site and even my online shop I don't use GA or any other third party tool (hosted on Shopify so they inject some things).
There are ways to estimate advertising ROI, but so much of what ads are attempting to do is inherently unmeasurable without a tightly controlled study, particularly given the highly chaotic nature of the economy and people's behavior.
Fundamentally, there are two "sides" of advertising ROI you can try to measure: the individual, and the aggregate. They each have serious challenges.
At the individual level: sure, some people see an ad, say "hey, I want that!" and go buy it. But that's definitely nowhere close to the majority of purchases—even online—and it's very unlikely to be a very high percentage of ad-influenced purchases. Much more often, the ad gets the product into your brain, and then later, when you're shopping, you see it and say, "hm, maybe I should get it," never consciously realizing that the ad caused or contributed to that. Thus, any kind of consumer survey will be unable to capture more than a tiny fraction of ad-influenced purchasing.
At the aggregate level: what you're trying to measure here is simply your sales numbers over time. And except for the very largest of companies, sales numbers will be pretty noisy. That means it's very hard to tell for sure whether this ad or that one worked, or if any of them had any direct effect on sales at all. (That's particularly true for the subtly-influenced purchases I described above; those might happen months after an ad campaign starts.)
Is it totally impossible to measure ROI? No, it's not. But it takes a lot of work, and has a fairly high degree of inherent uncertainty. The old saw "half of all advertising dollars are wasted—we just don't know which half" is very relevant here.
"Work" is loosely correlated with "Make money out of it" (especially for the ad seller)
The buyer of ads does (usually) keep track of conversion (as correctly pointed in the article, also the seller) and when optimizing their strategy they're usually ok with the results they get from it (because they sell and they have sales attributable to the online ads)
But here's the catch, making a more efficient ad probably doesn't make the sellers more money.
The way to measure whether the data-driven ads are a scam is not by saying "this ad is irrelevant to me". It's by the advertiser looking at their ROAS (return on ad spend). If the algorithm is optimizing for showing you an ad, there's a pretty fair chance that the advertiser has done that math and decided that in a world of imperfect data and algorithms, you approximate enough the type of person who is worth showing an ad to... statistically.
In fact, getting to a world where ads are perfectly targeted would be one hell of a privacy nightmare.
> there's a pretty fair chance that the advertiser has done that math
As an occasional admittedly naive experimenter with ads – it’s far more likely that the platform has done the math and showing you the ad is profitable to them, not the advertiser.
Google et al love nothing more than to burn your entire ad budget on cheap clicks from countries that will never buy anything. But the number of “ad conversions” looks high and graph go up!
Getting these platforms to do what you want is a literal full-time job. Hell, in bigger companies you have entire teams doing nothing more than bending over backwards to get these platforms to do the useful thing instead of setting money on fire. The algorithms love setting money on fire. It’s a lot like giving your 3 wishes to a genie, you have to be very specific.
I’m not sure this is true. Facebook makes a lot more money from me if my ads are converting, and I continue to spend money day after day. If they just burn my money and I’m not seeing enough conversions out of it, then I kill my ads and start trying my luck with Twitter or Pinterest ads..
I can assure you it is very true that it takes a lot of skill to get ad platforms to do what you want. They do not care about you. The algorithm will make The Metric go up and that’s a threat. It’s your job to make it a good metric.
While people relying on a 3rd party platform are under the whims and methodologies that an opaque 3rd party platform may provide, in first party AdServing (which I am involved with), the algorithm is not tailored to "your" metric. Data driven analysis is done by all major online retailers. It's quite effective and part of the reason that features and products seem to disappear, more often than supply chain disruption would indicate or the common perception might belie.
While I'm sure some companies have some "reliable" 3rd party + 1st party metrics that affect their bottom line, they don't need more until the trends start to invert, and then they just don't have the data for marginal short-term experiments. This causes these companies to make sweeping changes that a 3rd party will not help tool around...leaving the company to decide the changes either 1) help climb returns (overall), 2) hurt returns, 3) most commonly, show nothing at all. It's flailing in the dark, hoping for an external force to turn things around. The company that is still relying on someone else's platform are dinosaurs and small players. Sometimes they are able to climb out of it due to externalities or developing their own platform.
I don’t think so as only the advertiser has data you need.
Let’s say sales increase but your advertiser just burns 20% of your spend because it can and you won’t know. That’s the scam part.
It’s not that your spend doesn’t increase sales. It’s that part of your spend is wasted and you could increase sales more if it wasn’t scammed away as wastage.
I just bought a dozen bagels for $20. I’m happy, pretty good deal. But if the store was throwing away a dozen bagels for every one I bought, that’s wastage. Just because I’m happy doesn’t mean it’s not a scam.
Before reddit shit the bed, I wanted to advertise there, only to learn that the subreddits I wanted to target were too small to be able to be targeted. So much data wasted.
It’s funny because it’s true. Often one needs to check the dedicated sub and a more general one to get unbiased takes/reviews, e.g. ubiquiti and networking
Whether by social group principals (grassroots), or self interest promotion (corporate), this is the thing that has been infecting the Reddit platform for years making it less fun to use.
Small subreddit targeting is just a matter of shilling. I’ve seen it on certain niche subreddits, brands becoming the subreddit darlings and added to the faq and wiki and sidebars, constantly recommended in the comments under any “what do I buy for x” thread. Even if its a junk product. Sometimes the brand even has an account active on the subreddit giving away coupons or free stickers and such.
There’s one thing that advertisers agree on: ads work in the aggregate.
You can tell by turning them all off. Ask Kraft Heinz.
Beyond that it gets fuzzy. Is it inefficient? Yes. Does anyone agree on the actual mechanics? No. Is the data wildly inaccurate? Sure. Is there grift? Totally.
People also forget how inexpensive and ephemeral individual ads are. A display banner on a quality site costs less than a tenth of a penny.
And it’s like a room on a cruise ship — the boat is gonna sail one way or another. So it has to be sold.
Advertisers might choose to let Facebook arbitrage those impressions into a cost per click model — at pennies per click instead of hundredths per view. Tradeoffs galore in that model.
When you aggregate these numbers into a $600b industry, you start to see how sweating some of the finer details just doesn’t matter.
>it’s like a room on a cruise ship — the boat is gonna sail one way or another. So it has to be sold.
super shaky reasoning. the less rooms that get sold, the less likely future ships are to sail or even be built. individual decisions matter in aggregate
> you start to see how sweating some of the finer details just doesn’t matter.
I think it’s an opportunity. The original premise of google is that targeted ads would let you benefit from “sweating some of the finer details.”
If 1/3 of a $600B market is wastage or fraud, then that’s an opportunity to give advertisers better response for their spend. Really massive.
I’m sure advertisers would like more sales by being able to remove the waste ads. Currently they can’t do that. But if some new company or tech allows for that, it’s worth trying to figure out.
That is a poor comparison because its not a comparison with a single variable changed. Kraft and Heinz went from two independent companies to a single one controlled by new ownership engaged in organization-wide cost cutting. The world also did not stand still over that time, it probably also changed in many ways.
Its probably impossibly to conclude with any confidence that the results were simply due to cuts to the ad budget, cuts to the budget and cuts to operations, or simply cuts to operations. Ads are also not commodities. An ads effectiveness is also not the result of pure spend, cheap ads can be very effective and expensive ones can also not be effective and vice versa.
Many here already pointed at the fact that likely P(buying mattress in the future|googled mattress) is much higher than P(buying mattress in the future) thus for an advertiser with a finite budget it makes perfect sense to target users who googled mattress irrespective of whether they bought it already or not.
The author's point is that, with the data available to the platform owners, they could easily deduce the subset where that P value is nearly zero (having already bought the mattress).
Places that would also make sense to advertise mattresses: around universities and dense rental housing, or new developments of single family houses. I required no spy networks to make this list.
It seems that P(returning a bought mattress) is more than 10%, so it probably makes sense to continue advertising to those people.
Also, there are probably people who buy multiple mattresses (eg. parents), that might buy a second mattress.
The point of the article is that platforms know that both your examples aren’t true. The author isn’t a parent and isn’t in the market for buying a second mattress or returning.
I think the issue is that the platforms don’t want to actually target, they just want to pretend and allow for the narrative for why.
I would expect that the search pattern of multiple mattress buyers is very different than single (ie, single searches and buys one and stops searching; multiple searches, buys one, keeps searching). And for returners have identifiable patterns (ie, searches, buys, searches for “how to return”).
Remember that google knows pretty much every single search you run and every page you visit and every checkout you complete successfully (if you’re using chrome).
> The author's point is that, with the data available to the platform owners, they could easily deduce the subset where that P value is nearly zero (having already bought the mattress).
Let's think carefully before we argue that Google and Facebook should expose (indirectly or not) even more of the information they collect to third parties.
It only makes sense if you consider that the end all be all to target your customer. Maybe late night television ads are even more effective than google searches, because they are cheaper than prime time slots and are perhaps enriched for people with poor sleep. Maybe right now some subreddit is shilling mattress penny stocks and that is masking the true signal from your internet search data. You have no way of knowing that though, because you’ve reduced all of reality down to a single factor, an internet search, without considering how much of the observed variance could even be explained by this factor nor how the results might be confounded with other factors.
Of course its tested "rigorously". Performance marketing is always about testing and optimizing CAC/CPA. It can make or break your company. If you can acquire customers at 10% lower rate(at the same volumes) than you competitors and you have similar COGS, you will dominate them and will ultimately will be the market leader. Monetary incentives are there to ensure efficiency just like there are incentives around COGS for efficiency.
The only evidence the author offers are her personal experiences. While Im no fan of advertising and question its efficacy as well, I don’t really think this is a convincing argument against them.
Statistically, these advertising segments will be very fuzzy and there will be outliers (probably many of them?).
Since June 2022, we have stopped sending conversion signals back to our Google Ads account, and have completely disconnected Analytics and any Google-related tracking code from our site. We also started a new Ads property and retired the old, "enriched" one, starting 100% fresh (we were worried about breaking things, so we isolated the move and were ready to rollback quick).
We have a yearly Gads budget around ~$5-$6M, of which is 95% SEA. We are selling an unchanged service in a very stable, slowly declining market with heavy ads competition for the top keywords. Roughly 60-65% of our customers are paid and come from Google.
Since then, our ROI on that spending has not changed whatsoever. Paid traffic remains stable. Our CAC and conversion rates are virtually the same (even slightly better). Our new customer's LTV has increased compared to the previous cohorts as well, so it seems we're actually getting better quality leads (though I wouldn't attribute it necessarily to Google).
So if reporting more data to Google was supposed to help us target our users more cost-efficiently, it did a piss poor job at it. In hindsight, we have wasted so much time trying to help Google make "smarter" decisions by providing it with more and more data (hello server-side tracking), while the most efficient thing to do would likely have been to do nothing at all.
We have been working with GAds selling the the same service for more than 8 years now, so we have a large basis of comparison to be able to make that case.
Maybe we're lucky, maybe this will turn out badly, but since then I have stopped blindly believing whatever the ads platforms are pretending to do behind the scenes with all this data. If they do anything with it, it doesn't seem to be for the advertiser.
Our next step will be to gradually cut our SEA spending and follow the impact.
And don't get me starting on Facebook's borderline fraudulent ad performance reporting that doesn't match our reality...
> Statistically, these advertising segments will be very fuzzy and there will be outliers (probably many of them?).
Someone who goggled matresses and then bought one is hardly an "outlier" - it's probably more common than someone who googled mattresses and hasn't yet bought one. I guess the ad-tech companies don't let their advertisers de-select prospects that have already shot their load?
So much of advertising is lies that I don't do any advertising online any more.
I've pulled all my google ads, facebook ads, instagram ads and podcast ads.
After about 10 years of doing ads (for my coffee club) and really attempting to track what was happening (from a technical point of view) I couldn't actually attribute any of it (the holy grail of digital marketing). "Word of mouth" is still our biggest channel. And all the time whenever I speak to "marketing" people it's the same low value analysis with zero mathematical or scientific vigor. They look at what happened (High CTR on this ad) and draw some conclusion from it, and then next week when CTR flatlines on the ad they have no comment and are busy looking at the non-significant noise coming from another ad.
And that's before even talking about the pure lies coming from the podcast people who swear up and down they know exactly how far into a podcast someone plays, when the spec they quote explicitly states it doesn't really work on apple products and 90% of the listernship is Apple. So many lies. It is a grift built on grift. Somebody in the organization knows it's a pile of lies, but the sales people actually believe it I think or don't care.
Remember, the primary job of an advertiser is to sell their advertising services to companies, not to sell a company's products to consumers. That comes second and is only done insofar as it facilitates the first.
Same is true for consultancies. They effectively offshore the brainstorming for you, but the ideas they sell you on don’t have to work and the consultants usually aren’t even incentivized for it to work after the duration of the contract.
I wonder if there are any good books out there for these sorts of businesses in modern times, doing a kind of shovel selling during a gold rush sort of business plan, not really caring if the ore vein is even real, just serving the lemmings their desires.
I really hope in the near future there is some shift in the advertising business where they realize the ridiculous amounts of money they are lighting on fire and decimates the ad driven internet as we know it.
Their job is to sell advertising to advertisers, not to please their users.
I thought it was common knowledge. The more complicated and graph-filled the dashboard is, the more work that the advertiser can claim to have done to their bosses
> Marketing theory is never tested rigorously. The <del>common sense</del> incredibly sound scientific view based on heaps of scientific evidence view – showing your ads to people more likely to buy your product is more efficient because they’re more likely to buy your product anyway – is entirely untested.
Before this, the classic life style segment in media analysis was magazine audiences aged 15-25. Notably, this was not the part of the population that could afford the goods advertised immediately, or even anywhere in the near future. The idea was that this would be formative, that this would effect in an invaluable baseline of expectations and anticipations that could not be competed with, when it came to real acquisitions. Well, nowadays it's all just-in-time, often even just-after-buy, which seems to be the most efficient.
"Amazon knows exactly when you have an outbreak of aphids because you buy things to kill the nasty little beasties, and it probably also knows when you’ve had a nasty breakup because nobody listens to Fleetwood Mac’s Rumours on repeat at 3am when they’re in a good place."
Ehh I find it hard to believe that a multi-trillion dollar industry is a grift based on this single person's collection of anecdotes. If online ads didn't result in measurable customer conversions, I have a feeling that the millions of business using them would stop.
There are so many industries that exist like this. The concept of a car dealership was obviated in a few short years once car manufacturers grew enough to bear the burden of production as well as sales and service, so car dealers essentially got their business protected by law in many states. Something the horse stable owners were never able to do.
Yeah I have and they provided guidance and knowledge surrounding financing, bidding, regulations, etc that I did not have.
Can we build a society without the need for real estate agents? Perhaps. Would that society be better? Probably. But saying they are simply a grift is obviously untrue
Remember, we're talking 'targeted advertising' here, where I imagine it is not easy for general business owners to understand whether it is more, or even as effective as 'traditional' advertising.
Easy money for advertisers just by showing some fancy irrelevant graphs.
So OP claims that data-driven ads are a scam using the anecdata that she is getting served irrelevant ads.
I'm not going to argue about whether ads are "worth it", but working on the other side (serving and optimizing ads), I can tell that yes, we run the numbers, and the math (mostly) works out.
As per your examples, we have run experiments by applying targeting ads based on narrow segments, and others without. As it turns out the data shows that "casting a wider net" works better, statistically.
It's good to point out this possibility, but outside of an institutional purchaser, people don't buy 2 mattresses with the purchases separated by a week.
Knowing that you bought a mattress is useful for the ML to find the mattress-purchase-likely cohort. Filtering those people out just saves the advertisers money. Hence the grift.
What if they bought a mattress and found out they didn't like it after one night? A lot of stores let you return them if you don't like them within a number of days. That would mean they are in the market for another one.
I'm not saying it's likely, but there are scenarios.
Really 0%? It seems like some of the buyers would return their mattress and buy another. If 10% return and 90% of those buy another brand, that seems to me like an above average audience to market mattresses to.
>The platforms aren't as dumb as the post author thinks they are.
Maybe the platforms are not that dumb, it is the mattress seller that doesn't properly make use of that feature, or there could be other reasons why they insist on suggesting over and over again a mattress.
The experience described has happened to everyone for any (not very often replaced) item bought online, be it a printer, a wash machine or whatever, so if the platform is not dumb, it lacks data and the sellers are dumb, in any case the final effect of this "targeted" ads is 0 added sales.
It's more likely that the mattress company couldn't be bothered creating a separate segment from their purchase order completion page and then excluding that from ads, because 95% of people who come to the page don't buy a mattress, and 95% of people who have bought a mattress don't click on ads when you show them, and the cost for AdRoll retargeting is per click not per impression.
Well, according to Douglas Adams, somewhere in the Galaxy is a swampy planet populated by sentient mattresses named Zem:
“No one really knows what mattresses are meant to gain from their lives either. They are large, friendly, pocket-sprung creatures that live quiet private lives in the marshes of Sqornshellous Zeta. Many of them get caught, slaughtered, dried out, shipped out and slept on. None of them seems to mind this and all of them are called Zem.”
Maybe we should boycott mattress ads for the sake of preserving the species.
I am willing to bet that statistically (with the number of mattress companies offering X night returns) that a recent mattress purchaser is more likely to purchase a different mattress within X days than the random public.
Some % of the population buys more than one mattress in a 3 month period, intentionally. Perhaps they are landlords, or have 2nd homes, or want to stock their new home with new furniture.
Maybe, but (thank goodness) the mattress ads bombarding does not last 3 months, only a week or so, after 2 and 3/4 months have passed that % of population will either look where they bought the previous one (if they are satified with it) or make a new google search, in both cases the week long targeted campaign won't have much effect.
Maybe advertisers don’t actually get signals when people click pay in some third party site? I mean, why would that third party want to expose your purchase to the advertisers?
So, you’re left with this great signal that someone is shopping for a mattress. Seems like a very good reason to recommend other mattresses to them.
Maybe it is just so easy to point idiocy in ad biz ? Look: you buy X so you do not need X a n y m o r e - and then you are spammed with ads for X. Brain deadness no 1 proved. But ads still ar payed for, LOL ! I see time machine just right here :>
I can’t find her LinkedIn and the about page has no info, but with this blog post she definitely does not work in digital marketing as she has no idea how it functions. It’s just wild baseless claims that everyone is marketing is either dumb or not trying at all.
We can argue about whether data-driven ads work for the OP or for buying an extra mattress, but in aggregate they absolutely work. Contra OP's anecdote, if FB ad targeting stopped working, then FB would notice the difference in engagement within a day - probably within the hour.
I have a friend with a business in a niche market. His business is driven almost entirely by ads place with google. He’s tried other platforms and had much worse results. Something must be working.
At some point we tried using a major network…. That predicted our target market to be 50% female, and heavily present in middle east. Had a laugh and moved on.
Contrary to the belief that advertising is less data-driven, the complexity and dependency on feature-rich data sets has increased over time. Redundant targeting sometimes happens due to the ad objective of maximizing expected revenue over cost. However, it's more nuanced in practice.
Advertisers aim to optimize expected revenue over cost within a 'payback period', which is typically 1-3 years for large advertisers. This is calculated through customer acquisition cost, retention, and incrementality (the probability of an ad causing conversion).
Only advertisers themselves can effectively calculate incrementality due to access to their specific conversion/retention metrics. This incrementality, along with the optimal mix of channels and ad spend, is the ongoing challenge for sophisticated advertisers. It involves multi-objective optimization across millions of ad assets, campaigns, and targeting criteria.
Privacy regulations since 2015 and subsequent laws like GDPR and CCPA have led to more reliance on probabilistic modeling for targeting. The entire pipeline of targeting, engagement, conversion, and retention forecasts are now based on probabilistic models.
While ad networks offer simplified scaling solutions like 'target roas' and 'campaign budget optimization', they're more useful for average advertisers with limited internal resources - eg, they can't justify hiring ML SWEs, quant traders, technical PMs, etc.
Advertising has become even more data-driven and arbitrage gains for sophistication have increased. Profound gains can be made with investments in marketing and forecasting science, similar to the operations of a quant trading firm.
Source: I've managed $10B+ through automated ad spend systems since 2012.
How much more effective would ad spend be if ads were solely served to people with a decent bit of disposable income? I would guess the return on investment is almost all due to buyers with cash to spend more than anything, and those 40% of Americans with like $400 in the bank (if those popular stats are to be believed) are merely an expense serving them advertising content for products they are unlikely to ever buy.
You already see this with some goods so its certainly at least somewhat effective, e.g. luxury watch ads in beverly hills contrasted with military recruitment billboards in south LA.
This entire essay is just full of ideas I disagree with.
First is the old chestnut: Why does company keep advertising product to me after I've already bought it!? This is one of those child-like criticisms that keeps rearing it's ignorant head. Literally, you are ignorant as to why they are doing it. Your ignorance as to why doesn't imply their stupidity. Amazon does it. Google does it. I'm sure Facebook does it. So either three of the biggest companies in the world are full of stupid idiots or perhaps you are just ignorant to the reason they are doing it.
> Their ethos is that if a human can do a task, a machine can do that task better, and not cost them anything such as salary, pensions or or a basic level of respect
No, their ethos is - we can't possibly have any reasonable number of humans do the task. In what universe could we imagine hiring a sufficient number of humans to sift through the data Google has collected? They have immense data on literal billions of humans. How much time would it take any person to understand you as a person sufficiently by painstakingly reviewing your data so that Google could send you perfect ads? The very premise of this is ridiculous beyond reason.
So the conclusion is based on the premise that having humans do the task is impossible. We have no other option than to use machines and algorithms to attempt the task because it is the only viable alternative.
> Advertisers, then, are getting served a steaming turd on a plate rather than the medium-rare filet mignon they were promised.
Advertisers are making a value judgement based on the options they have available to them. They can advertise on TV, radio, or magazines. They can advertise on podcasts, YouTube channels or Instagram feeds by directly contacting the creators and doing sponsored content. They can put up billboards and wrap public buses. They can put people in public squares and hand out flyers, do personal demos and offer prizes. I couldn't even scratch the surface of what is available to modern companies for promotional purposes. The fact that Facebook, Amazon and Google are as large as they are suggest there is some value being derived by companies given that they have plenty of alternate options available to them.
> There isn’t any evidence to suggest that an ad targeted to 35 year old men with children with an interest in football is any more likely to result in sales of Football Dad socks than a poster for Football Dad socks at a bus stop. But an entire industry is based on pretending that this is the case.
I really don't think they are. But let's compare the idea to the other advertising avenues I listed. TV ads are broadly targeted based on the time of day the show runs at as well as the content of the show. Magazine ads are broadly targeted based on the content of the magazine. Junk mail and flyers are broadly targeted to people living in a particular geographic area.
But I guess this MSc in marketing knows more than the entire industry they claim to be from.
I think of it as an appeal to consensus but if you want to dismiss an argument based on some debate bro tactics then at least be more substantive than naming some so-called fallacy and leaving it at that.
The organizations that have the data and that are using that data to sell advertising to the largest number of buyers are doing a particular thing. Traditional media, as I point out, is doing a softer form of targeting as well. So who is right, the person who flashes their MSc in Psychology as if that grants them authority? Or do we instead look at the overwhelming evidence based on the behavior of the entire industry?
The Global Homeopathic Medicine Market size was estimated at USD 25.85 billion in 2022, USD 30.95 billion in 2023, and is projected to grow at a Compound Annual Growth Rate (CAGR) of 19.89% to reach USD 110.41 billion by 2030.
We have absolute proof that homeopathy is grift.
But by your logic, You ask people engaging in grift whether they have consensus
The argument against homeopathy (if there is one, I admit I have no interest in the subject and have no direct knowledge) is that there is significant contradictory evidence that shows there is no correlation between the treatment and the outcomes. It isn't: "I thought about it and don't see why it should work so I think it mustn't"
I would love, just once, for yet another commentator on the advertising practices of Google, Amazon, Facebook, et al to provide a shred of evidence for their baseless conjectures. But of course, there is no evidence. There is, in fact, no correlation based in fact or reality between homeopathy and the advertising practice of said companies. Just an attempt to use rhetoric to smear them based on personal agendas.
You can hate the companies and their practices. You can hate advertising in general. But I see no reason to doubt the effectiveness of the techniques on rational principles alone. I have no doubt that there is a clear signal in their data that shows their practices achieve outcomes aligned with the needs and desires of their customers: the advertisers. You don't understand why because you lack of access to the data and are therefore unable to independently see that signal with your own eyes. To conclude that the companies are stupid or deceitful based on your own lack of independent confirmation is a kind of hubris that is outstanding.
I can't speak to her specific observations, but a lot of Meetup events specify an age range of, roughly, 20 to somewhere between 30 and 40. I think most of them are implicitly (or explicitly) dating events, although there are group-interest events that do it too, which I'd think would be less picky about who attends.
I was curious and started looking at Meetup lately…
At first I was amazed at the number of interesting local groups and their member counts. I felt I missed out on so much, but activity seems to have not recovered since 2019.
Then after browsing around, I realized that there a huge number of single women living in the region, in their 20s-40s, whose main interests are hiking, real estate, and *Kubernetes*.
This seems incredibly odd since I do not live in California…
The dating scene must be brutal for “on-prem” kinds of people…
You can construct experiments to demonstrate damn near anything is true, and without (or even with) a rigorous background in statistics and scientific methodology, it's very hard to examine the supposed evidence.
I know I say this a lot, but constructing solid experiments is time consuming and difficult. It's hard even if you you're trying to be fair. Professors routinely get this wrong despite a literal decade of formal training in the scientific method and half a lifetime of practical experience with this exact craft; despite how fucking up puts their career, reputation and livelihood in jeopardy. Not only does this happen a lot, bad science routinely slips through peer review as well.
That's the state of affairs in the spaces that in general aren't extremely adversarial, where the people involved in constructing the experiments in general don't have a a vested interest in trying to mislead you. (Scientific fraud still happens of course, but Jan Hendrik Schön is the exception and not the rule)