No, we don't do anything. Theoretically we could judge several times with different ordering.
We could measure order bias really easily though; we just need to look at the average score by rollout position across many runs. I'll add that to my list of experiments!
> While the specific internal workings of DeepSeek LLM are still being elucidated, it appears to maintain or approximate the self-attention paradigm to some extent.
Totally nonsensical. Deepseeks architecture is well documented, multiple implementations are available online.
I love to hate on google, but I suspect this is strategic enough that they wont kill it.
Like graviton at AWS its as much of a negotiation tool as it is a technical solution, letting them push harder with NVIDIA on pricing because they have a backup option.
Google has done stuff primarily for negotiation purposes (e.g. POWER9 chips) but TPU ain't one. It's not a backup option or presumed "inferior solution" to NVIDIA. Their entire ecosystem is TPU-first.
Pretty sure the answer is yes. I have no direct knowledge of the matter for Gemini 2.5, but in general TPUs were widely used for training at Google. Even Apple used them to train their Apple Intelligence models. It’s not some esoteric thing to train on TPU; I would consider using GPU for that inside Google esoteric.
P.S. I found an on-the-record statement re Gemini 1.0 on TPU:
"We trained Gemini 1.0 at scale on our AI-optimized infrastructure using Google’s in-house designed Tensor Processing Units (TPUs) v4 and v5e. And we designed it to be our most reliable and scalable model to train, and our most efficient to serve."
I feel like this is missing a really important factor: how likely are guests to use airbnb again after staying at a listing?
A listing could look great online and receive a lot of bookings (so high LTV), but ultimately drive users away from the platform.
A certain ad platform I worked on cared a lot about this - offensive ads could get you to quit the site altogether. You might want to count every ad as positive for the company since you make money, but some might actually be negative expected value! As a side note, I think this is a really undermeasured problem. There are many sites I won't use because the ads are so overwhelming or are often offensive.
> A listing could look great online and receive a lot of bookings (so high LTV), but ultimately drive users away from the platform.
I am precisely this kind of churned customer. I have personally booked maybe 3 AirBNB stays, and stayed with family in them on other occasions. The units I pick are always well-reviewed.
But in the cities I've stayed in (LA, SF, Rome), the price is really no cheaper than a hotel, and the quality is extremely variable. You have to really carefully read those 5-star guest reviews to read between the lines.
And you feel pressured not to leave a negative review, as that would negatively impact your ability to book in the future, since the hosts (I have heard) can see your average review score.
My impression has been that AirBNB's customers are actually the hosts. You, the guest, are an expendable commodity. You will use AirBNB until you have a severe enough problem, and experience them siding with the host over you. Then you'll be churned permanently, and by force if you do a chargeback.
If I were going to disrupt AirBNB, I'd offer hosts a better percentage with the requirement that the experience is standardized and high-quality. There would be an in-unit noise & vibration sensor, reporting directly in the app. 24 hour check-in and check-out. A minimum set of amenities, minimum WiFi speed. The bedding would be standard. Cleaning fee standard. Every unit subject to a surprise multi-point inspection at least once per year. Essentially, make it no worse than an average hotel, and maybe some units as good or better than high end hotels.
> If I were going to disrupt AirBNB, I'd offer hosts a better percentage with the requirement that the experience is standardized and high-quality.
I've noticed more and more apartment rentals appearing on booking.com. I haven't used any of them but I wonder what the tradeoffs are. My impression is overall booking.com is more guest-friendly as their userbase has grown from people staying at hotels, who expect stuff like being able to cancel and complain about cleanliness.
From my experience booking.com is the most guest hostile platform I’ve experienced to the point I try to actively avoid it. I travel A LOT and can book anything from 20 - 40 stays per jaunt. Whenever I have encountered a problem on booking it has always been met with ¯\_(ツ)_/¯ tough shit energy.
I remember one specific incident when a hotel had purposefully put me in a noisy room (I saw a sheet on the reception desk labelling it as ‘ruidosas’ which means noisy in Spanish). I took a video of it against the receptionists will and ensured I got the logo of the hotel in as well. When I reported to booking they essentially told me to jog on.
On the contrary, anytime I’ve had an issue with Airbnb they have sided with me and provided a refund/assisted me with finding better accommodation.
This isn’t a ‘you book more with us thing’, either. I have substantially more bookings on booking due to the prevalence in Asia but they literally could not care less
I'm Booking.com's Genius Level 3, so perhaps this only applies to frequent users, but I've been refunded in full for a booking after the date of the booking without even giving the reason for the refund request.
I've shifted 100% away from AirBnb and to Booking and am still able to book whole apartments/houses. It used to be AirBnb=houses/apts, Booking=hotels. Now Booking covers both, is usually cheaper (especially considering the cleaning fees AirBnb listings usually charge), and has good customer support (in my experience).
> My impression has been that AirBNB's customers are actually the hosts
Yes for sure. Avg # of transactions per host dwarfs avg # of transactions per guest. Same with revenue. A frustrated host who pulls a unit (often, multiple units) off the platform is much more detrimental than an individual customer leaving the platform.
This could also be played the other way around if the customers are completely churned. (ie: airbnb will lose a full unit with XX bookings but can also loose XX customers multiplied by XX Life Time Value).
The duty cycle (number of business days per year) is less than 1% for typical guests but around 100% for a sizeable chuck of hosts, and in the 20-80% range for a lot of the rest.
Hence I expect the host LTV to be a couple of orders of magnitude greater to that of guests.
If you want standardized professional places to stay while you travel, why not just stay in a hotel? When I go to a hotel (and we travel a lot), I know I’m getting a standard level of service and if not - especially with a chain hotel like one franchised by Hyatt, Hilton, Marriott etc - I can complain to someone at corporate and get a refund and book somewhere else, usually it doesn’t even tags that. Just talk to the front desk. They don’t care if you get a refund - it’s not their money.
Airbnb “hosts” treat it like their “home” and are emotionally invested in what should be a business transaction.
This isn’t to mention how many hosts are running an illegal AirBnb.
> If I were going to disrupt AirBNB, I'd offer hosts a better percentage with the requirement that the experience is standardized and high-quality.
This is an interesting idea, but it puts the quality level way above what I want to pay for. Hotels are anti-septic and cold. I like staying in an apartment that feels like someone actually lives there. I don't mind a few dust-bunnies under the couch, nor a little dirt behind the toilet. It's even better when the kitchen is fully stocked, including a selection of non-perishable food (think cooking oil, salt, pepper, maybe a bag of ground coffee and a box of pasta.) Sure, the sheets and towels should be freshly laundered, but beyond that, I don't want much.
AirBNB allowed me to pay less to get a more human-feeling space. Hotels are like McDonalds, they are designed for the regular customer who wants to get that Hilton-feeling, regardless of if they are in Wichita or Cairo. I want to feel like I'm in Cairo, and if that means that I'm in a mud brick house with a single bathroom, no A/C, and no daily housekeeping that's great! AirBNB opened those worlds to us as travelers, in a way that hotel chains never did.
I think that's more common in other parts of the world. (i.e. getting places that people actually live in).In Europe most/all of the apartments I've stayed in are basically business apartments (more or less).
You don't feel that someone lives there because nobody actually lives there. It's like a long term rental apartment that in best case scenario someone is using as a savings/investment vehicle but on short term. "Worst case" it's part of an apart-hotel.
When I travel as part of a large group(i.e. more than 3 people) a short term apartment is great because we need a big affordable space (i.e. someone may sleep on the couch and we don't want the hotel "greetings & house keeping" experience) but nevertheless I don't want or expect a dirty place in any way, shape or form be it in Cairo or somewhere else. I'm pretty sure there are people living in less than sanitary conditions all over the world including Western Europe and the U.S. That doesn't mean I have interest in experiencing that kind of "living/sleeping".
All that being said I've stopped using Airbnb years ago. It seems a broken system. For short term rentals/apartments Booking.com is the only sane choice(IMHO).
It's not so much for a Hilton-like feeling. In most cases, it's because I'm looking for a predictable place to stay with, often, a 24 hour desk. I just don't care about the room most of the time so long as it's clean and comfortable. I'm generally not traveling for the purpose of staying in a hotel room. I do stay in more traditional B&Bs/inns though very rarely somewhere that's solely an Airbnb.
> In Europe most/all of the apartments I've stayed in are basically business apartments (more or less). It's like a long term rental apartment that in best case scenario someone is using as a savings/investment vehicle but on short term.
If that the only thing AirBNB offered, I would have much less interest.
> I'm pretty sure there are people living in less than sanitary conditions all over the world including Western Europe and the U.S. That doesn't mean I have interest in experiencing that kind of "living/sleeping".
Thankfully, AirBNB doesn't have to be all things to all people. As long as there are enough people like me to keep them afloat, they can provide a product that I am happy with, and you can stay at hotels that are immaculately clean.
I do hate the way that many AirBNB hosts have made hosting a business, and would fully support a limit on the number of listings per host. People renting out a space in their house, or a vacation rental in a vacation destination that they also stay in too is fine. People buying 3 or more apartments to rent them out (taking them off the long-term rental market) is terrible, and should be prohibited.
I just randomly looked at hotels.com for hotels in Cairo and I saw name brand hotels from American brands like Hilton for around $120 a night. Even high end hotels like the Waldorf are $284 a night (I don’t care about fancy hotels personally). Why would I stay in a dirty Airbnb?
Not that I would ever use a third party booking site like hotels.com either…
I want the place I stay at to be run professionally and it to be I give them money and they give me a clean place to stay without having to worry about my ratings, discrimination, etc.
Especially in another country where I don’t know the language and after taking a long flight. I wouldn’t want to take the chance on an AirBnb.
* Desire to have an experience and living environment I can't get at home.
* Desire to have living space beyond 2 beds and a bathroom crammed into the smallest available space.
* Indifference to dirt or wear and tear.
Thankfully, no one is saying that AirBNB should replace name brand hotels, in the same way that no one thinks that every hotel should be a Motel 6. AirBNB/VRBO is just another segment of the industry, and those people who want to stay in an AirBNB can, just as people who want a Hilton can without affecting those people who want to stay in the Ritz-Carlton. This is the beauty of the market!
> Hotels are like McDonalds, they are designed for the regular customer who wants to get that Hilton-feeling, regardless of if they are in Wichita or Cairo.
Stop staying at chain hotels. There are plenty of hotels out there that are what you're looking for, and have an actual business to engage with in case of issues instead of some anonymous lister on AirBNB.
> Stop staying at chain hotels. There are plenty of hotels out there that are what you're looking for, and have an actual business to engage with in case of issues instead of some anonymous lister on AirBNB.
That has not been my experience. It is very hard to find a robust selection of hotels in a major city that has a living room and kitchen. Even when you do, in a part of the city you want to stay in, they are often tiny kitchens, with limited kitchen equipment, and where the cabinets are completely cleared of every food item after every guest. These places also have very antiseptic, uncomfortable furnishings. They are night and day different from staying in most AirBNBs.
> My impression has been that AirBNB's customers are actually the hosts.
Fwiw, I listened to an interview with Brian a couple of years back where he said that, internally and strategically, they call hosts “partners” and guests “customers.” Which makes sense to me.
Many people try this. I think Sonder was most recent but they pivoted. Blueground will do corporate apartments short term. I think these aren’t effective against Airbnb customers tbh. It’s not a perfect platform but the others are not as good because availability is key.
>And you feel pressured not to leave a negative review, as that would negatively impact your ability to book in the future, since the hosts (I have heard) can see your average review score.
Source?
I did a quick search and couldn't find anything to confirm this. Various airbnb screenshots geared towards hosts also don't show anything about guests' average rating.[1]
An interesting distinction is between staying at a listing vs booking it. I booked an AirBnB in Texas for the eclipse last year. The booking was made months ahead of time. Minutes before I left for the airport, the host canceled with no communication and didn't respond to messages. I got a refund, but I wasn't able to leave a review because the stay hadn't begun.
I had only used AirBnB once or twice before and was leery of it, but after this experience I'm unlikely to use it again. The inability to review the host in such a situation is pretty much a dealbreaker. (Note that I want to review the host --- not the property, but the person who decided to cancel the booking at the last minute.)
We had an experience not as bad as that, but cancelling on us a week or two before we were due to fly (and had booked everything else). They wanted US to cancel, but I saw no advantage to us in doing so. I assume it was to maintain some reputation on their side. We declined.
No model is perfect, but some are useful. Their "baseline LTV" looks at the sum of listings on their platform (plus other features), then tries to forecast the next 365 days — so this should indirectly capture people coming and going. I think their cannibalization model is quite clever as well.
Going deeper with modeling users might yield some tighter estimates, but I imagine this gets estimates far closer than some simple accounting formula, and likely helpful for budgeting a year out — but it would have been nice to have seen some performance metrics.
I also think the cannibalization model is clever. The baseline model is however a bit underwhelming,
as bookings can be misleading by themselves - you have cancellations, impressions,
and so forth. For example if you only look at bookings in next 365 days, new listings will be penalized. But as you said no model is perfect :-)
AirBnBs are as expensive as hotels now, except you have to clean it yourself and deal with an often insane host and their random rules. I am back to hotels and resorts all the time now, you at least know what you’re going to get.
In the case of Airbnb, wouldn't that show up in the listing's reviews, requests for refunds, etc. and ultimately drive down the listing's LTV? Nobody leaves reviews on an ad, and I imagine that very few users report inappropriate ads, so you can only measure that indirectly. But if somebody books an Airbnb and has a bad experience, they are much more likely to give you direct feedback about it.
I think it could show up in those other places, but probably isn't fully captured?
Imagine an airbnb that's great for most guests but absolutely terrible for 1 in every 5, so bad that they quit airbnb. Maybe it's next to a music venue, so every once in a while it's very loud.
It's possible it could maintain a decent average star rating and LTV as described in the article but actually have negative (real) lifetime value for airbnb if the 1 in five that they lose would have spent a lot of money on the platform otherwise.
Bad reviews are often moderated away and refund requests are stonewalled (and few people know about chargebacks so that isn't factored in). It is easier for the platform to acquire guests than hosts, so the platform takes the side of hosts.
Edit: at least that was the situation a couple of years ago. A host below now reports that the situation has changed and they take the side of the guests; however, either way it's open to abuse no matter which side they take.
The issue is that in a marketplace where both sides can be dishonest, the only way to ensure quality is to do spot checks by trusted actors (aka company employees) where the penalties for failing such a check are dissuasive enough that it becomes more profitable to play by the rules.
This is similar to how law enforcement is supposed to operate - the reason the penalty for theft (for example) is more than merely returning the stolen items is that since law enforcement can't observe everyone all at once, the penalty needs to be enough of a deterrent to make the bad behavior unprofitable overall, to discourage it even in cases where law enforcement isn't there to witness it and enforce said law.
Good point and it affects far more than just Airbnb. I'm sure some companies internal ROI for app notifications is universally positive... but when you send me 10 app notifications a week for a product or service I only purchase occasionally, I'm uninstalling the app. Obviously I'm in the minority because the app still has millions of users but hopefully others have turned notifications off.
I basically uninstall all apps between two uses of them: Uber, AirBnb. Even the bank, I have a problem with the mandatory confirmation-on-phone for the yearly stock options plan I use. Wouldn’t want a thieve to see that app.
I think it's pretty hard to attribute bad stays to stoppage of the service though unless they use reviews in a model which is probably the best data they have to predict this.
I haven't booked an Airbnb in years, whether it be due to having hotel points to redeem or someone else in my party booking on their account or just the price differences in that specific city I am going to. I'm not averse to the service but my account is basically dormant
I had this experience years ago. The first place I ever rented in had Wi-Fi that didn’t work and after about a week the water stopped working. I cancelled my booking through support and the owner called me 3 or 4 times on WhatsApp and then left me a bad review. The second place had a shower that didn’t drain and the washing machine flooded the apartment one evening. The third place had black mold growing out of the kitchen ceiling. I tried to get them to waive the cleaning fee over it, but the owner claimed he had told me about it already and refused to do anything. When I left, he told me he had written me a good review and expected one in return. I pointed out that he had done nothing to remedy the mold issue and then posted my own review, and found that in his review he had accused me of bringing prostitutes and drugs into the apartment (which I did not).
It was nominally cheaper to travel this way, but for my next trip I’ll be staying in hotels.
> There are many sites I won't use because the ads are so overwhelming or are often offensive.
I hate to sound like the “Do people still watch TV? I haven’t owned a TV in 20 years.” Guy. But why are you seeing ads on the web? Don’t you use an ad blocker?
I used to use one but when I started working on ads it felt hypocritical. I want to support the sites that I use, especially those that don't paywall content. I don't work on ads anymore but have stuck with it.
I feel very fortunate to have worked other jobs before my first in tech. My last was hanging drywall and I will never forget how awful it was. I haven’t loved every part of every tech job I’ve worked, but I’ve always chosen to be there.
I think the author is missing perspective on what the alternatives are like, and seems to have a lack of agency when they say things like:
> How many of us have been forced to work on projects that make us sick to our stomachs - surveillance tech, data mining tools, algorithms that reinforce social biases - because we don’t have the power to say no?
There’s incredible mobility within tech - more so than almost any other industry. Vote with your feet! I’ve taken major pay cuts to have more choice over my work, and have never regretted it.
I think it makes sense when you look at it specifically with the "burnout" context in mind. The truth is, burnout can happen to anyone and it is often unclear whether there even is any real recovery.
And it can even happen not just necessarily due to overwork but due to lack of a clear goal, death by a thousand papercuts, complete riddance of passion and interest for certain reasons.
Sure it's nicer to work from your desk than to hang drywall but if you do end up burning out that will affect your brain and possibly permanently alter it. After that, who knows if you could even work in the industry at all anymore?
Personally I believe we way way underestimate it. Longevity is more important than exploiting the passion and overmotivation of potential burnout candidates for short term gains.
I can only speak for myself, but I would absolutely rather work burned out in tech than work in the trades again.
Drywalling was absolutely destroying my body, mind, and will to live.
Mobility is also a factor - I’ve had high stress tech jobs, but always had the option to quit and do something else if I felt close to burnout. That mobility doesn’t exist in many other jobs.
I think this probably isn't as big of a deal as people are making it out to be. But I find a certain kind of joy in Mozilla being judged on the worst possible interpretation of their terms of service, since they do that to others _all the time_ [1].
I tend to agree with you, but I’d hope that given how badly this has been received, Mozilla finds some better writers to put some better thought into the contents of their ToS and issue some corrections in the near future.
Ok this is way too long to read in one sitting, but it looks incredible? I've been looking for resources like this for """real""" model training at scale. The authors have worked on BLOOM (the first >100B parameter model), starcoder and fineweb so they actually know this space.
I wish it were not a single giant webpage, but I guess I can deal with that.
I think this misses the mark. We know LLMs can learn facts. There are lots of other benchmarks full of facts, and I don't expect that saturation of this benchmark will mean we have AGI.
The missing capabilities of LLMs tend more in the direction of long running tasks, consistency, and solving a lot of tokenization and attention weirdness.
I started a company that makes evals though, so I may be biased.
I'm not sure what they intended this to apply to. LLM based systems don't change their own operation (at least, not more so than anything with a database).
We'll probably have to wait until they fine someone a zillion dollars to figure out what they actually meant.
For LLMs we have "for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments".
For either option you can trace the intention of the definitions to "was it a human coding the decision or not". Did a human decide the branches of the literal or figurative "if"?
The distinction is accountability. Determining whether a human decided the outcome, or it was decided by an obscure black box where data is algebraically twisted and turned in a way no human can fully predict today.
Legally that accountability makes all the difference. It's why companies scurry to use AI for all the crap they want to wash their hands of. "Unacceptable risk AI" will probably simply mean "AI where no human accepted the risk", and with it the legal repercussions for the AI's output.
> We'll probably have to wait until they fine someone a zillion dollars to figure out what they actually meant.
In reality, we will wait until someone violates the obvious spirit of this so egregiously and ignore multiple warnings to that end and wind up in court (a la the GDPR suits).
This seems pretty clear.
