Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Alameda lost tens of millions because of a fat fingering mistake (adityabaradwaj.com)
235 points by miohtama on Sept 20, 2023 | hide | past | favorite | 187 comments


Honest question: How do you determine that something like this was an honest mistake?

Once, when I was much younger, I had a side gig dealing poker at an underground club in NYC. One morning at the end of 10 hours dealing, I accidentally exposed a burn card which turned out to be something one of the players was representing (bluffing), with about $10k on the table. It was an honest mistake - literally a fat finger mistake. I was tired and my dexterity slipped. The player jumped up, upset the chips and started shouting that I was cheating and in league with his opponent.

The manager came over to calm things down. He was the young son of a mob boss, whose portfolio encompassed a variety of entertainment franchises. In order to calm the player, he pulled me off the table, told me he believed it was an honest mistake, and then threatened my life if it ever happen again. I quit then and there. Only after I had quit and refused to come back do I think he actually knew for certain that I wasn't pulling a scam.

[edit] I'm really enjoying that people are enjoying this post, but I want to clarify what I was getting at: The original article assumes that SBF et. al. concluded by some internal mechanism that this was an "honest mistake", and that this is sufficient. That assumption belies the way criminal organizations actually work, which is to say, they wouldn't assume it was an honest mistake, but they'd happily say it was one and let off a patsy if they had orchestrated it. I kind of hope I can tilt the conversation toward this aspect of the OP's revelation?


> Only after I had quit and refused to come back do I think he actually knew for certain that I wasn't pulling a scam.

In interactions of bluff, bluff, and double bluff suspicion never dies.

Quitting indignantly on principal is either a sure sign of innocence or a true con artist cutting and putting distance from a 'scam', staying can be just as ambiguous.

I'm not doubting your innocence here, just doubting that there was any action you could take that would convince a suspicious mind that you were clean.

I dare say had you stayed then eventually something else would have happened around you down the track (that's just life). Parting ways and moving on saved you from that at least.


The value of a bluff - and the value of a hand - is all about what's in the pot. I made clear that my integrity and my life were worth a lot more than whatever fraction of $10k a crooked dealer would've gotten. There are plenty of guys who might try to do something like that for $5k, but I'm not one of them.

I'm sure the doubt remained. What you wrote kind of reinforces my point that it's very hard to be sure that an expensive mistake was an honest mistake.

Personally, though, I didn't quit to try to double-bluff. I'm a self-preservationist. I'm in the business of keeping myself alive, and I draw the line at working for someone who threatens me.

I'm aware that quitting could've seemed shady, but at that point it was the least of my worries. And I doubt it's the normal course of behavior, either, if someone were caught skimming. Someone who was guilty of malfeasance, who was offered to keep the job, would have almost certainly chosen to stay and lay low while finding some new way to scam them, because that person needs either the money or the frisson. I enjoyed 'em both but I didn't need either enough to risk my life on another accident.

Someone who lacks any trust will always find reasons to assume the worst, as you say. There's a bit of a corollary, though, in my experience, which is: People who choose to continue working around those people also mostly can't be trusted. And I feel like the trust-less parties are well aware of that.


It's also possible that the manager just didn't care whether you were guilty or not. By quitting and not coming back, you removed yourself from the equation, and so it's immaterial whether you were trustworthy or not.

There could be nothing more at play here than rational self-interest. You quit to preserve your life; this is rationally the best move for your risk-adjusted interests. The house gets to say (possibly very visibly - did you quit on the spot?) that the dealer no longer works here, which mollifies the customer, because the potentially-corrupt dealer is no longer in a position where they can influence outcomes. So it's the best outcome for the house. And then the hand has already been played, so the customer now has a sunk cost on their bluff, but you're gone, so at least they know that if you are corrupt, you won't be dealing any more hands to them. So it's the best outcome for the customer.

You don't need to trust people who aren't in a position to hurt you anyway. That's why walking away is sometimes so powerful: you just change a bad situation by removing yourself from the equation, and start over somewhere better.


> Quitting indignantly on principal is either a sure sign of innocence or a true con artist cutting and putting distance from a 'scam', staying can be just as ambiguous.

Who said anything about quitting indignantly on principle? The sequence of events goes like this:

1. Dealer deals a round of cards, does not intend to reveal a card, but does anyway.

2. Dealer is informed that if that happens again, he'll be executed.

No amount of indignation is necessary to get the dealer to quit. All that's necessary is that the dealer believe the assurance he's given in step 2. The mistake is guaranteed to happen again -- if the dealer was able to avoid doing it, that's what would have happened in step 1.


I once watched a new (I assume) dealer make two large mistakes at a blackjack table, in Vegas, of all places.

One, I had put down $500 in twenties, and he was giving me chips, except he was going to give me $1,250 in chips, like he thought they were fifties. Pit Boss looked at him, looked at the chips, "What are you doing??", let him fix the error.

A couple of hands in, dealer gets an Ace, so he checks the hole card. Satisfied, he continues, and we all bet and play out the hand, until he goes to turn over the hold card... and has Blackjack with a King.

Pit Boss again, situation explained, voids the hand and returns everyone's stake to them.

What blew my mind was that after two mistakes like this in a matter of minutes, the dealer wasn't pulled off the table.


Based on the second mistake, on pure speculation, I think that might've been a dealer who recently came from Eastern Europe, where typically dealer blackjacks are not announced until the end of the hand (a significant loss in EV for the players). Since never exposing a card early is the one thing that's most drilled in, exposing one is counter to training, and that might be a hard habit to break.

A lot of Eastern European casinos don't even deal the "down card" for the dealer until the players have finished.


> typically dealer blackjacks are not announced until the end of the hand (a significant loss in EV for the players).

That’s not just Eastern European, it’s European in general. In fact, that ruleset is well-known as ENHC (European no-hole-card) among blackjack players.


Then why would he have checked the card?


There are some places I've seen in Europe that do deal the hole card, and check it (for reasons unknown), but they don't stop the hand and flip it if it's a blackjack. They just ask for insurance, and the insurance sits there while they let the hand play out. It's quite disconcerting.


> after two mistakes like this in a matter of minutes, the dealer wasn't pulled off the table.

Does anyone know what the dealer training looks like in practice? I assume people will make silly mistakes for a while. Are they expected to practice until perfection before they start working?


Especially when the pit boss catches it and it costs the casino nothing except perhaps annoying some gamblers and a few cents of expected earnings. A trained dealer must be worth much more than a few voided rounds of blackjack.


Dealing requires an exceptional amount of concentration. You need to do every player's calculations faster than they do, while also controlling the game. But it's a lot of rote repetition, physically and memorization. Any person can be taught to deal over a week or two, if they can concentrate. It's just practice. But like suddenly driving a car on the other side of the road, it's hard to switch your automatic reactions. I grew up in a family of Vegas dealers, part of the genesis of my conversation with the guy who got me dealing poker. Blackjack is the easiest... mostly it's just simple additions, which you do the moment you deal the cards. Poker is much more about nuance and controlling and monitoring the game.

Would love to have a properly trained Vegas dealer weigh in here.


Would you mind explaining the second story? I don't play casino blackjack. Isn't the goal of the house to get blackjack? So if the dealer had Ace face up and King down, isn't he assured to win the hand and collect all the stakes?


If he has ace face up and checks and sees a king he is supposed to flip it over, show his blackjack, and take everyone's money. He instead let them play the hand out as if he didn't have the king.


Minor point of clarity... it doesn't have to be a king... in blackjack, a 10, jack, king, and queen all have the same point value (10), so any one of them paired with an ace makes a blackjack.


If the house starts with a blackjack they collect the initial wagers from anyone who doesn't also have a blackjack.


Yes... and this is a net benefit for players who might otherwise double down or split without knowing they were already beaten. Some parts of the world take away this player advantage.


I thought tie goes to the dealer, so they collect from everyone regardless?


A tie is a “push” where neither the player nor the dealer wins. Everyone keeps their money (you keep your money and the dealer doesn’t pay you).


Except in some European casinos, which don't check for dealer blackjacks before play. Under those rules - someone may correct me - a blackjack outranks any other 21, and if the dealer has a blackjack and you made a 21 with more than two cards, I believe it's not a push and the dealer wins. I could be wrong, it's been awhile since I played over there.

They often weight the rules back towards the players in other strange ways, like, being able to double down at any time (even on the 3rd or 4th card), resplit aces, as well as surrender.


Your $500 is such small potatoes to them they literally do not care. There’s probably someone betting $10,000/hand 20 feet away in the high stakes room. You can be sure that dealer never makes it to that room.


Not a poker player, so maybe I'm missing something obvious, but: How did he think you were cheating? You had no way of knowing what that card was, right? And if you did somehow know what it was, surely you could have pre-arranged more subtle ways to tip off a player you were conspiring with?


I guess they thought I produced the bluffing player's card to prevent the other player from folding. But you're right that a dealer who was trying to do that could've done so in any number of subtle ways. I couldn't really figure out why they were accusing me at the time, or what I would've gained from it, but it all happened very fast.

There are dealers who are true shuffle mechanics and either know for sure or have a very good idea of who has what. I'm not that skilled.

I re-watched the movie "Casino" recently, and there's a scene where the Joe Pesci character cans a floor man for not stopping a slot machine run after the first or second time someone hits a jackpot in a night. Paraphrasing the movie: "Either he was in on or he's a fucking moron, either way we can't have him here". To bring it back to the original topic, I think in the gambling world, the default assumption is that there's a scam and the employee is guilty until proven innocent. I wonder why that wouldn't apply to a shady firm like Alameda, too, unless management was "in on it". Come to think of it, I guess that's the point of my story, which I didn't realize when I wrote it.


> the default assumption is that there's a scam and the employee is guilty until proven innocent.

To answer your real question - my guess is that this was a place that had a history of moving fast and breaking things. But probably more importantly - there was no way for that person to benefit from the mistake. The BTC price recovered automatically through the quick arbitrage market. The only way they could have benefited would have been if they were the counter party in the sale, which is highly unlikely.

Now, why FTX was a place that operated like this is another question altogether.


Why do you think they didn't have a counterparty with a lot of purchases queued up at certain trigger levels on the exchange? It would have been an easy way to make a lot of money. Then blame a nameless employee for an honest mistake and come out looking as if they'd just absorbed a loss for moving too fast.


Namely because it would have been too obvious. To have this work, that person would have to have set limits well below anyone else. So low that it would have been obvious when someone just so happened to offer BTC at a much reduced price.

There is nothing that says the BTC “mistake” wasn’t a fraud. But for it to have worked (and you’d only have one shot), the setup would have needed to be clear to outside observers.

The other argument for it being a mistake is that if you really wanted to have exploited Alameda/FTX (not sure which, or if there was a distinction), the person could have tried to take more. The amount is small enough to be a “mistake”, but not large enough to make the higher ups question it. Maybe the gains would have been enough for a corrupt party, but something makes me doubt it.


You wouldn't necessarily need to have those buy orders in place the day before. You could time them to go in at $10k if BTC fell below $13k, for instance. Even the exchange might not detect that if it were done carefully.

Meanwhile, I just find it hard to believe that the accepted narrative - backed up by this story - is that SBF lost $10M in this little fuckup and pardoned the employee who made the error. Seriously? This article? How did he know the employee was innocent, how was that proven to be a mistake, and what happened to the employee? Seems even more plausible given this paper-thin explanation that the flash crash was deliberately triggered to move some money off book. The fat fingered employee, if they exist, gets the blame and is nobly pardoned.

I certainly would hope the FBI is putting together a list of all the unrelated counterparties who profited during that event, just to rule out related ones.


You're thinking very small. People who actually own global businesses think in terms of billions, and SBF had billions. $10M is rounding error to them.

I've lost $50M for my employer (a large, reputable, very by-the-book tech company). I've also gained $100M for my employer, on a project that was canceled because it didn't make enough money. (I half-jokingly asked "Well would you spin it out, I'd love to have $100M?", but it was too tied to corporate infrastructure.) My current project is on track to lose ~$10M/year for my employer, but it's considered strategically important, and so we've run it up the chain and every indication is that it's going to launch anyway.

It just doesn't matter when you make billions. It's like how you stop clipping coupons when you start making a six-figure salary, because the time and attention needed to sweat $0.50 isn't worth it when your paycheck is $10K/month.


my dude, my paycheck is $10k a month, and I still clip coupons. Why? Because something I learned from poker. Winning is as simple as never losing money you don't have to lose.

SBF may have been "worth billions," but he sure didn't have enough escape cash on hand when he needed it.

Your division, project managers, VPs may throw around 9-figures worth of investment capital on loss leaders, party on the Riviera and give everyone Christmas bonuses, but all that means squat to someone like SBF once he's under indictment.

A cold hard $10 million in pocket, off the books, earned illegally, that can't be tracked back is worth a lot more than controlling a billion dollar budget or even getting a $100 million paycheck.

The very fact that it's considered a rounding error would work to the benefit of whoever had set up the scam. The real scam is setting up a corporation where you can pass off a rounding error large enough to let you escape federal custody, and that's where he fell short.

I'm pointing out why it would have been a worthwhile scam for him from my own experience of launching the first serious Bitcoin casino and coming to the rational, purely cost-based determination that there was no way to make enough money from opening it to Americans to let me avoid the likely consequences of that if the feds decided to treat Bitcoin as currency and go for me, which I now know they would have. I determined I could probably pull $10M in a year, but it wouldn't be nearly enough. I'd need to have at least $50M to have a chance. But this is the thought process you can probably project upon SBF running a flash crash, not the "too small to worry about" idea. $10M of misplaced money is right in the ballpark for someone a little stupid who's considering running for the islands.


There's usually a ton of jokers who have $0.01 bid orders in a bunch of markets, just in case of a flash crash. I certainly do - if Bitcoin hits $0.01, sure I'll buy a few thousand dollars worth. I actually looked at the order books around 2019 (this is just a simple free API call) and it was O(high hundreds to low thousands) of people.

It's a bit rarer to have limit orders at say 1/4 the current price but not extremely low, but they still happen. Enough to have plausible deniability at least. Check the order books; my recollection is that this was still O(hundreds) of people.


> there's a scene where the Joe Pesci character cans a floor man for not stopping a slot machine run

Robert de Niro. Joe Pesci plays a thug.


Yeah. You're right, de Niro is the one who makes that analysis.


He said the player was attempting to bluff. In a given set of community cards, you're calculating the possibilities in which you can get beat. You're also watching your opponents actions to see if their lies (bluffing) are plausible.

Let's say you have a seriously strong hand, and there's only one way you could see that you could be beat. The opponent may have that hand now, or they may be hanging around in an attempt to upgrade to that hand while bluffing in the meantime.

In some cases, the above scenario is obvious. Everyone at the table knows well enough what each player is attempting to project. Every player knows which cards to look for to be revealed. It can be devastating to the bluffing player for that card to be accidentally exposed as a burner card.

Given there was a burner card exposed, then there was at least one more card to be added to the community cards. At this point, the bluffing player would be in a tough spot. The player with the stronger hand would likely be raising / re-raising to call the bluff and / or get more chips into the pot.

Continuing the game after the exposed card would have been pointless. You also feel stupid getting exposed on a bluff, when you otherwise might not have had to show your cards. You're probably going to be very pissed.

Edit: The dealer would have had no way of knowing what the burner card was, but the dealer could have known what card the bluffing player was looking for. However, I don't see how it would make sense to expose that card intentionally as a cheating move. Maybe the dealer could have used that as a signal to expose the bluff. My guess is the bluffing player was bluffing in an attempt to make their hand (fake it until you make it) and the other player sensed that. Maybe the other player wasn't convinced and would have won the hand regardless. But at the point of seeing the exposed card, they both knew it was over.

NOTE: We would have to know more about how this played out to do much more speculation than the above.


ftw, this is a perfect explanation of that hand and the betting / emotional situation of the players. Also, it was the river, and it was heads up. But I'll be damned if I can recall their hands or what card I flipped by accident.


You must not know many gamblers. They’ll invent any insane story to explain why they lost, no matter how implausible.


I guess. What I don’t know is whether you can call a misdeal and cancel the round. But if you’re the one who’s not advantaged by the situation will probably be upset if you had a good hand. I suspected there are rules for such a scenario since Im sure it was not the first or last time occuring…


In a casino the hand would not be canceled for that. The card would just be burned.


In this game, the hand was rewound and nullified. I can't speak for what a legitimate casino would do.


Let's assume the poker dealer is working a scam with one of the players, "Sam." On this particular hand, the scam is falling apart. Maybe Sam didn't pick up on a signal, or he has gotten out of sync with the plan.

The dealer sees the tragedy coming down the pike, and as a last ditch effort he spoils the whole game to prevent an even larger loss from occurring after the river card.

For example, on this particular hand, maybe the cards were preloaded into the deal to actually harm Sam. This was done deliberately because we need to keep some volatility across hands to make the cheating more subtle. So, on this hand, Sam is supposed to bet a little bit early, then fold. He loses some of his money, but the whole game seems on the up-and-up.

Except, oh no!, Sam appears to think this is one of the planned winning rounds, so he just keeps betting! Sam has made an error, and may lose big. I (the dealer) am going to suck it up and pretend to make a mistake. I'll get yelled at in public, but the bosses will thank me later for rescuing Sam.


If there is a bluffer on the table, revealing a burn card is going to be net neutral or negative for them. Then add in the meta game! A mob bosses son won't want to look like they did nothing in a situation like this. They need to show they are not a pushover.


It sounds like he didn't think he was cheating and that it was a honest mistake.

An escalation (though obviously not to violence) makes perfect sense for a second error of that type, however. It's the kind of mistake that seems innocent enough but if it's deployed in a pattern it can obviously assist the bluffee.

Fool me once, fool me twice, etc. It's not uncommon for dealers to be in on scams, obviously.


You can deduce odds by seeing a burn card. It could help more or less in certain situations


I'm skeptical about the "honest mistake" story as it's self-serving because an "honest mistake" isn't a crime, or at least it isn't as serious a crime.

I can easily believe that development was lax without proper monitoring, controls and testing. Typically for any kind of trading system there'll be multiple layers to help catch these kinds of errors or deliberate fraud. This could include some or all of:

1. A separate system that monitors and clears orders looking for pricing mistakes;

2. A risk management system that looks at your total position and can block orders if they would make your position too long or short;

3. Another system might monitor actual assets vs custodial assets to see if you ever "leak" assets.

To trade any US securities you'd have to go through SEC compliance to make sure you have sufficient systems and monitoring in place, particularly if your system is facing unsophisticated investors. Crypto has largely avoided this kind of scrutiny and rigor.

Part of all this is separate systems help prevent both accidents and fraud. A pricing check in a trading system could be broken by a code change or bypassed by a malicious change. Subverting an external system you may not have access to gets more difficult.


Yeah, and - someone correct me - aren't at least some of the charges against SBF related to intentional backdoors in the code?


> Honest question: How do you determine that something like this was an honest mistake?

Good question. It's not like the errant trader couldn't have had a friend of a friend put in a bunch of $10k limit buys the day before, knowing the price would recover nearly instantly...


Right, this was my original question and I'm wondering how SBF would've gotten to the bottom of it. What I'm speculating is that a gangster would've either punished the employee or been in on it.


How does the hand end in a case like that?


They take you out back and cut it off with a circular saw. At least then there’s no more “fat fingering” to worry about.


Or break your kneecaps. I remember it from a movie whose name I don’t recall


Answering the wrong question lol


On the house, especially if players are usual customers.


Unlikely; if it were lower stakes, maybe, but even high end poker houses are not such high margin enterprises that they can afford to comp $10k to an angry customer because of dealer error. Maybe some smaller comp, but not the whole pot.


So the house pays $10,000 to everyone who hasn't folded yet?


No, the person you’re replying to is just wrong. In a real casino the card would be burned and play would continue.


> let off a patsy if they had orchestrated it

the patsy in that scenario would have been the guy screaming the game was rigged

or you, if they had popped you in front of the gambler to prove they took his allegations seriously

the patsy is not "in on" what's happening. They might let you off if they thought it was intentional but that somebody more important had put you up to it

wiktionary

patsy: (informal, derogatory) A person who is taken advantage of, especially by being cheated or blamed for something.


I played poker at casino and these kinds of mistakes happen all the time.


Yeah, play just continues usually. High stakes dealers are usually more experienced and less likely to make these kinds of mistakes, but at 1/2 or 2/5 games (max buy-in around $500 at these stakes, so pots tend to stay under $1k), this sort of error isn't uncommon in my experience.


Funny that the top comment here is about poker - there's a rumor in the poker world that SBF was a whale in some of the biggest private app games, and owed the game runners north of $10m when he went down, which the winners in that game (apparently including legend of the game Dan "Jungleman" Cates) are now never going to see.


Don’t they generally have ties to the underworld/extralegal means of reprisals against people who owe them money? So wouldn’t they have acted by now if this were true (even if not “successful”)?


I mean, there are gangsters that play poker, and there are private games run by gangsters (I imagine), but the idea that the poker world is highly entangled with the underworld is pretty outmoded at this point. High level pros like Jungleman aren't going to risk their rep/access to other good games by breaking someones legs, even over a few million. He and his backers have the risk of losers defaulting baked into their EV calculus already when he plays those games.


I imagine right now while the spotlight is on him might not be a great time to carry out any kind of revenge. There will be time for that later.


That's an interesting coincidence... it wouldn't surprise me in the least. Never heard he had a bad losing poker habit, though. Sounds juicy if it comes out.


Great story! Thanks for sharing.


Heh. I love being able to pseudonymously tell HN my crazy stories from time to time. It's the only online chatter I indulge in. Sadly I have no colleagues who I can both talk shop with and also talk about my previous lives.

"Build what makes you happy..." is a darn good philosophy, btw.


I've played one of those spots by union square. Was fun to feel like a gangster for an evening but ultimately not a crowd I want to spend a lot of time around.


Hah. That's posh. They used to have apartments all around NYU stuffed with slot machines they brought up from Atlantic City. Club members got a key to let them just go play slots. But I never played or dealt downtown. This gem was in a basement in Harlem up on Malcolm X, steel door and heavy security, 2 blackjack tables, a couple poker tables and some slots. I knew the manager because he came every morning for breakfast to the restaurant on the upper west side where I was a waiter, and we got to talking poker every morning. He asked me to come deal, so they'd pick me up in a decked out black Lincoln Navigator with tinted windows after I changed from my waiting shift, I'd deal for 8-10 hours and make 5x as much as my regular job, and then they'd drive me back. I don't even remember the cross street.

Let's just say your money isn't going to anything good if you play in one of those places.


Wild. The slots thing - seems so easy for someone to rat on them and then so easy for the cops to break it up.

The Manhattan one was poker only. Steel door with a camera, get buzzed into a second door where you talk to somebody and they only buzz you up if they recognize you.

The poker room had two nice tables and a little bar where you could buy drinks. TVs playing sports. Seemed to be Asian-run.


Similar setup, this was Italian.. but yeah they used to get rooms busted by cops all the time. They had trucks full of slots. No one even sat in those places, they just went to pick up the cash every day (I know this because I accompanied someone on their rounds to pick up cash from the slot machine apartments once, and was kinda shocked that some of them had no one inside at all. Just like, some studio apartment on 14th and another on Canal and another on Ave B. Wall to wall with slightly used slots hah)

[edit] There was also one night I showed up to work only a few hours after the cops had raided the poker room, and they were just getting back up and running.


> "Build what makes you happy..." is a darn good philosophy, btw.

Thanks! I wish I'd say it's my daily driver, but I still work for money.


Cheers =)

Seriously though, I think making clients happy still makes me happy, if I like what they're doing and I get to impress them a bit. I guess it would be more fun to be able to live only on passion projects, but I know I'm lucky even to be able to keep the hope alive and keep creating.


Celsius lost millions in dumb mistakes like that too.

Someone at Celsius manually approved a bogus transaction on a hacked website...

https://www.coindesk.com/markets/2021/12/03/crypto-lender-ce...

Badger later reimbursed Celsius with inflation on their token that would be released slowly over several years. "restitution".

Someone at Celsius thought that they could sell that restitution token and ended up forfeiting it... which was another $22m loss...

https://www.thismorningonchain.com/articles/defi/celsius-mad...

Then they tried to get the DAO to vote to give them back their lost tokens, without even admitting who it was that lost the tokens... and everyone voted against it.


That moment where they got scammed by a hacked frontend was when I realised they were extremely incompetent and told everyone to get their funds out immediately.

Moving millions of dollars customer funds around via a browser wallet is insanely bad, they should have had well tested scripts that interact with the smart contracts directly.


How many people listened to you?


yeah a lot of "dumb" mistakes. whoops accidentally sent to wrong address, which i happen to own the private key too...


In this case, it wasn't anything like that.

Someone exploited a weakness in CloudFlare and was able to replace the Badger website. When someone clicked on the site to execute an approval transaction, it went to the attacker first, which gave the attacker full control over their wallet.

It did take a manual step on the part of Celsius though... which should have been checked more closely. The UX around that checking is really terrible though and when someone is trying to do something quickly, they aren't always going to check. This is a big failure of wallets these days.

Balancer.fi just had a similar attack happen to them where the .fi registry allowed a nameserver change.

https://twitter.com/Balancer/status/1704552288201883809

It is also clear that the frontends really need to be hosted in a way that they can't be modified.


> It is also clear that the frontends really need to be hosted in a way that they can't be modified.

Years ago the advice was IPFS and IPNS.

I agree. This is not the Web3 dream everyone was promising us when frontends and nft media assets themselves are mutably stored on some server relying on serveral entities in the DNS chain to maintain security, behave, and stay available.


I wish there was a way we could almost hash a website or a piece of the critical path code running the site so that you know the content on the page was not modified and that the code that is executed is what the site intended.

We kind of have the 'secure lock' with https doing part of the work, but it is kind of irrelevant if DNS is pointing to some hackers site.

This isn't just crypto... it is your bank too.


That's kinda what IPFS is - every webpage is identified by its hash. But then updates are impossible so IPNS steps in to give a mutable name -> hash relationship. Just like a git commit, if you have the IPFS hash you are guaranteed for it to be correct. How you find that hash - IPNS or some other method, has been the major challenge.

Further we used to have HPKP to further protect the security chain but it ended up being dangerous for various reasons. Monitoring certificate transparency logs for any re-issuences of your domain's certificates is the current detection method as http is pretty heavily penalized in todays browsers.


rookie mistake. have to send it to a family member’s address so you can claim it isn’t yours - truthfully.


Yeah, easy steal, no traces, Id never believe it’s just a mistake but the burden of proof is on me.


A lot of the "genius" credit given to FTX, Alameda, Sam are all during a time when crypto was soaring, probably partially because people were home more and got free checks in the mail. The fact that a single person could absolutely tank the entire company with a single trade shows how reckless they all were. But it worked because the market was going up. That's it. The market slumped for like a week and their behavior dropped them to 0.


> The fact that a single person could absolutely tank the entire company with a single trade shows how reckless they all were.

Sadly, this sort of thing happens in traditional trading firms as well.

https://en.wikipedia.org/wiki/Oil_futures_drunk-trading_inci... https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stoc... https://en.wikipedia.org/wiki/Nick_Leeson#Barings_Bank https://en.wikipedia.org/wiki/Rogue_trader https://www.foxnews.com/story/typing-error-causes-225m-loss-...


also don't discount sporting events being closed/reduced and sports betting dollars moving to crypto


Haha mild sports betting is my 2nd main use of bitcoin, other than store of value/wealth/whatever we call it when you buy something low and now it's worth a lot.


It also helped that they couldn't be margin called because of a secret system setup at FTX (which is why FTX imploded).

Just leverage long 50x forever until they're all billionaires or FTX is bankrupt.


This can't be the whole story because Alameda started a few months before the big 2018 BTC crash and made it through just fine.


> the "genius" credit given to FTX, Alameda

The hype cycle marketing was targeted by geography and social class.


This is my sense too. What's kind of funny is we weirdly get caught up and excited by the hype... every single time


“We” is doing a lot of heavy lifting. I am proud that I have never gotten burned by this sort of thing because I understand that if there’s a lot of hype it’s probably too good to be true.


Fair enough. I guess media cycles get caught up.

And for me personally, I will say every time the economy booms for a few years I start to ask: gosh is this ever going to stop?

For example with real estate, I haven't bought any because the prices always seem insane to me. But I'd be far wealthier (on paper at least) if I'd bitten the bullet and bought into the real estate market a few years ago.


I always felt like SBF was a massive fraud. When someone is inflated as a genius and "the kindest billionaire" for superficial things like not driving an expensive car and not dressing appropriately, it sends alarm bells ringing to me. Acting like the exaggerated TV stereotype of a tech nerd screams "I'm covering something up", just like what happened with Theranos.


Just something relevant in another field: this is exactly how people feel about John Fetterman demanding to wear basketball shorts and a hoodie in the senate.


Hmm my reaction to that is "good marketing." People who have some kind of shtick tend to get more media coverage.

Like, was Steve Jobs (turtleneck) covering something up? Zuck (t shirts)? Newsom with his Bruce Wayne hair?


Maybe it's just because I'm too young to have seen much of Steve Jobs prior to his death, but my impression of coverage surrounding him was that while he did have certain behaviors he insisted on and were often talked about like his turtlenecks, it didn't really fit the stereotype of a tech nerd of the time.

Back then "nerd" seemed to have a much more negative connotation and seemed to describe people who were closer to "gross" than "quirky", like the trope of them constantly having a runny nose or sneezing everywhere due to allergies. Steve on the other hand came off as very clean and insistent on dressing nicely, even if in his own style.

I feel like Zuck was/is similarly seen as a sham for his entire robotic persona. Perhaps not as a scam, but he certainly never seemed to be all that liked by people.

I don't know anything about Newsom to comment on that one. But one other example where I was telling people it was probably a scam beforehand was Nikola. I don't recall what about his behavior did it for me, but something was just off about him in the way SBF was that just screamed deception.


Here is a good analysis of the flash crash from that day: https://twitter.com/austerity_sucks/status/14523324727258194...

- It lasted 12 seconds.

- Total BTC traded was 290.54

- Only 2% of that was at price of 10K or below.

- Most volume happened in $20-30K range, over 50% discount to mkt.

Here is a tweet from someone (rightly) speculating that it had been Alameda: https://twitter.com/TheoryBitcoin/status/1452345411759398923


The good news is that the world's Bitcoin network can only process ~100 transactions in 12 seconds.


The Bitcoin blockchain processes a block of transactions every 10 minutes on average. Each block usually contains 2k-4k transactions. The maximum capacity is around 4k, equating to 7 transactions per second or 84 transactions per 12 seconds. (Note that 12 seconds is the confirmation time of the Ethereum blockchain).

That being said, the Lightning Network was deployed on the Bitcoin blockchain in 2018. It allows for thousands of transactions per second, thanks to a very elegant solution[1]. Unlike traditional blockchains, the Lightning Network is not blockchain-based, freeing it from the limitations we are all familiar with. However, it inherits all the security properties of its underlying blockchain. While the Lightning Network has its fair share of shortcomings, its scalability remains an open question. Nonetheless, it has been working great for over 5 years.

[1] https://lightning.network/lightning-network-paper.pdf


(Most) trading's not done on chain, it's settled later. Alameda could absolutely trade faster than that.


Doesn’t that just mean the trading is done by a trusted third party? Fine until the third party goes belly up.


This is how most asset trading works, but outside of crypto these third parties are actually trustworthy


Well, one might argue they’re trusted to grift retail. The money firms make doesn’t come out of thin air. SEC is fairly toothless in this area.


> outside of crypto these third parties are actually trustworthy

With multiple layers of mutualised failsafes.


Don't forget meaningful regulations and legal requirements placed upon these parties


Yes, the exchanges are 3rd parties that could lose, and often do lose, the money


I thought the (noncrypto) financial exchanges made a very small amount of money on every trade.


Has this improved as of late? Maybe with other blockchain systems?


Yes, it improved in 2018, but people are still stuck on the 12 tx/s. The Lightning network allows thousands of tx per second [1].

[1] https://en.wikipedia.org/wiki/Lightning_Network


Snarky (flamewar starting) answer is it improved massively since they made the blocks bigger in 2017.


> According to SBF, the utility we gained by moving fast outweighed the occasional costs we paid due to poor risk checks, hacks, and the like. This was SBF's work philosophy, and it drove the culture he created at Alameda and FTX.

Moving fast is great, but you need structure to support it. If you front load your development with guard rails that ensure you're always on track, then you "aim small, miss small" so to speak. If you do none of that and just hope that you can respond to problems as they arise then you've really not aimed at all and can potentially hit a target, but likely also hit a house.


And this is why this sort of thing will always keep happening, especially in new fields. It's like an anthropic principle of competition. If you're willing to cut corners in a way that could blow up the whole company, you can get ahead, therefore the leaders in a new field most likely cut a lot of corners and may be at risk of blowing up.


> The story of how a misplaced decimal point at Alameda Research caused a market crash that echoed around the world

Hang on a second. It caused a dip in the price of Bitcoin, that is not a "market crash". Bitcoin is a toy and not a real market, its price is set almost entirely by bullshit wash trades and other forms of fraud.


Bitcoin has a market cap of $530B and at its peak was over $1.3T. That is not a "toy" but a very real financial market.


> the total (ostensible) value of all coins that have been mined in a given cryptocurrency. These values should be taken with a hefty grain of salt, as they are considerably larger than the total value that could be realized if holders of a currency decided to try to cash out.


You realize that this is how valuations for publicly-traded companies are calculated, right? Also the networths of people whose vast majority of wealth is tied to publicly-traded shares.


Putting on a white coat doesn't make you a doctor, the years you spend studying and practicing does.

Similarly, printing a billion tokens and trading one for a dollar doesn't give you a billion dollar market cap, even though "all the stocks do it!" Those stocks also back productive systems, have financial statements, forecasts, and a million other things that make them a billion dollar (or larger) market capped company.


Hey, it's fine, we'll just make another cryptocurrency that serves to back it, and control the price of that to keep the value stable. What could go wrong?!


So, it's a bit different. In the simplest case, a company pays 1bn dividends every year, so if 5% is considered a good rate of return currently, and there's no reason to expect the company's profitability changes much, then the value of that company is $20bn. Now, obviously, there's a lot of complexity layered on top of this. Many companies don't do dividends, preferring stock buybacks, for various reasons. Some companies are valued on _assumed future profits_, not current (ie valued on potential). And then there's speculation. But ultimately, if half of the $20bn company is suddenly sold, well, the price, in a rational market, shouldn't actually move much, unless the future value of money has also shifted. The company could be taken private by someone who had $20bn and thought $1bn/year was a good return.

Bitcoin's really pretty different; the value is ~entirely speculation driven. It does not, and cannot, pay dividends, or buy back its own 'shares', or anything like that. No-one would ever consider buying all the bitcoins (in the same way they might take the $20bn company private); the sole value of bitcoin is in other people wanting to buy bitcoin.


Shares give you rights and have a fundamental value. If the market undervalues them, you can buy the company and make lots of money when proven right (by virtue of the cash the real business makes and pays out in dividends). That’s fundamentally different from crypto which has no fundamental value, but only sentiment behind it. When the sentiment is gone, the value is gone. Unlike, again, shares.


IBM is still worth money if their stock gets delisted. Real estate, buildings, equipment, patent portfolios. Bitcoins is worth $0.


Those buildings and equipment and patents may have value, but less than the bonds, loans, and other liabilities IBM also has. Not only could IBM be worth $0, it could be worth less than that.

Things are worth whatever people are willing to pay. Bitcoin may be the original sin of crypto bros, but it still appears to be worth money, because it's impossible to make one for free, or obtain one for free. And it has utility. You can buy drugs or avoid the banking system using it.

Silicon Valley Bank had buildings. It was literally worthless at the end. A company is just as fictitious as a bitcoin, only existing because people all agree that "company" is a thing.

(EDIT: I changed my example to SVB from Enron)


But IBM is a public company. These numbers are known. Its estimated liquidation value right now (selling off all the assets) is around $22 billion. That's pretty far from its market cap of just over $130 billion, but it's not nothing.


SVB became worthless by virtue of real changes in the real economy. Of course it can happen that a company goes bankrupt, and then its fundamental value is zero or negative. But the point is that there is a fundamental value that is realised over time. The market value is a good estimate of that fundamental value according to the EMH, but if you have a better analysis and better estimate of the fundamental value you can trade and then realise that difference over time if you were right.

> a company is just as fictitious as bitcoin

Nonsense.


Yes, that's how all stocks and every other market in the world works. Still doesn't mean the value isn't valid.


Not sure how wash trading sets price. My understanding is miners sit on BTC and cash, then occasionally collude using cash to pump price, short perps at the top, then offload BTC and ride the perps down.


Prices are basically set by takers, like in every other market.


Prices are set by people stupid enough to believe that the "market price" is a market price.


Then short it


"the utility we gained by moving fast outweighed the occasional costs we paid due to poor risk checks, hacks, and the like"

I think this may have potentially been true when everything was going up volume-wise, but as competition improves there's less margin for error.


It was true from Sam's perspective, where all of those holes provided cover for fraud, embezzlement, and self-dealing (possibly up to and including the large “oops, unattributable hack” loss right as Sam was about to be shoved out the airlock.)

But that's because the holes were actually directed added utility from that perspective.


it's not about the number of mistakes but the size of the mistake


This happens all the time at traditional finance firms also. For example, last May an investor in the Nordic region sold off hundreds of thousands of shares of an index fund leading to an 8% drop in dozens of stocks https://www.bloomberg.com/opinion/articles/2022-05-03/citi-d...


And if you are waaaay off the exchange just shuts down your port for the day.


Interestingly the Tokyo Stock Exchange did not countermand an order from Mizuho in 2005 which flipped order quantity and price in yen, a rather catastrophic thing to do when the intended trade was one share for about 600,000 yen.

Regulators were… not pleased.

(Quick English summary of incident: https://www.foxnews.com/story/typing-error-causes-225m-loss-... )


US exchanges tend to either bust or price-adjust clearly erroneous trades retroactively.

That is good for unsophisticated customers, but it creates a disincentive for market makers to provide liquidity when there is a fat finger event. That in turn leads to larger price dislocations on such events.

(The reason is that the market maker faces adverse selection. Providing liquidity is a bet from the market maker that the price dislocation will revert. Price adjustment reduces the profit the market maker makes from mean reversion. But if the price dislocation was caused by a real news, the market maker will eat the full losses.)


Reading stories like this makes me wonder why I don't have a long term trade in place for a low amount and hope to fish a good one during a flash crash.

Seen them on the stock market too. Back in my day I wrote a few trading bots, they never went anywhere positive after about a week of profit.

Seems to me now that the smartest robo-trader algorithm is...

    10 wait for human mistake...
    20 goto 10


I made 17.5k doing this by dumb luck.

I was trading crypto back in 2016-2017. Making a few thousand here and there. Mostly just from time in the market during the run-up of ethereum. The whole thing drove me crazy because I thought it was so stupid, like gambling. So in 2017 I withdrew my several thousand dollars and put in a trade on coinbase pro for 5 etherum at $100. (I had left $500 to do this, the most I was willing to risk at the time) On 2018-11-25 this trade was executed in a flash crash.

I had totally forgotten about this trade and when I was going to buy a house in 2021 I was looking for any change in the seat cushions thinking I may have left a small amount in these accounts. Well imagine my shock when the amount was just short of $18,000.


Glad it worked out for you! Thanks for sharing.


HFT algos are going to make quick work of the opportunity well before your broker can even think about executing your limit order.


You want your GTC order parked on-exchange, not sitting on a broker's private book. Find a better (more expensive) broker.


I thought the real market also has circuit breakers in place. If the market falls X% too quickly, trading is halted for some amount of time.


That applies to the whole market, not necessarily a single stock.


It definitely can apply to single stocks, depending on the exchange. For example I believe NYSE has single-stock circuit breakers that kick in around 15%, then subsequent further breakers once trading resumes.


If you aren’t paying attention you just might end up trading on a real crash too.


Which I guess is why it has to be so low you don't care.


Yes we call this a stink bid, and it’s a valid strategy. Keep one open just in case. Most exchanges have slippage protection in place but some are better at it than others.


I believe, the traditional exchanges either won't fill outrageous trades or would undo them if the counterparty can prove it was an honest mistake.


Running a system that allows a fat fingering mistake like that to go through is yet another sign of their gross incompetence.


This happens in “proper” firms too



I believe that knight was not a fat finger


Sorry, you are correct. It does demonstrate how brittle things can be at large funds though.


The number of things that had to go wrong at the same time in the Knight case was large as far as I remember.


I’ve seen straight up attempts to sell a massive amount of a major stock at ~0 due to a single bug and failure of backup checks.


> According to SBF, the utility we gained by moving fast outweighed the occasional costs we paid due to poor risk checks, hacks, and the like.

When I was young I "moved fast and broke things", then had brushes with the law that fortunately didn't result in jail. Silicon Valley prides itself on second and third order thinking, but do people ever consider what might come after "moving fast and breaking things"


Contacts who worked at Alameda claim that this guy did not work at Alameda, or if he did, they never met.


Turns out that the author of TFA did work at Alameda as confirmed by court documents.


In personal banking, at least of those I've experienced, the numbers are displayed in both plain language ("forty-two thousand sixty-nine") as well as numerals ("42,069"). Wonder if this would help prevent this kind of mistake as well.


Seems like a bunch of kids who were born into borderline fraudulent culture... and now damage control pieces (ordered from SBF's jail cell maybe, ha) are being tolerated... maybe with the tacit support of a culture which would rather think of these as things that 'everyone does.' Somewhat akin to the Stanford or Harvard data integrity issues, maybe? This is just business as usual and sometimes it spills over. Who cares as long as you are headquartered in the Caymans, right? To be sure, all of these 'oopsies! SBF is such a sweet, gentle soul' articles of late look a little weird. Dude threw his GF under the bus in obvious fashion.


on a sensible exchange that attempted match would trigger a circuit breaker and either halt the market for a short period or put it into an auction

(and members that create orders to do that a lot either get their membership removed or pay fines)


> What they missed was the decimal point was off by a few spaces. Rather than selling BTC at the current market price, they sold it for pennies on the dollar. The result was immediate. The price of BTC shot from a high of $65k to as low as $8k on some venues.

Wouldn't this sell just be gobbled up quickly by buyers? Why would it move the BTC price so dramatically to the downside? Surely this couldn't have been that much volume at near all-time highs of BTC in 2021?


Yes, it would, but at what price(s)? Think about what would happen if the size of the (very low priced) sell order was >= the size of all resting buy orders priced >= the price of the sell order.


So like ~0.5% of their total losses?


~0.5% only needs the error to happen ~200 times to account for 100%


Or just reject the trade


> So naturally, we had sanity checks in place to make sure that the orders being sent were reasonable relative to current market prices. Not so for manual trades, which were by nature discretionary.

This seems to imply that risk checks are somehow either unnecessary or impractical for manual trades, which is completely untrue. This is 100% a case of 'we chose not to implement that'.


>The tricky thing about risk is that it's usually invisible, right up until it comes around and bites you in the ass.

Saved!


Invisible can equally mean “actually invisible” but often means “we told the people shouting and screaming about this risk to shut up”


> we had sanity checks in place to make sure that the orders being sent were reasonable relative to current market prices. Not so for manual trades, which were by nature discretionary.

I guess we have different definitions of "invisible"


This description of SBF does evoke a Netflix "I was in a cult" documentary. A larger-than-life leader, with huge vision. I wonder how much of his success was due to his charisma and the ability to steamroll (and sidetrack) people with over-the-top world-changing vision.


I used to work in operational risk management at a major investment bank. This kind of thing is not at all unusual.

Fat finger errors would often result in unintentional gains. We incorporated them into our models as if they were losses of the same magnitude.


It's not a mistake - it shows the sheer lack of safeguards at a terrible company with a narcissistic child-leader and his cohort of children at the helm! Hope he rots in prison and is made an example of.


Alameda is a city (and an island) in Alameda County, California.

Perhaps the title could be updated to reflect that this article is not referring to the city?


I used to do some work for a company called Captech, then called Capital, I think it's called Hum Capital or something now, but I was always waiting to see how they'd handle it in the news if someone did an article on it when it was called Capital.


Interesting to see Binance lie and say it was an algo mistake as opposed to a human one.


Could be Alameda misrepresenting the situation. Doubtful that Binance would be able to tell whether Alameda's accounts were running an algorithm or manually trading


Never attribute to stupidity that which is adequately explained by malice.


i don't recall this crash. must have been nanoseconds. I guess in hindsight that was the least of its problem. billions dollars missing makes ten million a drop in the bucket.


They managed to successfully get money for cryptocurrency.


Is that a difficult thing to do?


isn't it fat finger and not fat fingering .... which sounds something nsfw?


I have a fair share of our story with Fat Fingers:

The Athlete's Glitch

Our client team had an amazing NFL season. But with the start of a new NFL season, but our NFL team was buzzing for a different reason – the unveiling of our off-season updates to the beloved iPad app used by one of the major NFL teams.

Our app, which was primarily used for training and past game analysis, had undergone a major overhaul, and we were proud of the sleek new UI/UX designs. But just as the athletes started to get into the grind of their training, our office started receiving some unexpected feedback.

"Hey! The app is taking me to the wrong sections!" one message read.

"I keep hitting the wrong button. Something's off," said another.

Confusion took over our team. We had spent months meticulously planning, designing, and testing these updates. Automated tests had been executed to perfection, internal testing hadn’t shown a single glitch. What was happening?

Then, a video came in. It was from a coach, showing one of the top athletes trying to use our app. We watched intently as his fingers moved over the screen. The problem became immediately apparent: his fingers, sculpted by years of athletic training and naturally larger than the average, were simply too big for our redesigned interface. Every time he tried to tap a specific function, his finger would unintentionally touch the adjacent ones.

Turns out, in our endeavor to create a sleeker, modern UI, we had inadvertently shrunk the size of the clickable buttons and packed them tightly in a grid. This might have looked aesthetically pleasing and worked perfectly for our testers, but for the athletes with their robust fingers, it was a recipe for frustration.

We convened an emergency meeting. Our lead designer, Marcus, broke the silence, "Our primary users are these athletes. We should've considered their physical attributes in our designs. It's my oversight."

Our project manager, Clara, nodded in agreement. "We need to fix this and roll out an update ASAP. We can't have the team struggling with this during their crucial training period."

The next few days were a blur of coding, designing, and testing. With feedback from some of the players, we reintroduced larger buttons and ensured enough space between them, all while maintaining the sleek look of our new design. It was a lesson learned the hard way, but it reiterated the importance of understanding our users' needs and physical attributes.

The next feedback we got was from the star quarterback, "Perfect! Back in the game with this. Thanks, TechTouch."

The NFL season kicked off with roaring crowds, and our app, now more user-friendly than ever, was right there with the athletes, assisting them every step of the way.


[flagged]


Sam Bankman-Fried was associated with effective altruism, not e/acc (although there's a link between the two).


The OP of this post has e/acc on his twitter bio


It's wild to think that just one fat fingering mistake was responsible for the collapse of the whole FTX and Alameda ecosystem.


It's wild because it isn't true.


Millions, billions, it's just one letter difference, right?




Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: