On one side, yeah - the work he did will be worth a lot more than $10k to Rockstar. They've had lots of people complain about & stop playing GTA online because of the slow loading times. And this issue would probably never be found or fixed otherwise.
On the other hand, he did it for fun (without any expectation of reward). $10k isn't massive and we can quibble over the reward amount, but its delightful that companies are rewarding people at all for fixes like this. A few years ago he'd be lucky to get sent a GTA t-shirt. Its definitely the right direction we should be moving in - bug bounties are a thing in security, and we should be celebrating any movement toward rolling them out in other circumstances.
And also they have to balance out internal politics as well. Putting massive incentives on the discovery of bugs could really mess up the motivations of internal staff.
This might be an argument for him to console himself (or just keep smiling) in the event that he received nothing. It isn't a valid consideration for Rockstar, which should take heed of the calculus you stated with your first hand only, because for the beneficiary of a windfall, karma awaits.
If some engineer at Rockstar had done this work, they would have gotten their regular salary and probably not much else, which probably would have worked out to less than $10k for the amount of time spent on this issue.
And yet, in practice, none of the engineers at Rockstar has done this work in 7 years and a half. I do not believe that it is fair to value *a posteriori* a one-million-dollar fix with "if" sentences.
Let us spin it differently: contact Rockstar before publishing the details of the fix, and offer to help them on this issue. How much do you believe that Rockstar would value the fix *a priori*?
The truth is this kind of fix is worth a lot, but it is always tempting for the company to be penny-pinching (and come up with justifications for being so) after the fact.
Engineers don't get to work on whatever they want (usually). The described problem doesn't stop people buying shark cards, the fix is not new content and players leaving the game because of it don't affect the bottom line significantly. This could be why the priority was low and why you should prioritise tech debt
A Rockstar employee would have had access to the source code probably. Running a profiler would have taken them significantly less time than the author.
As a company that's technically true. As an individual engineer they would probably have been sneaking time to work on it and have to be a pet project of someone senior enough to get it merged and released.
It's a really nice and wholesome use of HackerOne. Nice that they acknowledged and awarded as opposed to some companies that try their best to push a security disclosure out of scope and not pay out a bounty [0] [1]
I honestly expected Rockstar to do nothing at all about the situation. I’m genuinely surprised they’d implement his findings and give him some money as well.
At this point, I wouldn't have been surprised if a video game company sued someone for DMCA violations for something like this. Rockstar and Take-Two in particular are no strangers to handing out DMCA take downs, cease and desist letters, or lawsuits. I'm glad they're being cool about this.
The added value of the fix is in the upper 8 figures due to enormously reducing friction for players. If a consultancy was hired to fix the performance problem the cost would run over $1M. $10k is funny money to Rockstar.
In normal people's terms, if somebody did hours of labor that gained you >$10k, would you give them $10? That would be insulting, yet that's what Rockstar did.
Maybe you don't realize that taking in $1B means they were doing just fine without this bug fix. Faster load times amount to "Oh isn't that's nice".
After reading his account of finding this, how long do you think it took him? 10 hours? 20? 40? That's $250 to $1000/hour for something he did out of curiosity.
Oh my good lord that was a serious eye-opener. I’m not American but can’t imagine it’s drastically different in the UK. I’ve lost a huge amount of money in the last year and also my car decided to die on me, I’m currently scraping together 3 grand or so to get a new one at the moment. So many people I know are struggling at the moment too. Meanwhile Jeff Bezos is probably sleeping like a baby.
The original post, analysis and the actual fix of this bug is beyond impressive. Strangely, Rockstar awarding $10k for that is like someone searching for lost treasure, identifies historical references which leads to finding a ton of it and somehow leaves with 1/8th of it, even though they did 98% of all the work.
For that detailed effort, analysis and with a fix, I'd say that is worth around $80k. At least Rockstar was generous enough to award them in the first place; so I'll give them that.
> For that detailed effort, analysis and with a fix, I'd say that is worth around $80k.
No good deed goes unpunished. Rockstar went out of their way to thank this person and send them a check equivalent to several weeks of median developer salary. Can we not shame them for this goodwill gesture?
$80K would be on the order of half of the annual salary of most developers outside of SFBA.
I'm sorry to hear that you're not better compensated. I also emplore you to reframe your perspective. Consider the possibility that pay in many places outside tech hubs in the US is outrageously low. There's a race to the bottom wrt exploitative commodification of developer talent as the 3rd world comes online, and I find it abhorrent. This "SFBA / FAANG companies are the sole exception" narrative is incorrect, and harmful to you and to me and to everyone in our field. If you're living in a 1st-world economy, have solid technical chops, can communicate reasonably well in written and spoken English, working for a for-profit business, and have a few years of experience, and you're not making over $150k/yr, you're almost certainly underpaid.
Know your worth! Demand more! Capture the value you produce!
- Chris in Duxbury, MA
PS For context: I've been building web-related things for a living since 1998, and have always been paid fairly for it. Because I insist.
PPS Ofc YMMV... circumstances, personality, ambition, etc etc. But there is so much more opportunity to create and capture real wealth than you might imagine.
"and you're not making over $150k/yr, you're almost certainly underpaid."
Those salaries don't exist in the UK, no matter your position or the company you work for. I know senior programmers working for Faang companies in London and they don't make this much. Like, in general you won't see a position offering above £100k/year. I know someone with 25+ years experience in the industry and they are on an (incredible for the UK) £90k/year.
The only way to reach the equivalent of $150k/year in the UK is to work as a contractor and be good enough to justify billing that much. As a normal employee you won't see that.
My point is - it's impossible to judge everyone in the world by the American pay standard. Would I make more money in the US? Maybe. Would my standard of living be better? Also maybe, but I lean towards no with this one. But it's almost certainly not true that by making less than $150k/year you're underpaid in a place like the UK.
I worked for a US/UK company with its engineering HQ in London until recently and was earning drastically more than that. Now, I was an early stage employee and doing unique work for the firm. That position wasn't advertised - I created it. But some of the more senior engineers on my team (in the UK) were earning about $150k/yr GBP equivalent, a bit more in some cases.
Are salaries in the UK lower than in the USA, yes, absolutely. They are not that much lower than non-SF jobs in the USA.
Putting aside the question of whether there's an objective standard for whether someone is underpaid - it is not true that nobody in the UK makes $150k/yr as an IC SWE without being a contractor. Filter on the UK: https://www.levels.fyi/comp.html?track=Software%20Engineer
You'll see that mid-level (L4/E4) engineers at Google & Facebook easily break 150k; senior engineers (L5) start at ~225k and hit 300k. That's about 30-40% less than what Google & Facebook pay engineers in SV.
I also saw multiple other companies where engineers with less than a decade of experience were approaching or breaking 150k - Transferwise, Improbable, Palantir, Morgan Stanley, Booking.com, Cloudflare, Bloomberg, Yelp, Babylon Health, Goldman Sachs, Stripe, etc.
My entire network makes more than that in Dev and Arch roles in that part of the world. None of them have to live in London for them either.
Either you're underpaid, or C++ has been commoditized to the same point as PHP. If you value money over language, I'd switch. But given quite a few in my network are Java developers, I doubt language is the issue here.
Well then I have no idea how you and your mates do it. I know people who are in very senior positions at FAANG companies as well as couple big financial firms in the UK and not a single one of them makes over £100k/year. Two of them with PhDs in big data analytics and working in the industry. My own salary might be excused by being in the North East, but they work all over the place, in and out of London.
It sounds like you've got an academia background so if they're research or R&D positions, that might be why. Because they're competing in an already saturated field. Or there aren't many jobs but lots of graduates keen to work in your subfield who will do it for lower pay because they enjoy the work.
If you moved into in-demand fields, service delivery or consulting, you'd see 100k GBP as normal for seniors. The fields only require mastering the tools or domain and the ability to work with business stakeholders. That is from my experience, the majority of the software world.
I don't know what your friends told you, but either you misunderstood them, are talking solely about base (and see my other comment for why >100k base also exists in those roles), or were given inaccurate information. 100k GBP would be 45-60% of the credible range for senior engineer compensation at Google/Facebook in the UK (and maybe 55-70% for Amazon). That's so far outside of the standard compensation bands that other explanations are more likely.
You’re framing it the right way, and this is what we should work towards. But man, I long for the day when what you’re describing here is realistic.
In Norway, country of «holy crap, I’m an employer lobbyist and our salaries are so high we can’t compete in the export markets», you’ll only be around this level if you’re a freelance contractor billing 100% and doing all the sales work yourself.
There are some CTO and probably a few lead roles paying this much, but for an IC you’ll top out around $120k. Prove me wrong if other Norwegians have a better view :)
If you're reading this and are an startup founder hiring software engineers, one way to help kickstart the trend in this direction is to hire fewer engineers than you otherwise would have, and become _much more selective_ with who you hire, but pay them salaries that are competitive with not your local market, but with the hottest markets in the world (SF, NYC, etc), and actively advertise that you're doing so.
This will give your company a unique competitive advantage in capturing the best talent your market has to offer (and it likely has plenty of world-class talent to offer, but many of them end up getting brain drained away to those hotter markets today). It will also allow you to stay small for longer, dramatically reducing communication/coordination overhead and delaying the need to introduce management hierarchies, which is actually a substantial competitive advantage especially for early stage companies where agility and flexibility is the key to success. If all that's not enough, it will also do wonders for retention since it's basically a semi-permanent golden handcuff.
Over the long term, other companies in your market will end up having to compete with _you_ for talent, which raises the boat for everybody and helps your local market flourish with engineering talent and reverse the outward brain drain trend.
The unwillingness of employers in other markets to compete for talent outside their local maxima, optimizing for short-term self-interest over the long term health of the local talent market that they depend on for global competitiveness, is a true tragedy of the commons situation, and is a huge part of what makes Bay Area startups so dominant.
It's also what forced my hand into moving to SF myself. It would have been extremely financially irresponsible of me to stay. By leaving 5 years ago, I tripled my salary compared to competing local offers, and one job change later I'm now making more than double of that (at a startup, not at one of the big tech corps).
Nowhere in the world outside SF/NYC is there so much VC money that can support paying developers those salaries.
I would say over 90% of tech companies make their money from local markets. You can pay your developers as much as you make from your local market.
As for hiring. You have no idea how well someone is going to work out. Some people are really good at interviews but turn out to be terrible employees and vice versa. This is why the hiring process in tech tends to be pretty convoluted and broken in general. No one can figure this out. The laws regarding hiring are also much less liberal outside of CA. Hiring and firing is severely restricted in many places.
Very few companies are in the position of being "unwilling" to pay developers more. Some with plenty of cash are even simply limited by the tax laws how much they can pay. The cost of paying someone too much over local market rate incurs obscene tax costs.
> The cost of paying someone too much over local market rate incurs obscene tax costs.
This is fascinating to me, do you have any examples of jurisdictions where this is the case?
Here in Germany the “taxes” (they’re technically mandatory insurances) than an employer will have to pay on top of the gross salary all have a salary cap, the largest one being around €85k. After that any increase in gross pay has no additional cost. Of course my personal tax rate might take a hike but that’s not the concern of my employer.
For slovenia taxes are progressive, above 10k a month is the highest bracket and taxes are 61%. If you want your employee to make 10k a month, your company pays 24k. If you want your employee to make 5k a month, the company pays almost 12k.
I think that’s a foolish comparison. When talking about these super high SF salaries, folks are talking about _gross_ salaries. Nobody will reject a higher salary because they’ll have to pay more taxes.
A lot of the conversation is about the value of the developers work being high enough to warrant higher salaries. With that argument we should look at gross salaries + taxes/fees paid by the employer on top of the gross salary.
Your personal tax rate is between you and the state.
I was not talking about a personal tax rate. My point was that employers can not pay anywhere near SF salaries because some places tax high salaries so severely. It is not a matter of employers being unwilling. A rich company can't just decide to pay people at SF levels and expect to create a local silicon valley.
One key part of the suggestion that might have been overlooked was to hire fewer engineers than you would have otherwise.
This could mean hiring 1 really amazing engineer instead of 2 or even 3 average ones depending on the realities of your local market, and pay them based on what it would have cost you to pay the 2 or 3 engineers combined.
This is always a choice you have available, unless you're bootstrapping and can't even afford to make the first hire at those rates, at which point you're probably not looking to compete on the global stage to begin with, so this suggestion doesn't really apply.
If a company knows that it'd actually cost 300k to hire the local kind of engineer that could make 150k in the Bay Area, that company is either going to move to the Bay Area or hire Bay Area-like engineers as remote contractors to get their 150k worth of engineer for 150k.
The "able to pay" part will resolve itself if you manage to start a sufficiently successful startup. Eventually through high enough profits, since the startups that turn into global leaders in their field are more profitable per employee than anything a local non-tech company can compete with.
Getting to that point is a bit of a chicken-and-egg problem, though. And firing policies is a political barrier that's hard to avoid. I suppose you could keep everyone on as contractors, paying them standard rates but going to them directly, not through an agency. That would land the salaries in the region suggested, and you'd still have the ability to fire without cause.
There are more silicon valleys, like Tel-Aviv or Shenzen, but arguably the Bay Area is the only one merit-based. Good luck getting funded at the other two as a foreigner.
Interesting thoughts. I think you're onto something.
Going off on a tangent: I personally won't leave for California/SF since I have family here and won't put the obviously massive financial advantage ahead of that.
But I did consider this when I was in my twenties. It wasn't a super easy choice. Interesting to hear that someone have taken the leap. Not multiplying one's salary by 3-6x could indeed be described as quite financially irresponsible, and the alternative has to be pretty important to pass on that.
Companies that pay well in the US also provide good healthcare benefits. There are some weird setups like HDHPs where they give you free money at the start of the year, then immediately charge you so much for prescriptions that it gets used up again, but I think this is a way to let us farm credit card points.
I lived in the UK with all those things you describe, but I still made a point of billing my Bay Area rate because that is what the work product of a software engineer is worth on the market.
So, no. Even if you live in your parents basement and they cook your meals and take are of your other expenses, you should still approach your work life as though you were an adult.
Never lived in the US (and haven't lived in the UK for years) but the idea that you need a SV salary to get health care which meets or exceeds that of the NHS sounds a bit suspect.
From what I know, moving to US from a country with good socialised health care is a lot like selling a lot of put options on your health with different strike "prices". You might get rich, or if you are very sick, you are bankrupt.
Multiple studies have shown that Kaiser Permanente (an integrated health insurance/care provider—basically a non-governmental equivalent of the NHS in comprehensiveness—that is available in many US states) is more efficient and effective than the NHS for about the same cost.
You seem to have some simplistic view of how global economy works if you think all around the globe people should be payed the same for the same profession. Unless you work as freelancer in which case most of the time you have to pay your own taxes(which in most countries will amount to a big percent of your paycheck), you will get salaries according to the local market price standards regardless of what your peers in another country get. There are at least a few reasons for that one of which is that the market would not allow for huge disparities in salaries compared to other professions and to the same profession with local funding. You can try to negotiate good bonuses and/or equity but that is not mandatory and it is as much as you can get.
Actually, I have no intention to "retire". (In recent years I've taken the month of July off to take my family sailing; I can easily imagine taking longer stretches off, and/or shifting my time further from paid work towards hobbies... and at some point maybe even pivoting to writing and making music full-time.) My "ikigai" is, thankfully, mostly decoupled from making money per se.
When you realize how high cost of living is in cities paying those salaries, how much education (for potential kids) is, how you must save for your retirement entirely on your own... and then of course insane out of pocket medical costs... you soon realize that when you adjust for lifestyle you may be in fact much better off than the person with stratospheric salary.
I'm in the bay area. For what it's worth, my personal out of pocket max for healthcare is something like $2500/year, and they were only half-joking when I got the "million dollar work-up" last year at UCSF. I spent 3x that when my dog ate a corn cob... There is the additional cost of premiums for the rest of my family, but it's not absurd.
Real estate and property tax is rediculous, state income tax is high, and restaurants around here charge $18 for a hamburger these days. We're going to have to move once the kid(s) are old enough to be hindered by the bad schools. It's enough to perhaps make the high salaries not quite so competitive, and especially so if you need to pay for private school.
Salary is only a partial picture of compensation in the bay area, though. If you're a software engineer at a FAANG type company, you're potentially getting 1-2x your salary between bonuses and equity. If you joined early on at a start-up that's doing well, you're perhaps getting set up to spend your 40's mowing lawns like Forrest Gump if you want. It's enough that, even if you take very conservative career choices, you could work hard out here for a decade while living comfortably, then move somewhere else, buy a nice house outright, and work for fun while not stressing out about money ever again.
Right. A very, very ordinary apartment in NYC is $6000/mo. Figure 30-40% of your salary for taxes. If you made only $240K in NYC, your apartment would use up fully half of your take-home.
Even assuming this is true ($6k a month sounds absurd), this is a bad argument. The old "rent should cost no more than one third of your income" rule is basically for people around the poverty line. For them it's a question of whether they can afford to eat that month.
Just to give an extreme example: if Jeff Bezos uses half his income this year to build the largest estate the world has ever seen, he's still sitting pretty on the rest of it and has the largest estate the world has ever seen.
At a 35% tax rate, your take home income is $156K. At $6K per month ($72K/yr), you have $84K left over after rent. If this is you then YOU ARE WEALTHY. There is simply no getting around this. You are making more alone, after taxes and after rent than my partner and I have ever made together total in a year in our lives, and we live in a city with a relatively expensive rental market and are not "poor".
The absurd talk about "only" making over two hundred thousand dollars a year (plus whatever your partner makes if you have one) needs to stop. It's so out of touch with the norm in virtually every city in the world that it would be laughable if it weren't kind of sad.
People taking in $200k are not in any sense "very, very wealthy", except from a third-world viewpoint. $200k in today's dollars approaches what would have been called "middle class" a few decades ago, when there was one. Now you are only low, below $300k, and high, above $1M, with very few left between.
(In countries with regular health care and education, life costs less, so this about the US.)
The plutocrats have been very successful at deceiving American voters that they are not almost all now lower-class. The Powell Memorandum, 50 years ago, laid out the plan. It has been followed since then, and Americans now are routinely persuaded to vote against their own interests.
Wealth produced in the US has been rising as fast since 1975 as it did between 1945 and 1975, but normal Americans' income since 1975 has been flat. The whole difference has gone straight into the pockets of the extreme rich, and they are who are very, very rich. You and I have no concept of how different their lives are.
In 1975, regular parents could pay for putt their kids through college while paying their mortgage.
>>People taking in $200k are not in any sense "very, very wealthy", except from a third-world viewpoint.
Mate, I don't know what to say. You live in a bubble made out of solid stone granite. Literally anywhere else outside of US that kind of salary makes you "very very wealthy". Within top 1% definitely. So I don't know if you think anything outside of US is third world? Or what?
>>(In countries with regular health care and education, life costs less, so this about the US.)
Simple: Outside the US, $200k is very comfortable, except in certain places like Moscow and London.
In the US, it would be lower middle class, if there still were a middle class. So, instead, it is high in the lower class. If you consider it wealthy, then congratulations, the propaganda is strong with you.
I'd love to know where and how, because I have never seen a position offering £100k and above anywhere in this country. I have friends working as seniors for Facebook in London and they don't make £100k.
I have two guesses - either you were a contractor, or worked for a bank where I've heard it's possible to cross £100k salary. Or third option, you mean <£100k of salary + bonuses?
The one I worked for wanted me to move to London, and I said fine, as long as you give me a comparable standard of living.
Nope, they took my Irish salary and converted it to GBP, and that was it (this was post Brexit, btw).
Obviously, I didn't move. And working with any engineers in the London office was a massive pain as anyone without family commitments worked in London for a year, and then transferred to the US for a massive salary bump.
The core issue is that FAANG don't benchmark against finance in London, which means that pay is pretty out of wack.
For instance, I got a 150K offer from an investment bank in London around that time, and I'm definitely not as useful in finance as I would be in tech.
It just blows my mind that they haven't fixed this to be honest (I suspect that they'll have to, now that Brexit has actually happened).
Almost certainly the conversation is about total comp rather than base salary; at FAANG the difference is marginal (especially since at Facebook, there's no 1-year cliff; your equity starts vesting immediately every month).
But even putting that aside, most senior (E5) Facebook engineers in the UK make 125-160k base salary in the UK. Even if that's denominated in USD, that still leaves a significant % breaking 100k GBP base.
As I explained in other comments, salaries that high don't exist in the UK. And you can't just negotiate to have your salary be double the market rate where you live, no matter how good you are. Also, isn't Sydney the most expensive place to live in the world? My mortgage where I live is £600/month for an actual house, what good does making 2x the money give me if I have to pay 4x the money for rent?
Rockstar will make tens of millions of additional revenue from increased player retention over the years (GTA Online is a massive cash cow, it is pretty much the original hyper-monetized game-as-a-service as a continues to fountain cash to this day), slow load times are an extremely commonly cited reason for marks (excuse me, customers) leaving the casino. This is a product that had made six billion dollars as of two and a half years ago, it is likely the most profitable entertainment product ever created, $10k is an almost insultingly low fraction of the value he created for the company.
Better than nothing I guess, but suggesting that it's OK to exploit people because of where they live is kinda crass. This is an industry where you are (should be) valued based on what you can create, not where you live.
To flip this around, if this guy lived in Haiti would you say it would have been fine to give him $1k because welp, cost of living is even lower there? Insulting.
And again they had years and years they could have fixed this themselves, at in-house prices if they wanted to. They had to be shamed into it by a third-party developer pointing out their "could be fixed in a day by a single developer" tier mistakes. O(n^2) algorithms causing minutes-long load times, repeatedly reported by thousands of customers across the game's 10+ year lifespan.
It's like giving a waiter a 1% tip. Is it nice that you gave them a tip at all? I guess. Technically nobody asked them to get you your food. Is it a reasonable fraction of the value delivered by customary standards? Not really. And in this case it's like a 0.01% tip.
> $80K would be on the order of half of the annual salary of most developers outside of SFBA.
If the author of the bug fix is indeed from Latvia (.lv), the median annual income of a senior software engineer in the Baltic states is currently around 30K-40K after taxes.
Half? I live in the Midwestern US. Have been dabbling in software for most of my life, have some decent working examples and am finishing my BS:CS this year.
I can't even get close to my current salary in my current unrelated field ($85k) to even consider a career switch. Are the jumps from junior to mid-level/pm that high? I am a PM in construction so that may end up being my path.
Junior to Mid level is pretty significant, even outside of the bay area. At a finance company, Junior to Senior for me was about $45k base salary over ~5 years, with another $20k in bonuses annually. If you can only find say a $70k job, I would think its not unlikely that you would be able to work 2 years at that, and jump for $90-110k.
As someone searching for employment in the Midwest, I feel your pain. I saw a position yesterday that wanted 15+ years of experience and paid, I kid you not, $17/hr.
Compensation in the Midwest is a joke. If my family weren’t here, I’d return to one of the coasts.
They're looking to fill the position with a foreign contractor, almost certainly. I wrote some PHP sites 20 years ago and get contacted every now and then about taking a 2 month contract for $16 an hour writing PHP in Pittsburgh, which I'm sure has its charms but is also halfway across the country from me.
I think this is a great example of a fallacy that's pretty popular on HN: That work should be rewarded based on "value created" rather than effort.
I don't think this line of thinking is helpful. Should Amazon increase the price of their VMs if you made more money with them? Should a doctor charge a lawyer a higher price for a flu shot because their time is worth more?
Of course not. Because in all these cases, the clients would just go buy someone else's service.
Of course it is helpful to consider the value of the work you produce, and you can use that in negotiations. But you are not entitled to a higher rate just because your client is big or successful.
If the developer had offered a two day consultation for $80.000 to Rockstar they would most likely have declined and said, sorry, for that money we can hire someone else to do a lot more performance improvements.
If you want to get paid based on value created, you need to negotiate. But you don't have a moral right to get more money for the same effort just because your client is rich.
>For that detailed effort, analysis and with a fix, I'd say that is worth around $80k.
Do you think the fix is actually worth 80k in terms of additional revenue for Rockstar? I wouldn't be so sure. If not though then no it's not worth $80k.
"I'm a developer and I'm paid a lot to do my job, therefore everything I do must be very valuable" is not the right line of reasoning to use here.
It’s absolutely worth that much and more. GTA5 Online pulled in well over $500 million in 2019, and even more in 2020. It’s now the 3rd best selling game of all time, behind only Tetris and Minecraft. https://www.tweaktown.com/news/69666/gta-made-half-billion-d...
The only relevant fact to justify $80K is -- would slow loading times have resulted in at least $80K of lost sales going forwards? Or at $50 a copy, then starting today would there be 1,600 people who would buy GTA V, but won't because they know loading times were slow? Maybe because a friend or reviewer mentions that as a con? (And you can also add lost DLC revenue from people who gave up playing earlier than they would have otherwise.)
Honestly, that's an incredibly hard question to answer. But it is in no way obvious that it is "absolutely worth that much and more".
In contrast, $10K is a nice round number that demonstrates gratitude for their charitable work. And keep in mind this was probably a weekend's worth of investigation or so. Not many months of dedicated work, which is what $80K buys on the job market.
If the 500 million figure is correct, you would need only a 0,02% (1/5000) increase in spending for a change to be worth $80.000.
Diminishing the notoriously bad load times of GTA:Online by 70% could probably do that. I actually never got into the multiplayer because at the time my HDD was taking several minutes to load the game.
It's not about $50/copy, it's about lost revenue from GTA:Online which makes money off micro-transactions and continued spending from existing customers. And yes, I think it should be obvious that very long load times has made a significant number of people spend less time and money than they would have with fast load times.
This reminds me of my 13 year old sibling yelling at our mother when we were teenagers for not buying them the exact item they wanted, but something pretty close. Disappointing.
One should not hold people responsible for acting "non-perfectly". Any behavior exceeding "average" should be met with admiration and gratitude, average behavior with "respect and appreciation", and anything somewhat sub-average with "mild disappointment but no surprise and some understanding".
Having a company spend money they are technically obligated to allocate to share holders, to a developer who, without any pre-existing contract or even verbal or commonlaw understanding, helped them, is way beyond "average response" for modern businesses.
Please, don't hate on the vanishingly small number of counter-examples blazing a trail in this space.
They tend to get a lot of community accolades but yea - spending time fixing someone else's intellectual property isn't a great place to start your proposition if you want to make money.
Even traditionally mod friendly companies will tend to sweep in and either block any derivative work that's monetizing their improvements or else negotiate with the creators to take the lion's share and republish the content under their company label. There are a bunch of examples of this but Counter Strike is a pretty famous one.
That all said, it's pretty hard to argue that making minor alterations to a product should allow me to resell it as a novel work. Fan Fiction exists - but it doesn't (legally) exist as taking a novel and taking on a few new chapters then reselling it without giving a portion to the original author.
He got paid a good month salary for spending a bit of time fixing an issue (which is tricky but not extreme if you know re). I feel like people having strong opinions that's not enough should just open a GFM campaign to give him more and add their own money. It easier to say how others should spend.
The only reason he got paid at all IMO was the widespread exposure the original article got. When you find something this significant you’re almost obligated to do a song and dance and tell the world about it, so you can get the attention you need and be paid.
Frankly for an introvert with no public online presence or audience that’s just not very appealing. There’s easier ways for software developers to make $10k: just wait for two paychecks.
I think the author was motivated by intellectual curiosity rather than money. Thankfully he was enthusiastic enough about it to share it with the rest of us.
I'm not sure if "embarrassing" is the right word here.
This bug has existed for years and has lead to hundreds, if not thousands of people quitting the game citing this exact reason yet Rockstar did nothing.
I think "shameful" is really more appropriate. It's not like they simply didn't know this existed. I think they have known all along it existed and didn't give 2 shits to fix it.
Watch out not to condemn them no matter what they do. Sure, they messed up in the past, but given the situation they found themselves in when they saw this report, this response is still good.
You risk making discourse difficult when you continue punishing people who are attempting to do the right after doing wrong for a while.
I disagree. They ignored the issue for years and that part won’t be erased from memory. They should be given credit for finally fixing it, but nothing more. What happened before this week was and remains forever shameful.
Very likely that internally there were people who had an inkling this was going on, but could never convince product to devote time to put it in the backlog. This change is a win for that engineer.
This example will be ammunition in internal debates about how much to weigh product pressure vs engineering suggestions that improvement is technically possible; the next time an eng says "let's look at this slowness one more time, it may be fixable more easily than you think", they might be listened to.
Losing customers over such a simple, known problem, over years… is tragic for a company like Rockstar. This is like Microsoft ignoring a bug that made Windows take 10 minutes to start and just letting that happen for 5 years. Now tell me that’s not tragic.
> You risk making discourse difficult when you continue punishing people who are attempting to do the right after doing wrong for a while.
On the other hand, a few days of doing the absolute bare minimum of quality control doesn't undo a decade of actively hostile practices. They're still very very deeply in the red when it comes to their relations with the playerbase and for good reason.
Paying $10k to the person who documented this bug is a good start, but you need to do a whole lot of work to make up for the impressions caused by that kind of prolonged systemic negligence.
Right. I'm not saying eval(their net behavior)>0; I'm saying that if your response function is always negative, you lose the ability to influence their response.
Have you heard the one about the country with the death penalty for all crime? It just leads criminals to maximally try not to get caught, without regard to damage done [i.e. removing all witnesses to even the smallest crimes]. This isn't what you want; even within "negative" relationships both sides lose when you don't have a gradient of responses.
The crux of this is that most readers presumably found out about both the bug and the fix in relatively short proximity, and their response is negative to the combined events, not to the knowledge of just the patch.
A criminal prosecuted for murder and petty theft simultaneously can't really go around complaining that they got 25 years in gaol for stealing an apple, but if everyone had already known about the murder and dealt with that accordingly it'd be pretty harsh.
(not that fixing a bug is exactly a criminal act, but the general principle remains)
An alternative possibility: there were 100 bugs of similar consequence, all of which they cared about, and they got 99 of them. This one escaped, because stuff happens.
> I think "shameful" is really more appropriate.
When you make an inadvertent mistake in the future, karma is coming for you: people will speculate that you "didn't give 2 shits".
This bug existed for a very long time and was the #1 outcry from gamers. It is well documented all over the internet.
Every company I’ve ever worked at and every code base I’ve ever worked on would have made this a #1 priority. If it were a technical limitation of the devs I would agree it is “embarrassing” but this was not outside of their ability.
I simply don’t blame the devs, I blame management. Management should have allocated the resources to fix this truly game breaking bug but they didn’t because money was still flowing. Is it any coincidence that ads played during the 6+ minutes you would wait to get into a game?
Yea, if it was a 6 month old bug or it was a hard to repro edge case, then sure it’s embarrassing. Oops! But years long, and highly visible/reproducible, without correcting? That’s veering into shameful.
Blame should always flow up, but that doesn’t mean there aren’t other causes. For all we know there has been a team of devs putting their kids through college “working” on this for years.
More likely, there were 99 other bugs that they had clearer metrics related to monetization that they prioritized (not that this bug didn't impact monetization - I bet it did, just not in a way they were measuring). Or that their QA engineers identified and flagged as high priority or major new features with their own bugs and regressions or whatever. These things get lost in the shuffle. More likely there were 1000s of bugs in the tracker, probably 10s of thousands.
Yes, I agree with your elaboration. Also, of the 100 bugs, they didn't get 99 of them, because there are surely some left. Maybe they got 90, and there are 9 still hiding out.
The game is played by millions of people, we don't need to wonder if there were a lot of more important bugs, this has been by far the most complained about issue for many years.
> This one escaped, because stuff happens.
No, they mostly just don't listen to their players when compared with other game companies.
I understand your frustration, but anyone who's run (or been on) a dev team knows you have a million things you could/should be doing, and many, many priorities. Often supporting new launches/features is the priority.
Sure, ideally there would be someone internal advocating for a fix like this on behalf of customers. But Shameful isn't the right word. It's a miss like any other.
The egg that's on Rockstar's face has more to do with how much money they likely left on the table by not diagnosing and fixing this issue early on. (And assuming they had a team overseeing this aspect of performance with adequate tools to observe and debug it should have been trivial.)
I personally feel like the magnitude of this -- especially compared to how simple the solution is -- makes it a miss that's really much more extreme than most others.
EDIT: Though I agree that "shameful" isn't the right word to use. Rockstar wronged themselves much more so than any other party.
The cause doesn’t actually matter much, usually one doesn’t know the issue until they investigate. So often issues remain open for lack of interest, then someone finally does and it’s a missing comma. This doesn’t affect the fact that the issue has been ignored.
I’m sure GTA has unlimited bugs to look into, but a bug that turns people away before they even start should have probably been given higher priority.
> If the fix had been some super application specific super obscure thing, we would all understand.
Would we?
Even if I don't understand what it's doing, devs doing nothing about an O(n^2) algorithm that takes multiple minutes every load is clearly a big mistake.
Even if it's something that has to be that slow, cache the result.
Oh absolutely, I have a whole host of stuff on my todo list for projects I'm developing that has been there for months and will remain there for months longer.
That said, they're all things like "refactor the frobnicator code so it's more obvious what it does" and "make templating system log changes to server". Nothing on my list says "Look in to that major performance issue that literally thousands of customers complain about every single day". That stuff gets sorted immediately.
There probably is a story behind it. Like some developer looked into the bug in the beginning (when when the load time was not that bad because the json file was smaller), saw that the time was spend parsing json, and then, without thinking too much whether this makes sense, wrote in the bug ticket that, to make it faster, they would have to switch from json to something better/faster/different.
Since switching the file format is probably some work, maybe involving many differnt teams and network protocols, the bug was closed because it too much efford.
And then, whenever somebody complained, the old bug was referred which unfortunately never got a proper analysis.
Sad, but I saw this happening many times in the past in some projects. Since the old and wrong analysis will be in many powerpoint slides and heads, it is incredible hard to argue against or get even time to work on such a problem. Usually, you need someone who just takes some time form other projects without telling anyone and ninja fixes the problem to solve it.
Based on my own experiences, I'm absolutely certain many people over many years tried to get the load time fixed, and then were beaten down for their earnestness.
Just as I'm certain that this unsolicited 3rd party fix was used to settle some old scores.
I think "embarrassing" could still be the correct word. If you were to ask the development teams responsible they might feel a little embarrassed over the solution being so small (yet technical).
And if they didn't care at all given what you say then why would they care to implement a fix for it all these years later?
This is akin to that guy buying a whole-page ad to complain about his ISP recently, resulting in enough shame to trigger action. It's a bad look for Rockstar considering the income that title generates and at most it was a few weeks worth of dev time. Very interesting to watch the public reaction to this in comparison though, seems to be a lot more sympathy for Rockstar than for ISPs.
They still made like a billion dollars in shark cards so apparently all the people buying shark cards didn't give a shit about the load time issue and that's the real problem.
The game supports 8 platforms, has virtually miles of terrain, and now Rockstar is suppose to feel ashamed because they didn't fix a big performance issue that affected startup times?
The game probably has tens of millions of lines of code.
I'm glad they shipped. If they can enhance the startup time, great, bonus for them. Additional credit for thanking the third party that uncovered the perf issue.
I don't think you've been following this story. If you looked into the technical details [0], and surrounding context, you would understand why your comment isn't particularly insightful. For example, terrain has nothing to do with this bug and it isn't a complex fix that interacts with "millions of lines of code." A single run of a profiler would have caught this.
I also think it's relevant to note that GTA V has sold 140 million copies, 20 million last year alone, and Rockstar makes hundreds of millions of dollars a year on the title. For a multi billion dollar product to have a known, relatively simple, problem is shameful. I don't know what went wrong, but I don't think it was the competency of engineers or the complexity of the problem that allowed this to fester.
I think you missed previous news about this. This was an incredibly trivial issue that would show up in any use of a profiler on the slow startup screens. The fact they didn't is that they're either unable to run profilers on their code at Rockstar (which would be shameful indeed) or they never bothered to do so. Either case shows egg on face at Rockstar.
Strange that this was this downvoted. It is an issue that a single dev with a profiler should've been able to identify very easily. That they hadn't addressed it despite it being such an annoyance for so many users for so long is embarrassing.
Not to take away from the feat and initiative of doing so without access to source code.
I’m pretty shocked at all the defending of the game company here, honestly. Highly visible bug, affecting high percentage of users, relatively simple fix. I take pride in the software I wrote and would be personally very embarrassed if I caused this AND didn’t fix it for years. I guess companies don’t have feelings and can’t feel shame but I would personally feel shame as a supposed professional!
There are a lot of excuses for why this wasn’t fixed but no great reasons.
The idea that screwups are "shameful" is poison in any organization. I fight it in the companies I work for, I fight it here.
Bugs happen. Serious bugs happen. They will always happen. Shaming people for bugs undermines institutional robustness against bugs because it incentivizes people to hide mistakes. Furthermore, companies with blame-n-shame culture typically fault users when those users make inevitable "mistakes" statistically guaranteed by bad UX design.
In contrast, transparency is good, acknowledgment is good. Rockstar acknowledged fault and rewarded the reporter, which is difficult. That's a positive institutional arc I'd like to encourage, at Rockstar and in the wider industry.
If you stop at "embarrassing", I'm with you. But when you proceed to "shameful", that's quite different and we part ways.
Just to clarify my point: The bug was a screwup. We’ve all done screwups. My worst bug was an amateur hour royal screwup that resulted in DDOS. Nobody should be shamed for bugs, including serious ones that get to production.
Not fixing a serious, visible, known bug for six years is what’s shameful. There is nothing positive about neglect like that. It’s kind of admirable that they actually rewarded the guy but it shouldn’t have even got to that point.
Totally agree you should not shame for bugs. The shame is for ignoring this bug until one of your customers finally had enough and fixed it by patching the binary. If it was my company I would feel ashamed.
I haven't read all the comments on these articles, but I haven't seen anyone claim that the bug shouldn't have happened, that anyone should lose their job, that no one "good" makes those sort of mistakes, etc. But this was a bug that lasted long enough and irritated enough customers that one of them finally diagnosed and seemingly fixed it without ever looking at their source code. That's one hell of a process failure. And splitting hairs over whether we would feel professionaly "embarrassed" or "ashamed" in a similar situation is pretty silly.
Managerial problems that allow high-impact, low-fix-time, high-prevalence bugs to exist for 7 years are shameful and are in fact poison to an organization themselves.
Don't mistake the shaming for really being about the bug itself here. The problem isn't the bug, it's the organizational culture that allowed such a massive, easily fixed, widely encountered bug to persist for 7 years in a game that has made them $6 billion dollars over the last few years.
They probably just didn't have anyone working on it that had the knowledge (or incentive and/or permission) to fix it. AFAIK they've just been adding content to the online portion for the past 7~8 years, and you don't need engine programmers to do that.
"They had never gotten any programmer with the requisite expertise to take a look at the code in nearly a decade of opportunities" sounds more or less like a verbose phrasing of "They didn't care."
The fact that they didn't make this a priority to fix shows exactly how much this company actually cares about it's customers and it's reputation. It's unbelievable that despite the quite public outcry about the issue NOTHING was done until some 3rd party guy figured it out for them.
I'm quite surprised they are even going to patch the game because the level of "we dont give a shit" I thought they had about this issue certainly would have prevented them from doing anything now.
Imagine all of the quadratic poorly written code sitting on your code base at your workplace. It's probably not a handful. People need to stop pretending that their code is optimized and magically linear all the time. No one looked at this because they didn't know it about. There is no reasonable way for any person to fix things like this unless they are actually looking for it. How many people at Rockstar would be responsible for engine details like this and have the skill to find and fix it?
A game video company employees artists, sound effect engineers, and etc. They aren't full of programmers combing over the source code for quadratic footguns.
Noone is complaining that they did not look through all their code for quadratic footguns. There probably are bunch of places like that, but they don't matter and doing that would probably be just a waste of time. Premature optimization is the root of all evil and it makes all sense to dedicate all your optimization on the 1% of code that actually matters the most - but this was the 1% that obviously should have been the target of optimization and was not.
The issue is that slow loading times was a contentious issue with many, many customer complaints (IMHO it was the #1 complaint) and during many years not once did they spend even trivial effort on trying to look into that complaint, and that's shameful. As the tostercx's article shows, if they had ever assigned any engineer a couple hours to look into why loading is slow with a profiler, that issue would have been easily found; the fix as described was a bit more tricky because the author did not have access to the source code, but within a company that would have literally be done within an afternoon by a single non-exceptional person. But they never cared enough to have someone look into it.
You have customers complaining about everything all the time (and non-customers with a high profile "edgy" title like GTA), so filtering out the actually relevant topics might be hard (esp things like "bad loading time" when it might be the users computer).
The thing is that this loading time issue probably didn't exist at release as it was part of something that got expanded on over time and only got bad worse by the years.
Assuming somewhat linear growth of the data the loading time would have been at least 4x shorter only half the games lifetime ago (3 years) and 16x shorter 1.5 years after release. After that period they would probably not consider there to be issues in those places in the code so nobody really looked or realized anything was amiss there.
The problem is not that they suddenly hit this performance drop and solved it in some reasonable timeframe, or it was some rare occurrence affecting .01% of users
The problem is that it’s existed for 6 years, affected anyone and everyone, and would have been caught and resolved by basically anyone who attempted to look into the issue with fairly low effort/experience-needed. That is, the problem is that no one bothered to look into it, despite good reason to do so.
> No one looked at this because they didn't know it about.
What are you talking about? Nobody knew about it? Is that a joke?
This long loading issue was posted EVERYWHERE and was the reason so many people gave up on GTA v. It’s simply impossible that they didn’t know the issue existed.
Looking back, I realize it's wrong to assert opinions as facts with no basis at all. I definitely shouldn't have said that when I don't work at Rockstar or know anyone who does.
But I still think people are demonizing too much Rockstar and are trivializing the bug when it's far from obvious. In my opinion, there is a lot of hyperbole statements about this issue when it's more useful to think if your organization could or would catch bugs like this.
Ok, to be fair I am doing to same thing (asserting opinions without facts).
I don't work at Rockstar so it's true I have no idea how this bug was handled.
I just have a hard time believing they didn't know about the bug, and I also have a hard time believing they didn't understand that it was one of the chief complaints from their end users (a simple check of any forum or discussion group is all thats needed).
So, I an only assume short of an actual explanation, that they intentionally sidelined a fix for this bug. Why, is anyone's guess. If you had to ask me, management put all devs onto money making tasks like new items and store support, and fixing quality of life bugs even ones as egregious as this one, was a low priority.
Frankly it might've just gone unnoticed however strange that sounds. Not privy to any details but a probably scenario is probably something like this.
1: Some dev gets tasked with implementing the system, at the time there probably wasn't that many ingame purchases so that 10mb json maybe was a 500k file tops? So going by the O(N2) factor it would've buzzed by in a second (probably faster since being smaller there would've been more cache hits giving another factor or 2). Being at about a second or less they don't even consider optimizing it since it's not noticed among all other assets being loaded.
2: The content/ops teams adds more IAP over the years (it's a datafile so that devs don't have to implement everything), since they're adding big models,etc they don't notice too big changes and probably just chalk it up to bigger models,textures,etc.
3: Engine devs (those most often tasked or obsessive about optimizations) might be running a cut-down server for most testing (not noticing the huge loading time) and probably presume it's other assets if testing in a full env.
So in reality it's probably just an organizational oversight rather than "bad developers", now considering this public "shaming" they received they might actually place some people in an production optimization role for a while and optimize other parts if it's possible (the question is.. will they find it worthwhile to keep up in a longer period?).
meta comment and completely feeling based : sometimes it really seems like companies, politicians, bureaucracies, don't have a process for acknowledging mistakes and/or lack of knowledge (and/or are seemingly not allowed to).
Ah that's because at many companies if you make a mistake you are a mistake.
Which is BS tbh. Everyone makes mistakes.
I think with politicians and political appointees, it's a little different because that's very much a public trust. Nobody deserves to hold political office, they should instead be deserving of it by being ethical and having integrity.
Amen. And avoiding a culture of blame (taking inspiration from the NTSB) makes it much more straightforward to put systemic measures in place which reduce the impact of those mistakes. People still make mistakes at the same rate, but more mistakes get trapped and more negative outcomes can be avoided.
The vitriol of this discussion shows just how difficult it is to establish such a culture. People really want to see mistakes as evidence of moral inferiority.
This is very insightful. If you make a mistake = shameful, you probably aren't going to meaningfully decrease the number of mistakes made, but you will ensure that no one wants to admit they've made one.
And I guarantee anyone who's written code long enough has written plenty of footguns of their own.
I'd say that by talking about mistakes without blame, I think we actually avoid people making mistakes because folks can learn from other's mistakes, and see them as something they could just as easily do, and so internalize the lesson.
> Apologizing just gives fuel for revenge. In this case I think because the harm done was so small, it was economical to apologize.
Yeah, it's always a tradeoff.
I worked at a non-tech fortune 500 for a few years and was able to see the inside of repair/warranty/disputes. These were very high ticket items and the brand was/is very highly regarded for after sales support.
In the employee handbook for repair support staff (the people communicating with clients about the status of their repair/claim) there was a passage similar to the below:
"Apologies MUST NOT acknowledge anything other than the customer's explicit feelings without written acknowledgment of fault from $MANAGER".
I asked a member of that team to decipher WTF that meant, and they showed me an example email. They were only allowed to apologise in the below way:
"Hi bob, we're sorry you feel our product is faulty".
I always found these weasel apologies infuriating in my personal life.
Yep that’s awful. The reason for that rule is probably legal. If your company is taken to court over damages resulting from a faulty product, a written acknowledgment of blame will probably make that worse for the company.
It’s still awful though. I would hesitate to do business with a company that doesn’t take any responsibility for their own products.
Right, so the company is saying "I will never acknowledge our mistakes because if I do we might have to fix them." or "I am trying to trick you into thinking I am apologizing even though I'm not".
That's more offensive and insulting and time-wasting than saying nothing.
More charitably, people have an instinctive tendency to bend over backwards to apologize for things they're not responsible for in the interest of cordial relations. This is good if you're casually trying to make friends, but bad if the other party might take you to court. Companies therefore very reasonably explicitly correct for this behaviour.
You're right about it being a tricky attempt at faking an apology.
But the below isn't correct:
> Right, so the company is saying "I will never acknowledge our mistakes because if I do we might have to fix them."
I probably didn't word it perfectly in my original comment, but the company did acknowledge mistakes quite often after assessments of the claimed faulty/broken/whatever product (roughly 30% of faulty claims were approved and either repaired or reimbursed from memory). The rejected claims were often taken to court (like 15%) due to the value of products.
I was mostly commenting on the deception of only allowing pseudo-apologies.
Well, this HN hate fest of a discussion certainly bears out that saying! The internet is a terrible place. You will get dragged mercilessly no matter what you do. The incentives are all towards sweeping mistakes under the rug.
There are at least some examples of startups choosing to apologize, and try to make things better in the future.
I think the real advice is “don’t say sorry unless you actually mean it.” People see right through that, because your actions and follow through won’t match your words. But people seem to appreciate sincere apologies that come with substantive corrective action.
as-if that's the worst they could do. Suing the guy, pressing for charges with CFAA or DMCA anti-circumvention. There were far worse ways this could have gone.
It is a common misconception that the victim can "press charges" against someone for a criminal matter. This is entirely false.
Only the prosecuting attorney (someone employed by the government) gets to choose when and how someone is charged with a crime. The victim has no say whatsoever in the matter, and indeed the DA can charge someone even when the victim doesn't want them to, or decline to prosecute even when the victim requests it.
I did not say "press charges". I specifically said "press for charges" deliberately. In that context meaning to attempt to persuade someone (in this case a prosecutor/police/etc) to actually press charges.
Most of the time when people simply use "press charges" this is what they mean. I specifically used "press for charges" to try to avoid such pedantry. I guess I failed.
If shorter load times there translate to higher conversions in the same way this connection is observed by Amazon [0] this guy just made Rockstar a little fortune. Hiring the blogger outright and giving them a VP of Something Something Role would be more adequate.
Ya know on second thought, it's so unbelievable that he did this whole thing that I wouldn't be surprised if someone reading his resume thought he was bullshitting.
Gamers playing this game have all kinds of tricks to try to bypass the load screen itself because it takes a long time to go through it but the game by design forces you to look at the load screen multiple times just to do one task. One way that I remember was to play in a Solo session that way you don't have to load every time you come out of multiplayer. They even make a GTA online software tool that blocks GTA on the firewall at very specific intervals to force an online session in order to speed up load times.
There should be such tricks for Codemasters' games specifically and PS1/PSP/Vita games in general. All those intro screens and warnings are absolute bullshit, just publishers spitting in gamers' faces.
It's truly amazing that he did all this while decompiling the source code, but for engineers working on a large system it's not that uncommon to fix large impact performance bugs.
Is the resume where all cool accomplishments go after they've faded from the public eye? Your comment struck me as somehow extremely (and unintentionally) fatalistic... that this cool hack will somehow end up churned through the capitalistic society into a bullet point to earn a wage. Actually kind of depressing.
Millions of people play this game every day. This lone man, driven by his love of solving puzzles, is single handedly unlocking potentially hundreds of millions of hours of “lost time” across the globe.
Reminds me of a Windows installer I working on many years ago, that seemed to take a long time using CPU but otherwise not doing any real work. Turned out the SDK we were using tried to update the progress bar every kilobyte or something like that, and our near-gigabyte sized packages filled the message loop with a bunch of pointless updates (30.001% -> 30.002% but both displayed as 30%).
Once the vendor fixed the issue it shaved a few minutes off the install time that had previously just been a UI glitch (processing the large backlog of progress bar update messages).
> Turned out the SDK we were using tried to update the progress bar every kilobyte or something like that, and our near-gigabyte sized packages filled the message loop with a bunch of pointless updates (30.001% -> 30.002% but both displayed as 30%).
Had a similar situation with an Electron app I worked on. App’s window was getting very stuttery at some point during a long-running operation done by a third-party library, even though it happened in a background worker thread and wasn’t supposed to block the UI (which simply showed the progress of the operation).
Turned out, that third-party library called my progress notifier function (in which I used Electron’s IPC to notify the GUI) on increments that were fairly spaced out most of the time, but really frequent during some operations. Mere overhead from IPC calls and GUI updates added up to a point where everything lagged, even though the operation itself wasn’t that intensive. I wrapped my progress notifier in a throttler, which I frankly should’ve done much earlier, and that solved it.
This must be why when you turn off the progress bar for Invoke-WebResource in PowerShell your download speeds become reasonable, as otherwise it updates the terminal every byte and takes a million years to download stuff that takes a second in Firefox or via wget.
I want to see more of this from giant companies.
This gives me hope that there are real people amongst the mindless drones in these hulking corporations.
I know what you meant, but with those words I hinted, that the decision from Rockstar was likely more a move out of PR motivation, not moral standards, that you assumed.
I wonder how much money the load time slowdown has cost Rockstar -- given their revenue from "Shark Cards", which are bought by players who want the fun of playing with all the expensive in-game items without spending time grinding for them.
What's the correlation between being averse to spending time on less-fun aspects of the game (to the point that you'd pay Rockstar extra money to get only the more-fun time)... and being averse to waiting for long load times, when you just want to play a quick session of more-fun?
Given Rockstar's investment in a steady stream of new big-ticket in-game items, and their huge revenue from that, I wouldn't be surprised if the slow load times cost them many millions.
Red Dead redemption has the same load time issue I bought that game for five bucks. Which was rockstar's cry for help in a dying online game.I tried it out for about 15 minutes realized it suffered the same issue turned it off and never turned it back on
Tell me that is not cool.. being able to say you had that much of an impact on million of players across the globe. Not to mention, bragging rights and cool achievement you can put on your resume ( that anyone should be able to understand ).
What I find extraordinary is that none of the developers who play the game never bothered to fix this problem.
If I was a developer at Rockstar and played this game I would have have spent my own time to fix this problem just so that I didn't need to sit around and wait every time I played the game.
It used to be a practice in the 90s that companies would insert Sleep instructions or other means to delay program execution e.g. to make appearance of heavy computations being done or to make people buy a "pro" version that was without such elements. Given how frankly stupid is the error, could there be some merit to that?
For example to give themselves some ammo for the next versions - if they won't find anything to improve they could pull the rabbit out of the hat and said hey we optimised our code and it loads much faster, buy the new version!
Did you actually read the code. I think if they were going to do that it would be a literal sleep instruction rather than a naievely used way of looping over data
edit: Akchually, the DRY method of doing this would be to make a function like:
/* Remove this when it's time for a performance improvement! */
function Optimize(int N)
/* do stuff N times (so you can limit the amount of performance improvement) */
end
and then put that in the places you want to be able to optimize in the future
I wouldn't be surprised if they found a faster turbotax made people trust it less. You want to see it chugging away to "find your deductions" and "catch your mistakes".
Ironically the hardest thing I needed TT to do this year was handle my wash sales and it didn't even blink as it imported the 1099s. Wait, maybe it didn't handle them correctly... See what I mean? Now I better double check.
[citation... wanted] I mean I know it happened, but beyond one or two famous examples which get repeated on tdwtf, was it really a practice? (As in something done regularly, and not in shareware which had explicit visible delays)
It's the one and only reason my friends and I stopped playing. I hope they take this patching opportunity to resolve a few more load time problems, because even a 70% reduction isn't really enough in my opinion. The load times in GTA are at least 5-10x longer than most games I play.
We used to call it Loading Screen Online. Really was a shame because I made some long-lasting friendships from that one game, many of whom I still play other games with.
Had they bothered to fix this early on as many of us begged them to do from day one of online's release, many of us would have stuck around far longer.
Oh well actually the shark card garbage coupled with ever-increasing item costs, as well as releasing the casino ~5 years late probably would have made me stop anyway.
Imagine a world where companies open-sourced their most boring bits of their software so we, the users, could fix the problems we find with them or extend them in new ways without having to rely on the parent company to get around to it.
"Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?"
Reminds me of the bridge story in Margin Call [1]. Or if you'd rather read it, here's the script [2], just search for "Did you know I built a bridge once?". Or condensed:
"It went from Dilles Bottom, Ohio to Moundsville, West Virginia. It spanned nine hundred and 12 feet above the Ohio River. 12,000 people used this thing a day. And it cut out 35 miles of driving each way between Wheeling and New Martinsville. That's a combined 847,000 miles of driving a day. Or 25,410,000 miles a month. And 304,920,000 miles a year. Saved. Now I completed that project in 1986, that's twenty-two years ago. So over the life of that one bridge, that's 6,708,240,000 miles that haven't had to be driven. At, what, let's say fifty miles an hour. So that's, what, 134,165,800 hours, or 559,020 days. So that one little bridge has saved the people of those communities a combined 1,531 years of their lives not wasted in a fucking car. One thousand five hundred and thirty-one years."
I really was thinking that it was intended... I remember watching GTA V on Xbox 360 and being impressed by how good it looked, how it performed in such a "limited" device, so something like this would resonate in a team so good. But it was almost 6 years ago, so probably a good part of that team has moved on to other companies, and the ones left, maybe are in other projects, hopefully, GTA VI.
Glad to see Rockstar addressing this; I will try the update once it is released, and maybe I can play some online games without getting frustrated.
Off topic: Has Rockstar considered also making improvements to their identity management? Currently you can get the account details of any player from their social club. [1]
I am actually impressed R* followed up with this. I thought it would be sent to the backlog blackhole. I guess the social media impact really pushed it up the chain.
glad to see I am wrong. many GTAO players will definitely be very happy once the new update is live.
> Yeah, that’s gonna take a while… To be fair I had no idea most sscanf implementations called strlen so I can’t blame the developer who wrote this. I would assume it just scanned byte by byte and could stop on a NULL.
Darn, I got the exact same issue ten years ago, with sscanf on Linux/glibc. The sscanf was forcing the huge XML mmaped file region to be entirely read.
The reason behind is that the glibc library was using some "smart" streams (to have generic functions for file/string streams, I guess) that basically all do strlen as first step to buildup this stream structure.
For all the amazing things Rockstar has made, for doing what no other studio could do on the Xbox360, it's crazy that they haven't fixed this before now.
World size? Lots of open world games are as big or bigger.
Graphics? I thought this too, but then I had another look (been a few years since I played it) and its quite dated looking now. Not bad, but definitely not on the same level as more recent games. Understandable though, it came out an entire console generation ago!
Even gameplay-wise, yeah, it has much better driving mechanics than most non-racing games, but shooting and such is fine, but nothing special.
Its a great game and was absolutely fantastic for when it came out. I'm not at all surprised that its as popular as it is! But "most games still don't come close" just isn't true, at least for large AAA titles.
On each individual point, you're right. But I'm curious which titles you think do better than GTA V on all of those points? Red Dead Redemption 2 and The Witcher 3 are the only two that come to mind.
Ghost of Tsushima and Horizon: Zero Dawn immediately springs to mind. Both are huge, graphically fantastic, mechanically great. Days Gone, too, was an incredibly good game, large, pretty and had fantastic gameplay. Not quite the same replayability as GTA though.
Does it have to beat it on all points for it to be valid? I mean, since the games aren't trying to be GTA, they're not going to have the exact same priorities.
What about large, beautiful, but not open world games? God of War 2018 and Dark Souls 3 meet that criteria for me.
Or open world games that aren't as large as GTA but hit all the other points? Spider-man 2018 is a great example.
Or maybe large, graphically gorgeous, but the driving is terrible? Cyberpunk 2077. (Buggy too, but I assume that will eventually be fixed)
Those are just some games that spring to mind. I'm sure I can think of some more, especially on PC (the games I listed are all ones I played on PS4 -- so I'm comparing GTA V on PS4 against these PS4 games, perhaps GTA V on PC could be modded to visually beat these games)
GTA 5 on the 360 runs at 720p@30 FPS and even then it drops to the low 20s any time something happens even considering the graphics are cranked extremely low by today's standards.
How well they had it running on the 360 is absolutely amazing given the limitations but nostalgia is taking over if you think most games on 4th gen Xbox don't even come close.
Because it's a game that has been in developpment for 8years, the game as of now is very different from 8 years ago from when it released. You can't compare a game just released with a game that had that much iteration with thousand of people.
I've found that when writing something like this which iterates over and over many millions of times a helpful test can be to place a simple counter in your loop. If you have an algorithm which you expect to loop only one million times and it loops ten million times, that is something you want to investigate.
None of the programmers at rockstar who had familliarity with the codebase and the platform ever thought "There's no reason these load times should be so long, I'm going to look into this"
Good job on the person who found they could reduce GTA Online's abysmal loading times; it shows how greatly they could improve the game.
Keep in mind Rockstar has almost 0 interactivity with its fanbase and doesn't listen to any feedback. So the possibility of negative attention forced them to do something about it.
Yup. I'm not deeply familiar with the situation and circumstances but it sounds like this person should be very well compensated for what they did.
The amount of free skilled labor businesses benefit from and abuse while not donating back to society is laughable. It's great they helped the GTA 5 community and a fun read but keep your pride and fame, I'd take cash anyday over a pat on the back.
I don't think it was simple. It came down to the odd specifics of scanf making the algorithm O(n^2) which wasn't obvious at all from the programmer standpoint. A lot of people pan Rockstar for being incompetent but in reality, it's a C footgun that any programmer would make.
Edit: I reread the article and I overlooked how there was a second issue of a poorly implemented "hash table" that was a linked list that checked the entire list for dups in every update. Since we don't have access to the source code, it's hard to say if this is poorly implemented piece of code on Rockstar's part or someone else. It could be some deeply buried piece of code that no one ever looked at for years until now.
On the gripping hand, if someone had run a profiler on the code in all of these years, it would have popped out immediately that some investigation was warranted.
It seems like the original GTAV is a efficient and well optimized game. I played it and it looks amazing for how fast it runs. They certainly ran a lot of profilers to nail the performance down. It's amazing what video games can do, look at the Doom series.
But this is online GTA. I can imagine that they handed it off to another department that didn't have the same amount of discipline and expertise. it's probably an off shoot team that creates content for the engine but don't necessarily need to dig down into how the engine works.
Maybe not fixing it was on purpose, but I doubt the original bug was. They could've just added a more straightforward sleep/delay in the code. If anyone finds it they can say it was accidentally leftover testing code because they needed a delay to see ads in the office to make sure it works.
Yep; we tried to play with some friends over the pandemic, but rather quickly we nicknamed it "Loading Online" and preferred to do anything else where you, well, get to do stuff instead of waiting for loading. Perhaps it was better back when GTA Online was just starting, but last year (whan I tried this) the loading times on good PC hardware were just outrageous compared to any other game.
Rockstar awarded them 10k for it.