Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The state of debugging for graphics code is deplorable. When you run code on a GPU you almost always have a gigantic closed source binary blob underneath you, the graphics driver. Graphics drivers contain complete compilers that munge your code with a focus on speed over correctness. The graphics driver is often years out of date, and you can't even run it yourself to reproduce issues unless you go out and buy the exact hardware that each user has.

The state of the art in debugging GPU issues is to have a closet full of random hardware purchased used on eBay that hopefully contains something close enough to what you need to reproduce the issue you're working on.

Most fancy debugging tools pretend that GPUs don't exist and GPU-focused debuggers are incredibly buggy and feature-poor, not to mention hardware specific in many cases.




It’s not that bad, at least on Windows.

> contain complete compilers that munge your code with a focus on speed over correctness

If you’re a developer, the right tools are available. Set D3D11_CREATE_DEVICE_DEBUG when creating the device, also run debug builds of your HLSL shaders. You’ll get huge amount of error checking and diagnostics this way, including very detailed and specific messages about incorrect API use.

> GPU-focused debuggers are incredibly buggy and feature-poor

I indeed found and reported bugs in GPUs and their drivers (mostly Intel GPUs), but it only was a few times over my career. The majority of times they were bugs in my code, not in GPU drivers, and renderdoc.org helped a lot fixing them.

I generally agree GPU debugging is harder than CPU debugging, but I don’t see how open source driver gonna help. It’s an unfortunate combination of execution model (many thousands hardware threads running your code in parallel), external piece of hardware completely separate and asynchronous from the CPU, and very limited ways to interact or even output data (you can’t printf messages, nor allocate dynamic memory, from GPU code).


> The majority of times they were bugs in my code, not in GPU drivers

the Servo/WebRender folks maintain such a list of driver bugs with workarounds, you may find useful:

https://github.com/servo/webrender/wiki/Driver-issues


Vast majority of them aren't for Windows.

The majority of their Windows bugs aren't GPU driver bugs, they are bugs of compatibility layers like Angle and SPIRV-Cross.

IMO when using native stack i.e. D3D11 or 12 and HLSL, the quality of GPU drivers is usually OK on modern Windows.


> It’s not that bad, at least on Windows.

Funny you say that. I've seen many developers praising the open source stack under Linux, as you can step right into the driver and fix it yourself, instead of landing in symbols land.

Feral Interactive mentioned it there, for instance: https://www.gamingonlinux.com/articles/interview-with-feral-...


Note that Feral is comparing it with their macOS experience, not Windows (they do not develop for Windows).


They do not develop on Windows, but presumably they also build and trace on Windows to debug ports. Source: I have worked on multiplatform desktop software.


On the bright side the situation is at least better on Windows - D3D11 and D3D9 both have well-specified behavior and full software rasterizers that can be used to render a reference version of a given frame and then debug it at a pixel level. Wish I had something like that for OpenGL... console game developers have had toys like this for ages.

It's 2020 and I still sometimes swap my renderer from OpenGL to D3D9 when I need to debug a graphics issue because the D3D9 graphics debugging tools are that much better than cutting-edge GL debuggers. (I'd use the D3D11 debugging tools but my 11 backend doesn't work right)


I've never understood the logic behind closed-source drivers. Nobody pays for a driver, people pay for hardware. If you want your hardware to get better support, and therefore become more popular, aren't you incentivized to open-source the drivers?


> Nobody pays for a driver

This is not true. Driver quality is a major differentiating factor between GPUs and absolutely influences purchasing decisions.

GPU makers don't want their competitors to be able to take parts of their drivers and benefit from them. They want to protect trade secrets. They want to avoid revealing how many of their competitors' patents they are violating. They don't own the rights to some third party code or trade secrets that are incorporated in their drivers. They want to avoid revealing future product plans.

Of course it's debatable whether closed source drivers actually help with all of the above reasons, or whether they are good reasons to begin with. But those are the reasons.


AMD Radeon has made multiple of their performance-enhancing projects of theirs open source only to have nVidia copy and implement their work without reciprocating the favor.


Is it mandatory to reciprocate? If not, then that's a problem of their own making, isn't it?


It's not mandatory, but it sure isn't polite, especially when projects like HairWorks are killing AMD because it's a driver handling issue more than a hardware one.


> aren't you incentivized to open-source the drivers?

No, Not at all.

If we go all the way back to 3Dfx era, we have seen dozens of GPU ( or 2D/3D Graphics Accelerator before the term GPU was coined ) trying to compete in the PC space, 3D Labs, Matrox, PowerVR, Intel I740, S3, ATI, Nvidia, Rendition, ( These are on top of my head, sorry if I missed any of your favourite ). None of them had much problem with designing hardware, their maximum triangles, ( If I remember correctly it was one of the unit used to market graphics card before GFLOPS were used ) were all on par with each other. And some even exceed the current dominant players, but in the end they all failed.

Why? Because they cant get the drivers / software to optimise the use of their Hardware. They do not perform any where close to their potential in Tomb Raider or DOOM. There are huge amount of optimisation built up over the years. As Nvidia once said ( That was even before their huge bet on CUDA ), 60% of their engineers are Software.

To point where the I am going to say, the Driver IS the GPU. You are just paying the hardware to use it.

* And now I feel very old just typing this up. :/


Don't feel old, I still remember doing Blitter stuff on Amiga, or how sad I was that the newly bought 3Dfx card wasn't compatible with my PCI version and had to exchange it for a Nvidia, hardly did I knew.


Graphics drivers contain competitive optimizations that make a significant difference in the performance of the hardware. Since these optimizations are both costly (in R&D) and allow for the vendors GPU to perform better than a competitors, they do not share these optimizations (that a competitor may or may not use to improve their own driver). And since people pay ultimately for performance, not hardware, the driver is just as important as the silicon.

Your second point is correct, that open source drivers do get better support. AMD has better support on Linux than NVIDIA because it has an officially supported open source driver (AMDGPU). However, the support they receive from open-sourcing the drivers is not always the R&D required to gain a competitive performance advantage, limiting the competitiveness of an open source driver (at least compared to proprietary).


They do sketch things, like keep a database of third party shaders that warranted hand optimization by their driver team, and their replacement shaders. I'd be surprised if they could open source such a thing. It's an IP nightmare.


Why would that be sketchy? I'm sure there's lots of bad things in there, but hand-optimizing common third-party code paths doesn't seem like it should be one of them.

Would there be some sort of uproar if people found out that Firefox had special optimizations for common websites, for instance?


If I found out my competitors made sweetheart deals with hardware manufacturers so that their software worked better on their hardware than my software does, I'd be pissed.

> Would there be some sort of uproar if people found out that Firefox had special optimizations for common websites, for instance?

It would ruffle a few feathers if Chrome was optimized for Google properties, but ran their competitors' code with a performance penalty.

One of the issues in United States v Microsoft was that IE used optimized APIs that Microsoft's competitors, like Netscape, didn't have access to.


> It would ruffle a few feathers if Chrome was optimized for Google properties, but ran their competitors' code with a performance penalty.

It is: https://9to5google.com/2018/07/25/youtube-chrome-firefox-edg...


Thanks for pointing this out. I remember that the EU were conducting antitrust investigations into Google, do any of them pertain to this?


> If I found out my competitors made sweetheart deals with hardware manufacturers so that their software worked better on their hardware than my software does, I'd be pissed.

From 2002: https://www.nvidia.com/object/nv_gdc2k2_02.html


>If I found out my competitors made sweetheart deals with hardware manufacturers so that their software worked better on their hardware than my software does, I'd be pissed.

As long as they aren't actively sabotaging your software, I don't see why this is a problem. Adobe is clearly the market leader in photo editing. If adobe pays nvidia to improve performance of photoshop, why is that an issue? They aren't paying nvidia to make gimp worse.

Also "sweetheart deal" is a loaded term. What is your definition of sweetheart deal here?



All of the examples in there involve the government. What does a "sweetheart deal" between two private companies mean?


> A sweetheart deal or sweetheart contract is a contractual agreement, usually worked out in secret, that greatly benefits some of the parties while inappropriately disadvantaging other parties or the public at large.

> A "sweetheart settlement" may also occur in a legal context. For example, in a class-action lawsuit the attorneys representing a class of plaintiffs may reach an agreement with the defendant in which the primary result is a lucrative fee for the attorneys rather than maximum compensation for the class members.

> Noted instances and allegations

> In a 1947 unionization dispute, San Francisco area grocery store owners claimed that other stores who had "given-in" to union demands had signed sweetheart deals with the unions.

> In a 1949 dispute between the rival American Federation of Labor (AFL) and the Congress of Industrial Organizations (CIO) involving unionization of the laundry industry in Indianapolis, Indiana, a lawyer for the 42 laundry and dry-cleaning plants testified before the National Labor Relations Board that an AFL union contract was not a sweetheart deal between the employers and AFL union officials, as alleged by the CIO.


> Would there be some sort of uproar if people found out that Firefox had special optimizations for common websites, for instance?

There was a comment on this site a few days ago that claimed Google was being anti-competitive by hard-coding a list of their web properties into Chrome for it to send diagnostic info when visiting them, so yes, anything where there's a special path for some 1st/3rd parties but not others could be construed in many ways that look bad for the author.


Because they use partial verbatim copies of other people's shader code to make that happen.


There is a disincentive: patents/IP as one thing (competitors could deduce how your stuff works from oss drivers), and the other thing is exclusivity with games: especially nvidia has deals with game developers to tweak their games in drivers (!) - things like hand optimized shaders or other rendering settings, that provide superior quality for the gamer, so that gamers prefer to buy nvidia cards over AMD.

With open source drivers, AMD or Intel could just walk in and steal whatever optimizations nvidia has made for a game.


Everything that is patented is already fully disclosed to the public. It's a requirement of getting a patent. Keeping drivers closed-source is more likely to be due to the presence of trade secrets or that the code includes licensed bits that the manufacturer doesn't have the right to disclose the source code to.


With patents you're afaik only required to describe the process itself - but not the implementation in code, and especially not optimizations fitted to your architecture.


From the USPTO:

> The specification must include a written description of the invention and of the manner and process of making and using it, and is required to be in such full, clear, concise, and exact terms as to enable any person skilled in the technological area to which the invention pertains, or with which it is most nearly connected, to make and use the same.

You don't have to provide code, but you have to teach the reader of your patent how to make and use the invention.


In practice, patents are written in specialized legalese that is so far away from actual practice that it's difficult to translate. Additionally, due to treble damages for willful infringement, standard industry practice forbids engineers from reading any patents whatsoever.


> standard industry practice forbids engineers from reading any patents whatsoever.

It does? Exactly zero of the companies I've worked for in my career have forbidden me from reading patents.


The thing is, if you've read a patent, then you (and your employer) now "know of" the patent. If you're later found to infringe the patent, the patent owner can ask for triple damages, since it was wilful infringement. But if you never read it, they can only ask for damages.

Given how many patents are vaguely worded, and how, um, interesting the decisions in patent cases sometimes come out, many companies decide that it isn't worth the risk.

Note well: IANAL. I've just been in companies that had this practice (don't read patents), and I remember why.


Yes, I understand the logic of it, and I have heard of companies that include this requirement. I just doubt that this is anything like a standard industry practice. I've been in the industry for a very long time, and have never once encountered such a restriction, even at very large corporations. I would think if this were close to being a standard practice, I would have encountered it at least once.


> Nobody pays for a driver, people pay for hardware.

You'd think that, but understand that Nvidia's primary advantage is actually in the drivers. In the graphics industry it's known that AMD GPUs often perform better at various tasks as they become highly optimized. However Nvidia GPUs most often benchmark better. As it turns out Nvidia drivers have an enormous set of optimizations it can enable based on heuristics which end up being the key to making up most of the difference.


Companies think open-source drivers will reveal "special sauce" implementation details of their graphics hardware to their competitors.


No, they think it will prove their use of bog-standard techniques to people who laid patent minefields, instantly fueling a hundred lawsuits.

The USPTO is terrible at performing prior art searches and terrible at rejecting obvious claims, so there are legions of trolls waiting in the wings to jump on them if they go open source.


One possible factor: graphics drivers are full of app-specific hacks to make benchmarks look good, on the level of "if game == GTAV then run super special fast code instead of whatever the game is telling us to do"


they definitely do that, but they don't keep it secret anymore. they even advertise it as "game ready drivers" from nvidia.


> aren't you incentivized to open-source the drivers?

Opening the source to drivers as they exist now might expose IP that might not be yours to share, IP that you aren't licensed to use or open you up to liability because of how bad your drivers actually are.

That said, I do believe hardware companies should prioritize open-source drivers to benefit themselves and their customers.


I think the incentive to keep drivers closed is not huge, but the incentive to open source them is effectively nil, especially from the point of view of "management types".

It may on may not end up improving support, but you can't really know that ahead of time. Then you have the risk of people using your code against you, finding vulnerabilities, patent/IP violations or simply reverse-engineering your hardware more easily. You also won't be able to cheat in benchmarks anymore.

There's a cultural aspect to it too. Paying all these engineers to write that code was expensive, and now you want to give it all away for free? Most companies never do that for anything, why would they make an exception?

It's dumb, I too would love for these drivers to become open source, but it doesn't look like it will be happening any time soon.


> Paying all these engineers to write that code was expensive, and now you want to give it all away for free? Most companies never do that for anything, why would they make an exception?

I'm not saying it doesn't exist, but I've been doing professional software development for more than twenty years and have never encountered this. There's plenty of "we should be paid for our work and innovation" but nothing so reflexive as what you describe here.


I can understand that AMD and Nvidia have spent a ton of time and money developing their drivers and that ML/GPGPU performance and features are highly competitive.

But embedded GPUs are so fucking obnoxious. There’s about a million of them and they pretty much all have proprietary binary blobs. Pretty worrying for something that could potentially allow any device to be remotely compromised via some malicious webGL or something. Good luck getting the manufacturer to continue supporting it 3 years from now though


It'd expose trade secrets and give away the details basically giving away any competitive edge.


I guess, it makes it harder to copy the hardware design, which is often off the shelf parts. Once you have the finished design it's relative cheap to mass produce.


As someone who debugs graphics drivers for a living, I assure you it's the same from the other side. Except the gigantic closed source binary blob is some AAA game, benchmark, or a popular game engine demo. Also often the developers are using the API against the specification, but one of the GPU vendors is more lenient than the other so it works for them and they are already shipping the game, too late to fix.




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: