Has anybody experience with testing code that produces graphics (e.g. 3D engines, etc.)? I saw some articles stating that mocking the API is a good approach. But how can you test shaders etc. Tests based on image comparisons seem very cumbersome. We currently rely heavily on our QA which does automated integration tests based on image comparisons. But there is no immediate feedback for developers with this approach.
I'm working on a project where I'm doing boolean ops on 3D meshes that need a lot of tests.
I do a mixture of traditional unit tests w/asserts along with visual feedback. I have a rig set up that will dump final (or intermediate) results of operations to the screen with several tests being presented at the same time.
If I'm actively developing something and am going to be spending a lot of time in the debugger I can solo a test. Having the additional visual feedback makes everything go a lot faster.
For higher level stuff having a number of tests on screen at once gives visual feedback about regressions, which again speeds things up a lot.
This combined approach is the most useful I've found so far.
I have problems with this as well. Sometimes there are obvious things that can be tested on the resulting image. For example when applying a blur on a grey image with one white pixel in the middle, I know how I expect the sum of the pixel values to change, and I know the upper and lower bounds of valid pixel values.
After the fact unit tests that verify that abs(expectedImage - resultingImage) < e are also helpful. But unless you write some modeling application that has to produce exact results, a lot of code is typically about what looks good, not about what's mathematically correct. Those parts are probably impossible to properly test automatically.
Whatever computations you are making should be composed in such a way that distinct, reused computation types are implemented in units, perhaps conceptually one type of transformation per function. Each of these functions should be tested by a collection of unit tests which verify that it performs the correct computation on hard-coded, mock data, both good and bad, and that it fails when it should.
In cases where multiple transformations are applied together and there is more complex behavior, the application should be designed so that those combinations of "unit" behaviors are done for a specific purpose, with a specific goal (e.g. warp this image and then project its shadow); you should have an integration test suite that verifies this more complex behavior performs the function it is supposed to. Here, you can probably be less strict about checking inputs since bad inputs should be covered by unit tests, but you should verify that there is proper error handling of multiple units failing.
These should all run using mock data as part of your CI process, and no code should ever be merged that has failing checks; new code should not be merged without code review verifying that testing of comprehensive.
Finally, there should be a QA process that verifies the final product when being considered for release, using real data and manual validation, ideally with acceptance criteria in hand that was written before the code changes were begun.
In the end, it shouldn't even really come down to trying to mock your APIs; your abstractions should, if at all possible, be decoupled enough that they perform transformations of data that they are given, regardless of whether it came from another API or not - so instead of considering "mocking the APIs" you call the relevant functions with example data of the sort you receive from the APIs that will be actually used.
