The article touches on basic testing strategies that apply to general software as well as compilers. I have been involved with compiler testing projects on production LLVM and GCC in my past life. One thing that makes compiler testing specifically more difficult than general software is the Oracle Problem: how do you verify the output is in fact correct? Crashes are relatively easy to find by random fuzzing, but in the general case, proving that the output program is not miscompiled is non-trivial.
There are a couple effective techniques in the literature that might be useful here:
- Differential testing[1]: generate a bunch of random, correct, deterministic programs; run them under different compilers or under different compilation flags and check if the output of the program is identical
- Equivalence Modulo Inputs[2]: a class of techniques that can be used transform a program to a distinct program that is supposed to be equivalent to the original for a specific input. (shameless plug)
What about using multiple open source projects with already existing tests? You check if results of execution are the same between compiler versions? This should also provide you with a better coverage of various features
Sure that counts as differential testing. The issue is there are limited number of them compared to the amount of code you can generate mechanically, and especially in the case of C, it is not straightforward to write a standardized script to build and run a bunch of random projects.
There are a couple effective techniques in the literature that might be useful here:
- Differential testing[1]: generate a bunch of random, correct, deterministic programs; run them under different compilers or under different compilation flags and check if the output of the program is identical
- Equivalence Modulo Inputs[2]: a class of techniques that can be used transform a program to a distinct program that is supposed to be equivalent to the original for a specific input. (shameless plug)
[1]: https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf
[2]: https://web.cs.ucdavis.edu/~su/publications/emi.pdf