I work as a software developer in a hardware company. We have A LOT of tests.
But testing costs a lot more when there is hardware present. It's a whole new dimension in your testing matrix. You can't run on cloud provider CI. Hardware has failures, someone needs to be there to reboot the systems and swap out broken ones (and thing break when they run hot 24/7). And you need some way to test whether the hardware did what it is supposed to do.
Although some kernel driver developers have testing clusters with real hardware (like GPU drivers devs), there is no effin way Linux could be effectively tested without someone paying a lot of money to set up real hardware and people to keep it running.
Of course the hardware can be simulated but simulation and emulation are slow (and potentially expensive). At work we run tests on simulation for next gen hardware which doesn't exist yet. It is about 100,000x slower than the real deal, so it's obviously not a solution that scales up.
Testing purely software products without a hardware dimension is so much simpler and cheaper.
I think people have forgotten or indeed never learned in the first place that testing terminology borrows from physical device testing culture. A “fixture” for instance is a mock environment that you place a piece of hardware into to verify it.
Hardware testing takes up a pretty sizable portion of the development cycle. Why do we think we are special?
kind of this -- I can't imagine any hardware company moving forward without fixture test flows of all sorts and a lot of effort put into them. It really seems like an area where software tries to stand on the edge of "art" forgetting that there is a science and craft needed when teams revolt against testing.
>
Although some kernel driver developers have testing clusters with real hardware (like GPU drivers devs), there is no effin way Linux could be effectively tested without someone paying a lot of money to set up real hardware and people to keep it running.
Just crowdsource this testing: I am sure that there exist some people who own the piece of hardware and are willing to run a test script, say, every
* night
* week
* new testing version of the kernel
(depending on your level of passion for the hardware and/or Linux). I do believe that there do exist a lot of people who would join such a crowdsourcing effort if the necessary infrastructure existed (i.e. it is very easy to run and submit the results).
I think you are vastly overestimating the willingness and capability of volunteers and underestimating the effort needed to coordinate such an effort.
And having any kind of manual intervention required will almost certainly reduce the reliability of the testing.
This is further complicated by the need to reboot with a different kernel image. Qemu and virtual machines can't do all kinds of hw testing needed.
And in fact, the kernel is already tested like this. Just very irregularly and sporadically. The end users will do the field testing and it is surprisingly effective in finding bugs.
> I think you are vastly overestimating the willingness and capability of volunteers and underestimating the effort needed to coordinate such an effort.
> And having any kind of manual intervention required will almost certainly reduce the reliability of the testing.
Perhaps I am underestimating the necessary effort, but the willingness and capability problem can in my opinion be solved by sufficiently streamlining and documenting the processes of running the test procedure.
If the testing procedure cannot be successfully run by a "somewhat experienced Linux nerd", this should be considered a usability bug of the testing procedure (and thus be fixed).
The NixOS community would be perfect for this, since "nothing" (i wouldn't wanna do experimental filesystems) can break my system, I just atomicly roll back to my previous generation and report it broken :)
But testing costs a lot more when there is hardware present. It's a whole new dimension in your testing matrix. You can't run on cloud provider CI. Hardware has failures, someone needs to be there to reboot the systems and swap out broken ones (and thing break when they run hot 24/7). And you need some way to test whether the hardware did what it is supposed to do.
Although some kernel driver developers have testing clusters with real hardware (like GPU drivers devs), there is no effin way Linux could be effectively tested without someone paying a lot of money to set up real hardware and people to keep it running.
Of course the hardware can be simulated but simulation and emulation are slow (and potentially expensive). At work we run tests on simulation for next gen hardware which doesn't exist yet. It is about 100,000x slower than the real deal, so it's obviously not a solution that scales up.
Testing purely software products without a hardware dimension is so much simpler and cheaper.