At EuroBSDcon 2022, Allan Jude gave the presentation "Scaling ZFS for NVMe":
> Learn about how ZFS is being adapted to the ways the rules of storage are being changed by NVMe. In the past, storage was slow relative to the CPU so requests were preprocessed, sorted, and coalesced to improve performance. Modern NVMe is so low latency that we must avoid as much of this preprocessing as possible to maintain the performance this new storage paradigm has to offer.
> An overview of the work Klara has done to improve performance of multiple ZFS pools of large numbers of NVMe disks on high thread count machines.
[…]
> A walkthrough of how we improved performance from 3 GB/sec to over 7 GB/sec of writes.
> Learn about how ZFS is being adapted to the ways the rules of storage are being changed by NVMe. In the past, storage was slow relative to the CPU so requests were preprocessed, sorted, and coalesced to improve performance. Modern NVMe is so low latency that we must avoid as much of this preprocessing as possible to maintain the performance this new storage paradigm has to offer.
> An overview of the work Klara has done to improve performance of multiple ZFS pools of large numbers of NVMe disks on high thread count machines.
[…]
> A walkthrough of how we improved performance from 3 GB/sec to over 7 GB/sec of writes.
* https://www.youtube.com/watch?v=v8sl8gj9UnA
When ZFS was created spinning rust was still the main thing, and SSDs were gaining popularity, so ZFS created "hybrid storage" pools:
* https://cacm.acm.org/magazines/2008/7/5377-flash-storage-mem...
* https://web.archive.org/web/20080613124922/http://blogs.sun....
* https://web.archive.org/web/20080615042818/http://blogs.sun....
* https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-...
* https://en.wikipedia.org/wiki/Hybrid_array
Still useful for use cases that lean towards bulk storage (versus IOps).