With the gradual accumulation of parallel projects, I've recently adopted a constantly-jump-around approach to development to keep everything braincached. About four projects in flight that I can talk about at the moment:
1) Neural approximation of surface light scattering (learned BSDF approximations, in other words). Historically, real time BSDFs used in games have been statistical models that make quite a few simplifying assumptions. GGX and other modern BRDFs do an admirable job and get about 90% of the way there, but creating new surface models is very time consuming. You need a mostly-unique model for retroreflective cloth, another for faraway swaying grass, another for plastics, another for skin, and so on. It's doable (and much has been done), but the process is far from easy.
So, an attempt at a workaround: trace trillions of statistically modeled paths and use the resulting distributions to train a small network that can be baked into an efficient ALU-only shader representation. Results are promising so far: https://twitter.com/RossNordby/status/1241224996518838272
Added bonus: as a part of this process, you get a PDF/importance sampling network for ~free.
2) ML applied to physically simulated character animation. Lots of neat work in this field already- the recent motion matching -> RL balancing paper seems like a very promising path for reducing the size of RL's responsibility: https://montreal.ubisoft.com/en/drecon-data-driven-responsiv...
My own work is still in 'side project' territory and I haven't exactly caught up to SOTA (https://www.youtube.com/watch?v=Haz9o3lbJJQ), but there are a bunch of things I'm looking forward to trying. Different heuristics (muscular control delay, effort minimization, etc.), some variants of off policy learning, different exploration approaches, some low level architecture tweaks, and so on.
3) Gradually resurrecting and modernizing a bitrotted renderer from a few years ago (https://www.youtube.com/watch?v=mIax_ProQ8c) and expanding its functionality. Beyond just fixing the massive problems it has accumulated, I'd like to flesh out a full surface space rendering pipeline. All shading would be computed on the surfaces of objects (and, where appropriate, in volumes) rather than in screenspace, so temporal reprojection is made far easier and there's no geometric blurring. Final rasterization is extremely cheap (~1 texture sample read) and can be decoupled from shading, so with 144hz+ geometric rendering in combination with late latched camera input, you can get application contribution to display latency below 10ms pretty easily.
The surface space caches and decoupling also give you a path towards tons of other interesting approaches- surface space shadow maps, low resolution directional occlusion and global illumination caches, conetraceable prefiltered surface representations, and all sorts of cool stuff.
4) Continuing to improve the physics engine (https://github.com/bepu/bepuphysics2). It exited beta last year, but there is always more to do. Next target is the tree structure used by meshes and the broad phase. While it is more than an order of magnitude faster than v1's, it should be possible to extract more parallelism, eliminate the need for a separate cache optimizer step, and cut down the aggressiveness of the static tree refinement. Simulations with hundreds of thousands of sleeping or static objects could see some significant gains, among other things.
Was also considering training a network to compress in-memory solver data since, on manycore AVX2 capable systems, the solver is always memory bandwidth bound, but down that path may lie madness... maybe not everything needs to have DL thrown at it. Maybe.
1) Neural approximation of surface light scattering (learned BSDF approximations, in other words). Historically, real time BSDFs used in games have been statistical models that make quite a few simplifying assumptions. GGX and other modern BRDFs do an admirable job and get about 90% of the way there, but creating new surface models is very time consuming. You need a mostly-unique model for retroreflective cloth, another for faraway swaying grass, another for plastics, another for skin, and so on. It's doable (and much has been done), but the process is far from easy.
So, an attempt at a workaround: trace trillions of statistically modeled paths and use the resulting distributions to train a small network that can be baked into an efficient ALU-only shader representation. Results are promising so far: https://twitter.com/RossNordby/status/1241224996518838272
Added bonus: as a part of this process, you get a PDF/importance sampling network for ~free.
2) ML applied to physically simulated character animation. Lots of neat work in this field already- the recent motion matching -> RL balancing paper seems like a very promising path for reducing the size of RL's responsibility: https://montreal.ubisoft.com/en/drecon-data-driven-responsiv...
My own work is still in 'side project' territory and I haven't exactly caught up to SOTA (https://www.youtube.com/watch?v=Haz9o3lbJJQ), but there are a bunch of things I'm looking forward to trying. Different heuristics (muscular control delay, effort minimization, etc.), some variants of off policy learning, different exploration approaches, some low level architecture tweaks, and so on.
3) Gradually resurrecting and modernizing a bitrotted renderer from a few years ago (https://www.youtube.com/watch?v=mIax_ProQ8c) and expanding its functionality. Beyond just fixing the massive problems it has accumulated, I'd like to flesh out a full surface space rendering pipeline. All shading would be computed on the surfaces of objects (and, where appropriate, in volumes) rather than in screenspace, so temporal reprojection is made far easier and there's no geometric blurring. Final rasterization is extremely cheap (~1 texture sample read) and can be decoupled from shading, so with 144hz+ geometric rendering in combination with late latched camera input, you can get application contribution to display latency below 10ms pretty easily.
The surface space caches and decoupling also give you a path towards tons of other interesting approaches- surface space shadow maps, low resolution directional occlusion and global illumination caches, conetraceable prefiltered surface representations, and all sorts of cool stuff.
4) Continuing to improve the physics engine (https://github.com/bepu/bepuphysics2). It exited beta last year, but there is always more to do. Next target is the tree structure used by meshes and the broad phase. While it is more than an order of magnitude faster than v1's, it should be possible to extract more parallelism, eliminate the need for a separate cache optimizer step, and cut down the aggressiveness of the static tree refinement. Simulations with hundreds of thousands of sleeping or static objects could see some significant gains, among other things.
Was also considering training a network to compress in-memory solver data since, on manycore AVX2 capable systems, the solver is always memory bandwidth bound, but down that path may lie madness... maybe not everything needs to have DL thrown at it. Maybe.