I like the 3D datasets in these papers since, like a scientist in a lab, you can setup the experiment and explore the domain. Adding time in would be cool too (e.g. Are the blue and red ball going to collide in 10 seconds?)

It also helps to be able to show you can answer some of these questions in principle with your model. It gives you hope that it might be able to cover real world images.