You use SfM to find the first point cloud. However SfM is based on the hypothesis that the same point 'moves' linearly in between any two views. This hypothesis is important because it allows you to match a point in two pictures, and given the distance between the two images, you can triangulate the point in space. Therefore find it's depth.

However, non-Lambertian points move non linearly in viewing space (eg a specular point depends on the viewer pose).

So, automatically, their positions in space will be false, and you'll have floating points.

Gaussian 'splats' may have the potential to render non-Lambertian stuff using for example the spherical harmonics (even if I don't think the viewer use them if I'm not mistaken). But, capturing non-Lambertian points is very difficult and an open research problem.