I think your nervousness is well founded, and that's why Microsoft won't implement WebGL [1], but Chrome's developers seem to think they can make things secure [2].
I don't think nervousness about a particular implementation approach is a reason not to implement WebGL, just a reason not to do it in what I gather is Chrome's way.
On the other hand, Chrome's strategy might just be getting explained poorly. My nervousness comes partly because the way it's described sounds like a filtering strategy. If it's more like a whitelisting strategy, it could be considerably less scary, and more in line with what I was kind of expecting from browser vendors.
[1]: http://blogs.technet.com/b/srd/archive/2011/06/16/webgl-cons...
[2] : http://games.greggman.com/game/webgl-security-and-microsoft-...