Perhaps, you can start by learning what WASM is:

https://developer.mozilla.org/en-US/docs/WebAssembly

As for vulnerabilities, here are nccgroup's slides about WASM explots:

https://i.blackhat.com/us-18/Thu-August-9/us-18-Lukasiewicz-...

Here's an example vulnerability in WASM parsing leading to RCE:

https://labs.f-secure.com/assets/BlogFiles/apple-safari-wasm...