You need to flush the addresses out of the cache in order for the branch predictor to speculatively execute and load the address back into the cache. This is where things get very tricky, because lets say you have some other process that accesses that address on a regular basis - it will get reloaded into the cache so your timing attacks have a lower chance of success.

So overall, putting together an exploit with this through JS becomes a matter of lots and lots of research and testing, for a specific target - i.e not worth the effort for anyone but a state sponsored agency.