Is it the the size of the buffer in samples, or in milliseconds that matters?

In my experience, samples, although I don't know exactly why. But it's why you can get lower latency glitch-free audio with higher sample rates.