> which should roughly cancel out the error over time.

To be pedantic, if the errors of size e are as likely to go one way as the other for each of N steps, the expected magnitude of the total error will be sqrt(N)e. It's a random walk.