do you use any tool for this, or simply crunk up slightly the dial each day

There are multi armed bandit algorithms for this. I don’t know the names of the public tools.

This is especially useful for something where the value of the choice is front loaded, like headlines.

We've used this Python package to do this: https://github.com/bayesianbandits/bayesianbandits