For a user of which you have voted on many posts, we can define "interesting" directly. If we use I(you, u) to denote our proxy of the chance that you find user u interesting, we define I(you, u) = (number of upvotes to this user) / (number of total votes on all users).

For any user u on which you have not cast a lot of votes, calculate I(you, u) as the sum of I(you, u') I(u', u) over all users u' (other than you or u), plus the above formula.

There are some issues with this: it'll take some work to make it perform, and it strongly favours established accounts. But if you can solve the first issue, you can probably live with the second - karma has the same problem.