The only think that might confuse people is if you start to recycle numbers (because you don't want to keep paying for them), and the call gets connected somewhere totally unrelated.
I thought you might be using caller ID, so the number is only relevant to people with a specified phone number. (Similar to how Rebtel works.) But it doesn't seem to be the case here, as I think numbers can be shown publicly.
All phone numbers have some ambient activity -- wrong numbers alone account for at least some randomness, though some obviously have more than others, and even very established carriers sometimes distribute numbers with a "legacy". So it's sort of a known problem.
For Burner, each number goes through a quarantine and is monitored for activity before we use it.
EDIT: Also worth noting that SMS seems to be more popular than voice calls, at least initially. "Wrong number SMS" doesn't seem like an actual problem as of yet.
The only think that might confuse people is if you start to recycle numbers (because you don't want to keep paying for them), and the call gets connected somewhere totally unrelated.