So, what is the actual threat from this? That someone now has my phone number (already public) and knows that I have called or texted with some other numbers? What is the risk in that? It’s not clear.
Well for one thing they can start figuring out who is not yet registered on Signal but would likely be in the phone contacts list of a rich person's number that they know. Social engineering attacks succeed with less.