It's not just that.. anyone you know who ever imported their contacts list would be connected to you. With enough data points about them, FB/IG can infer quite a few things about you pretty easily.

If your university's alumni association imported their contact list, it's even worse.