I imagine it could take some minor contributor deciding they don't like what you do and taking extreme interpretation of AGPL including trying to insist everything that talks over network to it is derivative work, vs interpretation of yours (let's say you're not modifying an AGPL project at all, just running it vanilla, and it's part of some bigger offering, and your lawyers said it's not crucial enough for the bigger offering to be considered derivative)

As far as many people are concerned, AGPL is vague enough it would end up in court, so the question becomes "do I want to have this sword of Damocles of unnecessary complications and upsets?"