Critical portions of the server are not FOSS. Also the core software forces you to join their servers.
Also we have no proof that they are running the server software published on GitHub. This concern is exacerbated by the fact they didn't publish server code updates for many months.
I mean sure, but also, the client app source code lets us know that unencrypted data is not sent to the server. So at best they could perhaps be collecting some additional metadata, but I don't think it's a whole lot
The signal-server repository is open source