I know the redphone is library is just a binary blob in the github repository:

    https://github.com/WhisperSystems/Signal-Android/tree/master/libs/armeabi

But I always thought that .so was just a compiled version of this C++ source, which in the same github repository:

    https://github.com/WhisperSystems/Signal-Android/tree/master/jni/redphone

I haven't compiled it myself so I can't be 100% sure, but the C++ entry points matches the API the Java code is using. I presume it's written in C++ for speed. There isn't much to the C++ bits. It just pumps data through an encrypted RTP connection - CPU intensive but not particularly complex.

The server code is up there too - in fact it's all up there. AFAICT, Signal is completely open source.