Telegram can be built locally too, and there are independent apps for it, and they have taken efforts within the security community to prove that it is secure.
You may be able to run Signal server from your build, but you won't be able to talk to users on Signal.org. Signal's official position is that it will never federate with others. So I don't understand how building and running a server locally is a step up in any way.
They've also gone to extreme lengths to prove that the software running on their servers has not been tampered with: https://signal.org/blog/private-contact-discovery/#trust-but...
I think your doubts create very unreasonable expectations for Signal.