Signal is centralized, so it becomes a huge target of all kinds of hackers and three-letter agencies. This alone is sufficient for me to never touch it. And then, there is this:
The vast majority of people using "end to end encrypted" messaging systems fail to verify the identity of their contacts. So those running the servers can fairly trivially MITM the messages. So in practice it does matter who controls the servers.
The good thing is that verifying the other contact is invisible to the server in Signal. This means that it's stochastically sufficient that a few people do check their contacts in order to see whether there is any widespread MITMing going on.
https://news.ycombinator.com/item?id=42788647
https://news.ycombinator.com/item?id=39445976