With Signal, you can't really validate the code running on the client. Signal insists on distributing only via Google Play Store or Apple App Store, so usually updates are automatic and uncontrolled by you. And Signal has a history of not releasing timely updates of their client code, so even if you would do your own builds or compare their released code to their public updates, you would have at least a few weeks latency. And I doubt anyone would notice, since the Signal people tried hard to piss off everyone who did reproducible builds of their code.
In theory you can do the same with Signal, as they source dump their server code every now and then.
If you reject that on the basis of "we can't know if it's what they're running" or "it's a partial dump", then I don't see how Matrix is any different. Not only we can't know if Matrix servers have modified software, but we also have to trust/verify several servers instead of a single one.
