Even worse than C++? That has never been my experience. I maintain C++ projects that take hours to build from scratch. Most Rust projects I install, even the very big ones, are ~3-5min max.
Are you building in release or debug? Do you have lto enabled? Rust compile time is absolutely worse than c++, and I'm not sure many in the community would disagree. You need to aggressively split large crates up to make it sane. A lot of this comes down to the fact the crate is the compilation unit rather than individual files.
