Does your motherboard have auto-oc enabled? Have you checked what voltage it's using? Have you tried setting a negative voltage offset and stability testing? Some motherboards will apply 1.3v+ when 1.2v is plenty.
The 5800X didn’t ship with a stock cooler IIRC. Mine is cooled with a 360 AIO + PTM7950, the thing just runs really hot when all cores are hitting ~4.4GHz.
The 5700X has the same 8 cores as a 5800X3D but with a slightly higher maximum clock speed (the X3D CPUs tend to have lower maximum voltages because the extra cache die doesn't tolerate voltages as high as the CPU cores do). The only reason the 5700X is running cooler for you is because it comes with a 65W "TDP" setting out of the box rather than the 105W "TDP" setting used by the 5800X3D. If you configure a 5800X3D to operate at the same power limit, it'll give you generally better performance than a 5700X.
In general, buying a power-limited desktop CPU has never been a good strategy to get better efficiency. You can always configure the full-power chip to only use that extra headroom for short bursts, and to throttle down to what you consider acceptable for sustained workloads.
It’s impossible to keep my 5800x below 90C under full load.