What I really need is this + monitor input switching. I want to use both my monitors with both computers and just be able to hit a single button to swap everything.
Many KVM switches do this, just search Amazon for KVM switch + HDMI/Display Port, but they're often not great in my experience. My last one only lasted a couple of years and would sometimes fail to pick up my monitor.