> I believe all cached Docker layers will need to be downloaded onto the build machine before it can be rebuilt

Docker making some sort of layer-sharing mechanism that constantly distributes layers to all build runners would be worth some cash, I reckon.