This is a good point, something that I've been struggling with in my own personal projects is keeping track of parameters as I tweak and play with hyper-parameters and model structures.

A few parameters are fine, you can pull them out into constants, but you quickly end up with a lot of variables to keep track of.