Yes and it was never a point, it was just much much denser but still infinite. And so uniform there was no net direction for collapse before expansion took over.

Since there are conserved quantities like energy and angular momentum it is impossible that everything just collapses. If something collapses, there is usually a large amount of matter which does not collapse to carry away the energy and momentum of the collapsing stuff.