@MK2boogaloo@Jim@graf You have to keep making new housing or else they won't start new families.
You could also take a gamble with the nomads that show up periodically, which I have done (at one point, something like 100 nomads showed up at once) before... but they bring disease and can easily crash your economic equilibrium because you have way more mouths to feed suddenly.
Which has also happened to me, because I wanted the labor. Oops.