The most common worry is not about having a smaller population, but about the process of getting there. As birth rates fall, each new generation is smaller than the one before. Over time, this could mean too few workers to support all the pensioners