Yeah, if you have enough people to fit some town in Montana you still can't run New York City, not even in part, nor as some strange analogue of that Montana town. There's this thing called infrastructure that is not very good with being forced to operate differently. A city planner, or civil engineer, could give a heck lot better talk than me, but the end point is that, generally speaking, there is a certain cut-off point for the type of sectors you have. Er, in other words, no, having 5000 people who can work fields in some rural region does not mean you can run your city like that. For one thing, there's a lot more to run in a city to make it satisfy the minimum...level. Yeah, my technical language in this is lacking, meh. Maybe Medieval towns could downsize, who knows. But the "does it make sense" part can run both ways.
And if nothing else, it's a game mechanic that basically says, "Dude, use your brain before you take actions."