Besides time and money for construction and maintainance like all other buildings? I don't see the need for more downsides.
Mostly, I think it should be made such that a choice is necessary, and building everything is a losing proposition; otherwise most people will actually build everything, and the only thing achieved will be to create a new gold and time sink when you take a new region before it is productive again.
If the price for building climbs exponentially to extortionate prices, this should do the trick.
