What if having a building in estate required a certain amount of peasants to function with top efficiency? So some are taken away from actual gold production, or food production or whatever, and the more buildings you have, the less efficient each of them is.
So the estate size would matter in this too.
