While I certainly agree with not having a huge building component, things like Tower > Keep > Castle > Citadel, or Manor > Mansion > Palace would be a major part of being a landed noble. How else do we show our control, our prestige, our wealth, and our power? Remember, many of these are homes for our families, not just military installations.
And it could add to the strategic element. Nobles would be less impressed by a Citadel than a Palace, but a Palace is much easier to take via conquest by armies of your rivals.
