Then why have a separate lord for each? An estate with a fortified home, maybe, but the point of the lord is that he was needed to manage lands further out and protect them. If they can't protect their lands worth a damn, then they don't have a purpose.
I see several logical fallacies in here, but I will leave it at this: BM is not a medieval simulator, nor does it aspire to be one. There are times when we make decisions based on what was most historically accurate, or what would be most realistic, but there are other times we make decisions based on game balance or fun, even if it goes completely against history.