That is a woeful amount of power to hand to every rural lord when their supposed fealty in most realms is to their Duke or their King.
Feudal oaths go both ways. If your liege is basically robbing in terms of food, the links that bind you are already broken.