Well, maybe it doesn't need to affect inactive lords more than others. As long as there's enough options for players to generate turnover, there will be turnover. And either the inactive ones return to defend themselves, or they lose their positions.
