Past clarifications have said the following: Inactivity does not mean immune to persecution for failure to one's duties.
If a knight is just always straggling behind in movement, then sure, that's inactivity. If a knight never responds to letters, nor obeys any orders, then that's not inactivity. He might *also* be inactive, but his inactivity doesn't make him immune from reprisals for his other failings.