If the players are talking about inactivity of the ruler's player, I would have very grave concerns.
If the characters are talking about inadequacy of the ruling character, I would applaud fun politics.
If both are happening simultaneously, I would get a very annoyed expression on my face, make a blanket warning about respecting the IRs, then wait for a reaction.
Players just don't need to talk about "activity." Just ignore it. Have your characters talk about dereliction of duty. As a player who has been in numerous realms with "inactive" rulers, and as a player who has PLAYED a council member who was intentionally engaging in dereliction of duty and feigning "inactivity" through non-responsiveness, I am pretty confident that, if the realm is suffering, IC reasons probably exist, and the characters involved should work with those reasons. Nothing else.