Well, actually, to implement this at all, we'd have to go voice. We can't deny a lord a vote in the referendum system by removing him from a message group. Even so, simply switching to voice doesn't change the fact that they aren't contributing. We need the feedback to determine if it's a specific aspect of the referendum that's being turned down or the whole idea.
Imagine you want to form an alliance with another realm. You are neutral right now. Several people speak in support of it. Nobody speaks against it. Referendum results show 10 in support, 13 opposed, 5 abstain. Are they opposed because they want to go to peace first? Are the opposed because they'd rather go to war?
When we raised a fuss about it most recently, we did have a noble or two state that they voted no, but I don't recall them giving a reason why they voted no. Instead, they just said that they always vote silently. I would be just as suspicious if a referendum passed without any discussion, but it's harder to tell since the referendum isn't created at all if there's no reason behind it.
Now, all that said, the reason I bring up the question of IR at all is that there are likely inactive nobles that will be caught in this. It's not the intent, but are the IR always treated depending on intent? That seems to be very open to abuse. Even if it's a vaguely defined "reasonable attempts to accommodate inactive players", I would feel more confident in just putting this forth.
Oh, and for the record, this has been proposed, but ironically, nobody seems to want to discuss it. (We had an OOC chat about how to make it work, but that's it so far.)