I'm not aware of anybody who left because people didn't answer letters.
I'm aware of one or two who left for other reasons...but not a lack of letter responses.
One look at my message feed would prove beyond a shadow of a doubt that lack of letters is not a problem. 
You are a ruler though. Rulers usually get more letters than knights.
