The only downside that I can think of is that people would pausing characters all over the place, just to get a character there. Some realms would become veritable ghost towns! Even if we had quite a few more players, I believe that would always be the case.
So why not do it anyway, if the character count doesn't matter so much?