Wow, how much population were you losing that you though it would make a noticeable difference to tax income? I've never though that recruitment would exceed the population growth.
It can. A smaller city/stronghold with several good, larger centers that are heavily recruited can slowly drain the population. Used to happen more often when there were more characters in a realm, making sure the RCs were never full. Also, we didn't have the peasant migration code, either.