The contacts list has always been something of a kludge.
Due to the way it is structured, it can only keep track of one way that you know any given person—you were in the realm with them, or you corresponded with them, for instance—and it has no way of directly keeping track of how recently you corresponded with them. Thus, when you leave a realm, it just clears out everyone who's a member of the realm from your contacts.
None of these things are insoluble problems; however, it's not a particularly high priority to fix at present.
