I've seen that expression very often, and always wondered what it meant, "servant of the realm, not of the King/Queen". So far it has seemed to be "servant of oneself", to me.
What it means is, "I know how to run this place better than those idiots! If only everyone would listen to
me, and do what
I say, we wouldn't be having these problems! I'll show them all! THE FOOLS!!!!"
(So, basically, it means he's a traitor, especially in a Monarchy. In a Republic-style realm, he could make a case that he wasn't, but if he failed to win an election, then he was also not just against the Ruler, but against the people, so it kind of loses its punch.)