IMO, the reason that knights shrug it off and leave is that getting kicked by your lord would suck. It would make you think your lord is an ass. And who wants to play a game as vassal to someone you think is an ass? I would much rather find a different lord that I think is fair and honest with me, and not likely to kick me out again at any time.
Let's face it, it's just gold. And who really cares about gold? Given the right reasons, you can get all the gold you really want just for the asking.
