It's also easy to lock the banker out of your regions...
True, but if he's good and quick you'll be too late for that. Food can be moved within matter of an hour.
There are also other tricks you could pull as a banker. Say you need money to ship food and send some of it to your family every time, or keep it but claim you need new funds again. Unless you do it extremely foolishly, people won't catch you really. You could also transfer food to your own region and subsequently sell them for much more if your realm is producing a surplus etc.
But the 'starvation' as a pressure point won't work so well, since they'll chase you out. But you can use it to gain a lot of gold, or as some sort of end-game kind of style as described above. In fact, if you have 2 or 3 bankers cooperating together towards such a goal, you'll get complete chaos in a large area, giving power to those boys. Now that I think of it, maybe I should become banker one day :p