The amount you have on hand should not be relevant. The charge is for sending small amounts, not having small amounts on hand.
Next time you get a chance to bank, record the numbers *before* you do anything, then again at each step in the process, along with the actions taken. If you do notice anything missing, then that should show us where to start looking.
