And Dukes collect taxes from Lords.
I don't think it's so very unreasonable if the taxes of someone who gets exiled get confiscated entirely by the person next up the chain.
Lords send a portion of the taxes collected to the Duke, but the Duke doesn't stand between that Lord and the taxes of his region. The crown's purse isn't filled before the Duke's.
I could see giving Lords the option to pay their Duke's share directly to the King, if that Duke is exiled.