So, the challenger could lie, make up whatever he wants, misrepresent the entire situation, or do whatever they want. And the challenged noble doesn't get to respond at all with their side. And the results get permanently attached to that character? No thanks.
Why not put in a box that say reasons Duel refused?
So you have both sides of the story.