Or rather, because you usually don't know what everyone else's skill with a sword is. All you know is that yours sucks, so others' either sucks just as much, or is so much higher than you so that you don't stand a chance. So you either get a 50% loss chance or a 95% loss chance. Most people would not gamble such things (death or, alternatively, a time-out that may last few a few days) over such odds.
In the modern age, maybe not. When honour is at stake as medieval noble? sure they would.
