Your swordfighting skill helps you fighting tough monsters and undeads; your adventuring skill helps you do a variety of things, some of which I can guess at, none of which am I entirely certain about.
Your gear, however, limits your skill, i.e. if you have a 40% skill as verified at the academy, but 20% fighting gear, you will fight like a 20% skill noble. Buying more gear than your skill level, however, does not help you.