They are untrained/unskilled in infiltrating a well-armed camp. Bandits/robbers only strike nobles with no units or very few men. Adventurers don't attack monster/undead hordes (the equivalent of a strong unit), only when they're smaller sized (though the text does refer to attacking undead armies, which isn't entirely sensible either).
I think this is the best counterargument as far as the Adventurer point goes. I'd also add that in no way are Adventurers portrayed as sneaky; they walk up to a noble's camp and they'll be turned away like any other peasant. (Yes, the can try to sneak up on undead and monsters, but that's a "oh god, I'm fighting these unnatural beasts... I'd better be careful".)
All things considered, I'd have to agree with the idea that nobles have the experience to tie someone's presence to the crime. Even if you want to say "well, there's no direct proof", the fact stands that the noble (and his camp, if any) is the only variable that is different. Maybe I'd accept one of that noble's men being the scapegoat, but that noble is also responsible for his men. If our nobles can't recognize that, then it's a miracle that trading works at all.
