I don't know that anyone has really done an exhaustive study of this. It would be very difficult, due to the differences in unit stats, and the fact that you really can't ever know your opponent's unit stats as anything more than an abstracted CS rating. Also, you have to be wary of any guides or analyses that were written under the old battle system. A lot of those were written to take advantage of vastly different system of overcrowding and other things.
1) Box formation should help you defend against anyone, especially those that are much stronger than you. It should help your men hold together longer. They should be better against monsters, and more able to resist a cavalry charge. Conventional wisdom seems to be that you should use box when you want to: a) Hold out longer against a superior foe, b) Resist a cavalry charge or are fighting monsters.
2) Skirmish formation will help *any* unit resist the effects of ranged attacks. The problem is that it also greatly reduces your close combat attack strength, and resistance to close combat damage. If you try and put your melee unit in skirmish, you will find your men slaughtered by units that are much weaker. I would *never* put any kind of melee unit, infantry or cavalry, in skirmish. This formation should only be used by ranged troops who are confident that they will not be entering melee combat. For example, we used it extremely effectively during the BT invasions when facing the ranged daimons that were shooting fireballs of doom. Our archers on the walls would be in skirmish formation and take much, much fewer casualties, allowing us to deal out many more rounds of archer fire against the daimons.
If you try and put melee units in skirmish to reduce your ranged damage, you will find yourself slaughtered by nearly any enemy melee force you encounter. If you are pretty sure that your lines will be overrun by the enemy melee forces, and your ranged troops will be forced into melee combat, then you may want to put your ranged troops in Line formation. Ranged damage is unaffected by the attacker's formation, so you will not be reduced the amount of damage you deliver. You will take more from ranged attacks yourself, but your ranged troops will be able to much better combat the enemy melee troops, especially if your ranged units include MI and SF troops.
If you think your melee troops will be taking too much damage from enemy ranged troops, then use Line formation as a good compromise. It's generally the safe choice in any situation.
3) I almost never see infantry in Wedge. I suspect, though I have no evidence, that wedge may help you break through the lines and penetrate the enemy's formation. So with this, if your enemy has a lot of archers and a thinner infantry line, you may be able to use some infantry on Wedge to try and break through the lines to engage the enemy archers in melee and stop their ranged attacks.
In any case, proving any of this, other than the skirmish stuff, will be next to impossible. The randomness of battle, and all the hidden factors, will preclude put any kind of rigorous analysis.
In addition, it's not as simple as one formation canceling out another. "Inflict more/take more" vs. "inflict less/take less" doesn't always equal out in the end. Switching to box and taking less damage may let your men stay in battle for one more round, letting you get in three hits, instead of four. That's a 33% increase in the number of attacks you get. So even if each hit is individually 10% weaker, you're getting more hits in total, which could tip the battle. However, because you're doing less each round, you may not knock out your enemy in the third round, thus letting *him* get in a fourth swing.
These effects may be multiplied in scenarios where the forces aren't equal. Getting slammed by a monster group may make your men retreat immediately when you're in Line, but let you go a few rounds if you're in Box.
