The only issue is that having an invasion to the north would be the same as having the invasion from the western island to the eastern island, the distance and lack of ability to refit would cause the whole thing to crash and burn, especially since there is no quick way north aside from sea lanes which would grow expensive and increase the potential of a disorganized invading force. Unless of course Swordfell opened up their borders to allow people in the south to skip around the giant desert and mountain ranges, but then you still will arrive with high equipment damage only to run into giant stacks of militia.
