Because like we said Madina culture didn't really push unity. Probably find that those that had the gold didn't want to give it up for some stronghold that wasn't theirs. Given that we have already seen some regions jump over it is also entirely possible that many in Madina WANTED the realm to fall.
Actually the defenses of the Tower were greatly reduced because the newly appointed Marshal of one of the armies decided to take his army on maneuvers away from the Tower and no one objected.
