Yeah, the magic was a bit of a letdown. I maxed destruction on one of my chars and yet almost all enemies would take multiple hits with the most powerful spells in the game.
With my new main char (level 51 now), I don't bother with destruction magic much, except for a fireball or so to hit enemies around a corner and draw them out where I can shoot them. :-)