Yeah, the magic was a bit of a letdown. I maxed destruction on one of my chars and yet almost all enemies would take multiple hits with the most powerful spells in the game.
With my new main char (level 51 now), I don't bother with destruction magic much, except for a fireball or so to hit enemies around a corner and draw them out where I can shoot them. :-)
Morrowind had better magic. Levitation, silence, chameleon, ... I've only used flames to ignite puddles of oil
(which I have been doing consistently ever since I discovered the feature)