Temple size are actually less important than having good neighbours. If all (or even some) your neighbours are fully converted their influence will be greater than the loss you get from not having a large enough temple. Sirion, EC is a good example. For years the largest religion there had no larger temple than a small shack (Due to lack of elders.) but still had 100% in most of heartland Sirion. If you place your bigger temples strategically you'll find that regions where you have never even went will be converted without preaching, shrines or any temples at all.
