In order for a religion to have staying power, and keep the interest of its members, it has to get involved in what is important to them. I.e. politics and war. You have to choose how you do this very carefully. Too aggressive, and you will attract too many enemies too quickly.
Religions that try to exist solely on RP find their members bored too quickly, and uninvolved. The rich, diverse religion is a labor of love of the founder, and no one else really cares.
You have to give your people something to do, and a goal for which to strive. That's how you get them involved. You do need the religious "window dressing", but it can be useful as a way to drive external conflict.
But no matter what, relate it back to what is important to your members: war and politics.
The Blood Cult managed to accumulate significant power, despite being clearly antagonistic. It got up due to intensive OOC recruiting, though (IRC and friends), and pretty much fell when most of these people got tired of BM as a whole. It also got many strong members purely IC, but these people seemed somewhat more prone to getting themselves persecuted, and were less likely to return if deported or make a new character if executed.
One of the problems, I think, is that nobody wants to, overnight, become enemies with a bunch of people they don't know, especially for so little return. Because really, when you join a religion, you get your reputation tied up to the religion's, and you accept yet another guy above you against which no rebellion is possible. If it's an antagonistic religion, it's likely not an official state religion, and as such probably has limited means to repay your sacrifices. So what incentives are there for players to join such religions?
I think the number one cause of the lack of hostile religions is that characters have no interests in joining hostile religions to begin with.
