I think you are underestimating the power of the prophet. After that proclamation, !@#$ just got real. Try announcing your intention next time. With that trip Farronite tried to make to Balance's Retreat, you can't seriously be surprised when people suspect you when you show up at their doorstep, with a huge army.
The Prophet does make things tricky, but nothing more. I think you might be overestimating him. After all, there are many newcomers who haven't had time to be doctrinated in the "look upon the Holy Prophet in awe" aspect yet.
