I have a feeling you'll get some of the converts back. Not everyone who got dragged along seems to be very happy with it.
I know from some guys that will come back, but I guess that most of them will just stay at Toupellon. It is the most promising realm at Far East Island, so I understand them. Had it not been for the personality of my character he would have ended surely there.