Yes, the middle east did have a big influence on Medieval Europe. So did migrant steppe tribes. Frankly, so did India; even China had periodic influences.
But the game is not about those places.
Moreover, even in places with "religious freedom," it would mostly be for commoners. You won't find tons of Christian knights running around holding lordships in Fatimid Egypt. A very few perhaps, but not many. The Byzantine Empire had a few high-ranking Muslims in its later days, if I remember correctly.
If a realm RPed itself as allowing religious freedom, but nobles of religion X received 1/2 has large of oaths, or perhaps paid 2x as high of duchy taxes, that might be realistic. But even that is a stretch, given that most Medieval powers, even the Arab ones, had state religions. Some limited exceptions do exist in China and India at times, and I'm unclear what the religious status of nobles in Moorish Spain might be. But, by and large, religious diversity among the nobility was tightly restricted, even more than among the commoners.
