We just send the save to each other, so everyone can use their own graphic pack right? That shouldn't be a problem.
Some graphic packs have particular requirements as far as the raws go. Ironhand, for instance, adds in llamas.
I've been using Ironhand, too, and I think I like its latest iteration better than the current Mayday graphic pack.
If I do end up being the starter, I think I can twiddle the raws some to ensure that they will work with Mayday, Ironhand, or (naturally) plain ASCII. A least-common-denominator raw set, if you will, so that all we'll have to do is change the settings in the init.txt file to switch smoothly between graphics packs.
I can handle ASCII if I have to, but I very, very much prefer a good graphics pack. This is why another of my side projects (back-burnered due to bugs in DF) is a client for dfterm2 that lets you use a graphics pack...