Forum merging / new structure

[Tom]

At this time am running two complete forums (BM and MaF) and they both have sections about other topics as well.

I very much want to merge them into one. Each game would still have its own seperate section, it can even have its own look, but administration of one forum would be much easier than administration of two.

There would be some issues with user accounts and such. I haven't figured out how exactly to do it and if someone wants to dig into SMF docs and plugins that could help, it would be much appreciated. But ultimately, I cannot sustain several forums.

[GundamMerc]

Regarding the conflicts with user accounts, would it be possible to code a one-time account merge, with the option to choose which username you wish to keep? Most people will probably be using the same email for the accounts, so that could be used as a trigger for the code, "if account email duplicate" or however you would write it (python skills are rusty as !@#$ right now).

[Andrew]

I said most of this over on the M&F forum, but there are tools for account mergers, even tools for forum mergers, but there are particular requirements for most of them: forums must be running on the same [logical] server, tools require some trial and error to get the process to run right.

Most of it comes down to requiring some understanding of php and databases, and the time to do it all. It will almost invariably involve taking the forums down for at least a few hours to ensure everything works right.

The bigger issue is what the new forum layout will be. This forum is already pretty bloated as far as sub-forums go, with entire categories that haven't seen use in years, and forums and forum locations that don't make sense when you've got two different major titles that the board is servicing.

[Tom]

Would it be easier to keep both forums in read-only, merge things into a completely new forum, and continue from there?

I could simply open forum.lemuria.org and make both M&F and BM sub-forums of it. Maybe that is the easiest. It would allow for the trial-and-error process without destroying an existing forum.

[Andrew]

Assuming everything were to go down without a hitch, you could probably do the entire process inside of an hour, assuming you have the main forum setup wherever it is setup, all the forums are running on the same logical server, and nothing goes wrong with the server itself.

The plugins required would be mergeSMF found here and the SMF Admin Toolbox found here. The first will merge one board into another board, and the other will handle getting the accounts going.

mergeSMF does have warnings about the primary, recipient forum being larger than the secondary, merging board. To do this safely on a live/production server, you may want to clone the BM forum here into the new lemuria forum and then merge the M&F forum into it. Guides for restoring a SMF backup or moving a SMF forum to a different location should be available easily enough through Google.

That said, mergeSMF appears to be built for MySQL, based on the discussion, so your mileage may vary. What it does as far as topics are concerned appears to be pretty straightforward, so it could all be done as database actions if it came down to it. SMF itself has the ability to merge abandoned posts into a user's account, so you could even do that if you didn't want to import the user database.

The forums that mergeSMF merges into another, don't appear to be altered from what I understand of the plugin, they just appear to have their database copied, edited, then assimilated into the recipient forum.

[Andrew]

Oh, I should state, the admin toolbox won't have anything to work off of if you don't give people some time to sync their email addresses if they have accounts on both forums, since users can't change their own usernames. Should be something to give people a heads up about before you pull the trigger on any of this.

I can throw up an announcement on the M&F forum, but I have no such privileges here.

On that same note, I'll sit down tonight after I get home and start purging some of the never-used accounts from the M&F forum to cut down on database size and processing time. Anything not logged in after registration for 14 days will be considered fair game or never posted from AND not logged in recently will be purged.

After that, I can have the forum send out an email to all remaining users to give them an alert on what's happening, once you've got a firm timeline on when you want all this to happen Tom.

[Anaris]

I apologize for not posting before now.

Tom, please do not merge the forums until and unless you can guarantee that the BM forum integration will continue to work (even if we have to adjust the code some).

Losing that would be a devastating blow to the dev team and the Titans.

[Tom]

I think I put the forum integration into an actual package, so it should be easy to put that into the new forum.

But again, we can simply test it when we set up the test run. For this thing, we will do a test run.

[Andrew]

Alrighty, so I'm moving ahead with making this doable. I'll spend the next day or so getting the process for this down pat, and hopefully this will all happen sometime in the next few days.

At the worst, given the activity levels of the baords, there might be a few posts that get dropped in the move, but I don't foresee there being any other issues.

[Anaris]

Please confirm 100% before you make the transition that the forum integration will survive the merger.

[Andrew]

Right now, I'm still working with Tom on how to get the databases to mesh--one is running MySQL (this one), the other PostgreSQL. I did just setup an Ubuntu server next to my desktop here, so I'll give finagling the M&F database another go tomorrow, but this forum's database handled relocation just fine.

The integration is done through a file that I don't have access to, though based on discussion on the M&F forum with Tom, he seems to know how it all works.

Here's the intent: both this forum and the M&F forum will remain as-is until the new forum is online (longer if Tom wants to leave them on in maintenance mode or something). Neither of the two old forums will have any files removed or altered as a result of this merge. The databases from both will be merged into one larger database, and that will be imported into a brand new smf installation. In fact, given the BM forum is larger, it's going to end up forming the original database for the new forum, with the M&F database info getting added on to it.

The hangup right now is that I don't have a functional M&F SMF database to work from. If I did, I can use the MySQL Workbench to convert it to a MySQL database, and from there I can test already existing tools to see how much they actually merge. Though I am actively working on that, how I can, until Tom gives me a different means to access the M&F database. The backup I received earlier either doesn't work or I don't know how to use it, probably the latter and maybe part of why I now have an Ubuntu server running--though it was already on my "this would be a neat thing" list.

[Tom]

Please confirm 100% before you make the transition that the forum integration will survive the merger.

What is left of forum integration is posting to the boards, which is using SSI. The files in the BM repository will have to be updated with new board numbers and such, but that should be all there is. We should do this on the test instance and point it to the new forum before it goes live. The new forum will be set up in parallel to the existing ones, so we have time to test and then we will put the old forums to maintenance mode so no new postings can be made, copy over the databases one more time and that's it.

[Tom]

In fact, given the BM forum is larger, it's going to end up forming the original database for the new forum, with the M&F database info getting added on to it.

But on Postgres, please. In the long term, I plan to shut down MySQL.

[Andrew]

Alright, well, I know what the issue is well enough now to know how to fix it, but it's a lot more involved than I was expecting. Short answer is there's no easy way to fix the problems I'm encountering. The one fix I found involves exporting specfic tables, running them through a script, fixing identified errors, and then uploading that table back into the (live) database. Alternatively, since that method means I'm interacting with a live system, I might export them to .csv files, open them in excel, run a series of find & replace commands, and then import them straight to the functional cloned BM MySQL database. Once all the information is together and is proven functional, making it into whatever format is preferred is the next step.

Speaking of which, Tom, the more I look at this, the more research I do, the more I find information saying that SMF was designed to run on MySQL and then adapted to work on PostgreSQL. SMF's own support team says basically any time the discussion comes up that if you can use MySQL or PostgreSQL for SMF, the response is "use MySQL." A topic from last year, a few months before 2.0.11 of SMF was launch ends with the the SMF Support Lead saying "Not that PG is better or worse but the fact is that everything is developed towards MySQL and then "adjusted to work" in PG."

I'm not saying I'm not going to find a way to get this all in PostgreSQL if that's what you really want, assuming I can get the M&F forum database to actually be restorable, but if just trying to get a functional backup of a 2 year old forum is this much of a hassle with a PostgreSQL database behind the forum, I don't even want to imagine what would happen if you tried to do this with a forum the size of the BM and M&F ones combined.

[Andrew]

Actually, Tom, I just realized/checked this, but, are you aware the BM wiki is also running a MySQL database?