Do you need a QA Project Manager?

[Azerax]

I've noticed (and Tom has mentioned) that there's been quite a few bugs lately.  I have 10 years experience in the software field writing requirements, performing quality assurance, creating regression tests from the requirements, and doing general project management for software development. (I am formally trained in PM for IT and will be writing my certification in a few weeks)

If this is a skill set you need, I am happy to volunteer my time, if this is a platform that would support such a role. (and by that I mean all changes go through the test server and are promoted to live once QA signs off, otherwise, forget it, it's just an exercise in frustration)

Cheers,
Scott

[Ehndras]

+1

[Anaris]

If this is a skill set you need, I am happy to volunteer my time, if this is a platform that would support such a role. (and by that I mean all changes go through the test server and are promoted to live once QA signs off, otherwise, forget it, it's just an exercise in frustration)

I...don't think this is a project that can really survive such a model.

We are all doing this for fun, and that kind of procedure...kills a lot of fun.

Obviously Tom's the last word on this, but similar sorts of thing have been proposed in the past within the dev team, and soundly rejected because, basically, they would drive most of the devs to not actually write any code, because it would be too frustrating and too much like work.

[egamma]

I think it would be a good idea for someone other than the dev writing the code, to validate it on the development server.

At work, I'm a system admin, and whenever I install software updates on a server, either a developer or another admin has to validate the change--I can't validate my own patching.

[Tom]

What Tim said is important, but I'd love to hear from an actual expert in the field if you have suggestions that might work for us. How about we have a talk on IRC or by e-mail and I'll talk a little about what our current setup is?

[GoldPanda]

It would be nice if there was more focus on bug-fixing.

Edit: There are ways to improve code quality. However they come with costs.

1. Code reviews: If a Dev is doing something he's worried might break things, or working on a part of the code base he's not familiar with, get another Dev to look over his changes. However this will slow everyone down.

2. Automated functional tests: Every time there is a crippling, show-stopping bug (example: remember the time a hero died mid-battle and broke the battle script?), add a scenario to an automated test script to make sure that it never happens again. Run the test on every code check-in. However it's insanely expensive in engineering time to create and maintain the automated test script.

[Tom]

It would be nice if there was more focus on bug-fixing.

Uh, how much more? We've fixed over 200 bugs in the past 30 days, that is more bugs fixed than are currently open on the tracker.

[Darksun]

Similar to Scott, this is part of my job. I work in the enterprise world where downtime means lost sales and forgone revenue. I would expect in a free game that there is not as much pressure on pure QA test coverage as there is with a business application but when game shattering bugs appear (see Friday's travel breaking bug) it does take a lot away from the ability to enjoy the game and recommend it to others.

That said I think there are two things that can be done to firm up the code quality.

1) Scheduled Release Windows - Whether it is an recurring window at a certain time each week, month or just some date that Tom puts forth, features get batched and released together. This can give the players a bit of a heads up when things are changing and provide some guidance for the dev team. This doesn't prevent emergency fixes of course, but they should be the exception, not the norm.

2) Automated Regression or Functional testing - Gold Panda's idea is spot on. Collect a list of the critical areas of the application and use a tool to automate execute or write a manual script to make sure they work. This will be your criteria to actually release the code to the players. Example areas could include: All PHP pages, Turn Change Script, Battle System, NPC faction actions. Is it as fun as developing new features? No. Does it keep your user group much happier? Yes. The earlier you catch an issue the cheaper (in this case, sweat equity and street cred) it is to fix.

[Tom]

Take into consideration that parts of the BM codebase are almost a decade old. This stuff was written long before unit testing was en vogue, and wasn't written in such a way as to be testable.

The BM code was largely written on a totally different conceptual framework than modern code is. That makes applying many methods quite troublesome. It's not that we haven't looked into unit testing before.

[Azerax]

What Tim said is important, but I'd love to hear from an actual expert in the field if you have suggestions that might work for us. How about we have a talk on IRC or by e-mail and I'll talk a little about what our current setup is?

Sounds good.  I'm about to take the family out for the day, I'll send you an email later tonight and we can go from there. (I'm in Canada, GMT -5)

There are many valid points and concerns here and Project Management has a really bad name because too many people try to force the Company to fit their PM Model instead of changing the PM Model to fit the company.  It's the very reason when new commercial buildings are constructed that they don't put in the sidewalks until a year later.  They want to see where people are walking instead of them create their own paths and ruin the grass. (Put the path under the people, don't try to put the people on the path!)

Tom - should do you have a preference of email address? (if not I'll use @battlemaster.org)

Cheers,
Scott

[Foundation]

Go Canadian CS people! GMT high -5 ftw.

[Tom]

Tom - should do you have a preference of email address? (if not I'll use @battlemaster.org)

All my various e-mail address go into the same inbox anyways. tom@battlemaster.org is just fine.

[egamma]

I like the idea of a scheduled release date--maybe Tuesday. That gives a couple of days for bug fixing before the weekend.

[Indirik]

Mmm... Patch Tuesday....

[De-Legro]

My two cents

We use automated regression testing and unit testing at work, at least for some projects. However I work as a Automation Engineer and Robotics programmer, so my experience isn't exactly the same as for pure software. 90% of the troubleshooting and bugs I encounter will be from physical interactions with the sensors and actuators that our software controls and monitors. When designing a system for critical infrastructure such as power generation or for major manufacturing companies we use automated testing systems, but in general we are talking months of man hours to set up the initial test system and generally 2-3 days when implementing a new test case. Maybe its different for a project like this in terms of time required, but I've never seen what is essentially a hobby project dedicate the resources required to implement and maintain a rigorous testing system.