Ricebug on 16/12/2013 at 17:33
I'm pretty far along in my current project and am already anticipating the beta phase. However, there's a couple of questions for those who do the testing.
On a first test, would you prefer a mission without enemy AI? You'd get the completed mission with no enemies to deal with (unless they're goal-related). This would give you a greater range of freedom to explore, check, etc instead of always having to look over your shoulder.
On the second testing phase, after the author has fixed everything, you get the mission with everything included. This time, you can focus on tactics, AI interaction, etc.
But I'd like to hear from the beta testers themselves. What's on your wish list? What do you dislike (other than an author ignoring your recommendations)?
After a thousand FMs, this probably sounds like a mute point, but there are still new authors coming on-line (and testers, also).
Elentari on 16/12/2013 at 18:02
Personally, as someone who's tested a few missions, I would prefer the mission to be as complete as possible each time.
It can be kind of boring without AI - and even if the mission is 'broken', and you have to watch out for issues, you're still playing it for the first time. SO, yeah, personally, I prefer to play it as complete as possible from the first run.
If the AI starts to get in the way of checking an area or something, you can, usually clear the area or the mission yourself by killing or KOing the AI so you can check the area more thoroughly. So, unless there's an objective to not kill or knock out anyone, having no AI in the mission isn't really necessary. (If there is that sort of objective for some good reason, then, yeah, maybe a clear-run first out would be good, since you can't clear it yourself without failing the mission.) Also, if you suspect there is a trouble spot, the tester can also point it out and other players or the author can check it out - or a test-release without the AI could then be released.
But, no, over all, I like to play the mission as the players would see it. You only get to play for the first time once, and even if the mission is glitchy, and you'll be playing several times (or many times, as the case may be), you still only get that first impression once, so having it as complete as possible for best effect is still preferable. :)
Tannar on 16/12/2013 at 18:38
Leaving out the AI is more similar to what we would call an Alpha test, though Alphas usually don't have everything else in place yet either. Having tested close to a hundred missions of all types and sizes, I can say with certainty that I wholeheartedly support Alpha testing. It isn't so much the relief from always looking over your shoulder for AI, though having to deal with AI is time consuming when you would rather be focused on more important things, especially when you must test a specific sequence multiple times.
What is more to the point, though, is that Alpha testing allows the focus to be on more mission-critical aspects without the testing reports/discussions getting bogged down in less critical issues. The relationship between author and tester can be more collaborative in finding and fixing specific, potentially show-stopping issues. At the very least, Alpha testing facilitates this process. And I strongly suggest having only a few alpha testers, at least some of whom have experience doing alpha tests. You won't regret it. Then bring in a few fresh faces once the Beta starts.
As for my dislikes, I don't have many. Most authors are receptive to input and diligent about fixing issues. My only real dislike is when an author is in a hurry to release a mission (not counting things such as contests where there is a deadline). It takes time to properly test a mission and sometimes testers must put in an inordinate amount of hours in order to cover everything that needs covering. I appreciate authors who understand that.
EDIT: I just read gnartsch's post below and while I agree that the first impression of the full-fledged mission is priceless for a tester, that really only needs to be in the Beta test. That's why you bring in new people for the Beta test. Let the Alpha testers do their thing, then bring in the Beta testers.
gnartsch on 16/12/2013 at 18:44
I agree with Elentari, having a first unprejudiced & full impression is priceless.
So, the first run through a mission is always about PLAYING it as you would do as a player.
Only like that you get the feeling for the mission. Whether it is too confused, obstructed by too many guards and other obstacles, etc...
Looking for problems & details comes in a second run.
Having a first glance at the mission without any AI would perfectly destroy the chance to provide as much feedback on the gameplay as possible,
since the testers would already know where what needs to be done on the next run.
bedwine on 16/12/2013 at 19:00
I agree with Tanner. His analyst is correct. It might be interesting and fun to do an alpha test first then a beta test knowing the lay out well. Sort of like playing a mission for the second time but concentrating on gameplay and the AI.
nickie on 16/12/2013 at 19:17
I agree with Tannar too. But to me it depends on the size and how complicated the mission is. A large sprawling mission with lots of puzzles would definitely benefit from an alpha, I think. A lot of things that need testing are a pain with AI around, particularly sound testing. Checking all the architecture, textures, sounds etc. and probably story as well, before testing out the AI, does help. Testing can be such a long process that dividing it up can make that easier.
I agree that you only get to play it once but I'm afraid I'm the type of player who thinks AI just get in the way of the main event. :)
Ricebug on 16/12/2013 at 19:17
I've always heard of alpha testing but typically in the scientific community. Good to know, Tannar.
gnartsch on 16/12/2013 at 21:48
Total agreement on the potential benefits of an alpha test !
Focussing on story, puzzles, textures, etc. typically makes up most of the issues you run across
and not having to bother with loads of AI would defintely speed things up in that phase.
Just make sure to have a couple of fresh faces around for the beta phase. That would be an absolute must. No discussions!
For the average tester (I would guess) it would be just too hard to pretend not knowing what to do if you did alpha already.
So, one may get easily sort of lazy and easily miss even some obvious new and critical bugs then.
Renault on 16/12/2013 at 22:32
For most missions though, is an alpha test really necessary? This isn't a brand new game we're creating here, it's just a new mission for an already tested and released commercial game. Sure, if the thing is super complicated and has tons of scripting, maybe (hello, Mission X). But for 90% of most FMs, I don't think an alpha test is necessary.
I guess I just wouldn't want authors to start thinking this is the norm or standard and consequently start overtesting their missions.
Tannar on 16/12/2013 at 23:38
You're right, though the fact that Thief is an already tested and released commercial game is, for the most part, irrelevant. Yes, there is no need to test the things that were already tested in the OMs, in the actual game itself, but each new fan mission is essentially a new, untested game of its own. We aren't testing the Dark Engine or any of the fundamental elements of Thief, but rather the author's unique implementation of them (not to mention custom material and scripting and the fact that many authors push the engine past all original uses of it). Time and time again I've seen potentially mission-ending situations which would not have been caught without thorough testing. Granted, this certainly doesn't occur with every mission, or even most missions, but how do you know unless you test thoroughly?
And you're right, an alpha test isn't required for thorough testing; it can all be done in the beta test. I'm not sure an alpha test is ever completely necessary, but it can often make the testing process much easier. Especially for the testers. I'm not sure if I agree that 90% of FMs couldn't benefit from an alpha test but I do agree that many missions probably don't need one and I would certainly say that, in general, the more complex a mission is the more it could benefit from one and vice versa.
I also don't agree that conducting an alpha test constitutes "overtesting" a mission. Anything caught during an alpha test is one less thing to deal with in the beta test, so it's not as if it is adding more testing, it's just dividing up your testing into stages. And it can actually save time in the beta test. And though they can sometimes run long, alpha tests are typically short, often only a matter of a few days or a week. And while I agree that the average author, especially new authors, shouldn't feel obligated in any way to conduct an alpha test, I would say that for many authors, alpha testing is already the norm.
