Suggestions for Testing

This is an article I wrote back in 2003 or so, which used to be on my previous website. Recently I was asked to post it again, so I’m doing so, with a handful of minor edits.

The following are some suggestions based on my own experience of beta-testing, both from the tester’s side and from the author’s. Not all of the advice it contains will be universally applicable, of course. Pick as you will.

The Tester’s Side

Rule 1: Do not release the beta you’re given.

This should be obvious, but I mention it because it is the most important aspect of the beta-tester’s trust. Also, do not discuss the game you’re testing in public before it is released. And do not, ever, mention the bugs you encountered in the pre-release version, unless you have the author’s explicit permission.

Establish a form for feedback.

If the author doesn’t make this clear, and it’s not an author I’ve worked with before, I usually ask what kind of feedback he would like to receive. There are two things to settle on, form and content.

Content: Some people only want reports on bugs in the function of the game; others want a wholesale critique of the entire gameplay experience. The former comes up especially frequently when the author is re-beta-testing a new version of a game that has already been released — say a competition game that he has reworked a year later. There’s good reason to test again before re-releasing the game, to make sure that no new bugs have crept in while the old ones were being fixed. In this situation, though, it’s too late for remarks about the plot and so on to be really useful. In other contexts, however, authors may welcome feedback on the non-bug aspects of the game. I have in the past gotten comments about puzzle difficulty and cluing; about the efficacy of my UI design; about additional verbs that would be nice to have, or additional comments that might flesh the game out more;

Form: Some authors want to receive transcripts of your entire play experience, while others want just a message summarizing the bugs you found. If I’m beta-testing a game, I play with scripting on — whether or not the author has requested transcripts as the form of his feedback — and then I have the transcript to refer back to. Frequently, I send the annotated transcript along with a few paragraphs of summary that contain my overall feelings about the game and any things I think are most important to change or improve. (I also try for a few lines of encouragement about the parts of the game I did like, but this is not an essential aspect of the beta-testing job. The chief task is to get the game into playable shape; keeping the author happy is sometimes not perfectly compatible with that task.)

When I say that I annotate a transcript, what I mean is that I type my comments directly into the game as I play, usually prefaced (consistency is the key here) with asterisks or brackets, like so:

>* Missing space in the second paragraph.
>* I dropped the featureless cube but it's still in my inventory.

(TADS and some generations of Inform actually provide for this annotation so that no game time will elapse after a turn that starts with *.) Typically I read the entire transcript through at least once, but sometimes later when I want to review specific bugs I find the searchability important.

This is pretty standard beta-tester behavior (inasmuch as anything can be said to be standard in this community). Some of my testers get quite chatty with these comments, and I also get things like this:

>* Yay!
>* Wait, what??
>* ARGH
>* I thought the Marquis was older than that.
>* Rowr.

…etc. It’s sort of as though the tester is having a running conversation with me about the game as he or she plays.

Follow your own style.

What you actually do while testing is largely up to you. As I mention elsewhere, some of my testers play as though they were experiencing the game as regular players — which is useful, and gives them a regular-player sort of perspective on the game. I have also had or heard of testers who

• Tried every possible direction from every possible room, even if there wasn’t supposed to be an exit there.
• Closed every door behind them when passing through an area, and/or otherwise experimented with restoring things to their initial conditions.
• Ruthlessly repeated all NPC-related actions — showing them things twice, asking them the same questions over again, and so on, to see whether critical triggers fired twice or only once.
• Tried to attack, break, cut, burn, or otherwise destroy all important objects to see whether they interacted appropriately and whether it was possible to lose game-critical items.
• PUT ALL into each container item to see whether capacities and sizes were handled intelligently.
• Ran around using any special new verbs I’d written on the most implausible objects — scenery, other characters, the player, etc.
• Disassembled the gamefile in order to find things that should have been accessible, but weren’t.
• Set their monitor or screen size to different settings to see how a graphical UI behaved at different sizes and aspect ratios.

Testing a game often involves deliberately trying to break it. As you play a game, you may get a sense of where the code is likely to be most complicated. Anything that involves an important plot trigger, any complex simulation, any added verbs or language: these are things that can use special investigation. But ultimately, you should do what you like when testing. It is part of the author’s job to find several beta-testers and, if possible, to get several testers of different temperaments. No one tester will do or think of everything — so do what you do best, and trust that someone else will catch whatever you don’t think of.

Send reports in chunks.

Unless the game you’re testing is a short one or the author has made it clear he has a different preference, it often makes sense to send your reports serially rather than in one big batch. You may only have a few hours at a time to concentrate on a game; the author would probably prefer to get that feedback immediately, so that he can start fixing those bugs, and also because waiting in limbo for feedback is nerve-wracking. Also, a huge email containing hundreds of bugs is much more daunting than five emails each with thirty or forty. In my larger games I have fixed thousands of bugs.

The general operating procedure for my testers has usually been that they will play my game until they’ve worked up a few transcripts and have gotten stuck or come to a major plot break-point. Then they send me the files, and ask any questions they have about how to solve puzzles that are daunting them. I never send walkthroughs to my testers in advance, because finding out how they interact with the puzzles, and whether they get frustrated, is part of what I use the testing process for.

Assuming the author is interested, share your reactions.

I always like to know how my testers feel about a game. Sometimes they have specific complaints about the way I’ve arranged the windows, or about my writing, or about the way a puzzle is clued. Sometimes they only have vaguer impressions — a sense that the gameplay is too rushed or too slack, or that the narrative is too confusing. Even that kind of information is useful, especially taken in combination with the feedback from other testers: it gives the author an idea of how the game is likely to be received, and may allow her to diagnose the problem, even if you don’t know what exactly is wrong.

If you have to stop, let the author know.

It’s not at all infrequent for a tester to find herself swamped for time and unable to finish testing a game. If you’re testing, you’re doing so at your own option, and there’s no shame attached to quitting — either because you’re busy, or because the game is not to your taste. (If you really dislike a game, you’re unlikely to be a good tester for it anyway: too much of the process requires goodwill on your part. It’s one thing to find a work flawed but intriguing — that you can work with. If you don’t think you’ll ever like it however well-executed it is, though, you should think about putting it aside and working on something else.)

If you do quit, you should let the author know that you’ve decided not to go on so that he can find a replacement tester. Simply ceasing to send reports may leave the author in an uncomfortable limbo position.

The Author’s Side

Contact the beta-tester and make your requests clear.

In particular, it’s a good idea to mention some or all of the following points, if they’re important to you:

Time-frame. Mention when you are planning to release the game (and thus what your time-frame is), if you’re working on something that has to come out at a specific time. This doesn’t guarantee that your tester will be able to finish your game in the time you indicate, but it does mean he has a chance to write back and tell you that his schedule isn’t going to fit that.

Format. Say whether you need your feedback in a specific format. By this I don’t mean anything really complicated — there’s usually no point in arranging annotations in a special form — but it can sometimes be useful to ask that testers send you their transcripts from game play. In fact, this is so frequently valuable that I recommend everyone do it. Not only does it make it possible to reproduce bugs that occur under finicky or rare conditions, but it means that you can see things your beta-tester might not actually report as bugs: places where phrasings didn’t work, where the tester was obviously having trouble, where he tried something that you might like to implement. I try to implement responses, as much as is reasonable, for whatever my testers try: any objects or parts of objects that didn’t have descriptions, any verbs that are suggested by the narrative, and so on. The Last Lousy Point in Savoir-Faire comes from an action that a beta-tester tried to perform, and I thought it was funny enough to implement.

Specialty verbs. Some authors add special verbs to their games to make annotation easier, and/or to dump the game state to transcript after a bug has occurred. If you’re using anything like this, you should obviously clue the beta-tester in about them.

Any important disclaimers. I sometimes begin testing the opening phases of a game before the endgame is complete. There are two reasons why: one is that it makes things go faster if I’m working against a competition deadline or something similar; the other is that sometimes tester feedback necessitates structural changes that are easier to implement if the game is not completely finished. Technically speaking, this is pre-beta-testing, and it requires extra understanding and forebearance from the testers — so I always try to let them know what they’re in for. If there are any bits of the game that are unfinished, I let the tester know that this is the case; I also try to put an ending in the game after the final (currently-solvable) puzzle, to indicate to the tester when he’s gone as far as he can go.

Do your best to pre-test the game.

Try not to send your testers something whose bugs are obvious to you even on a trivial playthrough. Even if you’re sending over a game which isn’t complete in every aspect, the parts you’re asking your testers to test should be playable: the puzzles should be (as far as you know) complete and well-clued, the prose should be (to the best of your ability) devoid of typos, and so on. You’ll be wrong about this, naturally, but you should try; there’s no point in asking your testers to tell you things that you already know, and you’ll be wasting their time and patience.

With that in mind, there are a couple of fairly trivial things you may want to check yourself before sending.

• If you’re not strong on spelling, dump your game text to a file and spellcheck it. Your testers may still turn things up that the spellchecker missed, but it will at least lower the number of trivial things they have to concentrate on.
• Check over your default responses, the ones you borrowed from the existing library. (Inform puts these in English.h.) See if there are any that leap out at you as totally inappropriate for your game. Likewise, if you use Inform and you normally use American spellings, be sure and set the constant DIALECT_US so that the library will be consistent with you.
• If you’ve added new verbs with new functionality, consider whether there are any reasonable synonyms you haven’t accounted for. Also, make sure that you’ve accounted for the problem cases: does the verb routine make sense when applied to an object that’s untouchable? out of reach? alive? unmoveable scenery? the player character herself?
• If you’ve added new grammar, is it consistent with the old stuff? One problem I’ve seen — more than once — occurs when the author has added grammar to handle an indirect object (>HIT MAN WITH STICK, say) but hasn’t changed the default response to >HIT MAN. So hitting the man with the stick may be the only solution to a puzzle, but >HIT MAN will turn up “Violence isn’t the answer to this one.” Severely misleading.
• Of course you’re not going to get everything. These are places where it’s particularly easy to generate errors, however. The less your testers have to fight their way through simple errors, the more time and energy they’ll have to devote to the trickier ones.

(You may also be interested in my more recent comments on preparing a game for testing.)

Ask questions.

Once you receive your report, feel free to contact the beta-tester about anything you don’t understand: the circumstances under which your game crashed the interpreter, the reason he doesn’t like your lead NPC, and so on. If he got bored, find out where he got bored. Try to avoid being defensive or justifying your decisions at this stage: you may have a good reason for doing things the way you did, but in most cases I at least find it’s actually counter-productive to convince the beta-tester that I’m right. What I want is to find something that will fix the problems the beta-tester experienced while not ruining my original concept.

Respond to reports.

Different authors have different approaches to this, but wherever possible, I like to respond in some way to the reports I receive even if I don’t have any questions — not just by saying, “thanks, I got it,” but by letting the tester know about any significant changes I made in response to his suggestions. I’m not sure I recall any cases where this was important to the development of the game per se, but I think it does improve tester-author relations. Obviously, it’s not always possible to do this in great detail, especially if you’re doing your testing on a tight, pre-competition schedule. But it’s nice. (I tend to just send my testers email or speak to them in person, where appropriate; another author for whom I beta-tested handled this aspect by sending all his testers a list of changes he’d made when he distributed out the second beta-release of his game.)

Don’t take criticism personally.

This is obvious, but important. Bear in mind that your testers are there to help you. They are on your side. What they say is in aid of your project. If you’re upset or discouraged by their reactions, that’s understandable — but rant to someone else, like your boyfriend or your dog or your journal. Don’t blame your testers for feeling the way they feel. If you really think that someone has criticized your work unreasonably, get second (and third and fourth) opinions on the same issue from your other beta-testers. This may help give you enough perspective to tell whether you should drastically rewrite something, or just ignore that tester’s advice.

Expect to do several rounds of testing.

Once you’ve fixed your first set of bugs, you’ll probably need to test your game again to make sure that you didn’t miss anything and that your changes didn’t themselves introduce bugs. (This happens a LOT. There’s no shame in it.) The more complicated your code is the more likely that you’ll need to iterate this process a number of times. Your original tester(s) may or may not be up for multiple testing sessions; if not, you should be prepared to ask for a second group.

Credit all testers by name.

This should go without saying, but I know of an occasion where it was forgotten and the omission caused some grief. If you’re releasing a competition game under a pseudonym and feel that revealing your testers’ names would be too obvious, you can release a post-comp version with the testers listed — but let them know that that’s what you’re going to do, if you can. Most people will be content as long as they feel their efforts have been appreciated. (Similarly, if you get bug reports after release that you then incorporate into subsequent releases, it’s nice to mention everyone who sent them to you. This will make it more likely that people will send feedback on your subsequent games, if any.)

Take responsibility for any bugs that remain in the game at release-time.

They’re not your tester’s fault, they’re your fault. Never ever blame the testers for missing things. Testing interactive fiction is a very difficult thing to do well, and more or less impossible to do thoroughly.

Cultivate the testers who work well with you.

Some combinations of tester and author just seem to click especially well. Also, some people are better at testing certain kinds of games than others. I have some testers who really like trying to break a simulation, and poke in all the corners looking for extreme cases to experiment with. I have others who like to play through the game as though they were an ordinary player, and then give detailed feedback on how the pacing, structure, and thematic development of the game succeeds. Both of these are incredibly useful to me — the former especially for a game with a lot of finicky materials systems, the latter especially for games with a lot of multilinearity where I might not anticipate the exact experience of Joe Random Player. When I’m inviting testers to work on a new WIP I have in progress, I tend to choose whom to ask first based on my knowledge of their testing style. This is something I couldn’t even have tried to do when I was writing my first couple of games (though, as it happens, I was fortunate enough to accidentally ask people who turned out to be extremely helpful). The more familiar you are with the community, though, the more likely you are to be able to select people who will have useful insights on your work — by, for instance, noticing what sorts of work they’ve themselves written, what they tend to say in reviews, and what other games they have tested.