Incarna Play Evaluation Model
Roles
- Coordinator: This is a game designer, project manager or designated representative of the Incarna Gaming Network. They will manage communication
and other specifc duties as dictated by their responsibiltiies between the group and Incarna interests.
- Tester: These are designated and assigned people who will be testing rules and materials as assigned by the Play Test Coordinator.
Initial Design
The final responsibility for good game play lies in the hands of the game producer.
Prior to any code or materials being written, the designer must look at the proposed game through
the filters of " game play" to assess its playability. This can be difficult if the
game designer, developers and producers are not the same party and do not have adequate measures
to enable a dynamic, fast response development process. If the initial game materials arrive
to play testers in a hap-hazard way or format, without any thought to playability prior to
initial testing, the design will be flawed and doomed from the start. Incarna went through a
year long concept phase wherein players and designers discovered that was important to players
and what approach designers should take to accommodate the priorities set by players. The proper
approach would allow the game a level of flexibility/adaptability for easy alteration.
Starting and Controlling the Test Process
Play testing will be done in two different ways: Observe players while using the materials (in-house),
and granting access to the game and elicit feedback via questionnaires and interactive online elements
(email, chat, etc.).
In house testing is very time consuming, and requires a lot of formal and involved direct observation
of players using the materials and playing the game of a prolonged period of time. The focus of this is
playability, and not market or demographic impact. The recruitment of in-house play testers in usually
accomplished by canvassing friends and known elements. This usually allows a reasonable degree of cooperation and
lack of tension as the participants are known and social interplay is reduced to elements which can be filtered out during test
result review. Using community (game community) resources (computer shops, game shops, college campus) such as news boards
can also be an effective means of finding play testers with like minded interests.
All potential testers should be questioned about past experience and games they like to play. People may enjoy
RPG games, but only within a strict set of guidelines such as a particular genre (sports, fantasy
or sci-fi perspective) or game technology or setting and may not be good candidates. While playability
does not focus on demographics, initial play testers should be pulled from the your most common demographic
element. Also, play testers must understand that in the case of testing, character development and the value of
their actual role playing skills is reduced, while the emphasis is stronger on technical knowledge, timeliness,
and effective communication. All play testers should be made aware of the commitment, preferably in a play testing contract,
prior to their acceptance of an offer.
Review Timing
A game needs to be stable enough that the play tester doesn't spend too much time noting bugs in the mechanics,
yet not developed to the point that changes cannot be made without affecting the entire underpinnings of the game.
A minimum of 3 months time should be allotted for each play testing cycle, as the availability of play testers
may vary. As much feedback mechanisms, and online surveys and questionnaires should be made available so that in-house play testing
can focus on important aspects during the time the group is together. Having more than enough people for a single play testing
group is usually a positive factor, as scheduling that many people can be difficult.
Testing Oversight
In-house testing should always have a a high ratio of developers and designers to testers.
These staff should be available to answer questions, observe, and note issues that come up in play for later review.
-
What materials cause the most confusion? Where do the testers ask for the most assistance?
This may vary by tester, if one person continually does not understand a concept or know how or where to find help, it should
be noted this is not likely to be a problem with the game materials but lack of adequate skills of the tester.
-
What kinds of features do the play testers have the most questions about?
Identify key categories of questions and ask if the layout, presentation, organization or complexity can be changed.
-
Do play testers get frustrated with the game easily? How closely does their frustration level relate to their skill level?
Benchmarks need to be established prior to bringing in the play testers; additional benchmarks will be added to as testing proceeds.
If a game scenario uses puzzles which directly relate to game facets, establish a minimum and a maximum amount of time for the play testers to solve each puzzle.
Timing tester answers and ability to find information needed for the game can be an important tool for evaluation.
-
Do play testers like the game?
On test is their ability to cite specific pieces of information about a setting or rules in the game that they liked.
Requests for more materials will give you an idea of a measure of success. If there is an aspect of the game that is near universally
disliked, a Quality Control meeting needs to be held and the assessment and impact of changing the aspect needs to be
measured against the usability test results.
-
If the play testers echo sentiments made during the earlier testing phase, and the items criticized were not fixed or changed, not enough attention has been paid to the testers.
Again, a Quality Control meeting should be arranged to discover where and why the process has broken down or the materials have not been changed.
Testing Goals
-
Play testing should provide the materials producer/owner with as much information as possible for making the necessary game play changes.
The designers and producers need to hear overall opinions which may shape the action or playability of the game.
-
The play testing coordinator needs to make sure that the groups have adequate time to review all materials assigned.
Oversight and communication between test groups and producers/designers are the primary concern.
To this end, they must make sure all the right tools are in place and working properly. The end goal for them
is to establish new and refine exiting processes to make sure this happens.
Standard Evaluation Guidelines
Testing Results
-
Game tweaks and changes. Information needed for the game to enhance or maintain its viability in the market.
-
White Papers: Out of play testing should come several position papers (called "white" papers) which clearly spell out decisions
made to persist game aspects, change game aspects and explain or justify the underpinnings of the game mechanics.
Guiding Principles
Although the general flavor of these principles is from the Play Test Coordinator perspective,
they are good general guidelines for testers to adhere to when dealing with other members of their own group.
Listen
Do not lead testers to solutions or opinions that they have not professed. Listen to what is being said and review
with other designers or producers. It is important to try and solicit answers to problems, like "What would you do instead?",
but do not provoke leading questions or answers until you have had a chance to discuss with others.
Do Not Be Defensive About Criticism
While there may always be those testers liking nothing and advocating a re-vamp of the entire game,
most play testers will have constructive feedback. Some of this may be negative. Not all of it
will have easy solutions. Do not mistake frustration with personal attack - it's easy to do if a tester
has problems and their patience runs low as a result of lack of response or general disaffection with
an aspect of testing or the game itself.
Give Clear Instructions
Make sure all instructions you give the play testers are clear. If possible, have documentation to distribute
or some other form of persistent (web pages, emails) information which can be referenced by the tester to
understand what is expected of them and the scope or goal of their involvement.
Do Not Waiver
In the face of the bombardment from play testing resources, do not change positions or play soft to multiple parties.
When expressing opinions or rulings, stand firm on positions and be able to back up the official position
with a knowledge of the materials and any documentation about them.
Do Not Perpetuate Legacy Problems
Just because it's difficult or a lot of work, do not discount feedback for testers on any aspect.
If past responses to change requests have been that it would take too much time, change the entire game engine,
or cause a lot of problems, that is no reason to not accept and review them as equally as any other.
A lot changes in technology and ideas from one year to the next, and a refactoring or engineering may not take as much
as previously thought.
Reward Testers
Testers provide the feedback for the product to remain viable in the market place.
Some form of reward is always necessitated. The exact form will depend on the closeness
of the group, the financial situation at the time and the specific desires of the individual play testers.
Offering reward options is usually better than granting everyone the same reward.
Teamwork
If every piece of the play test process is not running well, the results and responses
to them will be flawed. Much of the entire process boils down to all involved working well together.
Play Test Coordinator Responsibilities
Assign product and specific rule reviews. Manage play test group assignments.
Reporting summarized results back to game designers in a clear and concise manner.
The coordinator must consistently send out
notice regarding where and when the next session will be for each group. This may include what will be tested but
will certainly include what you will need to prepare beforehand and bring to the
session. This notice may state specific instructions (e.g. Play Tester A prepare a magus with Magic Aptitude 7
and SCH 55), freeform instructions (e.g. make a fantasy based character), or
something in between. Play testers should follow the instructions provided so that when the time comes
the session may begin without undo delay.
The coordinator will create and supply materials for feedback.
These materials will be made available online in the play tester section
of the Incarna web site or some other shared venue.