Statistics are Misleading, Especially for People who don't Understand Statistics

Steven Trustrum Commentary

I was recently involved with a discussion on RPG.net about the effects of piracy in the gaming industry. Within this discussion, some people made a habit of posting statistics and the like. As frequently happens when this is done, especially in relation to people trying to make an argument as to why people taking my stuff illegally is supposedly good for my business, I could only shake my head at how badly these people were misinterpreting the statistics (and not just the people in the thread, but the people writing the original articles being quoted.) I would not be unhappy if the Internet somehow gave people a mild electrical shock every time they tried using statistics in an online discussion without actually knowing how to make sense of statistical data.

Anyway, during this discussion some people started contacting me off-forum to debate some of the finer points of my responses, including to ask me what made me think I knew anything about statistical data and other information gathered during surveys and other means. The resulting back and forth inevitably brought up WotC’s 2000 survey (yes, it’s that old but is still the most extensive, if erroneous, instance of publically released market research into the gaming industry.) Rather than take on this other gentleman’s points on the matter, I merely directed him to the Misfit Studios website where I had posted my observations about said research years ago.

This is not the only recent instance I’ve seen of this survey still being used by people debating this or that topic about the gaming industry. As such, I thought my rebuttal to the research would make for a good blog post (and if you want more reading on the subject, I suggest you track down a post made 7 or 8 years ago by Ryan Dancey on ENWorld about other WotC research practices that I likewise took issue with.) Both the survey and Mr. Dancey’s ENWorld posts regarding WotC market research indicates how even the big companies sometimes get things wrong–especially when the results these studies produce miraculously turn out biased in favour of their products.

I think what a lot of people don’t realize about market research in the commercial sector is that, as often as not, it is used as a tool within the company to prove a theory or opinion to people in power, and thus are rife with purposeful bias towards the desired result, rather than actually looking for honest answers to honest questions. And sometimes even those seeking honest answers go about it the wrong way. That said, here’s the analysis of the WotC survey I wrote WAY back in 2007 after getting tired of having to repeat myself every time it was brought up.

My original examination of the survey

One of the things that constantly comes up in discussion of the RPG industry, especially with regards to its future, is the Wizards of the Coast market research survey of 2000. Whenever I see this pop up in such conversations, I’m compelled to point out how unreliable this particular study is and then need to explain why. Frankly, I’m tired of having to type the same thing over and over so I’m going to summarize the issues here so they can merely be referenced in the future.

First, let me state my qualifications for what I’m about to say. I’ve earned my primary income in the market research industry, working for various companies in various positions at various levels of involvement, for over a decade. Now, most people will say “that’s great and all, Steve, but Ryan was president of WotC, a much larger company, so obviously they know best.” My response is a rather emphatic “no, quite the contrary.”

If there’s one thing working in market research teaches it’s that executives, especially in larger companies, know pretty much zero about actual market research methodology. When they hand in a desired survey/research project to a market research company the latter is inevitably going to point out frequent biases and discrepancies that are going to cause issues with proper data gathering and analysis, usually brought on by the fact that the company has its own pet theories and the like that slips into their project design, resulting in questions that, by form or content, shall illicit responses that support and justify the pet theory rather than being designed with an unbiased, entirely open-ended purpose in mind (by intention or otherwise.) Often this is the result of the client company’s ignorance of how to construct a proper research project. As I deconstruct the following survey, you’ll begin to see some of what I mean and why I think this is certainly true of the WotC survey.

Keeping that in mind, I’ll state my observations are presented under the caveat that WotC states they’ve kept back some information, including the totality of their methodology, so I can only comment to the extent of what’s been released. That being said, let’s dive in.

http://www.rpg.net/news+reviews/wotcdemo.html

Adventure Game Industry Market Research Summary (RPGs) V1.0
Release Date: February 07, 2000

Summary prepared by:
Ryan S. Dancey,
Vice President, Wizards of the Coast;
Brand Manager, Dungeons & Dragons

Permissions: This file is Copyright 2000, Wizards of the Coast. This file may be freely redistributed or quoted in whole or part, provided that this attribution remains intact.

Methodology: Wizards of the Coast regularly surveys various aspects of the adventure gaming channel; distributors, retailers and consumers to better understand their preferences, concerns, and needs. That data is regularly reviewed and distributed internally to senior management. The contents of this file are excerpts from those sources; the source materials themselves
are confidential internal documents and are not available to the public. You have my assurances that to the best of my ability, the information presented in this document represents a fair and accurate representation of the data.

Sources: The primary source is a market segmentation study conducted in the summer of 1999. No confidential information provided by non-Wizards companies was used in the preparation of this report.

Exclusions: The internal information gathered by Wizards is considered an important competitive advantage. Therefore, not all the information available to Wizards is incorporated in this document, and there may be areas where substantial, significant information is purposefully not included. An effort has been made to ensure that the absence of any portion of this confidential information would not render the material provided herein inaccurate or invalid.

Pokemon Effect: As this study was conducted just as the Pokemon TCG phenomenon was gathering speed. For this, and several internal reasons, I have elected not to present information on the TCG component of the industry at this time.

Updates: From time to time, I intend to revise and update this file to reflect our ongoing efforts to understand the industry. When an update occurs, the version number of the document will be changed, as will the “release date”. Interested parties can write to me at ryand@frpg.com to request an up to date copy of this document.

Although not related to my issues with the survey’s validity, keep this part in mind when considering for yourself what parts to believe. This is an old survey that is often quoted by people to prove points in today’s market. Even if this was the ideal research project, conducted in the utmost perfection in 2000, the relevancy in 2007 (and onward) has greatly deteriorated on a quarterly basis, let alone annually. 7 years is a LONG time when talking about the relevancy of ANY market research results.

——————–

Section 1: The Segmentation Study

Since so much of this data is derived from the ’99 Segmentation Study, it is important that the reader understand how this data was gathered.

For the purpose of the 1999 study, the following methodology was employed:

A two phase approach was used to determine information about trading card games (TCGs), role playing games (RPGs) and miniatures wargames (MWG) in the general US population between the ages of 12 and 35. For the rest of this document, this group is referred to as “the marketplace” or “the market”, or “the consumers”.

This age bracket was arbitrarily chosen on the basis of internal analysis regarding the probable target customers for the company’s products. We know for certain that there are lots of gamers older than 35, especially for games like Dungeons & Dragons; however, we wanted to keep the study to a manageable size and profile. Perhaps in a few years a more detailed study
will be done of the entire population.

To be clear, choosing an age range for one’s study is common in research, but it is usually derived from more carefully defined data. The admission that it’s an arbitrary age range (wide though it may be) leaves a great potential for skewed data, especially in a market most publishers agree has an aging consumer base. It isn’t 100% certain this is going to be a problem, but it does open a wide door for possibly flawed results, as it cuts out a lot of people who could still be gaming after joining during the hobby’s early days, let alone older people who came to the hobby late and whose spending habits will vary from those of younger adults, etc.

Information from more than 65,000 people was gathered from a questionnaire sent to more than 20,000 households via a post card survey. This survey was used as a “screener” to create a general profile of the game playing population in the target age range, for the purposes of extrapolating trends to the general population.

There are several problems here.
1) Earlier Ryan stated the data is gathered from “distributors, retailers and consumers” … although the above statement seems to indicate the data presented here excludes the distributors and retailers, it’s not exactly clear. If the data is solely consumer based, then the statement regarding distributors and retailers is wholly irrelevant and confusing. If the data is indeed from mixed sources, then we’ve no idea how the “households” break down between retailers, distributors and consumers. (Personally I believe it is the former issue and not the latter, but I can’t be 100% certain based on the way it’s presented.)

2) There are major issues to be found in gathering 65,000 data samples from 20,000 “households.” [First, a side comment: Although I’m sure these numbers are rounded off (normally a HUGE no-no when presenting an analysis like this) rather than being EXACTLY 65,000 and 20,000, they are the only numbers we have to go with so I’ll speak in the same terms as the research without it really affecting what I have to say.] This means that, on average, 3.25 people were surveyed per household.

To accurately present a typical consumer, only one person per household should be sampled in most consumer research of this sort. There are exceptions, especially of the sort where certain group dynamics are relevant, but most of the data presented here is not of such nature. If the survey was asking about what goes on during a game session, for instance, asking 3.25 people in a house may prove relevant, as it’s likely those people game together. Asking people in the same household to answer questions about buying habits as part of the same research creates flawed results because budgetary concerns and purchasing interests are more likely to overlap from one to the next in cases of the respondents defining their purchase habits from a combined cash flow (such as a married couple with a shared bank account and budget, for example.)

To explain what I mean, consider this example. Car Manufacturer A (CmA) wants to do research on who buys its new mini-van and why. To do so it contacts households and asks them about their reasons for buying their particular mini-van (not necessarily the one that CmA manufactures) to get an idea of what people want. If they call and get the wife, taking her information, they have completed one result and now know one data set for their project. If they were then to ask to speak to the husband and asked him the same questions they would then introduce a second data set into the results that should be identical to that which had been provided by the wife. Because the results will be reduced to mere numbers for the sake of statistical analysis, it will appear as though there are two people, with no relationship, who own the same mini-van for the same reasons, etc. This weights the results towards households rather than general trends and individuals (and also allows the results to cater to pet theories by asking to speak to more people from households that present data that support preconceived notions) and misrepresents the sort of accuracy you’d get by limiting your results to one person per house.

So, when looking at WotC’s data we see an instance of 3.25 results per house. While it’s entirely possible (even likely) that those 3.25 people own different games (say, Bob likes Werewolf whereas his sister, Jane, doesn’t and only owns BESM, which Bob hates), there is an incredibly noteworthy chance that purchasing trends will be affected. Think of your own gaming group: if one of the other regulars owns all the splat books for your game of choice, the chances of your own need to buy them likely decreases. (This is supported by publishers reporting that splat book sales never reach the amount of sales shown by core books.) So, if each of the 3.25 people in each gaming household are looking towards the game collections of the other 2.25 it is safe to say that there are some decisions being made along the lines of “I don’t need to get that book because he/she already has it and I can easily borrow it from them,” whereas if the results were gathered on the basis of one result per household we wouldn’t see the data sets overlapping, nor would we see the issue of intermingling purchasing habits within a household affecting the gathered information. Instead, we’d have one person speaking on behalf of those 3.25 people.

This “screener” accurately represents the US population as a whole; it is a snapshot of the entire nation and is used to extrapolate trends from more focused surveys to the larger market.

How is it an accurate representation of the US population? Aside from the whole “3.25 respondents per household” issue previously brought up, how was the sampling weighted versus population densities and regions? How many were from City/State A and how many were from City/State B, for instance? How many are urban and how many are rural respondents? How does the consumer sampling correlate to the concentration of consumer outlets for purchasing gaming materials? (In other words, how do we know how much of the data was taken from people with easy access to gaming materials and gaming groups versus those people who don’t have quick, easy access to the ability to purchase game materials or get together with fellow gamers?)

Because we know that the respondents are already weighted to 3.25 respondents per household, seeing how this is further weighted by location is incredibly important because any flaws don’t skew the data by an instance of 1, as would be the case on a one person/household survey, but rather by an instance of 3.25 each time such a discrepancy enters the data because the location data the information was gathered from isn’t balanced. This is yet another reason why the data should have been gathered from one person per household. Period.

[NOTE: I’m hoping none of this survey is based on information gathered from those post cards inserted into some of WotC’s game books because, even as a pre-screening point of contact, such a method means there was absolutely no control over how the sample was initially gathered, by person (meaning there’s nothing to stop someone from sending in more than one post card) or region (meaning the post cards are merely accepted based on who bothers to send them in rather than a properly weighted representation coming in from various regions of the US.) It also means the general population wouldn’t be drawn from for the sample, but rather only WotC customers who purchased their products with said post cards in them would be used in the sample. There’s no indication this is the foundation of this particular survey, other than stating that “post cards” were used for the questionnaires, but I thought I’d throw this out as something you should consider on your own.]

A follow up survey was completed by about a thousand respondents from the “screener”. The follow up survey is an extensive document with more than 100 questions. The particular individuals chosen to participate in this expanded survey represent the population, as determined by the screener. In other words, the small detailed survey group can be reasonably extrapolated to the larger screener group, and the larger screener group can be logically extrapolated to the public in general. This is a common, standard, and accepted methodology within the market research field.

Clarification: It’s a common and accepted methodology IF the data a secondary screener is drawing its respondents from can be considered valid.

As I’ve pointed out, there were a lot of problems with the original 65,000 sampling, so the ability to say the secondary grouping of 1,000 respondents is an accurate representation of the US market is compromised to a great degree.

It’s also important to note that no information is provided on how the 65,000 sample was distilled to 1,000. Without that information, we have to fully trust in the fact that the process was done correctly. Considering my issues with the survey to this point, I’m not really willing to put my faith in that.

The data from the detailed survey was collated and prepared by the Wizards market Research Department, in conjunction with an external consulting firm. We believe that the data is a fair and accurate representation of the hobby game consumer profile and that it does statistically correlate with the population as a whole in the US for the target age bracket.

As I stated earlier, in most instances a company’s internal market research department usually knows very little about how to conduct proper market research. Truly, it would make you weep to hear some of the stories I’ve heard over the years about how badly companies that are so big that their decisions radically affect global markets screw up their research and shift their business policies based on those results. So, that’s why companies that are solely dedicated to market research are brought in to lend a hand, but my confidence that the one WotC hired did its job properly is not at all high based on the errors regarding rudimentary methodology seen thus far.

Such departments are almost always used to prepare the research on the client company’s end, and the research company hired then points out concerns, such as questions that will likely provide skewed results or will appear confusing to respondents (and thus likely result in flawed answers.) They are also usually the ones who take the results provided by the market research company, analyze them, and package them for internal use and presentation within the client company. They don’t do the field work and it is from such departments in the client company that errors or flaws in the research are most likely to be introduced.

——————–

Section 2: Basic Terms

As a part of the detailed survey, the following terms and examples were provided to the respondents:

Term Example

(*)Paper RPGs Dungeons & Dragons
Card Games Bridge, Solitaire,
Uno, Poker
Trading Card Games Magic, Pokemon
Word/ knowledge Scrabble,
Trivial Pursuit
Puzzle computer games Tetris
Non-competitive problem solving Sim City, Myst
Puzzle table games Jenga, Dominoes
Class board games Chess, Monopoly, Go
Action/Shooter/Arcade Doom, Mortal Kombat
Miniatures table-top fantasy/sci-fi Warhammer
Games that use miniatures Battletech
War games Historical
Simulations Flight/car
Simulators
Strategy games Risk, Civilization
Social/party games Charades,
Pictionary
Strategic sport simulations Madden, MLB
Other non-sport games N/A

Specific questions were also designed to separate users of “computer Role Playing Games” vs. “paper Role Playing Games”.

(*) For my own purposes, I choose to use the term “Tabletop RPGs” in this document; the term “paper RPGs” was used in the study. The terms are synonyms; my choice is simply personal. I believe that in the fairly near future “paper” RPGs will hybridize with computer assistance – not becoming “computer RPGs” as that term is commonly understood, but not being games played simply with paper anymore either. Consider this a “forward looking” terminology.

As before, Ryan’s irrelevant side commentary introduces doubt in the unbiased nature of the survey and its results.

Again we see an issue with Ryan’s preferences coming into play with the survey results. Is it relevant to the survey, which should be a presentation of hard data, what Ryan prefers? Did this preference influence how the survey was designed? Was it biased at all towards his belief regarding paper/computer hybrids? We simply don’t know. We can’t see the original survey so we have to hope that it didn’t and that all we’re seeing is hard data, but we just have to take this on faith.

In other words: Ryan should not be stating his personal opinions in this survey, a survey that is supposed to present material wholly independent of the opinions of him and every other person at WotC.

The term “D&D” is used herein to describe all flavors and types of D&D play; from old “white box” players up to people playtesting 3rd Edition.

Okay, here’s another point of concern. If playtesters of 3.0 were included in the dataset there is the possibility of a brand bias being used because we don’t know if these playtesters just so happened to be contacted during random sampling or if, WotC already having contact information on these people, WotC dropped a message in the playtest email list (or whatever) along the lines of “hey, we’re running a survey. Any of you guys and gals want to participate?” Again, there’s no proof the latter happened but I am concerned with the possibilities brought up by the above statement.

So, why would this even be a problem? Because it would mean that this survey isn’t based on a sampling of gamers at large. It would mean a sample skewing towards WotC’s own customers (granted this happens anyway considering their market share, but it certainly doesn’t need any more help in this regard) because it’s highly doubtful other publishers gratned WotC equal access to its own playtesting groups so that WotC could make the same offer to them. In short, it would increase the chance of WotC customers carrying more weight in the final data gathered.

——————–

Section 3: Basic Demographics

The study provides the following information about the basic demographics of the tabletop RPG marketplace:

Size: 6% play or have played TRPGs (~ 5.5 million people)
3% play monthly (~ 2.25 million people)

Gender: 19% are female (monthly players)

Crossover: 17% of the total play MWGs monthly
46% of the total play computer RPGs monthly
26% of the total play TCGs monthly

The study provides the following information about the basic demographics of the computer RPG marketplace:

Size: 8% play or have played CRPGs (~7.3 million people)
5% play monthly (~4.5 million people)

Gender: 21% are female

Crossover: 33% of the total play tabletop RPGs monthly
21% of the total play TCGs monthly
13% of the total play MWGs monthly

The study provides the following information about the basic demographics of the MWG marketplace:

Size: 4% play or have played MWGs (~3.7 million people)
2% play monthly (~1.8 million people)

Gender: 21% are female

Crossover: 37% play tabletop RPGs
40% play computer RPGs
29% play TCGs

The age breakdown of players within the marketplace is:

Age TRPG MWG CRPG All Gamers(*)
12-15 23% 27% 23% 11%
16-18 18% 17% 16% 7%
19-24 25% 24% 23% 13%
25-35 34% 32% 37% 29%

(*) “All Gamers” means people in the study population who reported playing >any< of the game types monthly, not just TCGs, RPGs, MWGs or CRPGs.

And here’s where we see my previously mentioned issues with sampling 3.25 people per household rearing their heads.

If those 3.25 people are all playing tabletop games together, for instance, but don’t play CCGs because one of them refuses to play anything even resembling Pokemon, that skews the data towards tabletop gamers and away from CCGs rather than showing a more “pure” result.

The 2000 US census also shows 50.9% of the American population is comprised of women (http://www.census.gov/prod/cen2000/dp1/2kh00.pdf) but we have no idea how many of the original 65,000–or final 1,000–respondents are women. To be an “accurate snapshot” of the American market, 50.9% of the respondents in both groups would need to be women. The way the data is presented indicates that, most likely, this was not so and results by gender were “gathered as they fall,” meaning information was taken as it came in and looked at afterwards, pointing at the incidents of response by gender as indicators of how the market broke down by gender. This is problematic because a study looking to see what percent of the market is female should have been done seperately from one looking to examine the overall market’s purchasing trends. This is because looking to find a gender demographic breakdown within a market should employ a “gathered as they fall”, open methodology using weighted, random sampling whereas examining market trends should use weighted gender requirements (meaning that, if a previous survey showed 21% of the market was female then 21% of the answers should be from females, even if they are harder to find and male respondents need to be turned away if data is already completed for their 79% of the research.) This is because methodologies necessitated by these two results are almost certainly in conflict if the results are to be considered reliable, and so combining them into one study is almost certainly going to provide a self-fulfilling result.

The most obvious way I could see WotC getting around this conflict is using the 65,000 original sampling to find out how much of the market is this or that gender and then using the results of that to determine how many males and females should be included in the second group of 1,000. This is, again, affected by the problems already mentioned regarding how the sampling was undertaken, along with the fact that this would be a guess on my part because Ryan doesn’t differentiate between the information gathered from the original 65,000 and that gathered from the 1,000 to follow.

Yet again, all we can do is hope for the best.

Conclusions:

1. Few “General Gamers”:
The first, most notable conclusion we can draw from this information is that the mythical “hobby gamer” who plays TRPGs, CRPGs, MWGs and TCGs comprises a very, very small portion of the total market. A minority of gamers play more than one category of hobby game; very few play all three. The largest overlap, though still a minority, is with CRPGs and TRPGs.

Let’s return to my previous concerns regarding the hypothetical Bob who, along with the other 1.25 people in his household, likes TRPGs, CRPGs, MWGs, and TCGs, but his sister Jane only likes CRPGs and TRPGs, and thus as a group this is all they can agree to play. We see in this example how sampling more than one person per household influences the results because, despite contrary interests, Bob and Jane would likely answer that they play CRPGs and TRPGs, and not MWGs and TCGs because that’s all the 3.25 of them can agree on as a group brought together by circumstances, even though the resulting data will portray them as individuals without relations of any kind expressing the merit of their personal preferences.

It would be different if the information stated what these people BOUGHT or were INTERESTED in, as we know gamers also collect game materials they may not necessarily get the chance to play but still find interest in, but note how the data states what people PLAY. If the question was phrased the same, some people may assume that “play” really meant “what do you LIKE” while others took it literally and excluded their interests and restricted their answers to what they actively played. Adding a reasonable time frame would be a good idea, as well.

For example, if someone asks me “Steve, what games do you play?” I’d likely think in a context of recently (or even currently), and would exclude games I own but haven’t touched in a few years, even if the completionist in me is still seeing me buy products in those game lines. I don’t imagine I’d be the only person who would think this way if asked this question, whereas other people would think in a context of what games they’ve played EVER or what games they haven’t played but would like to. Considering the wide possibility of gamer interpretation of such a question, a timeframe/context within the question itself would be necessary for clarity’s sake.

This is a good example of what I was talking about when I mentioned the sort of thing a market research company should catch and point out to a client because the language the client uses within it’s own industry (WotC may commonly use the word “play” as a synonym for “interest” around the office, for example, which isn’t necessarily true for the people surveyed) isn’t necessarily the same language/turn of phrase the respondent is going to view it in.

In either scenario (the issue with responses from 3.25 people/household representing collective play habits rather than that of individuals versus confusion over use of the term “play”) there are problems that can affect the data. The chance of the respondents navigating these two issues to result in data without bias or errors is incredibly unlikely, to say the least.

This is an exciting conclusion, because it indicates that a company can successfully create brand in one of the three hobby categories, and extend that brand into the other two without significantly cannibalizing sales. In other words, the people who buy the RPG are not likely to be the ones buying the MWG or the TCG.

And here’s where we see the sort of conclusion a client company will draw from market research results.

Considering all the problems brought up at this point, I hope you see how it can be bad for a company to make conclusions like the above based on data that has problems with how it was gathered. Sure, there’s a chance the conclusion may still have some validity, but that doesn’t mean it’s necessarily so. As they say, even a broken clock is right twice a day.

2. There are “Women in Gaming”
Second, it is clear that female gamers constitute a significant portion of the hobby gaming audience; essentially a fifth of the total market. This represents a total population of several million active female hobby gamers. However, females, as a group, spend less than males on the hobby.

I don’t think anyone disputes there are women in gaming. I think most people would be shocked to find out that 1 in 5 gamers are women, however. As I stated before, this data is likely flawed based on equally flawed data gathering methods regarding defining gender demographics.

3. Adventure Gaming is an adult hobby
More than half the market for hobby games is older than 19. There is a substantial “dip” in incidence of play from 16-18. This lends credence to the theory that most people are introduced to hobby gaming before high-school and play quite a bit, then leave the hobby until they reach college, and during college they return to the hobby in significant numbers.

There are BIG problems with this part of the study.

The sampling was, by WotC’s indication, arbitrarily taken from people ages 12 to 35. We don’t know if the 65,000/1,000 respondents were weighted equally across this age range or selected at random. We need to know how the sample was broken down by age to believe any of these results, especially if we don’t know if the age and gender results were using cross-referencing requirements (meaning their 21% females were used to ensure each age group was comprised of 21% females.)

Here’s another example to see why this is so important. Let’s say we’re looking into a parking lot of 100 automobiles, broken down into groups of 10 cars of 10 different colors, one of which is red. There is no color with more cars than any of the other nine colors, correct? But we can look at those cars and form the conclusion:

“90% of the cars in the lot are not red, so clearly red is not a popular color.”

The first part of the sentence is empirically true but the latter part is an objective statement that ignores the remaining data (meaning it ignores the fact that all the other cars of different colors can just as easily and factually replace “red” in the above statement.)

We don’t know how the colors were chosen, nor do we know if the 10 cars/10 colors ratio developed on its own or was artificially enforced (meaning an 11th car of any one color was turned away so there will always be 10 cars each of 10 different colors.) We don’t know, so we can’t really relate what we’re seeing in this parking lot to the overall consumer market for cars and the colors people choose to buy them in.

The same goes for the age breakdown. Clearly, when talking about a sampling of 12 to 35-year olds the group of 19 to 35 is much bigger than the 12 to 18 group, so if the sample was divided evenly amongst the ages rather than using the “gathered as it falls” method there’s obviously achance for results that skew towards the larger age bracket.

I also have a problem with employing opinions about “going to college.” How do we know they’re going to college and not just joining the job market, and thus gaining more liquid income? While the results indicate many of the respondents have post secondary educations, we again don’t know if this is a natural result found during random sampling or is the result of flawed sampling methods (such as more answers coming in from college towns than not, and not being filtered by qualification demographics, for instance.)

It may also indicate that the existing group of players is aging and not being refreshed by younger players at the same rate as in previous years.

A statement made in the absence of any data regarding “previous years.”

——————–

Section 4: The Role of Computers

There is an intense, ongoing discussion between publishers and customers about the use of computers and the interaction between computer game play and adventure game play. The market research study presented some revealing insights into this ongoing debate.

Internet Gaming: 51% of the TRPG players report that they have ever played a game on the internet. 28% report that they play an internet game monthly.

% Who want to buy software to help manage game and speed up combat: 52%
% Who want to play D&D over the internet with others: 50%
% Who read newsgroups, mailing lists and web sites: 37%
% Who currently play with computer assistance: 42%

What computer do gamers use?
Wintel Platform: 63%
Macintosh Platform: 9%

(The question was essentially “What platform have you used in the last month”, and “none” was an option, probably accounting for the missing percentage.)

What’s sitting at home?
Wintel Platform: 54%
Macintosh Platform: 7%

Three quarters of the sample use the Internet at least once a week, but only two thirds have access from home.

“Who plays electronic games?”
Computer Console/Handheld Both
Average Age: 26 23 20
Education
% 6th-8th: 5 20 27
% 9th-12th: 23 52 37
% College: 53 26 31
% Post Grad: 20 2 5
Marital Status
% Single: 52 65 76
% Partnered: 46 29 22

Games electronic gamers play monthly:
Computer Console/Handheld Both
% TRPGs: 72 54 57
% CRPGs: 44 21 50
% Puzzle Comp: 39 41 49
% Classic Board: 39 48 44
% Action/Shooter: 32 55 61
% Simulations: 25 36 40
% Strategy Games: 26 26 32

One conclusion we draw from this data is that people who play electronic games still find time to play TRPGs; it appears that these two pursuits are “complementary” or “noncompetitive” outside the scope of the macroeconomic “disposable income” competition.

This data suffers from many of the problems I’ve already mentioned. I will comment on this part because it’s one that is often quoted in discussions regarding this survey’s relevancy.

The computer industry, especially with regards to RPGs, has changed dramatically since 2000. Indeed, anything to do with electronics and computers has changed dramatically since then. Although it’s possible these numbers haven’t changed, it’s incredibly unlikely, especially when one looks at the proliferation of online RPGs and online gaming communities.

——————–

Section 5: Tabletop RPG Business

We asked questions of people who play TRPGs to get a better and more detailed picture of that category. This section explores some of that data.

The market research study provides some useful information on the games TRPG players play when they’re not role playing:

51% play a non-TCG card game monthly
43% play a puzzle computer game monthly
43% play a classic board game monthly
58% play an “action/shooter” computer game monthly
41% play a “simulation” computer game monthly

The >least< played game types were: 26% play a TCG monthly 24% play a puzzle table game monthly 17% play a MWG monthly 17% play a social/party game monthly When asked how likely a person was to be the DM/GM, the responses were: 2+ Sessions as DM/GM: 47% Don’t DM/GM: 41% When asked to describe a variety of past game experiences, the market provided the following data: Question: Result Used detailed tables & charts: 76% Included Miniatures: 56% Used “rules light” system: 58% Diceless: 33% Combat Oriented: 86% (*) Live Action: 49% House Rules: 80% (*) Looked at in reverse, this interesting answer tells us that 14% of the gamers who play an RPG >have never played< a combat oriented RPG.

Again, aside from previously mentioned problems we need to keep in mind there are a lot of differences between 2000 and 2007, especially when one considers the changes brought on by the introduction of the Open Gaming License, something that caused massive changes throughout the entire industry and market.

Of the people who reported playing a TRPG, we further screened for people who played D&D and asked those individuals some more detailed questions. This data comes from people who have played D&D, not necessarily those who play monthly.

Here is an obvious problem. It shouldn’t surprise anyone that WotC is looking out for its own branding (who wouldn’t?) but combining such an interest in a survey meant to also find general information about the market will run into trouble.

For instance, because WotC also wants to gather info on DnD, how do we know they didn’t say something like “500 of our 1,000 people in the second group need to play DnD”, meaning people with otherwise valid and relevant data were rejected from the final 1,000–which was also used for general industry results–on the basis of getting DnD specific data. This would mean the data gathered from the sampling wouldn’t be “people who game in general” but rather largely “DnD gamers who may also play other games.” The distinction is incredibly important.

Just how and where the lines of bias are drawn cannot be seen, considering the lack of clarification in WotC’s survey presentation, but the fact that ANY branding-specific data gathering was combined with general data gathering introduces, by DEFINITION, a WotC-based bias to any data gathered, even if we consider the obvious existing WotC branding presence in the market at large.

Age:less< likely to quit playing D&D the longer they play, not >more< likely.

1-5 Years >5 Years
Expect another Year: 40% 75% 88%

We asked what the frequency of play was:
Total D&D 1-5 Years >5 Years
Monthly: 7.2 4.9 13.2 5.9

So we see that the longer a player is in the game, the fewer times per month they play after the 5th year. Once the “acquisition” period (1st year) has passed, frequency of play accelerates tremendously, then drops. One explanation for this fact may be that since acquisition happens most often at age 15 or less, “new players” may have a lot of time available for gaming, but as they age, they have less time per month to play.

We looked at a few other questions based on how long a person had been playing the game:

[ if this chart gets mangled in the formatting, it has three columns of data ] Typical 4 or More Average Sessions
Session Gamers In before Restart
5+ Hours Group (New Characters)
Total 28% 62% 15.4
1-5 Years 14% 60% 12.9
(*)>5 Years 42% 71% 19.6

(*) Remember that frequency of play is down sharply for these gamers)

This data tells us that the longer a person plays the game, the longer the game sessions get, the more people play in the game, and the longer the game progresses before a character restart. In fact, if you look at the >5 year group, you realize that the big jump in long sessions and in average sessions before a restart means that the 5+ year gamers are playing the same characters, on average, vastly longer than anyone else.

One conclusion might be that it takes 5 years for a player to really master the system and really figure out what kind of character that player likes to play.

All the previously mentioned problems aside, many people quoting this survey usually don’t notice that it’s DnD oriented and present its age-specific trends as industry-wide trends. Just because DnD may appeal to this or that age group doesn’t meant the same can be said of other games, which are likely to employ different themes, genres, and game mechanics. This is a flaw of the reader, however, and not WotC (well, beyond concerns already discussed), as the latter has clearly stated they’re talking about DnD.

The following financial figures are for TRPG players in general (D&D information, where available, is provided as well)

This data seems to validate the theory that young gamers, while very active, don’t spend a lot of money. (The following data is reported by for RPG expenditures) The big dollars come from adults…

Total spending by age:
12-17: $297
18-24: $850
25-25: $2,213

And, the longer they stay in the category, the greater their total outlays…
Play5 Years: $2,502

And if they can be induced to become a DM/GM, expenditures skyrocket.
Will DM/GM: $2,048
Will not DM/GM: $401

Some breakouts for the D&D population in particular…

Total D&D spending by age:
12-17: $164
18-24: $443
25-35: $1,642

Monthly D&D spending by age:
12-17: $10
18-24: $12
25-35: $14

Total D&D spending by time in game:
5 Years: 1,756

Monthly D&D spending by time in game:
1-5 Years: $22
5 Years: $16

(Interesting note: Monthly spending in the first five years after adoption of the game is higher than the spending beyond that point – though the older, longer gamer plays the game more, they spend less. This may relate to the frequency of a character/game restart.)

D&D DM willingness effect on expenditures:
Will DM: $1,444 total / $21 monthly
Will not DM: $187 total / $7 monthly

(Interesting note here: Even people who don’t DM buy a heck of a lot more than just a PHB…)

Effect of miniatures addition to RPG mix:

Few miniatures owned/used: $139 total RPG spending
Many minis owned/used: $4,413 total RPG spending

This is a good example of how even information that has been gathered improperly has the possibility of being right, usually because the trends being examined are so pervasive that they are going to stand out no matter how the sampling was done. For instance, it shouldn’t surprise anyone that the DM, being the person responsible for preparing and running the game, spends more money on game material than those people who refuse to DM (and thus only need to buy what is needed for their specific characters.)

[NOTE: another no-no. “DM” is very tightly branded to DnD and shouldn’t be used in a general survey as it will likely subconsciously bias responses due to its association with the DnD brand. The general term, “Gamemaster,” would have been far more appropriate. It may seem a minor thing, but it really can cause problems because it compromises any sense of neutrality.]

If that question is meant to be DnD-specific, then people who quote it need to realize that because it is yet another section people tend to quote as generally relevant. This, ladies and gentlemen, is why properly represented market research doesn’t mix it’s data on various points in the same section.

We found that players who were ‘lapsed’ – reported that they had played TRPGs but were not currently doing so; had spent more money than the current players, and had played more different games monthly – but interestingly, they had spent less money, on average, on D&D than players who were “current”.

(Current/Lapsed)
Mean RPG Spending Mean Total D&D Number
Spending RPGs Played
$1,273 / $1,667 $895 / $599 2.2 / 3.3

How does the information regarding spending by age cross over with the spending trends by current and lapsed gamers? This is an incredibly important thing not to have pointed out because where one is in one’s life, and thus the time and money to spend on games, is highly tied to one’s age and career. It also makes the information regarding current/lapsed gamers rather difficult to do anything about because one can’t track the trends regarding customer entry and exiting of the market.

One conclusion that could be drawn from this data is that gamers who don’t like D&D will spend a lot of money and try a lot of systems to find something they do like before they quit. Gamers who like D&D will spend less money and try fewer systems, but will spend more on D&D than those who don’t.

Point of interest: this is a conclusion we can draw only if we accept there were no flaws in how the data was gathered and that the use of the survey to gather both general gaming info and DND-specific info doesn’t create a bias. This conclusion can only be trusted if it is a natural result of the general data (such as asking “what games do you like?” and finding lots of DnD results) rather than being influenced in any way by the inclusion of DnD related questions.

When asked why a gamer lapsed, the answers (multiple choices allowed) were:

Got too busy with other things: 79%
Too few people to play with: 63%
Not enough time to play: 55%
Found a game I liked better: 38%
Unhappy with the game and the rules: 38%
Cost too much money: 32%
Burnt out from frequent play: 29%

Another instance of a breakdown by age would be incredibly relevant. A lack of an “other” option is also important to note.

Getting back to the people still playing the games, when asked what games TRPG players play monthly, the answers (multiple choices allowed) were:

D&D: 66%
Vampire: The Masquerade: 25%
Star Wars: 21%
Palladium: 16%
Werewolf: The Apocalypse: 15%
Shadowrun: 15%
Star Trek: 12%
Call of Cthulu: 8%
Legend of the Five Rings: 8%
Deadlands: 5%
Alternity: 4%
GURPS: 3%

Issue #1: Palladium is a publisher, not a game, so WHICH Palladium game? Issue #2: Which version of Star Trek, considering the license has been owned by more than one company? Issue #3: considering the % total is higher than 100%, we know this question allowed more than one answer per respondent, so why then was “other” not an option? Obviously there are more games out there than just those presented, so we’ve no idea if it’s simply a matter of an “other” option being available but nobody filling it out (HIGHLY unlikely) or WotC biasing the data by only providing the above options as possible answers.

So, the confusion between publishers and games, along with the restrictions of limiting the possible answers makes this information entirely worthless other than to see what we already know: regardless of how badly the data is messed up, we can still see that DnD is the industry’s major player (and who really needed this flawed question to tell us that?)

When asked to describe aspects of their games, on a scale from 1 to 5, answers were:
(normally/rarely)

Create Own Adventures: 42% / 11%
Create Own Campaign Material: 29% / 17%
Replay Adventures: 18% / 35%
Use adventures from magazines: 21% / 40%
Follow official D&D Rules 33% / 17%

Confusing point: are we still talking about the market in general or, by nature of the last line of data, are we back to talking about DnD specifically? If the former then the inclusion of DnD creates a bias in the general data. If the latter then we’ve no idea if this is true of games other than DnD, games that may be designed in such a way that engenders them to entirely different trends than those mentioned above.

When we asked RPG purchasers how many had purchased D&D at a particular retail type, the answers were:

(*)Hobby/game shops: 36%
Book Stores: 27%
Comic book stores: 18%
Specialty toy and game: 17%
Large toy store chains: 15%
Conventions: 4%

Again, there’s a noteworthy absence of “other.” What about gifts? What about such things as department stores, online purchasing, and mail order? The latter two are especially more relevant now than in 2000, which again shows us how dated this survey is, flawed or not.

We also have to note that it only speaks to DnD purchasing habits and doesn’t address gaming in general, as many publishers don’t have the distribution resources to sell in places like comic book or toy stores, go to conventions, or deal with the hassle that is the book trade.