A while ago, we wrote a story about how Naughty Dog had to specifically request that its focus group for The Last of Us include women. We all had a good laugh, a few of us got outraged at how the games industry treats females, and the rest of us went home satisfied that such a stupid mistake wouldn’t happen again. However, we all seemed to miss the point here. Focus group testing is -- in technical terms -- total bullcrap.

Think about what focus group testing is for a second. To help us, we have Professor Dictionary.com who says a focus group is, “a representative group of people questioned together about their opinions on political issues, consumer products, etc.” “Representative of what?” you may ask. That’s a good question. In the most basic focus group test, your focus group is meant to represent the population as a whole. Unfortunately, market research is a bit more complicated than that.

Say, for example, you are focus testing for new brand of baby wipes. You aren’t going to just take a random sample of the population, because the people who don’t have children aren’t going to purchase your product anyway. Moreover, people who do have children but don’t raise their children won’t give you good data either. Heck, people might have children, raise their own children, but always push the baby wiping on to their significant other. These people also wouldn’t be a good judge of whether or not your baby wipe is an awesome product or not.

So, in short, to make your focus group effective you have to narrow it down to only people who might purchase your product in the first place. So, in the case of the baby wipes, you are looking for parents that have to wipe their babies bottoms’. Unfortunately, this targeted marketing introduces several new problems into the mix.

First of all, focus groups can become much more targeted than they should be. This usually happens after several rounds of focus group testing. Eventually, you will whittle down your sample size past the people who “might” buy your product, to the people who “will” buy your product. For example, no one is surprised when a focus group made of shooter fans enjoys the latest Halo or Call of Duty, yet that’s what these focus groups are made of. As a result, their information becomes cyclical. They made things that their focus group likes because they focus on groups who like them. You aren’t getting any actual information from that. The only thing you have figured out is that you can successful find a small group of people who like your game, but whether or not that group of people is representative of gamers in general is a whole other story.

Second of all, focus groups cause genres and franchises to stagnate because they tend to appeal to gamers with only one set of preferences. If you show someone who likes the same old crap the same old crap they will like that same old crap and you will keep making THE SAME OLD CRAP! Even if your cross-sample of Call of Duty fans really does represent the majority of shooter fans as a whole, you aren’t going to get any other game information out of them but Call of Duty if they are the only people you ask. The real valuable information comes from the people who don’t like your game genre or IP. If you can get one of them to like your game, you are probably on to something. Unfortunately, these people are more often than not ignored and treated as lost sales to begin with.

Third of all, the people who you think might like your game, might not be the people who actually like your game. Like, who thought that the new My Little Pony was going to appeal to male 18-30 year olds? Similarly, if Spec Ops: The Line was given to just hardcore shooter fans, that focus group might come back exceedingly negative due to the flaws in its gameplay and the brutal slog that is its difficulty. But when shown to people who appreciate indie and art games, suddenly you get a much more positive reaction. You never know if you have a break out hit in a community you aren’t even marketing to. Maybe your baby wipes make good motorcycle polish? Who knows?

Finally, focus group testing turns us all into stereotyping jerks. Notice, when the baby metaphor was made, I never used the word “moms.” Why? Cause you might not be a mom. You might be a married or single dad. You might be a professional nanny with no kids of your own. You might be a nurse who needs the baby wipes to clean the butts of old people, I don’t know. These are all valid people who might purchase your new brand of baby wipes. Unfortunately, if you look at a baby wipe focus group, you are probably going to find middle aged moms. You assume that these are the people who want to buy your product because you are treating them as a stereotype and not actual people. If you keep believing that stereotypes are true, eventually the marketers holding gaming focus groups will believe the, “girls don’t play games” stereotype and soon we will see focus groups without any females in them … OH, WAIT!

That’s the big problem with focus groups testing; it becomes a self-fulfilling prophecy. Let’s say, for example, that the focus group for The Last of Us went through without including any females and they got positive results. They would make a game that only males like and it might do good, it might not. But if it did, then they are going to attribute the focus group testing to its success (at least partially). So they’ll make another game that satisfies that only male focus group. Then, they will find another focus group that is likely to enjoy the game they made. They’ll then be stuck in this loop of positive feedback without anyone stopping to consider, “maybe girls like games too.” Remember, focus group testing is supposed to provide you with a cross section of anyone who might buy your game. That means the people doing the focus group for The Last of Us, thought that no female anywhere would ever buy their game.

Maybe we shouldn’t be trying to make a game that “shooter fans” like, or that “guys” like, or even that “gamers” like, we should just try to make games that “people” like. Sure, you’ll never please everyone, but at least you’ll get information. And who knows, maybe you will please everyone. Maybe that 50 year old stay at home mom need to take a break from raising her kids to shoot some zombies. That’s information that you quite frankly would have never gotten if you listened to your focus group.

Angelo Dargenio is a freelance staffer at Arcade Sushi. His opinions are his own and do not reflect the opinions of the staff at Arcade Sushi or Townsquare Media. (He also may never get invited to focus groups.)

More From Arcade Sushi