Griefers in Virtual Research and the Truth Serum

Materials related to this program are now compiled here.

On this week's Metanomics panel, Mary Gordon of Market Truths Ltd. outlined the six psychographic segments she uses to track marketing trends in virtual worlds. These include team players, connectors, chameleons, apprehensives, entrepreneurs, and competitors.

I asked Gordon to speak to the methodological issues associated with psychographic research in Virtual Worlds. Specifically, I was curious about her approach to collecting data and how she ensures that people are truthful in their responses. Eliciting truth-telling in online surveys is a common challenge of research conducted in virtual worlds, and is the subject of a current discussion on Terra Nova started by Thomas Chesney about Pittsburgh economist John Duffy's recent working paper criticizing my research. Gordon outlined some excellent practices in collecting useful and reliable data in virtual surveying:

It's certainly something we've worked on for a long time. We've been perfecting... methods for detecting liars and cheaters on Virtual Worlds for a couple of years now, so we're definitely getting better at it as we go and more confident in our ability to do that.

We have our own research panel within Second Life. And, for that we, first of all, require that people had been in Second Life for at least 30 days. Second of all, [we] require that they have a verified account. So that already reduces some of the potential--you figure, if you're going to be griefing and that kind of thing, you're probably not having a verified account so that controls some of that. We also do a lot of quality control checks on our panel to check for exactly the kinds of things that people were asking about. We don't want people that are going to be untruthful with us. It's a bit like virus detection, in the sense that I don't want to tell you too much about that because it would make it easier for people to do it in the future. But we run a whole lot of different checks on our panel, and then, for each time we collect data for a new project, we run some additional checks.

For this particular project, we'd be sampling from that panel so we don't have anybody anyway. We're doing a bunch of different samples so different people in each sample. And we're looking to see if we're getting the same results across samples because that would be another indication if people were just making stuff up. It's hard to do that in a consistent way where you'd be getting the same answer all of the time. I used to teach mar-ket research at a university so I'm really good at checking when people have made up data, and that's proved to be helpful in this regard. So we do a lot of that kind of checking to see if the data is valid. We use a lot of different samples. For the psychographic categories, there's a lot of multi varied analysis behind it. We're using fact analysis, cluster analysis and dis-criminate analysis, so we're also running different algorithms to see if we get answers different ways. In this case, we've also used samples that don't come from our research panel be-cause we wanted to just make sure, well, maybe that's true for this group of people, but it's not true for others. So we've used Second Life participants that weren't part of our panel. We've also used people just from the general population, to make sure that those categories really were robust. And, within all of that process, we do implement a lot of quality control checks so that we can identify anybody who's not truthful and get rid of them.

In particular, I am struck by Gordon's observation that the majority of inaccurate data is provided by a small portion of survey participants. However, individuals who deliberately and reliably undermine researchers for no explicit benefit, provide one of many potential examples of segments that may not cleanly fit within Gordon's paradigm. Perhaps "griefers" would make a relevant seventh psychographic category and certainly exhibit behavior that could be profiled and even targeted for marketing or policy purposes.

Nevertheless, there are certainly additional opportunities for researchers to establish general techniques to detect liars and insulate virtual research from their damaging effects. One intriguing possibility comes from the literature on truth-telling that to my knowledge has not been previously applied in virtual worlds. In particular, perhaps there is an opportunity for a direct application of MIT marketing professor Drazen Prelec's mathematical 'truth serum' designed to reduce lies in subjective judgements, or an opportunity to develop a related mechanism tailored to the griefer profile. Perhaps my next experiment will investigate this concept further.