Spiegel

Reading Ben Goldacre, medical doctor and outspoken critic of bad science, has done something to my news reading sensibilities – in a pathological way. Now when I read a news story that involves science, statistics and surveys, my ‘bad science’ radar switches into tracking mode, ready to dissect possible flaws.

My sensors hit pay dirt when I reached page B6 in the Home section of the 28 November edition of The Straits Times. The entire page was devoted to a single story on the connection between poor academic performance in and youth addiction to Facebook and computer games. (See the full article here).

“Too much time online hurts your grades: Polls”, declared the headline. Lester Kok and Eisen Teo then open their story with a fairly definitive gambit: “If you think that spending too much time on the computer is bad for your grades – well, it is.”

They cited a study from the Ohio State University which “found that college students who used Facebook spent less time studying and scored lower grades than those who were not busy “poking” their friends online.”

They reported that “Facebook users in the study of 219 undergraduates had grade point averages (GPAs) of between 3 and 3.5, while non-users had GPAs of between 3.5 and 4.”

A fairly straightforward claim. But I’m a little too nosy for my own good, so I googled for this research. Funnily enough, it first appeared in April 2009 – in British newspaper The Times (12 April) and in Time magazine (14 April), amongst others. Pretty old stuff, then. Okay, to be fair The Straits Times reporters were using this research as a peg for presenting newer data which they had just gleaned, so maybe we can forgive these two decidedly stale paragraphs.

There is more than meets the eye about this Ohio research; it appears to be simply a correlational observation – when a group of people does A, they exhibit signs of B; another group of people doesn’t do A, and they don’t exhibit signs of B.

Yet nothing in this study indicates a definitive causal link between A and B. For example, we may observe that some people eat junk food and then become overweight, while we see others not eating junk food and not becoming overweight. But this doesn’t necessarily mean eating junk food causes people to become overweight.

It doesn’t even necessarily mean eating junk food is the most important factor. Other factors like failure to exercise, failure to balance diets with healthier food, underlying medical conditions, or lower metabolism could be just as, if not more, important.
The thing to remember is correlation doesn’t equate causation. And the Ohio researchers admitted as much in their press release (see:http://researchnews.osu.edu/archive/facebookusers.htm).

“We can’t say that use of Facebook leads to lower grades and less studying – but we did find a relationship there,” said Aryn Karpinski, co-author of the study and a doctoral student in education at Ohio State University.

Later on in the press release, the researcher goes to some length to clarify the superficial relationship her initial research has revealed.

Karpinski emphasized that the results don’t necessarily mean that Facebook use leads to lower grades.

“There may be other factors involved, such as personality traits, that link Facebook use and lower grades,” she said.

It may be that if it wasn’t for Facebook, some students would still find other ways to avoid studying, and would still get lower grades. But perhaps the lower GPAs could actually be because students are spending too much time socializing online.

As such, while the Ohio study can provide a hypothesis for further study into the phenomenon, it cannot be taken as proof that increased Facebook use would harm academic performance. Not that the Straits Times deigned it necessary to let you in on that.

What the Straits Times had to say that was new was their findings on computer gaming addiction and its effect on academic performance. This consisted of survey data collected by online entertainment company Sulake.

The same Sulake that is “focused on virtual worlds and social networking”, whose “long-term strategic aim is to be a leader in community-based entertainment with a portfolio of properties addressing a wide range of target audiences”.

If this isn’t enough to sound off the alarm bells in your head, your ‘dodgy surveys’ radar probably needs a tune-up.
I can see it now – smirking Sulake executives thumbing smugly through copies of the Straits Times, trying not to piss themselves with glee at an early Christmas. Free publicity on a national broadsheet, in an article which also slams one into the collective mugs at Facebook…oh the gratification!

Besides an obvious conflict of interest, there are more issues with the survey. The Straits Times seem to be attempting a study similar to that of the Ohio research on Facebook – looking for a relationship between gaming addiction and poor grades.

As their opening sentence suggests (see above: “If you think…), they suggest, rather strongly, that there is a causal relationship between addiction to computer games and performing poorly at school.

If intent of the exercise is to discover whether computer gaming addiction is a causal factor in falling academic grades, you would likely approach the problem in one of two ways.

You can do a ‘case-control’ study – comparing groups which have the outcome or not (in this case, deteriorating grades) and seeing how common is the exposure to the hypothetical cause is in each group (computer gaming addiction). Alternatively, you can do a ‘cohort study’ – comparing groups which have had the exposure or not, and then see if there is any variation in the hypothetical outcome.

The Straits Times went with the latter approach. The survey involved 653 teenagers, aged 12 to 18, on the social networking website Habbo Hotel. In paragraph 6, the reporters wrote:

Of those who were not addicted, 59 per cent said they scored mostly As and Bs, while 6 per cent scored mostly Cs and Ds and below. For addicts, 50 per cent scored mostly As and Bs while 14 per cent ended up with Cs and Ds and below.

Their choice is not surprising; a ‘case-control’ study would have been very difficult in this case. The potential causes of poor grades are a dime a dozen – isolating the sole factor which the study is interested in while keeping all other variables under control is well-nigh impossible in a simple questionnaire-based survey.

Also, deterioration in grades is relative – an intelligent person suffering from gaming addiction may achieve higher grades than someone not addicted but less intelligent. Trying to ascertain the relative grade deterioration experienced by survey respondents is difficult, if not impossible, given the way this survey was conducted.

Significant problems remain, nonetheless. The survey sample is self-selecting – respondents were internet-savvy teens who use a particular social networking site (Habbo Hotel). The possible influences this self-selecting sample has on the survey outcome are as problematic as they are numerous. For a start, it is not implausible to suppose that such teenagers might possess characteristics that make them more susceptible to addiction, and amplifies the consequences this has on their grades.

Also, the veracity of some of the survey results is also suspect. This survey counts on the honesty of its respondents, as well as the consistency with which they answer questions that involve significant value judgement, such as ‘When you lose a game, do you get frustrated, angry and upset?’ On addiction itself, we can reasonably expect that some addicts will not be self-aware and deny that they have a problem.

It is not clear if the survey is designed to remove the influence of other possible contributory factors like psychological conditions, quality of teaching etc. But given that it was conducted by an online entertainment firm concerned more with PR outcomes than the scientific integrity of their work, it is not unfair to suppose that it probably isn’t.

Regardless, this whole exercise remains fundamentally flawed if the researchers truly desire to prove causality; by nature of its design, this survey can only detect observational correlation. To test the hypothesis that gaming addiction causes deterioration in academic performance, the decline in grades must be measured as a process occurring in teenagers as they become addicted to gaming.

Nonetheless, based on this inadequate poll, the reporters make further tenuous claims:

The Straits Times poll found that gaming addiction was likely to hurt a child’s relationships with his parents and siblings, with just over half of them picking fights with relatives over the use of the computer.

According to the table provided, 54 per cent of those addicted pick fights with relatives over the computer. But this figure in itself means nothing – especially if a very similar percentage of the non-addicted are just as hostile to their relatives. To clarify their claim, the paragraph should have noted that only 38 per cent of those not addicted pick fights over the computer.

However, the 38 per cent doesn’t represent a reliable baseline figure from which we can compare to addicted gamers. In fact, neither is the 54 per cent figure.

We know next to nothing about the detailed personal background of the respondents. This is important because underlying factors, unseen by the survey, may have caused the alleged increased likelihood of being aggressive to their relatives. Gaming addiction could be a surrogate indicator of psychological inclinations that result in more hostile behaviour and poor interpersonal relationships, or of family issues – broken families, financial difficulties etc.

In essence, gaming addiction could be how teenagers with underlying personal problems cope with the issues in their lives. Rather than being the cause of their aggression and hostility, gaming addiction might be something that accompanies these traits, as part of their coping mechanism for their problems.

Only a simplistic interpretation of the data could back up what the reporters claim, but as we know, correlation doesn’t equate causation.

In any case, the real point of the story was not proving causality – the mere intuitive suggestion is more than sufficient. The thrust of the piece is the fingering of the new scourge of teens – online social networking – and insinuating that it siphons time and energy away from more useful pursuits…like studying.

In this context, such warnings are merely a natural extension to the old cautionary tales against the proliferation of computer games, mobile phones, tamagochi (remember that?), glue sniffing, cards, marbles…ad infinitum.

While the answer to addiction lies more with the addicted than the offending game, product or service, the idea that the problem is really within us, is understandably hard to swallow. The temptation is therefore to externalise and objectify it. We see addiction as an externally-introduced affliction that can be eradicated, rather than as a systemic “flaw” in our psychological make-up.

The castigation of Facebook and computer games is but the latest incarnation of this externalization. It is informed, perhaps, by an inherent phobia of technology and the rapid societal changes it induces.

Related posts:

1. Press Muse – Moving news and pushing pills
2. Press Muse – The good, bad and ugly (part one)
3. Press Muse – Singaporeans dreaming
4. Press Muse – Love thyself, ignore thy neighbour
5. Press Muse: Speaking “truth” for power