Originally published in journalism.sg. Reproduced with permission.
The Press Freedom Index of Reporters Without Borders (RSF) is once again making the news in Singapore, with law minister K Shanmugam dismissing the Republic’s low ranking as “quite absurd and divorced from reality” – even as he stated that the government’s approach has been to ignore criticisms that make no sense.
Indeed, the wonder is that the RSF rankings get so much airtime in Singapore ministers’ speeches. It is simply not deserved. On past occasions, ministers have referred to the rankings and then gone on to state that there is more to life than press freedom – without questioning the validity of the rankings in the first place. This time, at least, the law minister has questioned whether the rankings make sense. How, for example, could Guinea’s brutal junta rate higher than Singapore? (Another shocker: Indonesia lies at 100, below several Gulf states.)
The truth is that RSF’s Press Freedom Index is methodologically and conceptually flawed.
First, it lacks what researchers call “inter-coder reliability”. When surveying anything that’s subjective, you have to make sure that those taking the measurements (the coders) are all applying the same benchmarks. To the best of my knowledge, RSF does no such thing. Instead of using a common pool of rigorously trained assessors, it asks respondents within each country to rate that country on various indicators. Their responses help determine where the country ranks.
That’s like deciding the Miss World beauty pageant by comparing the scores given by the judges in each national competition. In such a system, if Miss Singapore judges gave the winner very high marks – perhaps because they were more easily impressed than, say, the Miss Venezuela competition judges – Miss Singapore could be Miss World faster than Ris Low could say “boomz”.
RSF’s method means that, at the end of the day, what it gives us is not really a “press freedom index” but a “national perception of press freedom index”. Actually, it’s not even that. Depending on who is asked to fill out the survey, it could really be “national press freedom monitors’ perceptions of press freedom index”. What we can conclude from Singapore’s low ranking is merely that whoever the respondents are in Singapore have a worse opinion of their country’s press freedom than 132 other countries’ respondents have of theirs.
Note that it would be wrong to blame the respondents for the result. Even though the end product is an international comparison, that’s not what the respondents were asked for. Indeed, the ranking would probably be quite different if the local experts were shown a list of countries and simply asked, “Where do you think Singapore’s level of press freedom places it relative to these 174 other countries?” My guess is that such an approach would cause Singapore to leapfrog over dozens of states in the table and land somewhere in the 50-100 range: a middle-ranking and far-from-ideal state, but hardly an international pariah.
These, however, are little more than parlour games, because there is a more fundamental conceptual flaw in such rankings. When compiling an index, which is an aggregation of different indicators, the tricky part is what weights to attach to each separate indicator. Nobody has come up with a clear way to compare the different types of violation that are involved in the denial of press freedom.
For example, which is “worse”: (a) killing a journalist to stop a single story? or (b) giving broadcast licences to only a small number of commercial broadcasters who pump out entertainment, leading to a long-term denial of the public’s need to be informed through news and current affairs?
To Amnesty International, which exists to focus on individual human rights violations, (a) would certainly be worthy of attention, while (b) would probably not appear in its annual report. Yet, Article 19 of the Universal Declaration of Human Rights upholds the right of the public to receive information they need. If your main concern is the public’s right to know, (b) may be more worthy of attention than (a).
RSF just ignores this conceptual problem and does not declare the relative weights of different indicators in its index. But, if you read into the results, Singapore’s ranking must mean that RSF considers the Republic’s lack of alternative media and lawsuits against foreign media to be more serious than, say, torture and kidnapping of media workers and blocking of political websites (not practised here, but common in some countries that RSF places higher than Singapore in its rankings).
This problem is intractable, which is why more credible NGOs such as Amnesty International and Article19 don’t bother to create rankings or ratings. They write detailed reports on each country without attempting to make international comparisons.
Like many media-savvy organisations, RSF benefits (at least in the short term) from the publicity payoff of publishing annual rankings. Looking at its success in getting noticed in Singapore, who can say they are wrong?
This is exactly why THE-QS’ university ranking is similarly flawed. But that doesn’t stop the media from quoting it all the time to prove NUS’ is world class.
In any case, this is an excellent article. Although I think it’s probably not a very popular one as it does not present an obvious government-linked punching bag. ;)
dear George
the index is still useful. because of limits to the index, what one cannot do is compare a country that hold 132 rank and another country that holds 133 rank and another country that holds 131 rank.
however the index is still valid to show the differences between countries by say three broad categories – those with the better standings, those with the average and those with dismal standings.
one may have a problem with those countries that fall along the boundaries of those three categories.
however the index is useful to show very clearly the best countries and the worst countries.
without doubt Singapore lies amongst the worst countries. even if the methodologies that you mention are rectified, Singapore will still remain with that dismal group.
So i guess the author is saying- the index measures how well/poorly perceived by the country’s own media observers.
Which is kind of true- we do not really like our media that much.
But i appreciate his point- if u ask the same person who gives SG media “2″ to rate Guinea, he/she will probably give Guinea much lower score- maybe “negative 3″? Who in the right mind will rank Guinea higher than SG ?
(see- Military authorities bar foreign journalists – IFEX) you can also google “journalist, guinea” to see accounts of journalists being attacked.
So yes, Singaporeans don’t like our media as much as Guinea (that we can tell from the rankings). But i don’t think our journalists will be migrating there anytime soon either.
In any case, I think the way forward is NOT for the govt to take comfort in the fact the ratings are inaccurate, nor for critics to cling onto a rating that is methodologically difficult to defend. Instead, it should be to come up with better indicators to track and make progress.
Good Morning,
“That’s like deciding the Miss World beauty pageant by comparing the scores given by the judges in each national competition. In such a system, if Miss Singapore judges gave the winner very high marks – perhaps because they were more easily impressed than, say, the Miss Venezuela competition judges – Miss Singapore could be Miss World faster than Ris Low could say “boomz”.”
This comment would only hold water if Singaporeans and residents dont have plenty of opportunity to read broadly, travel and interact with different nationalities inorder to compare, contrast and distinguish what avaliable to the status quo ante – but these days with globalization and the whole idea of a borderless world where states of minds and schools of thought are freely exported and imported.
How valid is Cherian’s assumptions?
We encourage him to travel widely and read broadly whenever possible.
SD (Internet Liaison officer of the brotherhood)
Thanks for this piece, Cherian. It does puts a new perspective on the whole ranking system.
But this comment of yours interests me.
“…whoever the respondents are in Singapore have a worse opinion of their country’s press freedom than 132 other countries’ respondents have of theirs.”
I accept your point that the rankings are not rigorous enough to debate on whether one country’re press is free-er than the other.
But i would think the respondents in Singapore would be a lot more well-travelled than the respondents in “oppressed” Guinea, as you claim.
Yet, for them to score Singapore so lowly, still does say something, doesnt it?
This is all very complex and i may not have the capability to understand it all, being so less-travelled.
But, Cherian, you were part of the media landscape in Singapore for the longest time, and a senior editor within SPH. With your hand on your heart, what do you have to say, from your own personal experience?
Let’s discuss just one example, a strictly “Singapore only” event that does not need gobal comparisons – Election Time and coverage on Opposition’s messages.
Were journalists free to cover anything they want? What happens to a journalist/ newspaper who actually writes good indepth coverage on opposition news? What happens to his piece? How does it get edited? Why does the opposition have to rely on their own newsletters to promote their manifestos and messages during election time? Why do the opposition complain they dont get fair airing of their messages during elections? Let’s check out the stats, if there’s any out there, of how much news were allocated to the ruling govt versus opposition?
Singaporedaddy,
I think even with a borderless world, human perception is an extremely subjective thing. For example, if I were to ask you to rate the Singapore press system according to “Excellent, Good, Not Good, Bad”, you idea of Excellent would be different from mine. I think with a borderless world, what we have is even more subjectivity rather than objectivity!
That is why research methodology is so important in ensuring results are credible.
Good Afternoon Terence,
“human perception is an extremely subjective thing.”
Come, come, that’s like saying water always takes the course of least resistance – we all know human perception is subjective – besides that’s already factored in; if what you have to say is supposed to lend credence to what Cherian wrote; then you could just as well apply the same yardstick to rubbish practically every survey conducted since time in immemorial with admirable skill.
My point is simply this: Cherians assumptions is a leaky string bag at best; at worst it embellishes; as what it fails to take into account is we dont live in a hermetically sealed state like North Korea; on the contrary, we live in a very porous country- where a person may very well read the ST, but in all likelihood he may also read the WSJ, Heral Tribune and that conclusively suggest not only is he empowered to decide wisely; but on the balance of probabilities; his assesment may even exceed the remit of what Cherian convenient chooses to elide so completely in his critique.
Tell me; do I need to wear snow boots and make friends with deadly polar bears somewhere in Sweden just to know that if I want to commit suicide in a car; its a bad idea to try to do so with a volvo? I think not!
Cherian should give both Singaporeans and residents more credit for their power of disccernment.
As for research methodology; yes I agree with you completely; cherian does show apparent disregard for such academic orthodoxy.
I wonder why?
SD (Internet Liaison officer of the brotherhood)
Singaporedaddy,
RSF’s research methodology is stated explicitly on its website.
Of course, subjective perception should not be used as an excuse to debunk every research result. The point is this: has RSF’s research methodology succeeded in minimising such subjectivity to obtain semi-credible results? It is not clear in this case who has been asked to do the survey — it is not stated on the website.
Is it local journalists, foreign correspondents, or civil society groups? Depending on who you ask, you’ll get different results. Also, some of the questions asked in the questionaire seem suspect. For example, respondents are asked: Frequent detailed investigative reporting on a range of sensitive subjects? The questionaire then proceeds to force them to give ‘Yes /No’ answers. How do you define frequent? How do you define detailed? How do you define ‘range of sensitive subjects’?
This is hardly rigorous research methodology.
But I think my main grouse with the survey is that we have no idea who did the survey, and whether the perceptions of these people compare with somebody in other countries. Also, we do not know which criteria is more privileged than another.
I am not a researcher, but perhaps the best way to go about such research is for the same group of people to evaluate the different countries with a qualitative approach. Or at least if they want to do questionaires, they ought to narrow the scope and ambition of their research by limiting their results to say “the perception of local journalists towards the local media” and stating so explicitly.
You point about globalisation leading to a conglomeration of perspectives makes sense, but globalisation has another face to it: global heterogenisation may result as well, leading to a varying myriad of perspectives, within the same geographical area. A person reading ST, IHT, NYT may potentially come to adopt a more liberal western perspective, but he may also end up going in the opposite direction, spurning the “influence” of Western liberal values altogether. I’m sure you know what I’m talking about.
Of course la Cherian will come and refute RSF – he works for the local mainstream media what! He has to justify the media that he writes for what. Haha, not a very convincing argument coming from you Cherian.
XD,
FYI, Cherian George has left media scene for a long time and he is teaching now.
http://www3.ntu.edu.sg/sci/about/profile_CherianGeorge.html
inter-coder reliability my a55.
Out of the 40 questions, I only found ONE(1) question which can be subjective. i.e where the respondent was asked to do a rating of 1-5. The rest are simple yes/no and counts of instances questions, influenced by how well they know what is happening around them. At no time (other than that 1 question) were the respondents asked to make a judgment/perception.
———
8) Terence on November 2nd, 2009 3.03 pm
I am not a researcher, but perhaps the best way to go about such research is for the same group of people to evaluate the different countries with a qualitative approach.
yes, and you never will be if you don’t READ available literature to begin with. The other TOC article has so many links, of which one of them is about methodology. It’s not very detailed, but at least it answers your question on who the questionnaires was sent to.. IF you had bothered to read.
And yes, it also had a copy of the questionnaire. Why don’t you go read it and see if it can be subjective or not? Don’t be like George who simply hid behind ‘to the best of his knowledge’ while never bothering to acquire more. I wonder how he became a journalist…
Hello Terence,
Nice to hear from you again,
“has RSF’s research methodology succeeded in minimising such subjectivity to obtain semi-credible results?”
Yes, I think it’s fair to say you may be correct here; but what you have highlighted is merely a failing that is common to ALL surveys and NOT just RSF’s.
Its important to understand surveys should never be confused with the quatrains of Nostradamus; one could even say its foolhardy to even expect a survey (even a very good one) to capture a high level of fidelity concerning what people may feel abt their respective objects of interest – as by it’s very nature there is always bound to be margin of error.
But the real question is can that margin of error derail the cumulative opinion to the point of rendering that survey so inaccurate that it’s useless?
Mmmmh….I don’t think so.
As here you need to thread very carefully; because when you say people don’t have the power to make an informed decision; then what you’re really doing is denying wholesale the human condition along with discounting its resilience and perhaps versatility in the face of lies. I dont deny ants may probably do that, but humans?
Bear in mind, I am not just churning out feel-good-and-happy sound bites here. If you’re looking for an anecdotal gold standard to verify what I just mentioned; then how do you explain : even at the zenith of the USSR when the common soviet had ONLY Izvestia and Pravda, they had already cultivated a healthy askance towards the state inspired apparatus of assimilation?
During those dark days the words were
“There is no Pravda in Izvestia, and there is no Izvestia in Pravda” (“There is no truth in News, and there is no news in Truth.”)
Consider this Terence: If the truth could somehow find a way to surface in those reprehensive regimes (could be through the underground, homing pigeons and the samzidat movement) in the collective consciousness.
How much more can it be expected to surface in the age of the internet these days?
I think this is something people need to consider very carefully when they propose to rubbish the survey results of RSF or for that matter any survey that causes them embarrassment.
My personal opinion as a tax payer is the custodians of power should take this as an opportunity to reflect deeply on the results of the RSF and even use it as a basis to improve their processes, systems and policies – of course that would mean a whole lot of people will have to stop whinning like a bunch of old women and get off their fat backside to do some real work.
That may not be such a bad idea after all.
Thank you Terence, I really must return to my conference call.
SD (Internet Liaison officer of the brotherhood)
Lobo,
that’s the problem: yes/no question forces respondents to make a choice, when there are many shades of grey.
Let’s take this question as an example:
“Being tortured or ill-treated?”
There are many ways to define “ill-treated”, and the survey fails to make any distinction at all, expecting respondents to answer either yes or no. A better way to phrase this question would be:
“How many journalists in this period of time (Sept 08 – Sept 09) have been arrested by the police?” or “Is physical torture frequently inflicted on journalists?”
Respondents should be given 4 or 5 options so as to produce a more reflective answer. The answers should not be vague yes/nos, but quantitative and measurable.
But even then, you will realise that such research is fraught with methodological difficulties. Very hard to come up with.
And lobo, I have read the literature. Plus I don’t plan to be a researcher.
Here;s what RSF stated:
“The questionnaire was sent to Reporters Without Borders’ partner organisations (15 freedom of expression groups in all five continents), to its network of 130
correspondents around the world, and to journalists, researchers, jurists and human rights activists. A scale devised by the organisation was then used to give a countryscore to each questionnaire.”
Extremely vague if you ask me.
Singaporedaddy,
I enjoyed the exchange. Of course, the RSF is fraught with methodological difficulties. But it is useful as a very rough gauge, IMO. And yes, it needs to improve in its methods. See ya around.
aygee, so its debatable whether he left the “media scene”. And that doesn’t change the fact that he was part of them for a long time.
And he STILL writes for Straits Times from time to time even as recent as , don’t tell me he writes those articles for free?
Hi Everyone,
Please focus on the topic at hand, and not the author’s background.
Cheers
Donald
Deputy Editor
The Online Citizen
even if the methods are straighten out, the fact remains Singapore media is one of the worst in the world in terms of fairness, transparency and independence…. its one of the most obvious things that every single foreigner has said of singapore when they pass by or live in….
And i also find Cherian’s last point a little harsh…
“Like many media-savvy organisations, RSF benefits (at least in the short term) from the publicity payoff of publishing annual rankings. Looking at its success in getting noticed in Singapore, who can say they are wrong?”
Is Cherian saying RSF is being sensational, and trying to get a quick media hit, benefiting from the publicity payoff?
So if Amnesty Intl publishes a harsh report on Singapore’s track record on capital punishment, are they also seeking quick publicity?
Mr George,
Thank you very much for your article. I do believe that the index was seriously flawed too, but couldn’t quite put my finger on it. I had suspected the result was because of the usual Caucasian tendency to take the moral high ground and talk down to us in the third world, but I see that it could simply be because of the flawed methodology of their survey.
Also, thank you TOC. I think you really prove that you can be the voice of reason and maturity – sometimes. Do keep up the good work with more postings like this. What we need more of in this nation is a reasoned discourse. Kudos to you.
Survey or no survey, the fact remains that SPH is headed by a former deputy prime minister and staffed by quite a few ISD operatives. Under the newspapers and printing presses act, the govt holds management shares to determine who are appointed to top editorial jobs.
It’s also a fact that Singapore’s editor in chief Lee Kuan Yew wants his “media” to control and manipulate the minds of the people. LKY has defined the role of the newspapers as disseminators of govt policies and programmes in a way that they could easily be digested and accepted by the people . His newspapers are not to engage in a crusade of a cause/causes unless approved by the ruling PAP.
Therefore, Singaporeans like me don’t need agencies such as RwB to rank our propaganda rag.
Terence,
Respondents should be given 4 or 5 options so as to produce a more reflective answer. The answers should not be vague yes/nos, but quantitative and measurable.
That is the subjective part….
If you ask an ang moh if he thinks he has done a good job, he will say so if he thinks so (gives a 5). Asian or Chinese, etc, may like to be more humble and will at best give a 4. So in a ranking, does the ang moh outperform the chinese?
A yes/no answer which forces the respondent to make a decision is instead more objective (for measuring purpose), because both the ang moh and the asian/chinese can only choose ‘yes, I have done a good job’.
I first need verification of this: I believe that Freedom House is also in the business of ranking the media on an international basis.
If so, I wonder if Cherian George can now study Freedom House’s methodology and explain why FH’s ranking of the media in Singapore is even worse than that of RSF’s.
To answer my own query, here it is: the FREEDOM OF THE PRESS 2009 Table of Global Press Freedom Rankings Rank by Freedom House.
http://www.freedomhouse.org/uploads/fop/2009/FreedomofthePress2009_tables.pdf
Singapore is ranked a grand 151st out of 195 countries globally, still in the lowest quartile of all countries surveyed and almost exactly the same percentile – approx. the 76th in the RSF rankings and 77th in FH’s.
Singapore is also 32nd out of 40 in the Asia-Pacific region, faring only better than the usual suspects with the surprising iclusion of Sri Lanka, probably because of their current security considerations; in this category, Singapore is ranked higher than Sri Lanka, Afghanistan, Brunei, Vietnam, China, Laos, Burma and North Korea.
Freedom House’s homepage is here:
http://www.freedomhouse.org/template.cfm?page=470
Over to you, Mr Cherian George.
“I had suspected the result was because of the usual Caucasian tendency to take the moral high ground and talk down to us in the third world, but I see that it could simply be because of the flawed methodology of their survey.”
The writer did say “national perception of press freedom index”, so apparently some degree of “local moral high ground” is equally being involved here..
The writer also did say “What we can conclude from Singapore’s low ranking is merely that whoever the respondents are in Singapore have a worse opinion of their country’s press freedom than 132 other countries’ respondents have of theirs.”
Now what does they mean ?
It could mean that the press freedom here may well be much worst off or better off than the current position depending on the “”"”quality”"”" of respondents, having the subtle message of alluding to the selection process.
“There is no Pravda in Izvestia, and there is no Izvestia in Pravda” (”There is no truth in News, and there is no news in Truth.”)
“If so, I wonder if Cherian George can now study Freedom House’s methodology and explain why FH’s ranking of the media in Singapore is even worse than that of RSF’s.”
Why bother about how our so local experts feel. We do know our own situation here.
Without the internet and relying on our own local media alone, there is a likely possibility that we may not even know how we are being rated (especially those not so flattering ratings).
If you were to do a research on the happiness of Singaporeans, who do you ask? A Singaporean whether he is happy or ask a Malaysia, Indonesian or American whether they think Singaporeans are happy. What crap is this guy talking about????!!!!
Dear Donaldson, the author’s background plays a big role in how we read a piece. For example, you’d expect 2 very different pieces on the war in Gaza from an Israeli soldier and a Palestinian.
28) XD:
“Dear Donaldson, the author’s background plays a big role in how we read a piece. For example, you’d expect 2 very different pieces on the war in Gaza from an Israeli soldier and a Palestinian.”
I hope you don’t mean to say we should not take Cherian George seriously just because he is a former journalist. Why not just focus on his argument? I’m sure his argument is not that water-tight that we have to focus on his background just to make a point.
In any case, I think your logic is also flawed. “FORMER” journalist- means he has both experience working in the sector and am no longer beholden to toe organization lines. If there is anyone in the world who could bitch abt press freedoms (and the lack of it), it will be these former journalists who experienced it first-hand and am no longer on dependent on the organization for livelihood.
You and I can talk on and on abt freedom of press without never knowing for sure what kind or how bad the censorship really is.
Plus, personally, I think one can learn more if he reads from another perspective. e.g. if an Israeli reads a Palestinian newspaper about the suffering of war. Makes life more than one dimensional!
Hi temasek patriot, aygee, XD, seebeng, Robox,
Your views seems popular. There is an article named ‘Temasek Holdings, Financial Disclosure and Charles Goodyear’ in the Young PAP website, Editorial tab, Economy category. Press Freedom was discussed in the comments, It would be exciting if you can go in and challenge.
The main issue with George’s criticism of the RSF ranking is perhaps the inability to recognise that his own criticism are not robust arguments either. In his arguments, he made several grave errors (ironically) those that he accuses RSF of – for example, asking respondents to rate countries’ press freedom based on their perception is a clear violation of his own assertion for the need of intercode reliability. While I think the RSF survey is biased (but which is a problem that plagues all surveys), it certainly is not flawed. I have further explained my arguments in my blog on why this is so.
To Unsure, comment #30.
Thanks for the heads-up. I only wanted to comment on Cherian’s piece where he says that the RSF Index is flawed. I have no inclination to comment elsewhere. I was hoping that Cherian engages us here.
The point i wanted to make is that, while i accept Cherian’s argument that the index is not rigorous enough, for us to score so low, there could be a semblance of reality.
I wonder whether he agrees to this.
The FACT is that the msm in this little red dot is CONTROLLED by the gahmen…………whatever ranking methodology cannot and would not change this basic FACT & that the msm is just another mouthpiece for propaganda.
As I’ve read in this forum, Cherian’s brother-in-law is Minister of Environment & Water Resources Ministry & his wife is a senior staff in ST.
@ 29) RW on November 4th, 2009 7.57 am
Sure, we’re all reading the one-dimensional news in the local media everyday – and had enough of it.
Cherian George was an ex-journalist that’s true. But that doesn’t mean we have to take his criticism as being valid.
You might be interested in reading what an Israeli soldier has to say about Palestine and perhaps even believe it, but I’d find more validity in something written by someone who’s not directly involved.
@ 33) Aiyah on November 6th, 2009 11.38 am
“As I’ve read in this forum, Cherian’s brother-in-law is Minister of Environment & Water Resources Ministry & his wife is a senior staff in ST.”
Even more reasons why I cannot take Cherian’s article seriously.
Yes, there is less freedom for the press in Singapore compared to other countries.
But freedom of the press is not the same as credibility of the press.
Freedom refers to the press being able to publish what it wants with minimal restrictions.
Credibility, on the other hand, refers to whether the contents published are factually accurate, balanced and thorough.
While freedom does have an impact on credibility, they are not the same.
It is like the internet – especially in the earlier days when there is practically no restrictions on what can be published on websites, mainly because it was new territory and the laws have yet to be updated. More freedom does not mean more credibility.
Now, the Singapore press is hardly infallible, and certainly has been guilty of wrong representations.
But vis-a-vis individual newspapers around the world, are articles published by the Straits Times less accurate than others?
Just because there is more freedom for the press in other countries does not mean that they will publish more truthful articles.
Take for instance the reporting of Michael Fay incident in the 1990s. Facts wrongly reported by the US press were not retracted. The Straits Times had a field day reporting all the factually wrong things published by US newspapers.
Having more freedom for the press only means opposing viewpoints can be voiced out more readily, if there is enough support for it and if the views are backed by one of the newspapers.
Credible press is only possible if the press can be held accountable by anyone for factual errors, not just by those who can afford lawsuits.