Peer Review



It was this time in 2006 when Gordon brown announced his plans to overhaul the Research Assessment Exercise (RAE); the way in which the government decides on funding for science and technology research in the UK. There have been loud cries of alarm from all sides of the scientific community since then, ‘what’s wrong with the way we do things now,’ they ask?

The RAE is based on the system of Peer Review. It’s a ludicrous idea when one first hears about it – in what other subject would competing experts in a field, be allowed to critically review each others work, and advise the politicians how good it is? Some conflict of interest you would think, no? Yet this is the system the government has used for a hundred years and it’s also the way the editors of scientific journals decide which work they print - and which they bin.

It has been said of Peer Review that it is to Scientists like democracy was to Winston Churchill, that is, ‘the worst type of government, except all those other types that have been tried.’ So maybe the time has come to take a fresh look at Peer Review, and see if we can do any better.



The faults of Peer Review

We scientists can admit that the failings of Peer Review are not inconsiderable. The main objection (on the part of the government at least, who must foot the bill) is that task of coordinating the independent reviews is bureaucratic and costly. Journals have to pay the expenses this brings too, and they account for them by charging scientists large sums to read the intellectually valuable scientific goodies they contain. This seems somewhat unfair for hard-up developing countries and isn’t all that much fun for UK Universities struggling in the wake of the credit crunch either.

More fundamentally, Peer Review has been accused of slowing down the development of science – which is just not cricket. This is because well-known and well-respected experts (the people best placed to review a journal) can be old fashioned and loathe to accept radical new ideas which contradict with their accepted hypothesis. This means bright, radical, young 21st century Darwins (i.e. people with brilliant ideas which unfortunately go completely against the grain of the current accepted opinion) can go disappointingly unpublished.

Against Peer review is also the fact that it is no use whatsoever at detecting major fraud. If a researcher simply makes his graphs up, realistically, a reviewer (who could be on the other side of the world) can have no idea. This was exactly what happened in the memorable of case of Hwang Woo-Suk, the (ahem) celebrated Korean researcher and his work on the cloning of human embryonic stem cells. He published his work in the high impact journal Science in two ‘landmark’ papers in 2004 and 2005. His written experimental section, conclusions and experiments appeared totally sound; it was just a shame he never actually carried any of these out.

The fact is though, Peer Review does work well in 99% of cases – believe it or not. It is an excellent way of professionalizing and shaping up a paper; stopping the authors drawing rash conclusions, or over hyping their results. Reviewers can even offer input on a particular experiment which might prove the results more conclusively and make the research more convincing. Most importantly, scientists trust Peer Review (and indeed it is this mutual trust which allows the system to work at all) – changing the system will always be met with healthy scepticism.

Finally, we should remember that as Irene Hames, editor of The Plant Journal put it recently, ‘Peer Reviewed journals are not records of absolute truth, merely records of work carried out,’ and that in science you can only ever be right until someone proves you wrong.



The solutions

Browns new framework for dishing out cash to scientists, the research excellence framework (REF) will do away with all that filthy bureaucracy in one swipe, to replace it with a statistical system. Instead of using Peer review directly, the REF will generate a bibliometric evaluation of how good each research application is using figures such as the amount of publications a researcher has achieved in the past year say, or the amount of private funding they have acquired.

Many researchers argue this statistical approach is unfair; probably much worse than Peer Review ever has been. The thinking behind this objection being that great scientists could be given a poor rating if they have taken a career break (to have children or get over an illness say) and thus haven’t published enough work that year. Early-career researchers could loose out too, if they don’t make the breakthrough they need to get published before the REF comes around.

In my opinion, we clearly need to opt for including some form of Peer Review in the new procedure - it is vital in deciding scientific merit, and avoiding unfair prejudices.

How could we make Peer Review even better, though? To stop the those radical young scientists with great ideas getting sidelined from reputable journals, some have called for the introduction of so-called double blind reviewing, in which neither the reviewer nor the author of papers know the identity of the other. This might mean prejudice against radical newcomers is minimised.

What about the price of the journals? Could we open up science to the developing world and make the whole process more transparent to boot if we adopted an open access system? This would mean all journals were freely available to view (on the internet for example), and authors themselves would have to pay to have them published. This would be a radical reform indeed, and forcing scientists to pay to be published might lead to authors simply creating blogs of their work online, which would be equally free to view, but somehow less trustworthy.

In conclusion, it appears that Churchill was right. Peer Review might not be perfect, but it is the best idea we have. Perhaps we can eventually learn to see it as what it really is; the (ever so slightly flawed) arbiter of scientific quality.

To read more about Peer Review, find out how it works and join the debate try visiting:

//www.senseaboutscience.org.uk/index.php/site/project/29/