Alessio Guglielmi's Politics Page
http://alessio.guglielmi.name/res/pol.html

Following the recent announcement of a new open access journal on a philosophy list, with strong emphasis on "simultaneous policies of blind submissions, double-blind review, and anti-plagiarism", I strongly welcome the decision to make the journal open access (to be added shortly to my list here http://www.cs.bham.ac.uk/research/projects/cogaff/misc/open-access-journals.html) but feel the emphasis on blind reviewing, shared by several other journals and conferences, is misplaced: a bad solution to a real problem.

Below I present some thoughts on an alternative way of addressing the problems, namely using post-publication reviewing. After writing those notes, I thought the idea of post-publication review was so good and so obvious that others must have had the same thought, and sure enough Google quickly responded to a '"post publication" review' query, by pointing me at a number of online papers and discussions, including these (this is just a sample):

• http://futureofscipub.wordpress.com/open-post-publication-peer-review/
  The future of scientific publishing: ideas for an open, transparent, independent system -- Synopsis
• http://futureofscipub.wordpress.com/2009/02/12/open-post-publication-peer-review-full-argument/
  The future of scientific publishing: ideas for an open, transparent, independent system -- More detailed discussion
• http://cameronneylon.net/blog/p-%E2%89%A0-np-and-the-future-of-peer-review/
  A wonderful example of post-publication constructively critical review of a mathematical paper.
• http://scholarlykitchen.sspnet.org/2010/07/14/post-publication-review/
• http://www.bmj.com/content/341/bmj.c3803.extract
• http://www.plosone.org/static/commentGuidelines.action;jsessionid=1FBA8B379EB321068D2536B35697AC3F.ambra02
  Post-publication commenting on Plos journals

The announcement of a new open access philosophy journal is excellent news
http://www.adrianpiper.com/berlinjphil/
but I think the emphasis on blind reviewing may be mistaken.

There are several different reasons.

1. The most important objection in the long term is probably that the original reasons for only publishing papers after a stringent reviewing process have gone.

It was once expensive to publish and readers (or their employers) had to cover the costs through subscriptions. So it was important to filter publications rigorously so that neither publishers nor readers wasted their money on low quality papers. The costs of publication on the internet with immediate access for huge numbers of potential readers are now very much smaller, changing the arguments.

A premise of the original process was that it is possible for expert reviewers to assess the quality and long term importance of a research paper. This premise has repeatedly been challenged in part by showing that different referees have different judgements of the same paper (I've seen this often with my own papers) and also by showing that referees can be influenced by knowledge of who the author is, and other background information, which blind reviewing attempts to address.

However, there are many products of human endeavour that are not fully reviewed until after they have been made public. Those include works of art (paintings, operas, symphonies, novels, plays, poems), commercial products for consumers (e.g. cars, washing machines, cameras, television sets, software, dress materials, clothing, building materials, and many many more), political manifestos, scientific/academic books, and others.

In view of the technology now available, and the very low cost of publishing on the internet, I see no reason why post-publication reviewing should not become the norm with ALL academic/scientific research publications. (I address some of the problems that could raise, below.)

Moreover, reviews would then be made public, with reviewers named (as they usually are in the case of published book reviews), and with full right of reply for the original authors (in the same or another forum).

Moreover, whereas some publications will be relevant only to a small body of experts, others will have different potential audiences (e.g. students who need good introductory material, people in other disciplines who need to pick up good summaries of key work in neighbouring fields, lay readers with an interest in some research area, students trying to select a research area, etc.). So there can be different reviews written by different sorts of people for different sorts of readers and published in different places, whereas now a research paper is typically reviewed only from a narrow viewpoint associated with the journal, even if its relevance is potentially much broader.

Post-publication reviewing, with multiple review sites for different sorts of readers, must be a far better system than one that, as now, tends to evaluate researchers on the basis of where they publish rather than the quality of what they publish, and which makes heavy use of citation counts -- ignoring the difference between citers arguing that the work cited had important ideas that inspired new developments and citers showing up flaws in the cited work. For example, a large proportion of citations of John Searle's 'chinese room' paper criticising artificial intelligence are attempts to point out flaws in his paper -- and many authors re-discover the same flaws, boosting the citation count of the paper they are criticising.

Public post publication reviewing by well qualified peers, can remove the weight given to arbitrary citations, and high quality published reviews can be used for assessing researchers rather than the current use of consequences of secret reviews that determine acceptance or rejection.

If papers are mostly reviewed after publication then hiding the author's name, institution, etc. from reviewers makes no sense.

But I don't think there ever were very good reasons for trying to hide those details: blind reviewing is a bad solution to a real problem about quality of reviews. The solution is bad because the solution can itself seriously affect quality of reviews, and can create unnecessary problems for authors. (Explained below from the point of view of reviewer and author.)

2. There is no longer any need for publications to be frozen in stone. Many works of art, software packages, mechanical designs, etc. are improved after initial launch because the original authors, users, critics, or others have found flaws and made critical comments.

The same should happen to research papers, instead of what happens now -- authors keep writing new papers with revisions of their previous evidence, arguments, theories, etc., often published in quite different places, so that a student or other researcher reading a paper has no easy way of finding out whether the author has corrected errors, provided new evidence, extended the theory, etc. That can lead to wasted effort on the part of the reader.

As wikipedia shows, there is now good technology to keep track of changes, so that readers who want to know when changes were made to a publication, and why, and by whom, can do so.

When papers that have received a lot of attention are revised, the new versions will be reviewed (sometimes re-reviewed by reviewers of older versions) and the reviewers can comment on whether criticisms have been answered adequately, whether the new theory is better than the old one, whether the revisions are out of date, because of work done elsewhere, etc. Of course, that sort of reviewing is incompatible with anonymity.

3. There seems to be a widespread assumption that, in order to ensure that publications are accepted or rejected exclusively on the basis of their quality, a policy of blind submissions and reviewing is necessary.

I.e. it is assumed that judging academic works exclusively on the basis of their quality (as opposed to reputation of authors or institutions, or influence of good or bad personal relations between reviewers and authors, etc.) is done best by hiding the author's identity.

That assumption may be true in special cases -- e.g. the first publication by a young author who is the child, or collaborator, or supervisee, of someone very famous, or someone very infamous.

But my experience as a reviewer suggests that blind reviewing is often a hindrance to assessment of quality.

That's because very often a paper builds on previous work and it is impossible to evaluate the latest work well without looking at previous or parallel work, by the author(s), that provides the context. Someone who is not already an expert in the field can therefore get a better estimate of the quality of a paper by reading related work by the same author which reveals the bigger picture of which the paper is a part. For example, it can show that parallel work in progress, or some previous work, addresses some potential criticisms of the item being reviewed. I have had some of my submitted publications criticised for not considering points that I had addressed in earlier papers, which I could not refer to without revealing authorship.

Another reason why the assumption that blind reviewing supports reviewing on quality alone, by hiding authorship, is often mistaken is that when a research community is fairly integrated it is often quite easy to recognize the author of a paper from the style, the assumptions, the work it claims to be extending or refuting, etc., especially when it is a paper by a well known, highly respected or highly controversial researcher.

Moreover, nowadays with search engines it is often not very hard to take some distinctive phrases from a paper under review and identify closely related work by the same author, or even a preprint version of the work submitted.

I have done that often, and as a result, the information about the background, far from reducing the quality or objectivity of my review. has improved it, including enabling me to write more helpful critical comments to be passed to the author.

The attempt to make heavy use of blind reviewing expresses the assumption that most reviewers are unreliable, prejudiced, dishonest, vindictive or have similar flaws. If that's true the faults in the community will be too bad to be remedied by blind reviewing and should instead be addressed by mechanism that identify, expose and criticise people who are guilty of dishonest, biased, or incompetent reviewing.

Making reviewing a public process with authors having the right to know who has done the reviewing is a far better remedy than depriving reviewers of relevant information.

Of course there will still be problems, such as people refusing to review papers by powerful authors whom they fear. But I think the process of post-publication reviewing will deal with that: people who have critical comments on a published paper will be able to express them and there may be many who are competent to do that well who would not have been asked by the journal editor before publication.

4. As an author, I have often found that requirements of blind reviewing have made my task very difficult because my work does not consist of a collection of separate reports on things done or discovered in isolation: most things I write extend or revise things I have written previously and trying to present a new paper without setting the context, which would make it very easy for any reviewer to identify me (and inspect the papers on my web sites), can be very difficult.

I have often received reviewers comments that were obviously a result of the reviewer not knowing anything about the background and making incorrect assumptions, or failing to see the point. In several cases I believe that being able to provide an introductory overview of my previous work, with references, would have produced better reviews.

5. A post-publication review policy would remove the problems of plagiarism related to reviewing procedures. Reviewers would not have access to the contents of un-published papers, only already published papers.

(Of course people may have access to unpublished work in other roles, so the problem of plagiarism will not go away completely.)

6. One of the problems of "self-plagiarism" is a product of the practice of freezing publications. People who have revised their work are understandably motivated to publish improved versions, and that can often include re-using material from the original. Various obstacles to making it clear that the new paper is a revised version of the old one exist, including journal requirements for work to be original, etc.

If publications were revisable then I suspect there would be far fewer publications and much less wasted academic effort. A much higher proportion of published work would actually be read, instead of being used mainly for CV-pumping, as happens now.

Of course, selection and promotion panels would have to change some of their evaluation procedures. Hopefully this will lead to more decisions being based on professional judgements of the quality of the work done by applicants rather than other spurious factors such as quantity, ratings of journals in which published, citation counts, etc.

I have often seen the current mechanisms, which attempt to use 'objective' measures lead to the selection of a more solid and dull, but productive, candidate over a younger much more exciting candidate with clear potential to achieve deep new results, but who does not yet have a publication track-record. In bygone days we would have preferred the second applicant. (Appointed on probation.)

How to control the flood

An obvious problem for any journal offering post-publication reviewing without pre-publication reviewing is the potential flood of publications. Part of the answer is that the process may be self-limiting because authors will learn that publicising lots of junk, and lots of repetitions of the same content can lead to a bad reputation through the review process.

It could also be useful to have two levels of status for published papers: namely (a) unevaluated and (b) evaluated post-publication by (e.g.) at least three well recognised reviewers who recommend the paper. Mechanisms for evaluating reviewers could vary, including using votes by readers of their reviews or other means. It would not be uncommon for different factions to be for or against different reviewers. But at least that would be out in the open and readers could make up their own minds.

A journal could have some not very stringent process for pre-publication reviews of the unevaluated papers to ensure that their content is relevant to the journal, the quality of the writing good enough, the presentation appropriate, and there are no glaring errors that authors need to fix.

Another limitation mechanism could be a small but reasonable publication charge, to be paid by authors (or their institutions), to help cover the costs of the journal, e.g. about USD$5.00 per page for the first 5 pages, and a lower rate for additional pages.

There are likely to be other complications I have not yet thought of but I suspect none that are fatal to the idea.

I welcome comments and suggestions sent to me by email (serious researchers will find the address easily). I may add comments to this web site, unless requested not to.
(I prefer not to use blogging mechanisms, for various reasons.)