Thursday, June 28, 2012

Science Regulation


At the center of the 2008 financial collapse was the failure of the regulators. Financial regulation was supposed to vouch that bankers are doing an honest job. Who makes sure that the scientists are doing theirs properly? 

Science Regulation

Should science be regulated? We need to control scientists who engineer deathly viruses or nuclear devices and to make sure that human and animal subjects are treated humanely. We need regulators that can vouch for the correctness and quality of scientific research and scientific predictions. We need mechanisms to rate the quality of different research institutions, scientists, and works of science.

Who are the science regulators? And how well are they doing their job?

At the government level much of the regulation is carried out by granting agencies. In the USA these include the National Science Foundation, the National Institute of Health, the Department of Energy, and the Department of Defense.

At the "community" level much of the regulation is done via scientific journals. If a publication appears in a respectable journal, it is supposed to be a good piece of science. Those in the most prestigious journals are supposed to be "groundbreaking". At the university level, committees play a crucial role in the hiring, firing, and promotion of scientists.
Is science regulation necessary? 
According to the AAAS, there are about six million science and engineering researchers worldwide. Among six million people of any kind, some will be corrupt, some incompetent, some will deny global warming and some will believe in the powers of homeopathy. If given free rein, such attitudes and views could cause irreparable damage to science and society.

Is the current regulatory system working?

Almost all of the scientific regulatory work is done on an anonymous voluntary basis. By and large the regulators have very little or no accountability. Let's see how this plays out at the different levels of scientific regulation. 

Granting Agencies

Scientific evaluation of grants is generally carried out by scientists on a voluntary and anonymous basis. Scientists are asked to review proposals and participate in panels. Panels then rank proposals, effectively choosing the ones that will get funded. The process lacks transparency and real incentives to invest hard work. I do not know of a single case where a panel member was criticized after the fact in allocating funds to the wrong research project, or lauded after promoting a successful one.

I have participated in some better and some worse panels. The quality of decision making varies dramatically. It is very difficult to locate and recruit the right reviewers and panel members. This crucial part of the process usually entrusted to government officials, which gives them an inordinate amount of influence.

In one of the worst panels I sat on, there were numerous proposals in a very wide area. The panel consisted of three scientists who also reviewed the proposals. For most of the proposals our expertise was inadequate. A government scientist who was supposed to review a quarter of the proposals "disappeared" and we were asked to cover for him by working extra hours. There were millions of dollars on the table. We did our best to allocate them fairly, but we had little confidence in our decisions. 


The review process for journals is, if anything, much worse. It is often hard to find good reviewers for a paper. Reviewers who are too close to a specific paper tend to provide biased reviews as publication promotes their area and their work. Some try to gain or maintain a leadership status in their fields by strongly criticizing their "competitors". Those who are more removed from the area may not bother to learn it before passing judgement. All of them are pressed with time. Even if they read the paper carefully, they cannot allow the luxury to redo experiments, calculations or simulations and rarely have the time to seriously study the work related to a paper in order to make an intelligent judgement. 

University Committees

And it can get even worse. When candidates are evaluated for university positions, there is rarely even a single expert in the relevant committee. Much of the evaluation is based on the candidate's claims and her recommendations. Some advisors will write for each and every one of their students that they are the best scientist ever. Some sign a blank letter and ask the student to fill it in!

Job applicants sometimes take credit for work that is not theirs. They can safely assume that more often than not, committees will not be familiar with facts. They may get caught from time to time, but it is enough to succeed once. Modest candidates lose and over-sellers win. Honest advisers lose and corrupt ones win. The outsiders lose and the well connected ones win.  

Regulations that is never wrong is also never right

What is common about regulation by granting agencies, journals, and university committees? Once decisions are made, they are never reviewed. Panels are never punished for funding bad research. Refutations of works published in reputable journals have little effect on their popularity, and even in public perception of their accuracy (see this ESA paper about refutations and a Wired Story about retractions). Hiring committees are rarely punished for hiring the wrong people. 
The Solution - Transparency!

There are many ways to address the problems in the regulatory system but there is one common approach which is appropriate to all levels of regulation: transparency.
All reports of regulatory bodies should be publicly available, including the names of the reviewers. Panel members and reviewers would have to be much more careful in their work if the reviews are openly available. They would think twice before getting into a conflict of interest. Those who are good in their role as regulators would be able to publicize their service and be acknowledged for it.
 If the reports are public, each review would take a significant amount of work, but the overall time spent regulating science would be much lower.

Today, scientists can submit the same paper to many journals, the same flawed proposal to many funding agencies, the same inflated job application to many departments. As long as one of them goes along with the story, the scientist is bound to reap the benefits.If all of the submissions and reviews were publicly available, there wouldn't be much of an incentive for such strategies. The regulators would be saved from needless work and would have open access to important information regarding their decisions.

It is possible that open reviews would make science more confrontational. But scientists should aspire for openness. Isn't science all about learning in order to make progress?  Isn't it better to see who punches you in the face than being stabbed in the back?

Thanks to Andrej Bogdanov for very helpful edit of a draft of this post. 

1 comment:

  1. The security conferences that I mentioned that are making reviews available for the accepted papers are and (full anonymous reviews appended to the end of the PDFs)

    The mobisys ones seem to publicly identify the reviewer, but are summary reviews instead of the ones that the authors received.