Science: The Importance of the Peer-Reviewed Paper
One cannot overestimate the importance of the peer reviewed paper in science. While it serves the primary function as a vehicle to transmit newly discovered information, it is used for many other purposes such as evaluation the skill level of a scientist, a predictor for future research success, etc. Because it is so important, it is imperative that the public understands what a peer-reviewed paper is, why it is important, how does it differ from other papers, how it can be abused, and so much more.
What is a Peer-Reviewed Paper?
When a scientist is given money to perform research (usually in the form of a grant), it constitutes more than just a gift. The grant becomes part of a contract where the scientist takes the money with the promise that new, useful, and scientifically important information will be generated in return. So how does the granting agency know the information being generated is accurate or even real? The answer is the peer-review process. While the peer-review process is not perfect, it is far superior to any other mechanism available.
The peer-review process revolves around the research journal. Whenever a scientist obtains enough information to tell an important part of their research, they write up all the information into a specific format and send it to the editor of the journal in which they would like to see it published. Upon receiving the submission/manuscript the editor first looks to determine if it is of an appropriate content and an appropriate length for the journal . If these are satisfactory, the editor consults their files of authors who had previously published papers in the journal and then finds several who have published on topics closely related to that being described in the manuscript. The editor assigns them as the reviewers for that manuscript and then sends the manuscript off to them with instructions concerning deadlines and such. Upon receipt of the manuscript, each reviewer would carefully read it with a very critical eye. They would look for errors in logic, spelling and grammar, unaddressed assumptions, whether too much or too little information was present, clarity such that all the work could be repeated using only the manuscript as a guide, if the work was of such significance to warrant publishing, the overall quality of the work, and so forth. After reviewing the manuscript each reviewer would write up their comments along with their opinion to publish, publish with revision, or not to publish the manuscript and then sends it all back to the editor who then transmits all the comments back to the author (the author is never informed of who the reviewers were). The author either addresses all the comments to the satisfaction of the reviewers and the manuscript is published or the submission is rejected. Typically the review process will take a few months to complete, but can go on longer for problematic manuscripts although the use of electronic communications has sped the process up.
Importance and Uses of Peer-Review Process
The importance of this process cannot be overstated. Without this process, every author would be crowing about how incredible and fantastic their research was for none of us can credibly or unbiasedly evaluate our own work. This process removes the ego of the author in the evaluation of the paper’s importance and quality and provides a quality check ensuring all that the information presented is of a high quality and important.
The importance of this process can be further stated by comparing the importance of peer-reviewed papers to that of symposia papers. Scientists love to gather together at conferences, present their work and discuss shop with other researchers. Frequently those who present at a conference are allowed to write up their presentation which are then combined without any sort of peer-review into a collection that is published as the symposium papers of the conference. The quality of symposia papers is often substantially less than that of peer-reviewed papers which is one of many reasons why symposia papers are rarely cited in peer-reviewed publications. Additionally, symposia papers typically have a short “shelf life” while peer-reviewed papers are carefully tracked and maintained in perpetuity. As an example, I have cited peer-reviewed journal papers from the 1880’s in some of my own publications.
The peer-review process serves yet another important function in that it provides a way to measure every scientist. In describing the process to get a manuscript through the peer-review system and published, one cannot help but notice how the process is quite involved and time consuming. It needs to be that way to ensure the high quality of the papers being published. When I worked as a postdoc I was informed that I had to publish as first author one paper a year minimum (the author list on a paper is a listing of who was involved in order of who contributed the most to who contributed the least) and that publishing two per year as first author would be a resounding success. It should become quickly apparent that the more highly skilled one is, the more publications one will obtain, but this comparison only applies for those working in a research position where one is allowed to publish. Scientists working in private industry where publishing is frequently discouraged are evaluated using other means such as patents awarded or some other internal measure.
But not all publications are the same. Some journals are very prestigious so becoming published in one is quite difficult and is usually reserved for only the very best submissions. Additionally, there are services present that monitor how often every paper is cited in subsequent publications resulting in an impact score. A high impact score represents a paper that has been cited a great number of times because it was so important and fundamental while a low impact score would indicate a paper that was largely inconsequential. It has long been argued whether or not a few papers with very high impact scores are better or worse than having a large number of publications with low impact scores.
As a result of all of this, a critical piece of information in a grant application is one’s publication list. A long career with a short publication list would indicate someone who is a poor producer and who is, therefore, a poor risk that the grant money will be well used. On the other hand, a short career with a long publication list or having some high impact papers published will serve as an indicator that awarding the grant money would be a wise investment. But evaluations like this are used for more than evaluating grant proposals. They are used in hiring decisions, promotion decisions, etc., thus showing the extreme importance of publishing papers and obtaining grants to support one’s research thus allowing one to publish.
Abusing the System
As with all things, people have found ways to abuse the peer review system. In my earlier post of what it took to become a research scientist it should have been clear that research scientists are extremely intelligent, Type A, problem solvers so, if there is a way to game the system, they will find it.
One of the most simple ways to game the system is to simply list all one’s publications in one group on one’s CV. (A CV [curriculum vitae] is like a resume but follows a different format that includes listing every publication and so much more.) I was trained to first list my peer-reviewer publications on my CV, followed then books or book chapters written, books edited, and finally a sampling of recent symposia. When asked how many papers I had published I would respond with only those that were peer-reviewed. When one lists all their publications on one list, it is difficult to tease out symposia, data dumps, and other non-peer-reviewed entries thus making one appear better published than is the case. In fact, many consider it a warning sign when they see a CV with all the publications listed together instead of separating out each kind of publication.
Another way to game the system is to gang up on authorship. Believe it or not, there are some papers being published that have dozens or more authors. Sometimes having all of these authors on the manuscript is appropriate, but other times it is nothing more than a case of helping others pad their publication list by giving them credit for little to no effort. This is sometimes exacerbated by them listing the publication on their CV as, for example, “Smith, Jones, McDonald, et.al.” where the extreme length of the authorship is somewhat hidden. I have personally seen this behavior when people sit in on conference calls, offer zero input, but demand to be listed as an author for simply being on all the phone calls. The outcome of this practice is a longer publication list for little effort that gives the indication that one’s accomplishments are far greater than what has really been accomplished.
Another way the process can be abused occurs when a researcher from a competing lab is chosen as a reviewer. This happened to me once when a certain reviewer made demands that were impossible to meet thus dooming my manuscript from being published in that particular journal. (The same manuscript was submitted and accepted for publication in another journal without the reviewers finding any comments - something that never happened again in my career.) It was later found that the reviewer who made the impossible demands was attempting to publish their own work that paralleled what was described in my manuscript. He was somewhat behind in the publishing process so he was attempting to slow my ability to publish my work in order to get his work published before mine. Upon discovering this, the editor informed me that he had placed mild sanctions on that reviewer as a consequence of their unethical behavior.
Another form of abuse occurs when peer-review is transformed into pal-review. In this situation, the editor will send one’s manuscript to one’s friends for their “peer-review” input. The author knows who’s reviewing the manuscript, the reviewers know the author knows, and so forth, with the result being a skewed reviewed that allows mistakes and such to get by thereby lowering the overall quality of the process and its output. The process is also used when a certain group is pushing their agenda for some reason.
Conclusions
Taken together the peer-review process has provided great added value to the publishing process. It goes a long way to ensure papers published are of a high quality while it also provides a tool to help evaluate the relative strength and ability of individual scientists, even though it is a tool that can be abused.
Coming Attractions
Now that this dead horse has been appropriately beaten, it is time to move on. Coming up soo will be how to tell a real expert from a fraud and no knowledge of science is required.