10 simple rules to format a preprint

Submitting papers to preprint servers (bioRxiv) is finally getting popular in biology. Unfortunately, many of these papers are formatted in a way that is very inconvenient to read, possibly because authors stick to the format asked by journals. Here are 10 simple rules to format your preprints:

  1. Format your preprint in the way you would like to read it. The next rules simply implement this first rule.
  2. Use single spacing. No one is going to write between the lines.
  3. Insert figures and their captions in the text, at the relevant place. It is really annoying when you have to continuously go back and forth between the text and the last pages. Putting figures at the end of the paper and captions yet at another place should be punished.
  4. Don’t forget the supplementary material.
  5. We don’t really need 10 rules. In fact the first rule is just fine.

Free the journals

These days, academic journals serve two purposes: to organize peer review, and to make an editorial selection. With internet and in particular “preprint” servers (eg biorxiv), journals are no longer necessary for distributing academic papers. It is also worth reminding that the peer review system has not always been organized in the way it is currently organized, ie with several external reviewers, multiple revision rounds, etc. For example, Nature only introduced formal peer review in 1967. Before that, the selection would be done internally by the editor.

These two missions, organizing peer review and making an editorial selection, are currently coupled. But this is neither necessary, nor a good thing. It is obvious that this coupling is not necessary. One could easily imagine a system where papers are submitted to a mega-journal (e.g. PLoS ONE, PeerJ, F1000 Research), which organizes the peer review, and then journals (e.g. Nature or Cell) make their editorial selection based on perceived “impact”, possibly using the reviews. Instead, authors must submit to each journal separately until their paper is accepted, and reviewers are asked both to check the scientific standards, which is the alleged purpose of peer review, and to judge the perceived importance of papers, which is an editorial task. This results in frustration for the authors, unnecessary delays and tremendous waste of resources. Thus, it is not a good thing.

Once a peer review system is organized (eg by mega-journals), the remaining role of journals is then editorial selection, and this could be done separately. Once we realize that, it becomes clear that very little infrastructure should be needed to run a journal. A journal issue is then just a list of papers selected by an editor, put online with the appropriate links and editorial comments. I propose that every scientist, or lab, or possibly group of interest, starts their own journal, which I will call “free journal” (see my related posts here and there). Free journals are not tied to any commercial interest; they are self-managed academic journals. Crucially, the editor makes a personal selection based on her readings, papers that she personally thinks are interesting. This means in particular that, in contrast with most journals, the editor is highly qualified. The selection is meaningful, not a collection of thumbs up/down made by disparate reviewers based on vague criteria (“impact”). It also implies that it is totally acceptable for the editorial selection to include “preprints”: the editor is a peer and therefore any featured paper is by definition peer reviewed (ie, as in Nature before 1967). I have made my own free journal of theoretical neuroscience in this spirit. I have collected some technical advice on how to make one, and I would be happy to receive any suggestion.

The bottomline is: it is very easy to make a free journal. On the technical side, one essentially needs to use a blogging system, eg WordPress. This automatically generates an RSS feed (eg one post per issue). I also put all the comments associated to the selected papers on PubPeer, with a link to the journal; this way, anyone using their plugin automatically sees that the papers have been selected in the journal when looking at them on pubmed, for example. On the editorial side, it actually represents little work, and I believe that this little amount is actually quite interesting work to do. Every scientist reads and annotates papers. All that is required to run a free journal then, is to set half a day in the month to put your notes together into a post. This helps organizing your bibliographical notes, and it is helpful for your colleagues and your students. One could also imagine coupling this writing with the lab’s journal club.

Please leave me a note if you are making your own free journal and I would be happy to link it from my journal.

My free journal: Brette’s free journal of theoretical neuroscience

My appeal to PLoS Computational Biology

I recently reported that one of my papers has been rejected by PLoS Computational Biology after 10 months and 4 revisions, on the ground of general interest. This has generated a little buzz. A colleague mentioned it on his blog. As a result, the editor of my paper contacted him directly to tell his version of the story, which my colleague has now reported on his blog.

Unfortunately, the editor’s story is “misleading”, to be polite. It is a shame that the review process is confidential, as it allows the journal to hide what actually happens behind their closed doors. Nevertheless, I have asked the journal for the authorization to publish the content of my appeal and their response, where I explain what happened in more detail (and more accurately). They have accepted. I have removed names of the persons involved. This illustrates one of the flaws of the current peer-review system (see this post for how it could work better).

(Just one note: the editor has apparently told my colleague that the third reviewer was a collaborator, so they could not take into account his review. Well, that’s a lie. I know because he chose to sign his review. The "collaboration" was the scientist sending me published data.)

So here it is.

Appeal

Re: Manuscript PCOMPBIOL-D-16-00007R4

So after 10 months and 4 revisions, our paper has been rejected, following the recommendation of one reviewer, because it is not considered of broad enough interest. I quote from the final editorial message: “We regret that the specific hypothesis that your manuscript is geared to dispute does not reach that level of general interest.”.

These facts being recalled, it should be obvious enough that the editorial process has gone very wrong. There were no more technical criticisms already after revision 2, on July 8th, 4 months ago, and the paper should have been accepted then. I have repeatedly asked the editors to explain why we were required to justify novelty and significance after having been required to do so much work on technical aspects. But the editors have refused to answer this simple query. Frankly, I was expecting a bit more respect for the authors that make this journal, and I do not think that explaining the journal’s policy and the decisions is so much to ask. All I know is Michael Eisen’s view, founding editor of this journal, who has cared to comment “I agree - a paper going out for review should mean it is of interest”.

This editorial process has gone beyond anything I have ever witnessed in my career in terms of absurdity and waste. Why scientists (“peers”) would voluntarily make each other’s life so unnecessarily hard instead of cooperating and debating is beyond my understanding. In the end it appears that the ego of one (important?) reviewer matters more than science, and that is very sad. This being said, I have been notified that appeals are only considered when “a) a reviewer or editor is thought to have made a significant factual error” or “b) his/her objectivity is compromised by a documented competing interest”, and since bureaucracy apparently beats reason and ethics, I will now explain how this applies.

I have already explained at length the factual errors of the first reviewer, who is apparently the only one that is trusted by the editors. This editorial message repeats some of them (no, we are not criticizing simulation results of a particular model, but the biophysical interpretation (what goes on physically), and we did so in several state-of-the-art biophysical models, not one). I will therefore focus on case (b), and attach my previous letter to the editors for reference; please also read the responses to reviewers as regards case (a), in particular to reviewer-editor Dr. YYY who has unfortunately not cared to reply to our point-by-point response that he had required from us. The editorial decisions that have led to rejecting the paper on the basis of general interest after 10 months are so bizarre that I am compelled to question senior editor Dr. YYY’s objectivity – I presume that Dr. XXX, who sent the paper for review in the first place, does consider the paper of interest. The sequence of facts speaks for itself:

- On June 6th (revision #2), the editorial message reads “We understand that Reviewer 2 was very enthusiastic, and Reviewer 3 had relatively minor comments, but we both stress that addressing Reviewer 1's reservations are essential. Indeed, it is only fair to say that it seems to us that it will be challenging to address these comments in the context of the presented results.”. The exclusive reliance on one reviewer and the presumption that we could not address the comments is rather surprising. Nonetheless, the editorial message that followed was exclusively about the match with experimental data, not about interest (“the reviewer's point about (apparently) unrealistic voltage dependencies of the currents […]”). We did successfully address these comments, pointing out that the reviewer had made factual errors (such as misreading the figure he was commenting, and discussing the results of an experimental paper he had not opened).

- On July 8th (revision #3), the editorial message was now asking to explain the novelty compared to what we had done in the past (and published in the same journal), blindly following the 3-sentence report of reviewer #1, and making no mention whatsoever to the fact that we had just answered the major (and flawed) criticisms on experimental observations, which constituted the previous editorial message. At this point we complained that we were asked to justify the novelty of our study 7 months after submission, especially when it was explicit in the introduction; nonetheless, we complied and explained again.

- On August 25th (revision #4), we were appalled to read that, instead of finally accepting the paper, senior editor Dr. YYY decided to nominate himself as a reviewer, admitting that “the latest revision is first one he has had the chance to read”. The report was not an assessment of the novelty of the paper, as would have been logical since this was what the previous editorial message was about. Instead, it was a 6 pages long report full of technical queries, making negative criticisms that, for most of them, had already been addressed in previous reports, and asking for substantial modifications of the paper.

- At this point, I replied to the editorial message and obtained no response; as the message stated “If you would like to discuss anything, please don't hesitate to contact either of us directly”, I emailed Dr. YYY, and he started his response as follows: “To answer your email, allow me to be brief, because this sort of exchange should really be going through the journal, and indeed that will be the case from now on.”. Nonetheless, we exchanged a few emails, in which he offered no explanation; in the end we agreed that I would write a point-by-point response to his six-page review, but not modify the paper. I submitted it, together with a response to the first reviewer, and a letter to the editors, on September 22nd.

- Three weeks later, on October 10th, I received a message where I was asked to edit the letter so that it could be passed on to the reviewers. Apparently the editors had not noticed the response to reviewers. It still took them three weeks to read a letter, which, considering the history of this paper, does not strike me as very respectful. I complained to Dr. YYY, who replied “We believe that you have been adequately notified by the PLoS administrative team concerning the status of your revision.”. I had to exchange several emails with Dr. XXX who realized the error. I received no apology from Dr. YYY.

- On November 11th, I received the reject decision, together with the response of reviewer #1 and, oddly enough, of reviewer #3 to which I had not replied (since there was no remaining comment). He also was surprised, since he wrote “I don’t have the expertise, authority or, honestly, the time to judge whether the new comments from Reviewers 1 & 4 are fair, or whether the authors’ responses have fully addressed them – this is clearly a job for the Editors (although hopefully not for the Editor who just became a Reviewer)”. But, editor-reviewer #4 Dr. YYY did not bother replying to my point-by-point response, which he had explicitly required.

- The final decision comes with excuses that are frankly hard to swallow. One is that the editors had failed to see the word “models” in the title. In 10 months and 4 revisions! Who can seriously believe that? And yes, the paper is about models – it is a computational biology journal (note that we have also successfully related the models to experimental observations, on request of the editors). The other excuse is that an anonymous reviewer (reviewer #2) had a conflict of interest and his reviews had to be dismissed. I am of course fine with that decision (let me simply state for the record that none of reviewers I have suggested are in such a position). But this happened in April, more than 6 months ago. Quite appropriately, the editor Dr. XXX asked for another reviewer who identified himself (Dr. ZZZ). Dr. ZZZ wrote a positive review, and in addition he read our responses to the other reviewers and wrote “the revisions of the manuscript in response to the other reviewers' comments seem entirely appropriate.”. At this point, given that no objection had been raised by any reviewer or editor on methods, results or clarity, the paper should have been accepted. Instead, the editors decided to follow a nonsensical comment from reviewer #1 alone: “unlikely to be of broad interest to the computational biology field, but could be of interest to computational neuroscientists”, which was not even consistent with his/her own first positive assessment (“this is an interesting paper”). Given that Dr. XXX sent the paper for review in the first place, this decision must originate from Dr. YYY (who at this point had not read the paper, by his own admission). I am compelled to conclude that Dr. YYY has not been objective, and in fact has been actively blocking our paper. Unfortunately, this is not the first time I witness the questionable attitude of Dr. YYY, as he has recently been a reviewer for an essay I wrote. The review process was extremely long, went over multiple rounds with massive lists of requests, where Dr. YYY basically wanted to rewrite the text to follow his own views and style. During the review process, Dr. YYY contacted me directly by email to discuss the paper, going so far as asking for co-authorship (“Indeed, the level of suggestions are approaching collaboration on this paper- something I would be happy with but I assume is not what you have in mind.”). In the same email, and while the review process was not over, he asked me for an experimental collaboration – which of course I have not followed up. I had to ask the editor to intervene to stop the madness – which he did: “Indeed your paper has been unduly delayed and I have asked the reviewer to answer me within 24 hours.”. I apologize for disclosing these email excerpts, but I have no other choice since I am asked to provide documentation. It is clear that, had I imagined that Dr. YYY could be chosen as a reviewer (which seemed unlikely given his recent track record), I would have opposed him. But I did not anticipate that he would nominate himself, or overthrow the editor’s decision without even reading my paper (by his own admission).

Therefore, I am asking that Dr. YYY is replaced by a new senior editor with a more reasonable attitude.

As far as I can see: 1) the three reviewers were initially positive on the interest of the paper; 2) the editor Dr. XXX, who as far as I can tell is the only scientist involved in this process who is a member of the computational biology community, supported our paper since he sent it for review; 3) one reviewer, who seems to be an experimental electrophysiologist (unfortunately he or she has decided to remain anonymous), reverted his subjective opinion on the paper’s interest after we have pointed out the errors in his/her report, and even then, still judged the paper interesting for the computational neuroscience community. I have failed to see to how the decision is “not trivial to reach”.

Best regards,

Romain Brette

Attached: Letter to the editors from September 16th

 

Letter to the editors, September 16th

Dear Editors,

In the previous revision, I raised serious objections regarding the abusive attitude of reviewer #1. These objections have apparently been completely dismissed, but what I have been most disappointed about is the total lack of response to these objections. I am writing this letter in the hope that this time it will be given some consideration.

This manuscript has been submitted 8 months ago. This is the fourth major revision that we have been asked to make. The responses are now totaling more than 25 pages, much longer than the article itself. We have now entered a phase where a large part of the responses consist in citing previous revisions where the issues have already been addressed. This revision reaches a new level, where a fourth reviewer is added and repeats mostly questions that we have already answered in previous revisions. Why a fourth reviewer is considered necessary after 8 months of revision is not clear, when none of the three reviewers has raised any serious concern.

I have officially asked a detailed explanation for this peculiar decision. The only response I have obtained so far is that there was “a tie” between “conflicting reviews”. So apparently the editorial decision has been based on a vote between reviewers. This is yet what I read on the journal’s website:

If reviewers appear to disagree fundamentally, the editors may choose to share all the reviews with each of the reviewers and by this means elicit additional comment that may help the editors to make a decision. That said, decisions are not necessarily made according to majority rule. Instead, the editors evaluate the recommendations and comments of the reviewers alongside comments by the authors and material that may not have been made available to those reviewers.

If one followed this process, then one would realize that:

- None of the three reviewers has any remaining objection about results, methods, or clarity of the text.

- Reviewer #2 and #3 have an overall very positive assessment of the paper and in particular of its interest. Rev #2: “This is a great revision. The authors have clarified and addressed all my previous concerns. […] I strongly believe the study is publishable as it stands”; Rev# 3: “This is a very clear and logically presented manuscript dealing with a key question in fundamental cellular neuroscience”.

- On his/her first report, reviewer #1 also made a positive assessment of the paper and of its interest: “This is a clearly written manuscript that addresses an interesting question regarding the nature of spike initiation. Specifically, the authors propose a plausible explanation […] This is an interesting paper.”.

- After two rounds of technical revisions, in which we pointed out the reviewer’s errors and to which no objection has been made, reviewer #1 changed his mood and now concludes, without any argument: “unlikely to be of broad interest to the computational biology field, but could be of interest to computational neuroscientists” (sic).

- Reviewer #3 has read our responses to the two other reviewers and concluded: “the revisions of the manuscript in response to the other reviewers' comments seem entirely appropriate.” From these facts, it appears clearly that there are in fact 3 convergent reviews. All 3 reviewers have concluded that results and methods are rigorous and the text is well written. All 3 reviewers have found the paper interesting. It might be that reviewer #1 has “voted” negatively; however I would expect the editorial decision to be based on the content of reviews and responses, which in this case is convergent, and not on the mood of one reviewer, which in this case is inconsistent between the reports. It is my understanding that an editorial decision should be based on arguments and facts, not on the reviewer’s emotions.

Nonetheless, we have replied in detail, again, to all criticisms. We have pointed out in particular the factual errors of reviewer #1. To help the editors, we have underlined the important points. We would appreciate if the editors checked for themselves whether reviewer #1 is right or not. We have also replied to reviewer/senior editor Dr YYY, although I deeply regret that this fourth version is “the first one he has had the chance to read”.

Finally, I would like to call your attention on the conclusion of reviewer #1, on which his/her recommendation is based, which requires in my opinion a clarification from the journal: “Finally, now in their third revision, the authors acknowledge that this work strongly builds on the previous resistive-coupling hypothesis, and tests whether this hypothesis is compatible with sharp spike onset (a view they have already proposed), vs the alternative proposed by Yu, of back propagation. This very specific theoretical result I feel is unlikely to be of broad interest to the computational biology field, but could be of interest to computational neuroscientists” (Please see also our response, pointing out that the said acknowledgement was clear already in the very first version.)

This recommendation makes some important presumptions about this journal’s editorial views. Therefore I would very much like to know if this journal:

- also considers that proposing a hypothesis is more important than testing one, and that only the former should be published;

- considers that interesting computational neuroscience studies do not belong to this journal. I would also very much like to know if this journal considers that it is ok for a reviewer to ask for substantial technical revisions when he/she has already decided that the paper should not be published anyway. This has been indeed a lot of work for a decision ultimately based on the mood of one reviewer.

As I have argued in this letter, it is very clear that, given the content of the reports of the 3 reviewers and of our responses, this manuscript should have been accepted already. After 8 months and 4 revisions, and no serious objection on the manuscript, I can only hope very much that this journal does not confuse rigorous peer review with author harassment.

Again, I am hoping that this letter will be seriously taken into consideration, and even perhaps responded to.

Best regards,

Romain Brette

 

Response of the editors-in-chief

Dear Dr. Brette,

Thank you for your response to the recent decision on your paper “The Basis of Sharp Spike Onset in Standard Biophysical Models”. The manuscript and your appeal letter have been carefully evaluated by Dr. XXX and the journal’s Editors-in-Chief.

We understand your frustration regarding the length and complexity of the review process, and we would like to apologize for the time taken to reach a final decision.

We would like to provide some further clarification on how the editorial decision was reached. The manuscript addresses the issue - how do cortical neuronal action potentials rise so sharply? – and after an initial evaluation, Dr. XXX found it interesting enough to merit sending out for review, so that the reviewers could assess the technical solidity of the work and the conceptual advance proposed. The paper received mixed reviews, and hence merited a revision. After several rounds of revision, Reviewer 1 remained unconvinced. In order to aid the review process, Dr. YYY volunteered to evaluate the paper in depth, and his opinion concurred with that of Reviewer 1. Dr. XXX also re-read the paper and came to the conclusion that this manuscript is critically close conceptually to the previous PLOS publications - in fact the idea was laid out clearly and beautifully in the 2013 and 2015 PLOS papers. The present manuscript is an implementation of this idea, showing that other biophysically realistic models used to examine the spike sharpness issue show the mechanism that was suggested in the 2013 and 2015 PLOS papers.

We regret that this did not become fully clear before the third revision, and we understand your disappointment with the final outcome.

However, we agree that the findings of the paper are not significant enough for PLOS Computational Biology, and we will not be reconsidering the paper. We are sorry not to be more encouraging, but we hope that you can understand the reasons for this decision.

My new year resolution : to help move science to the post-journal world

I wish I could make the world a better place. I would like to prevent climate change and wars; but that’s not so easy on a personal level. What I can try to do more modestly as a scientist, is to make the scientific world a better place. We have all heard the phrase “publish or perish”. We all complain that careers are made by publishing in “high-impact” journals who favor story-telling and grand claims, and generally select papers arbitrarily (let alone that they do not even predict the impact of papers they publish); a trend that has been increasingly strong and has very negative impact on how research is done, including serious ethical problems. But what do we do concretely about it? For most of us including myself, not much. We keep on submitting to those journals, and we say we have no choice because that is how we are evaluated (for positions or grants). But who evaluates us? Surely there are some political aspects to it (mostly for funding), but the truth is, we are evaluated by peers. In brief, we are inflicting this madness on ourselves.

So: let us stop complaining and try to change the situation. I have previously exposed a vision of how the academic publishing system could look like without journals (by the way, this is not an original thought, fortunately). How to make it happen?

Where we are now

We should be hopeful, because many good things are happening:

  • Preprint servers are getting more and more attention. In biology, a number of journals are now accepting direct submissions from biorxiv, including all the PLoS journals, PNAS, eLife, the Journal of Neuroscience. This ought to boost submissions of preprints.
  • A few journals have started publishing the reviews along with the accepted paper, for example eLife, eNeuro and Nature Communications.
  • More generally, open access to both paper and data is getting more and more common and enforced.
  • A new journal, F1000 Research, now practices post-publication review. The paper is indexed in pubmed once two reviewers have approved it.
  • Very significantly, the Wellcome Trust has opened a journal for its grantees, Wellcome Open Research, based on post-publication review (in partnership with F1000), with this statement “The expectation is that this, and other similar funder platforms that are expected to emerge, will ultimately combine into one central platform that ensures that assessment can only be done at the article level”.
  • Finally: Pubpeer, started just a few years ago. A simple idea: to let anyone comment on any paper, anonymously or not, and let the authors know and respond. You should install their browser plugin. This is an individual initiative but it has already made a big impact, in particular by showing that the “big journals” are not better than the other ones in preventing flaws or frauds. It also addresses the concern that open reviews would be too nice: anyone who finds serious flaws can spot them anonymously and the authors will have to consider them. Pubmed commons is similar, but with signed comments.

What we can do now on a personal level

  1. Put every single paper you write on a “preprint” server before you submit to a journal.
  2. Put all your data online, see eg OpenAIRE.
  3. Remove journal names from the publications in your website. People who care about them will find out anyway.
  4. Start a literature search routine that does not involve looking at tables of contents; a few ideas in this Twitter thread; you could also have an author alert on Google Scholar.
  5. Write comments on Pubpeer; including on “pre-prints”.
  6. Send your papers to a journal with open post-publication review. I know this one is difficult, because the community still cares about impact factors. But at least you can favor those with public reviews (eg eLife, Nature Communications; I would prefer the former as it is non-profit). Instead of sending your papers to Frontiers, send them to F1000 Research; or at least eNeuro.

At the local community level, we can advocate for post-publication review. For example, the INCF has opened a channel on F1000 Research. Maybe we could have a computational neuroscience channel there, sponsored by the OCNS. It is too bad that F1000 Research is for-profit rather than institutional, but currently I do not know of other options.

What we can do on a more global scale

Open post-publication review potentially addresses the issue of recognition, but it does not address the issue of visibility. One concern I have by submitting in F1000 Research (for example), is that my intended readership will not know about it. There are so many papers published each year, one does not even have the time to read the title of all of them. This is one role journals have fulfilled: to select papers worthy of interest for a given community. But since we do not need journals anymore to publish anything, editorial selection and publication need not be coupled anymore. So here is my proposition. We make an independent website which lets any scientist, or possibly any group of scientists, be their own journal. That is, make a selection of papers they find interesting (including preprints). We provide a number of tools to make this as simple as possible: linking to pubpeer and pubmed commons, searching/browsing, alerting authors whose work is selected, email alerts and RSS feeds, etc. Papers are preferentially linked to the preprint if it exists, so as to completely bypass the journals. We could also let authors suggest their own paper for editorial selection. Basically, we provide all the services a journal typically has. This will be made increasingly easier as public open reviews become more widespread. These new “journals” could be run by an individual scientist, or a lab (eg linked to a journal club), or possibly a scientific society or group of scientists. Let us call any of these an “editor”. I would be happy for example to follow the selections of a few authors I respect, and that would be probably more valuable to me that the selection made by any journal, of which very few typically catch my attention in a given table of contents.

I am hoping that it goes as follows:

  1. People start using these individual journals, because it provides relevant information to them.
  2. As a result, papers in less well-known journals and preprints start getting more attention, and more citations.
  3. People take the habit of putting their papers on preprint servers because they get immediate attention.
  4. Editors progressively stop selecting papers published in journals because they have already selected them when they were preprints.
  5. As editors are also committee members, journal names start to matter less in evaluating research.
  6. Traditional journals disappear; instead, we have direct publication (formerly known as preprints) + open public reviews, both anonymous and signed.

How to get it started?

One simple idea to get it started is to make automatic channels for the actual conventional journals. For each journal, we list the table of contents, linked to preprint versions instead of the journal website, and to pubpeer, etc. If it’s convenient, people might start to use it, especially if it allows free access to the papers (legally, since we would use preprints). Then to get people to use the non-conventional channels, we provide suggestions based on content similarity (ie “you may also like...”).

How about this resolution?

A vision of the post-journal world

In 2016, the academic publication system worked mostly as follows :

  1. Scientists do some research and write a paper about it to share their findings with the world.
  2. The paper is submitted to a journal, chosen for its prestige.
  3. The editor, possibly a professional editor (ie not a professional scientist), judges that the paper is not worth of review: go back to (2).
  4. The editor sends the paper to 2-3 reviewers, asking them to review the paper quickly (the more prestigious, the quicker). Reviewers are chosen based on the editor’s expertise and network.
  5. Reviewers have too many reviews to write already: go back to (4).
  6. The reviewers (or one reviewer) find the paper uninteresting, or bad, or they “don’t believe it”: go back to (2).
  7. The reviews are mixed: a third reviewer is asked; go back to (4).
  8. The reviewers find the paper interesting enough but have requests for the authors; they should entirely rewrite their paper; they should do new experiments; they should cite the reviewers; they should remove an uninteresting figure; they should add some discussion points.
  9. After some work, the authors submit a revised paper.
  10. Actually, reviewers didn’t think the paper was that interesting anyway: go back to (2).
  11. The reviewers have more requests: go back to (9).
  12. The paper is accepted: congratulations!
  13. The format of references does not follow the journal’s style: please rewrite the reference list.
  14. The author’s paper now belongs to Elsevier/Springer Nature/etc. It will be sold by the publisher to academic institutions, with a margin of 30-40%.
  15. (recent alternative option) The authors pay publication fees of about $2000. The paper is now freely accessible.
  16. A knowledgeable reader disagrees with the main conclusion of the paper: she writes a letter to the editor.
  17. After careful consideration, the letter is rejected.
  18. Another reader, not an expert of the field, reads the paper in a prestigious journal and believes it is highly regarded in the community, until she talks with colleagues and finds out that the paper is very criticized.
  19. The prestigious paper is commented in mainstream journals. Experts rant about the journalists.
  20. The first author applies for a position. The committee examines CVs of all candidates and selects those with publications in prestigious journals. The senior author sends an excellent letter of recommendation. The committee values it if the senior author is famous.
  21. The senior author applies for a grant. The committee selects candidates with a good “track record”, ie prestigious publications.

This publication model is wasteful, full of perverse incentives, opaque, and misleading. What I want to do here is not one more critique of this system, but rather give a vision of what the publication system could be and, in my opinion, will be, perhaps within a decade. This is a vision of a publication system centered on individual articles, rather than on journals. It is made possible today by information technology (aka “the internet”). What stands in its way is, in a word: conservatism. It works as follows:

  1. Scientists do some research and write a paper about it to share their findings with the world.
  2. The paper is uploaded to an archive site, together with the data. All is freely accessible to anyone.
  3. The authors invite an unlimited number of professional scientists (say, authors of at least one paper) to write a formal, signed, review. The reviews are public.
  4. Any other scientist can contribute a signed or anonymous review or comment, including comments on the public reviews (aka “scientific debate”).
  5. Authors respond to the reviews and may submit a revised version.
  6. Authors decide when to stop the revision process; reviewers express their satisfaction, or not.
  7. Any scientist can recommend the paper if she finds it worth reading. Tools are made available to search for papers, possibly using the number of recommendations, possibly weighted by scientist reputation (all data are freely accessible and third parties provide searching tools).
  8. A knowledgeable reader disagrees with the main conclusion of the paper: she writes a public critique, signed or anonymous, after reexamining the paper’s raw data. The authors respond publicly.
  9. Another reader, not an expert of the field, reads the paper. She notices that many knowledgeable scientists disagree with the conclusions and can read the detailed arguments and the authors’ response.
  10. The first author applies for a position. For every publication, the committee reads the reaction of the community (formal reviews, comments, recommendations) and uses it to evaluate the value of the work along all the dimensions addressed in the reactions (importance, originality, technicality, etc). The committee also asks the candidate what her main contributions are in her view and why. The candidate can point to published reactions to her work that support her claims.
  11. The senior author applies for a grant or a promotion. The committee reads the reaction of the community to the candidate’s previous work. They might also look at review invitations and reviewing activity.

In the post-journal world, a paper is submitted a single time (but of course it can be revised), and in contrast with the journal model, the reaction of the community is known. As a consequence, quality stamps (journal names) are obsolete. No review is thrown away. As a consequence, scientists save time on both submitting and reviewing. Scientists are not incentivized to make grand claims. In addition, the system saves several billions of dollars each year (eg US libraries spend $2 billion each year on subscriptions), not counting the time saved on wasted reviews and submissions.

All of this is clearly possible with today’s technology. The next question I will try to address is: how to move quickly to the post-journal world?

Do we need a new computational neuroscience journal ?

Where to send computational and theoretical neuroscience papers? A list of journals can be found on Jim Perlewitz’s page. First, many journals belong to one of the scientific publishing companies who make huge profits from the free work of academics (margins around 30-40%), with questionable added value. In computational neuroscience, we have:

  • Journal in Computational Neuroscience (Springer Nature)

  • Frontiers in Computational Neuroscience; in Neuroinformatics (Springer Nature)

  • Biological Cybernetics (Springer Nature)

  • The Journal of Mathematical Neuroscience (Springer Nature)

  • Cognitive Neurodynamics (Springer Nature)

  • Neuroinformatics (Springer Nature)

  • Network: Computation in Neural Systems (Taylor & Francis)

Then we have Neural Computation, run by MIT Press, which is non-profit. Unfortunately, papers are behind paywalls. In my case, the library does not have a subscription.

Finally, there is PLoS Computational Biology, a highly respected journal in the field. However, even in this non-profit journal, open access is very expensive ($2250 per paper). I find these costs hard to justify, when any scientist can publish on archives for free (eg arxiv or biorxiv). The solution to this is overlay journals: the papers are linked to an archive site; the journal only deals with peer-review.

I was also recently rather disappointed by PLoS CB, after a paper got rejected 10 month after submission on the sole basis of interest. Based on the reactions I received, this is not an isolated case. Anyway, PLoS CB practices editorial pre-selection, which is highly subjective. As for other prestigious journals, its prestige is precisely based on its relatively low acceptance rate. But this is a highly questionable model, because it is wasteful: wasting the time of authors who submit the same paper to many journals, wasting the time of reviewers who write reports that are thrown away. In my opinion, it is also perverse. It gives a label of quality to individual papers and scientists based on the opinion of just a couple of scientists, sampled by one editor, and this insignificant label is then used by various committees (grants, promotions, hiring). It also pushes authors into boosting their claims to pass the editorial barrier. The solution to this is post-publication review (see also Michael Eisen’s post).

So ideally, I would like to see an overlay journal practicing post-publication review in my field. The question now is: should we create one? What would be needed is the technical infrastructure, but I am not sure it is fully available yet (as regards the post-publication review system, indexing etc).

Alternatively, we could use a general post-publication review journal. There is for example F1000 Research. PeerJ also does this partially (publishing the reviews is an option). Both cost about $1000 per article; both are run by companies. But they are less expensive than alternative open access journals (PLoS ONE: $1495; eNeuro: $1950 for members, $2925 otherwise), and more open. None of these two journals is ideal, but they represent some progress over conventional journals.

There are two issues with choosing a general journal. First, there is the question of editors. I have no experience with these journals so I am not sure there is the relevant expertise in the editorial board. I have seen a couple of computational neuroscientists in PeerJ though. The second issue is visibility. One idea could be to manage a list of computational neuroscience papers, to which anyone could submit a reference or preprint, in addition to automatic additions (say from relevant journals + keyword alerts). One would simply subscribe to the list (RSS feed or email alert).

Any good ideas are welcome!

Update (6.1.2017)

Some more information and thought following comments on this text:

  • One issue for an overlay journal is to get it indexed in Pubmed. This seems quite important for a neuroscience journal, but the indexing requirements seem to be pretty much contradictory with the idea of an overlay journal, as it requires XML-tagged data of its bibliographic citations. But maybe this could be negotiable? (see "Optional arrangement"). Some more general information about indexing here.
  • Other issue is more technical; to get an automatic post-publication review system.
  • Finally a general thought; there is no reason why this should be specific to computational neuroscience. However, this could be a first step towards a more generalized system.

You are not a good scientist if you use impact factors for evaluating research

A lot has been written about impact factors, and why they are a poor indicator of article quality (just google it). Even for the one thing that the impact factor is meant to measure, ie, future citations, it performs badly. There are now many empirical studies on this. Sadly, in 2016, scientists in hiring or promotion committees still value impact factors very highly, especially in life sciences. Yes, in a domain of science that is meant to be highly empirical, scientists still prefer to look away from compelling evidence that impact factors should not be used for evaluating research. Recently, in a PhD thesis committee, we were discussing the grade and one member, an experimental biologist, raised the issue that the candidate had not published in the highest ranked journals. We opposed that we were there precisely to evaluate the manuscript itself as a whole, not some elements of CVHe admitted later that he had not read the thesis.

I could simply cite the overwhelming evidence. But I want to make a simpler point of statistics. Many things are wrong with using impact factors, but let us just consider the one thing it is meant for, ie, predicting the number of future citations of papers. We know that for any journal, the distributions of citations is very broad and highly skewed. According to Seglen (1997), “articles in the most cited half of articles in a journal are cited 10 times as often as the least cited half”. In practice, if you are given the name of the journal in which a paper has been published, you will not know its future number of citations. From a purely statistical point of view, journal name is just not very significant.

Examples are sometimes more convincing that hard facts. So from the paper I just cited, number of citations vs. impact factor for all publications of 4 scientists:

impact

Now look at my own publication statistics. To be more rigorous, I should calculate the citations per year, but the point is obvious enough. My top 3 cited papers were published in: J. Neurophys (IF 2.8); J. Comp. Neuro (IF 1.8); Frontiers in Neuroinformatics(IF 1.3). This third paper (on the Brian simulator) was published in 2008, and the same year I had a paper in Neuron, which has gathered about 4 times fewer citations. And this is in spite of the fact that papers published in big journals are obviously much more visible (ie there is an expected boost in citations that is unrelated to article quality).

Is this really surprising? In a big journal, just like in a small journal, decisions are based on n = 2 or 3 reviewers, selected by one editor, i.e. a biased sample of size 2/3; just after an editorial selection based on n = 1 editor, in many cases not a professional scientist.

It’s 2016 and life scientists have been recently warned repeatedly that good statistics require large unbiased samples, but impact factors, totally insignificant pieces of information, are still broadly used to distribute grants and make careers. So yes, using impact factors as a proxy to evaluate the quality of papers or of their authors is a display of poor scientific practice.

To me the problem with this is so not much that bad decisions are being made, but that it has a huge influence on the way science is done. Scientists know that they are being evaluated in this way, and this influences their research (especially for students and postdocs) so that it meets the editorial biases of the big journals. In 1952, Hodgkin and Huxley published 120 pages in Journal of Physiology, basically defining the field of electrophysiology. Who would take the risk of doing this today at the age of 35, instead of a career-making 4-page Nature paper?

A praise of post-publication review

We have all suffered from the inadequacy of the current publication system. As readers, we often cannot freely access papers; when we can, papers often become available a very long time after having been written, sometimes years. As authors, we spend a large amount of our time submitting the same paper to many journals, reformatting, addressing criticisms that are sometimes contradictory, only to be rejected for subjective reasons (“readership interest”). As reviewers, multiple submissions mean that the same paper gets evaluated multiple times only to eventually put most reports in the trash. Our reviewing work is free and we do not get any credit for it. As hiring committee members, we have very little information to judge CVs because we do not have access to expert reviews on the papers, which are kept confidential by the journals. As a result, we use journal names as a proxy for quality. As candidates, we suffer from the randomness of editorial selection and the fact that committee members use journal names as a proxy for quality. As tax payers, we pay for an extremely inefficient, redundant and expensive publication system, largely to for-profit journals that make huge margins.

There is a solution to these problems; it’s called post-publication review. Journals belong to the past. We do not need journals to publish anymore. To publish means to make your work public. You publish when you decide to put your work online. Nowadays we call this a “pre-print”. The term itself is awfully misleading: is printing a paper really the goal of the scientific publication system? When was the last time you read a paper in actual printed journal? If I put my paper on arxiv, bioarxiv or another site then it is published, because one can publicly access it.

An objection is: scientific papers must be peer-reviewed. But why? Because we want to know what the community, in particular the most expert scientists, think of the paper. Does the current system allow that? No, because in almost all journals, reviews and author responses are confidential. Peer review can be also useful to correct errors. Great, but why should this correction process stop at publication time? If you want to let other scientists make useful suggestions, it makes so much more sense to publish your paper so that anyone can do this. If all the process is public, then we do not need the journal name to judge the quality of a paper; we can directly read what the community thinks, objects, whether there might be alternative interpretations, whether the authors have counter-arguments, all useful things that are currently hidden by an opaque review process. Finally, journal names are used as a quality label by hiring committees and grant agencies. But that’s bad; we know that journal impact factor and paper influence (eg future citations) are poorly correlated. Committees use it because they are often not in the exact field of the candidate and therefore have little information to judge the quality of papers. If reviews were public, and ideally signed, each candidate’s paper would come with several detailed recommendations and criticisms, much more informative than journal name.

The new publication process would be: publish online, then ask for public reviews, answer and revise. How would you get to know the important papers in your field if journals disappear? Well, there are many ways: keyword or author-based email alerts, most positively reviewed papers, papers liked or reviewed by peers you respect, blogs that make a monthly selection of papers (= journals), PageRank algorithm, etc.

This is not a utopia: it is coming, although quite slowly because we scientists appear to be a very conservative group of people – still happily reviewing for private journals that make huge profits from our free work, most of which is thrown away. There have been many initiatives these last years. PLoS has pioneered open access. eLife publishes reviews and responses after publication. Nature communications just started the same thing. Pubpeer allows post-publication comments. F1000Research publishes papers and then does post-publication reviews. And now the Wellcome trust implements this same scheme for its grantees.

The question then is not, can we realistically switch to a post-publication process, but rather: what can we do to accelerate this switch?

There are a few ideas in Michael Eisen (founder of PLoS) “Mission Bay Manifesto on Science Publishing”. Some are more difficult than others. Here are some ideas that anyone could easily implement:

  • Remove journal names from your online publication lists, and from your presentations. Anyone interested can find out anyway.
  • Start writing post-publication reviews, for example in Pubpeer, but on papers you are interested in – not just pointing out flaws.
  • Post every paper on arxiv, bioarxiv , before sending it to a journal.
  • Think about what journal to submit to. Whenever possible, submit to open-access non-profit journals instead of for-profit journals. There is for example the PLoS journals. As much as you can, submit to journals with public reviews. For example, eLife.
  • If you are in hiring committees, try to resist the journal name criterion.
  • Start a blog and make an annotated selection of papers you like.

Journals belong to the past. We have the opportunity now to build a much better scientific publication system for all parties involved, and we should do it.

 

A few ideas to improve the grant system

In France, the success rate for national grants has gone below 10% (with the ANR). Not surprisingly since the budget has dropped in recent years. Unfortunately, there is not much else that is available for labs (apart from European grants, which have similar success rate). The situation is also quite difficult in the American's NIH (although not as bad). With success rates so low, many scientists complain about the time wasted on writing grants that eventually seem to be more or less arbitrarily selected. This is not just an expression of the frustration of losers: once again a study comes out showing that peer review of grants is no better than chance, when applied to the top grants. The public should be aware of this: writing a grant application takes a significant amount of time (not mentioning reviewers time), and 90% of this tax-payed timed is completely wasted, in a process that is not better than lottery. Does this really make sense?

How have we arrived to such an absurd situation? There are different factors. The first obvious error I see is to fund scientific projects almost entirely based on a competitive grant system. Only the best science should be funded, we hear. Makes sense. Really? Given that both the basic infrastructure (administration, space) and the salary of permanent scientists are paid for anyway, what this advertising slogan means concretely is that only a small proportion of paid scientists will be given the means to work normally. How does that make sense? So the first obvious thing to do is to ensure that there is a baseline that allows labs to work normally (eg pay for conferences, publications, basic equipment). Why would you permanently hire a scientist and then not give her the means to work? Absurd, but that's what the CNRS currently does (main French research organization). INSERM (medical research organization) gives a start-up package to junior scientists; it covers only two years though, but is certainly an improvement.

It is only quite recently that there is a competitive grant system in France. Many people didn't (and still don't) like it, partly for the obvious reasons I just discussed. Personally, I don't think it's a bad thing in principle. It has one value, which is to shift the power from lab directors to individual scientists, who can be more independent (that is, if they get a grant). The problem is not so much that it exists, but that it entirely replaces any other means of funding.

But a competitive grant system doesn't necessarily need to be based on projects. Great if you have a particular project that requires specific expensive equipment, for example. Or a project that requires simultaneous funding of several partners. But in most cases I know, scientists use the grant system to fund the daily work of their lab, and pitch it into a “project”, sometimes assembling different arbitrary “partners” just because the system requires it. Everyone knows that the best way to get a grant is when you already have the results but have not published them yet (so-called “preliminary results” that grant reviewers like so much). Then you get the grant to fund the “preliminary results” of the next one. It is not always like that; sometimes you really do have a good project idea – although in most cases you will use the budget parly for things that are not in the grant anyway, since it's research, not an engineering project. This happens because the funding system is built this way. Scientists have no choice but be cynical, which also results in a huge waste of resources. For programmatic funding (ie funding political priorities, say research on energy), a project-based grant system might make sense. But for basic research, all we ask the funding system is to fund individual scientists or labs because it is believed that they will do good science. For experienced scientists, the best way to predict whether they will do good science is probably to just look at what they did before. Anyway, that's what grant reviewers partly do when they judge CVs and “preliminary results”, with the added bias of story-telling skills that the project system emphasizes. So, there should be a competitive system that allocates grants based on previous work. This system already exists: it is used for hiring and promotion; it also exists at the level of teams and labs, except it has little impact on budgets. Let us just expand it and combine it with some long-term budget, and a lot of resources will be saved.

One possible objection is that it is difficult to estimate budgets if there is no specific project. True, but in reality this is exactly the same problem for project-based grants. Reviewers, who are often international scientists, cannot and do not judge budgets. In practice, the institution generally sets orders of magnitude for budgets, and applicants just ask for the maximum budget (there are of course exceptions).

A second objection is that it doesn't work for junior scientists, since they don't have much to show in their CV. I disagree, because they are hired through a highly competitive process, which is not much based on projects but on their achievements. So by design, the junior scientist did go through a highly selective process and should be granted budget. This is just called a start-up package and should be generalized. There is a valid concern, however, which is to compare scientists that are at very different stages in their career. But this doesn't sound like an impossible difficulty; one could use for example a similar system as the ERC (starting/consolidator/advanced).

All this being said, I believe there is still a need for project-based grants, but with a smaller perimeter. Projects make sense when they require setting up a collaboration, or buying some specific equipment, or within programmatic constraints. Not so much in general to fund a single team or individual. I would like to make a final suggestion. If the project-based system still has a very low success rate, so low that selection is random, then there is a simple way to increase it and waste less ressource. One suggestion made in the abovementioned paper is to use a lottery. Indeed if the process is no better than chance, we might as well dispense ourselves of the review process and save some time for reviewers. This is an improvement, but it still wastes the time of applicants; I don't think it would be received very well. There is another way that also saves the time of applicants: let them apply only every other year (say, if you are born on an even year then you can apply only on even years). The number of funded grants would stay the same of course, if the budget is the same, but the success rate would automatically increase. Maybe not by a factor of two, but probably not far from it. Note that this is not the same as the individual strategy to decide to apply only every two years; when selection is random, it is in your best interest to apply every year, because each application is independent. There is not even any reason to modify your application between successive years. Whereas if success rate is forcefully increased by blocking part of the applications, the same individual success rate is obtained by applicants with half as many applications. In addition, it makes selection less random. This system is clearly better than lottery since you know in advance that you will be blocked and don't have to write a grant. You can also prepare better for the next round.

In summary, I have made four complementary propositions:

  1. Ensure a baseline budget for each individual scientist or team, which allows them to work normally (conferences, publication costs, basic equipment).
  2. Have a competitive grant system based on past rather than hypothetical future.
  3. Have a smaller competitive project-based grant system.
  4. Increase success rates of competitive systems (possibly both) above chance level by letting scientists apply only every two years.

None of these propositions depends on total budget – which is clearly too low in France, but that's an obvious point.