My new year resolution : to help move science to the post-journal world

I wish I could make the world a better place. I would like to prevent climate change and wars; but that’s not so easy on a personal level. What I can try to do more modestly as a scientist, is to make the scientific world a better place. We have all heard the phrase “publish or perish”. We all complain that careers are made by publishing in “high-impact” journals who favor story-telling and grand claims, and generally select papers arbitrarily (let alone that they do not even predict the impact of papers they publish); a trend that has been increasingly strong and has very negative impact on how research is done, including serious ethical problems. But what do we do concretely about it? For most of us including myself, not much. We keep on submitting to those journals, and we say we have no choice because that is how we are evaluated (for positions or grants). But who evaluates us? Surely there are some political aspects to it (mostly for funding), but the truth is, we are evaluated by peers. In brief, we are inflicting this madness on ourselves.

So: let us stop complaining and try to change the situation. I have previously exposed a vision of how the academic publishing system could look like without journals (by the way, this is not an original thought, fortunately). How to make it happen?

Where we are now

We should be hopeful, because many good things are happening:

  • Preprint servers are getting more and more attention. In biology, a number of journals are now accepting direct submissions from biorxiv, including all the PLoS journals, PNAS, eLife, the Journal of Neuroscience. This ought to boost submissions of preprints.
  • A few journals have started publishing the reviews along with the accepted paper, for example eLife, eNeuro and Nature Communications.
  • More generally, open access to both paper and data is getting more and more common and enforced.
  • A new journal, F1000 Research, now practices post-publication review. The paper is indexed in pubmed once two reviewers have approved it.
  • Very significantly, the Wellcome Trust has opened a journal for its grantees, Wellcome Open Research, based on post-publication review (in partnership with F1000), with this statement “The expectation is that this, and other similar funder platforms that are expected to emerge, will ultimately combine into one central platform that ensures that assessment can only be done at the article level”.
  • Finally: Pubpeer, started just a few years ago. A simple idea: to let anyone comment on any paper, anonymously or not, and let the authors know and respond. You should install their browser plugin. This is an individual initiative but it has already made a big impact, in particular by showing that the “big journals” are not better than the other ones in preventing flaws or frauds. It also addresses the concern that open reviews would be too nice: anyone who finds serious flaws can spot them anonymously and the authors will have to consider them. Pubmed commons is similar, but with signed comments.

What we can do now on a personal level

  1. Put every single paper you write on a “preprint” server before you submit to a journal.
  2. Put all your data online, see eg OpenAIRE.
  3. Remove journal names from the publications in your website. People who care about them will find out anyway.
  4. Start a literature search routine that does not involve looking at tables of contents; a few ideas in this Twitter thread; you could also have an author alert on Google Scholar.
  5. Write comments on Pubpeer; including on “pre-prints”.
  6. Send your papers to a journal with open post-publication review. I know this one is difficult, because the community still cares about impact factors. But at least you can favor those with public reviews (eg eLife, Nature Communications; I would prefer the former as it is non-profit). Instead of sending your papers to Frontiers, send them to F1000 Research; or at least eNeuro.

At the local community level, we can advocate for post-publication review. For example, the INCF has opened a channel on F1000 Research. Maybe we could have a computational neuroscience channel there, sponsored by the OCNS. It is too bad that F1000 Research is for-profit rather than institutional, but currently I do not know of other options.

What we can do on a more global scale

Open post-publication review potentially addresses the issue of recognition, but it does not address the issue of visibility. One concern I have by submitting in F1000 Research (for example), is that my intended readership will not know about it. There are so many papers published each year, one does not even have the time to read the title of all of them. This is one role journals have fulfilled: to select papers worthy of interest for a given community. But since we do not need journals anymore to publish anything, editorial selection and publication need not be coupled anymore. So here is my proposition. We make an independent website which lets any scientist, or possibly any group of scientists, be their own journal. That is, make a selection of papers they find interesting (including preprints). We provide a number of tools to make this as simple as possible: linking to pubpeer and pubmed commons, searching/browsing, alerting authors whose work is selected, email alerts and RSS feeds, etc. Papers are preferentially linked to the preprint if it exists, so as to completely bypass the journals. We could also let authors suggest their own paper for editorial selection. Basically, we provide all the services a journal typically has. This will be made increasingly easier as public open reviews become more widespread. These new “journals” could be run by an individual scientist, or a lab (eg linked to a journal club), or possibly a scientific society or group of scientists. Let us call any of these an “editor”. I would be happy for example to follow the selections of a few authors I respect, and that would be probably more valuable to me that the selection made by any journal, of which very few typically catch my attention in a given table of contents.

I am hoping that it goes as follows:

  1. People start using these individual journals, because it provides relevant information to them.
  2. As a result, papers in less well-known journals and preprints start getting more attention, and more citations.
  3. People take the habit of putting their papers on preprint servers because they get immediate attention.
  4. Editors progressively stop selecting papers published in journals because they have already selected them when they were preprints.
  5. As editors are also committee members, journal names start to matter less in evaluating research.
  6. Traditional journals disappear; instead, we have direct publication (formerly known as preprints) + open public reviews, both anonymous and signed.

How to get it started?

One simple idea to get it started is to make automatic channels for the actual conventional journals. For each journal, we list the table of contents, linked to preprint versions instead of the journal website, and to pubpeer, etc. If it’s convenient, people might start to use it, especially if it allows free access to the papers (legally, since we would use preprints). Then to get people to use the non-conventional channels, we provide suggestions based on content similarity (ie “you may also like...”).

How about this resolution?

A vision of the post-journal world

In 2016, the academic publication system worked mostly as follows :

  1. Scientists do some research and write a paper about it to share their findings with the world.
  2. The paper is submitted to a journal, chosen for its prestige.
  3. The editor, possibly a professional editor (ie not a professional scientist), judges that the paper is not worth of review: go back to (2).
  4. The editor sends the paper to 2-3 reviewers, asking them to review the paper quickly (the more prestigious, the quicker). Reviewers are chosen based on the editor’s expertise and network.
  5. Reviewers have too many reviews to write already: go back to (4).
  6. The reviewers (or one reviewer) find the paper uninteresting, or bad, or they “don’t believe it”: go back to (2).
  7. The reviews are mixed: a third reviewer is asked; go back to (4).
  8. The reviewers find the paper interesting enough but have requests for the authors; they should entirely rewrite their paper; they should do new experiments; they should cite the reviewers; they should remove an uninteresting figure; they should add some discussion points.
  9. After some work, the authors submit a revised paper.
  10. Actually, reviewers didn’t think the paper was that interesting anyway: go back to (2).
  11. The reviewers have more requests: go back to (9).
  12. The paper is accepted: congratulations!
  13. The format of references does not follow the journal’s style: please rewrite the reference list.
  14. The author’s paper now belongs to Elsevier/Springer Nature/etc. It will be sold by the publisher to academic institutions, with a margin of 30-40%.
  15. (recent alternative option) The authors pay publication fees of about $2000. The paper is now freely accessible.
  16. A knowledgeable reader disagrees with the main conclusion of the paper: she writes a letter to the editor.
  17. After careful consideration, the letter is rejected.
  18. Another reader, not an expert of the field, reads the paper in a prestigious journal and believes it is highly regarded in the community, until she talks with colleagues and finds out that the paper is very criticized.
  19. The prestigious paper is commented in mainstream journals. Experts rant about the journalists.
  20. The first author applies for a position. The committee examines CVs of all candidates and selects those with publications in prestigious journals. The senior author sends an excellent letter of recommendation. The committee values it if the senior author is famous.
  21. The senior author applies for a grant. The committee selects candidates with a good “track record”, ie prestigious publications.

This publication model is wasteful, full of perverse incentives, opaque, and misleading. What I want to do here is not one more critique of this system, but rather give a vision of what the publication system could be and, in my opinion, will be, perhaps within a decade. This is a vision of a publication system centered on individual articles, rather than on journals. It is made possible today by information technology (aka “the internet”). What stands in its way is, in a word: conservatism. It works as follows:

  1. Scientists do some research and write a paper about it to share their findings with the world.
  2. The paper is uploaded to an archive site, together with the data. All is freely accessible to anyone.
  3. The authors invite an unlimited number of professional scientists (say, authors of at least one paper) to write a formal, signed, review. The reviews are public.
  4. Any other scientist can contribute a signed or anonymous review or comment, including comments on the public reviews (aka “scientific debate”).
  5. Authors respond to the reviews and may submit a revised version.
  6. Authors decide when to stop the revision process; reviewers express their satisfaction, or not.
  7. Any scientist can recommend the paper if she finds it worth reading. Tools are made available to search for papers, possibly using the number of recommendations, possibly weighted by scientist reputation (all data are freely accessible and third parties provide searching tools).
  8. A knowledgeable reader disagrees with the main conclusion of the paper: she writes a public critique, signed or anonymous, after reexamining the paper’s raw data. The authors respond publicly.
  9. Another reader, not an expert of the field, reads the paper. She notices that many knowledgeable scientists disagree with the conclusions and can read the detailed arguments and the authors’ response.
  10. The first author applies for a position. For every publication, the committee reads the reaction of the community (formal reviews, comments, recommendations) and uses it to evaluate the value of the work along all the dimensions addressed in the reactions (importance, originality, technicality, etc). The committee also asks the candidate what her main contributions are in her view and why. The candidate can point to published reactions to her work that support her claims.
  11. The senior author applies for a grant or a promotion. The committee reads the reaction of the community to the candidate’s previous work. They might also look at review invitations and reviewing activity.

In the post-journal world, a paper is submitted a single time (but of course it can be revised), and in contrast with the journal model, the reaction of the community is known. As a consequence, quality stamps (journal names) are obsolete. No review is thrown away. As a consequence, scientists save time on both submitting and reviewing. Scientists are not incentivized to make grand claims. In addition, the system saves several billions of dollars each year (eg US libraries spend $2 billion each year on subscriptions), not counting the time saved on wasted reviews and submissions.

All of this is clearly possible with today’s technology. The next question I will try to address is: how to move quickly to the post-journal world?

Do we need a new computational neuroscience journal ?

Where to send computational and theoretical neuroscience papers? A list of journals can be found on Jim Perlewitz’s page. First, many journals belong to one of the scientific publishing companies who make huge profits from the free work of academics (margins around 30-40%), with questionable added value. In computational neuroscience, we have:

  • Journal in Computational Neuroscience (Springer Nature)

  • Frontiers in Computational Neuroscience; in Neuroinformatics (Springer Nature)

  • Biological Cybernetics (Springer Nature)

  • The Journal of Mathematical Neuroscience (Springer Nature)

  • Cognitive Neurodynamics (Springer Nature)

  • Neuroinformatics (Springer Nature)

  • Network: Computation in Neural Systems (Taylor & Francis)

Then we have Neural Computation, run by MIT Press, which is non-profit. Unfortunately, papers are behind paywalls. In my case, the library does not have a subscription.

Finally, there is PLoS Computational Biology, a highly respected journal in the field. However, even in this non-profit journal, open access is very expensive ($2250 per paper). I find these costs hard to justify, when any scientist can publish on archives for free (eg arxiv or biorxiv). The solution to this is overlay journals: the papers are linked to an archive site; the journal only deals with peer-review.

I was also recently rather disappointed by PLoS CB, after a paper got rejected 10 month after submission on the sole basis of interest. Based on the reactions I received, this is not an isolated case. Anyway, PLoS CB practices editorial pre-selection, which is highly subjective. As for other prestigious journals, its prestige is precisely based on its relatively low acceptance rate. But this is a highly questionable model, because it is wasteful: wasting the time of authors who submit the same paper to many journals, wasting the time of reviewers who write reports that are thrown away. In my opinion, it is also perverse. It gives a label of quality to individual papers and scientists based on the opinion of just a couple of scientists, sampled by one editor, and this insignificant label is then used by various committees (grants, promotions, hiring). It also pushes authors into boosting their claims to pass the editorial barrier. The solution to this is post-publication review (see also Michael Eisen’s post).

So ideally, I would like to see an overlay journal practicing post-publication review in my field. The question now is: should we create one? What would be needed is the technical infrastructure, but I am not sure it is fully available yet (as regards the post-publication review system, indexing etc).

Alternatively, we could use a general post-publication review journal. There is for example F1000 Research. PeerJ also does this partially (publishing the reviews is an option). Both cost about $1000 per article; both are run by companies. But they are less expensive than alternative open access journals (PLoS ONE: $1495; eNeuro: $1950 for members, $2925 otherwise), and more open. None of these two journals is ideal, but they represent some progress over conventional journals.

There are two issues with choosing a general journal. First, there is the question of editors. I have no experience with these journals so I am not sure there is the relevant expertise in the editorial board. I have seen a couple of computational neuroscientists in PeerJ though. The second issue is visibility. One idea could be to manage a list of computational neuroscience papers, to which anyone could submit a reference or preprint, in addition to automatic additions (say from relevant journals + keyword alerts). One would simply subscribe to the list (RSS feed or email alert).

Any good ideas are welcome!

Update (6.1.2017)

Some more information and thought following comments on this text:

  • One issue for an overlay journal is to get it indexed in Pubmed. This seems quite important for a neuroscience journal, but the indexing requirements seem to be pretty much contradictory with the idea of an overlay journal, as it requires XML-tagged data of its bibliographic citations. But maybe this could be negotiable? (see "Optional arrangement"). Some more general information about indexing here.
  • Other issue is more technical; to get an automatic post-publication review system.
  • Finally a general thought; there is no reason why this should be specific to computational neuroscience. However, this could be a first step towards a more generalized system.

You are not a good scientist if you use impact factors for evaluating research

A lot has been written about impact factors, and why they are a poor indicator of article quality (just google it). Even for the one thing that the impact factor is meant to measure, ie, future citations, it performs badly. There are now many empirical studies on this. Sadly, in 2016, scientists in hiring or promotion committees still value impact factors very highly, especially in life sciences. Yes, in a domain of science that is meant to be highly empirical, scientists still prefer to look away from compelling evidence that impact factors should not be used for evaluating research. Recently, in a PhD thesis committee, we were discussing the grade and one member, an experimental biologist, raised the issue that the candidate had not published in the highest ranked journals. We opposed that we were there precisely to evaluate the manuscript itself as a whole, not some elements of CVHe admitted later that he had not read the thesis.

I could simply cite the overwhelming evidence. But I want to make a simpler point of statistics. Many things are wrong with using impact factors, but let us just consider the one thing it is meant for, ie, predicting the number of future citations of papers. We know that for any journal, the distributions of citations is very broad and highly skewed. According to Seglen (1997), “articles in the most cited half of articles in a journal are cited 10 times as often as the least cited half”. In practice, if you are given the name of the journal in which a paper has been published, you will not know its future number of citations. From a purely statistical point of view, journal name is just not very significant.

Examples are sometimes more convincing that hard facts. So from the paper I just cited, number of citations vs. impact factor for all publications of 4 scientists:


Now look at my own publication statistics. To be more rigorous, I should calculate the citations per year, but the point is obvious enough. My top 3 cited papers were published in: J. Neurophys (IF 2.8); J. Comp. Neuro (IF 1.8); Frontiers in Neuroinformatics(IF 1.3). This third paper (on the Brian simulator) was published in 2008, and the same year I had a paper in Neuron, which has gathered about 4 times fewer citations. And this is in spite of the fact that papers published in big journals are obviously much more visible (ie there is an expected boost in citations that is unrelated to article quality).

Is this really surprising? In a big journal, just like in a small journal, decisions are based on n = 2 or 3 reviewers, selected by one editor, i.e. a biased sample of size 2/3; just after an editorial selection based on n = 1 editor, in many cases not a professional scientist.

It’s 2016 and life scientists have been recently warned repeatedly that good statistics require large unbiased samples, but impact factors, totally insignificant pieces of information, are still broadly used to distribute grants and make careers. So yes, using impact factors as a proxy to evaluate the quality of papers or of their authors is a display of poor scientific practice.

To me the problem with this is so not much that bad decisions are being made, but that it has a huge influence on the way science is done. Scientists know that they are being evaluated in this way, and this influences their research (especially for students and postdocs) so that it meets the editorial biases of the big journals. In 1952, Hodgkin and Huxley published 120 pages in Journal of Physiology, basically defining the field of electrophysiology. Who would take the risk of doing this today at the age of 35, instead of a career-making 4-page Nature paper?

This week's paper selection (7 December 2016)


A praise of post-publication review

We have all suffered from the inadequacy of the current publication system. As readers, we often cannot freely access papers; when we can, papers often become available a very long time after having been written, sometimes years. As authors, we spend a large amount of our time submitting the same paper to many journals, reformatting, addressing criticisms that are sometimes contradictory, only to be rejected for subjective reasons (“readership interest”). As reviewers, multiple submissions mean that the same paper gets evaluated multiple times only to eventually put most reports in the trash. Our reviewing work is free and we do not get any credit for it. As hiring committee members, we have very little information to judge CVs because we do not have access to expert reviews on the papers, which are kept confidential by the journals. As a result, we use journal names as a proxy for quality. As candidates, we suffer from the randomness of editorial selection and the fact that committee members use journal names as a proxy for quality. As tax payers, we pay for an extremely inefficient, redundant and expensive publication system, largely to for-profit journals that make huge margins.

There is a solution to these problems; it’s called post-publication review. Journals belong to the past. We do not need journals to publish anymore. To publish means to make your work public. You publish when you decide to put your work online. Nowadays we call this a “pre-print”. The term itself is awfully misleading: is printing a paper really the goal of the scientific publication system? When was the last time you read a paper in actual printed journal? If I put my paper on arxiv, bioarxiv or another site then it is published, because one can publicly access it.

An objection is: scientific papers must be peer-reviewed. But why? Because we want to know what the community, in particular the most expert scientists, think of the paper. Does the current system allow that? No, because in almost all journals, reviews and author responses are confidential. Peer review can be also useful to correct errors. Great, but why should this correction process stop at publication time? If you want to let other scientists make useful suggestions, it makes so much more sense to publish your paper so that anyone can do this. If all the process is public, then we do not need the journal name to judge the quality of a paper; we can directly read what the community thinks, objects, whether there might be alternative interpretations, whether the authors have counter-arguments, all useful things that are currently hidden by an opaque review process. Finally, journal names are used as a quality label by hiring committees and grant agencies. But that’s bad; we know that journal impact factor and paper influence (eg future citations) are poorly correlated. Committees use it because they are often not in the exact field of the candidate and therefore have little information to judge the quality of papers. If reviews were public, and ideally signed, each candidate’s paper would come with several detailed recommendations and criticisms, much more informative than journal name.

The new publication process would be: publish online, then ask for public reviews, answer and revise. How would you get to know the important papers in your field if journals disappear? Well, there are many ways: keyword or author-based email alerts, most positively reviewed papers, papers liked or reviewed by peers you respect, blogs that make a monthly selection of papers (= journals), PageRank algorithm, etc.

This is not a utopia: it is coming, although quite slowly because we scientists appear to be a very conservative group of people – still happily reviewing for private journals that make huge profits from our free work, most of which is thrown away. There have been many initiatives these last years. PLoS has pioneered open access. eLife publishes reviews and responses after publication. Nature communications just started the same thing. Pubpeer allows post-publication comments. F1000Research publishes papers and then does post-publication reviews. And now the Wellcome trust implements this same scheme for its grantees.

The question then is not, can we realistically switch to a post-publication process, but rather: what can we do to accelerate this switch?

There are a few ideas in Michael Eisen (founder of PLoS) “Mission Bay Manifesto on Science Publishing”. Some are more difficult than others. Here are some ideas that anyone could easily implement:

  • Remove journal names from your online publication lists, and from your presentations. Anyone interested can find out anyway.
  • Start writing post-publication reviews, for example in Pubpeer, but on papers you are interested in – not just pointing out flaws.
  • Post every paper on arxiv, bioarxiv , before sending it to a journal.
  • Think about what journal to submit to. Whenever possible, submit to open-access non-profit journals instead of for-profit journals. There is for example the PLoS journals. As much as you can, submit to journals with public reviews. For example, eLife.
  • If you are in hiring committees, try to resist the journal name criterion.
  • Start a blog and make an annotated selection of papers you like.

Journals belong to the past. We have the opportunity now to build a much better scientific publication system for all parties involved, and we should do it.


This week's paper selection (30 November 2016)

Two review papers this week on autonomous systems – should be an inspiration for computational neuroscientists!


This week's paper selection (17 November 2016)

Some more patch clamp automation this week!