Saturday, 20 April 2013

Points of Authority: BlogSyn and Peer Review

Over at The Collapsed Wavefunction, Chad Jones is talking about a paper in the Journal of Chemical Information and Modeling (admittedly not one that's on my regular reading list...) which includes a straight-faced endorsement of traditional Chinese medicine. His discussion of the paper and issues associated with it is good and definitely worth a read if you're into bad science.

The crux of the problem is one of authority. Peer review is often peddled as a gold standard of authority when debating purveyors of pseudoscience. For example, any debate around creationism (or 'intelligent design') inevitably involves defenders of biology pointing out that few, if any, peer-reviewed papers arguing for design have been published. Conversely, proponents of creationism wave a handful of peer-reviewed papers around as if they refute the much vaster literature of evolutionary biology.

Peer review conveys authority. I suspect that Chad is, in part, frustrated that the hard-earned authority of his field has been lent to something as flaky as TCM. A proponent of TCM could quite plausibly cite this paper as an example of Science™ taking their woo seriously, which not only lends false credibility to TCM, but undermines the authority of the journal that published it.

Reading Chad's post reminded me of the debates about BlogSyn earlier this year. For those who don't know, BlogSyn is an effort by several chemistry bloggers to assess the reproducibility of methods from the chemical (specifically organic synthesis) literature, and achieved notoriety after failing to reproduce some results from the justly-famous Baran lab. No impropriety on the part of Baran et al. was implied by the authors of BlogSyn; the source of the problem was eventually identified, and they all blogged happily ever after.

Following the initial post, discussion at a number of blogs (particularly In the Pipeline) often focused on the perceived lack of authority or legitimacy of the authors of BlogSyn. A relatively common point was that work of this nature ought to be peer-reviewed to ensure the competence and identity of the critics. The fact that SeeArrOh and colleagues are pseudonymous was a point of much contention; Rich Apodaca offered a nice discussion of this. One counter-argument is that they are young scientists early in their careers, and criticising the work of senior researchers could harm their prospects.

In principle, peer review ought to solve some of these problems. In peer review, critics are anonymous and fear no reprisals from those being criticised, but despite their anonymity they are vouched for by the editor of the journal. This allows both the author being reviewed and those reading the journal to assume the work is sound and hence authoritative. From this perspective, critics of BlogSyn seem to be on to something.

These two cases highlight the strengths and weaknesses of peer review. The inclusion of TCM in a chemistry paper is startling because it's the exception; peer review is generally good at ensuring that published work is logically coherent, informed by the literature, and supported by evidence to back up its claims. On the other hand, BlogSyn highlights that peer-review in chemistry routinely fails to assess the key element of science: reproducibility.

There are good reasons that peer review does not assess reproducibility in organic synthesis. Reviewing is time-consuming, and adding more time to that to prepare reagents, fiddle with conditions to get them to work, and so on, would cost the reviewer time and money and delay publication. It seems unlikely that we'll be seeing routine replication in chemistry any time soon.

BlogSyn represents one approach to solving this problem. If pre-publication review is impractical, perhaps open, online, post-publication review of work that has had an impact is a solution.

The question, then, is can we trust BlogSyn? Is it authoritative? How should we assess it relative to a journal?

I suggest that we can trust BlogSyn more than the average peer-reviewed paper, provided we assume honesty on the part of the authors.

While there are exceptions, most peer-reviewed synthesis papers include relatively sparse experimental details due to constraints of space and the need to be easily legible. BlogSyn, on the other hand, takes an open-notebook format in which one can see every TLC plate, every NMR, and multiple repeats of the same reaction by different authors. This allows for a much more direct assessment of the competence of the chemists, the chance to pick up minor mistakes that aren't evident from a written summary, and a direct demonstration of reproducibility. Hence, if we trust the authors, I consider BlogSyn to be a more authoritative account of an experiment than a typical peer-reviewed paper.

Ought we to trust the authors? This brings us back to the problem of pseudonymity. Personally, I know several of the authors of BlogSyn to be people of integrity, and to have persistent identities online. Not every reader of BlogSyn has this knowledge, and they are probably justified in being skeptical. How can BlogSyn achieve a degree of authority which is acceptable to the average chemist, who may not be part of the online community?

No easy or ideal solutions are forthcoming, and it's likely that no solution would satisfy everyone. However, until BlogSyn can gain some legitimacy in the broader community it's hard to see how the project can flourish.

Two solutions spring to mind, and I welcome your criticism and suggestions. The first is to slog it out and establish credibility the hard way: continue to critique work from the literature, build up reputation by engaging with authors and marketing the project, and hopefully gain some acceptance. Alternatively, if an established chemist with a good reputation in the community is willing to verify the identity of, and vouch for the integrity of, the authors of BlogSyn, this may serve to lend legitimacy to the project by association. This third party would be fulfilling a role analogous to that of the journal in traditional peer review.

What other ways might BlogSyn gain credibility? Are there practical ways to solve issues of reproducibility pre-publication? Leave your thoughts below...

6 comments:

  1. Reproducibility before publication is in part a false premise. There are several reasons for a lack of reproducibility.
    The experiments may well be reproducible in the lab of origin at the time the work was done, but not reproducible elsewhere, due to some subtle difference in reagent batch and/or quality. In this instance the work is reproducible but the mechanism/reagents require further definition. At the extreme end of this problem is that someone took the "wrong" bottle off the shelf and persisted in the belief that it was something else. Similarly, the lack of detail in methods in papers means that reproduction is often a challenge.
    The experiments cannot be reproduced because there has been a "fatal" mistake. That is the result is in fact impossible.
    The experiments cannot be reproduced because there are several outcomes and the authors cherry picked one.
    The results were made up.
    So we need reproducibility post publication and there's the rub. Published work is looked on as infallible, rather than a step in a process. Journals generally do not accept such challenges and even if there is out and out fraud, often prefer to issue 'corrections'. Challenging published data from within the system is bad for your CV: publishing becomes more difficult and citations to your own work can go down.
    Blogsyn is a fantastic resource. Those who condemn it may be a little insecure. Labs challenged by Blogsyn who engage with it have their reputations greatly enhanced and Blogsyn's reputation rises too. So Blogsyn just has to stay civil and keep a list of work that was reproduced and work that has not been. Alongside, a note as to whether the lab that generated the paper actually engaged with Blogsyn.

    ReplyDelete
  2. Thanks for taking the time to comment. I agree to an extent with your comments about assessing reproducibility pre-publication, but there are ways to mitigate this. For example, I suspect 'cherry picking' the best yields and selectivities is fairly common, and could be avoided by asking authors to report a range of yields/ees/etc. rather than a single (and presumably the highest) value.

    You're spot on about the published record being seen as infallible, though. BlogSyn is one way to approach this and emphasise that, as you say, a paper is a step in a process, not a stone tablet from on high. In case it didn't come across in my post, I am a big fan of BlogSyn and would like to see it continue and grow.

    A possible alternative (or complementary) approach could be for journals and search engines to add more information to citation lists for papers. You could imagine looking at papers citing a particular paper and seeing them divided into those which actually used or reproduced the technique, and those that simply include the citation as part of the literature review, etc. I'm not sure if that's practical or technically feasible, but it'd be more systematic than BlogSyn can realistically hope to be.

    Again, thanks for your comment, great points.

    ReplyDelete
  3. Andrew, apparently the universe doesn't want me to respond to this. Twice, and on two separate days I have typed a lengthy reply to this only to have something happen.

    In short:

    "A proponent of TCM could quite plausibly cite this paper as an example of Science™ taking their woo seriously, which not only lends false credibility to TCM, but undermines the authority of the journal that published it."

    This is exactly what I wanted to say, and you did it in a much more eloquent than I could have. Thank you.

    Great post. Thanks for the mention.

    ReplyDelete
    Replies
    1. Technology is a pain! Sorry about that. Glad you enjoyed it, and likewise, thanks for your initial post. It prompted some thoughts I've been having for a while to come together.

      Delete
  4. Nice piece. I see BlogSyn as an interesting and timely idea, particularly as we move into a world where the results of science are becoming more open and scrutinised. BlogSyn or similar ideas may become a routine method for scientists to communicate and discuss their findings, although I think in certain fields (including chemistry) it will need to overcome a more conservative mentality.

    There are some "problems" though with a crowd sourcing approach such as BlogSyn. The strength of a BlogSyn-like community is dependent on the interest and involvement of that community and the free time of the bloggers to post contributions, verify comments and coordinate a response from authors. This could be a full-time job in itself! Presumably/hopefully at some point the bloggers at BlogSyn will be promoted into an academic position and no longer be working in a lab; who runs the corroborating experiments then?

    We also have to be aware that the active online community of scientists still only represents a small proportion of practising scientists. There could also end up being a large number of BlogSyn-like pages, from single bloggers to larger communities, posting content online that is critical of published papers. Engaging authors who aren't part of the online community or dismiss it will be an interesting challenge, although as the Harran episode shows, a challenge that is not impossible.

    Science, and chemistry in particular, is very hierarchical. Social media communities such as Twitter and blogs like BlogSyn, which are being taken up predominantly by the younger generation of scientists, are an interesting challenge to the established hierarchy. Will it end up making a significant impact into how science is "done" or will science co-opt these tools without necessarily making any changes that are skin-deep? Giving younger scientists a voice to critique established professors and creating dialogue between scientists that may not normally talk to one another are great opportunities. However, as you imply, if those doing the critiquing aren't respected or known then will they get listened to?

    ReplyDelete
    Replies
    1. Thoughtful comment, thanks. I think you're right that there's a real possibility of multiple 'blogsyn'-type projects springing up; the "May Require Mojo" page at not voodoo represents an even more informal version http://chem.rochester.edu/nvdcgi/mojo.cgi

      This is why I think it's important for BlogSyn to continue, expand, and establish some 'street cred' with people who aren't typically involved with the online community. Having a handful of good and well-known resources seems more effective than having many individual blogs or the like, and more likely to reach the wider community.

      One thing I find promising is that this is already happening to an extent. The editorial team at Nature Chemistry seem pretty keen on BlogSyn, and it's been featured in their 'blogroll'; Phil Baran engaged with BlogSyn very nicely, and his group now run Open Flask. It may be that as BlogSyn continues, these small interactions with the wider community will accumulate into a measure of legitimacy.

      I am very interested in what will happen if or when SeeArrOh, BRSM and co do end up in senior positions. If BlogSyn is still around at that point, it could be a lab hobby, or it might be that it could become a more formal project. Who knows?

      Delete