bioinformatics, genomes, biology etc. "I don't mean to sound angry and cynical, but I am, so that's how it comes across"

Citations are not a measure of quality

Edited 30/07/2014 – I got confused about who wrote what and when – apologies to Embriette and Jonathan

There’s a lot of discussion about Neil Hall’s paper on the Kardashian index, and this is classic Neil stuff – funny and provocative.  I’m not going to discuss this paper directly, enough are doing it already.

Over on the microBEnet blog, Embriette Hyde states:

A high K index indicates that said scientist may have built their reputation on a shaky foundation (i.e. the Kardashians, circled in Figure from the paper below), while a low K index indicates that said scientist is not being given credit where credit is due.

I think we’re coming dangerously close to making judgements about quality, and what we’re actually measuring is citations – and they’re not the same thing!

For me, the number of citations a particular paper gets is based on:

  1. The size of the field e.g. if you work in cancer (a large field), your paper has a higher probability of being cited than if you work on an obscure fungus that only grows in Lithuanian coal mines (probably a very small field, but I may be wrong).
  2. Given the size of the field, next comes the relevance of your work.  So, sticking with the cancer example, you may work with a high prevalence cancer, such as lung cancer (70,000 deaths last year in the UK) or cancer of the penis (111 deaths in the UK last year)
  3. Then, given the size of the field, and the importance of the work, comes relevance.  Perhaps you discovered an oncogene that affects 60% of the population; alternatively you have found an oncogene that affects 2% of a small tribe from the Peruvian rainforest.
  4. Finally comes the quality of your work.  It is quite easy to imagine someone who does incredibly high quality work in a niche area getting far fewer citations than someone who does really crappy work in a larger, more important field

To be fair to Neil, and Embriette, neither directly state that citations = quality, in fact Neil deliberately uses the phrase “scientific value”, which is ill defined.

However, the fact still remains that what Neil is implying is that scientists with low numbers of citations are somehow less than those with lots of citations.  And that’s just not right.

And Embriette’s wording implies that scientists with low citations (and high numbers of Twitter followers) are on “shaky foundations”.  I say the assumptions behind that wording are on shaky foundations!

Some really very high quality work never gets cited (we can argue another time about the value of such research).  And some really quite shoddy work gets huge numbers of citations.  So I get to have the last word:  number of citations is not a measure of quality.



  1. Note. 1) That is not my personal blog. It is a group blog for a collaborative project called microBEnet. 2) I did not write the post. Embriette Hyde, a PhD student and guest blogger for microBEnet did. Embriette is relatively new to blogging and I love that she has been posting on microBEnet. I don’t think she meant anything bad in regard to the K index. I think she just thought the paper was kind of fun.

  2. My apologies, will edit – written in haste!

  3. Yes, the # of citations varies a lot between field to field (which in turn means that impact factor of the “good journals” varies a lot, for what’s that worth). But there’s another way to get lots of citations that you didn’t mention — be spectacularly wrong. According to Google Scholar, the 1989 Pons & Fleischman “Cold Fusion” paper has been cited over 1000 times despite being the classic poster child of bad science. And more recently, the 2011 Wolfe-Simon “Arsenic Life” paper has been cited over 300 times.

  4. I agree with Jonathan’s point – we do also cite “bad” papers, or at least papers that have results contrary to what we observe/believe. Realistically some of these should be struck from the records, but with the current system they never are.

  5. Mick the quotation you posted as belonging to Embriette Hyde’s is actually not hers but Neil’s. Embriette was merely paraphrasing.

    As for my thoughts regarding this post, I agree wholeheartedly with everyone. I’d also like to add, that eminence is a double-edged sword regardless of whether or not its deserved. There’s many examples of Scientists who have made phenomenal breakthroughs but rather than use their new-found fame to extend human knowledge some more, they instead used it as a platform to promote really dubious theories. Just off the top of my head I can think of: Francis Crick (panspermia), Newton (corpuscle theory of light, flawed calculus notation), Kary Mullis (AIDS denialism, LSD).

  6. Another relevant scientist who I neglected to mention Linus Pauling, whose statements about Vitamin-C’s (supposed) benefits were apocryphal at best

  7. The original paper that suggested using BWT in bioinformatics in 2002 currently has all of 35 citations, and it revolutionised the field bringing about the short read aligners like BWA and Bowtie. The mean Nature Reviews paper gets more citations, yet contains no new results (I’m not saying it isn’t valuable, but I’d rather have the BWT paper than the mean Nature Reviews paper)

Leave a Reply

© 2018 Opiniomics

Theme by Anders NorenUp ↑