bioinformatics, genomes, biology etc. "I don't mean to sound angry and cynical, but I am, so that's how it comes across"

Month: March 2018

With great power comes great responsibility

Recently I published a blog post about a fairly simple test to find out whether you have “short” protein predictions in your bacterial genomes, and predicted that some of these short peptides may be the result of unresolved errors in long-read, single molecule assemblies.

Perhaps not surprisingly, there was a reaction from the PacBio community over this, and here is my response.

Before I begin, I just want to say that, whilst most people see me as some kind of Nanopore fan-boy, the reality is I am a fan of cool technology and that includes PacBio.  The hard facts are that I have spent around £200k on PacBio sequencing over the last 18 months, and about £20k on nanopore in the same time period.  I also encouraged our core to buy a PacBio Sequel.  So I am not anti-Pacbio.  I am, however, anti-bullshit 😉

In addition, my blog post wasn’t about problems with technology per se, it was about problems with people.  If you don’t know errors are there, you might not correct them, and if you believe company hype (for example that PacBio data is Q50 after polishing) you might believe your assembly is perfect.

It isn’t.  They never are.

Let’s dive into some data.  The three PacBio genomes I chose in the last blog post are:

AP018165	Mycobacterium stephanolepidis 
CP025317	Escherichia albertii 
ERS530415	Yersinia enterocolitica

if you search GenBank genomes for these, there is only one Mycobacterium stephanolepidis genome, but there are 39 Escherichia albertii genomes and there are 176 Yersinia enterocolitica.  Many of these will be sequenced on different technologies, which allows us to do a comparison within a species.

Escherichia albertii 

I downloaded the 39 genomes from GenBank, and ran the process I described last week.  For the complete genomes, this is what the data look like, ordered from lowest number of short peptides, to greatest:

Accession Technology Contigs # proteins # short
GCA_001549955.1 Sanger+454 1 4299 19
GCA_002285455.1 Sanger+454 1 4157 19
GCA_002285475.1 Sanger+454 1 4295 28
GCA_002741375.1 PacBio 1 (plus 6 plasmids) 5126 79
GCA_002895205.1 Illumina+454+Ion 1 4423 83
GCA_000512125.1 PacBio 1 4482 106
GCA_002872455.1 PacBio 1 (plus 3 plasmids) 4984 124

Sure is unlucky that PacBio assemblies are at the bottom of the table isn’t it?  Of course, Sanger is the gold standard, and many will be asking what the Illumina assemblies look like.

Let’s look at the next ten genomes, which are not complete, but are the least fragmented:

Accession Technology Contigs # proteins # short
GCA_002109845.1 454 25 4758 93
GCA_001514945.1 Illumina 43 4237 82
GCA_002563295.1 Ion 44 4531 143
GCA_001514965.1 Illumina 50 4390 86
GCA_001515045.1 Illumina 53 4480 110
GCA_001515005.1 Illumina 59 4963 117
GCA_001514645.1 Illumina 63 4470 84
GCA_001514685.1 Illumina 70 4209 68
GCA_001514925.1 Illumina 73 4464 84
GCA_001514905.1 Illumina 78 4622 113

What I think is worth pointing out here is that the PacBio genomes in the first table, which are complete, have about the same number of short proteins as the Illumina and 454 assemblies in the second table, which are fragmented.  We would usually expect fragmented assemblies to have more short proteins because the contig ends would interupt ORFs.   Indeed, compared to Sanger complete assemblies, the fragmented assemblies do have more short proteins.

They just don’t have more than the PacBio complete assemblies.  Odd that.  If there were no uncorrected errors in the PacBio assemblies, they would be more like the Sanger assemblies than the Illumina ones.

Yersinia enterocolitica

Of the 176 GenBank genomes for this species, my automated script could only detect sequencing technology for 35 of them.  Here are the 21 that have a claim to be complete or near-complete (<5 contigs)

Accession Technology contigs # proteins # short
GCA_000834195.1 Illumina+454 2 4161 17
GCA_000987925.1 PacBio 2 4340 19
GCA_001304755.1 PacBio 2 4344 20
GCA_002082275.2 PacBio+Illumina 1 4067 31
GCA_002554625.2 PacBio+Illumina 1 4384 35
GCA_001305635.1 Illumina 2 4162 35
GCA_000834795.1 PacBio+Illumina 2 4259 36
GCA_000755045.1 Illumina+454 2 4226 44
GCA_000834735.1 PacBio+Illumina 2 4138 48
GCA_000755055.1 Illumina+454 1 4095 53
GCA_000754975.1 Illumina+454 2 4083 53
GCA_000597945.2 Illumina+454 3 4640 59
GCA_000754985.1 Illumina+454 3 4282 65
GCA_002083285.2 PacBio+Illumina 4 4198 76
GCA_002082245.2 PacBio+Illumina 2 4521 106
GCA_001708575.1 PacBio 3 4673 221
GCA_001708615.1 PacBio 3 4615 224
GCA_001708635.1 PacBio 3 4659 224
GCA_001708655.1 PacBio 3 4654 226
GCA_001708555.1 PacBio 3 4674 230
GCA_001708595.1 PacBio 2 4733 235

Slightly different story here, in that the PacBio (and PacBio hybrid) genomes appear to have some of the lowest number of predicted short proteins.  This is what people now expect when they see PacBio bacterial genomes.  However, there are also six PacBio genomes at the bottom of the table, and so I don’t think you can really look at this data and think there isn’t a problem.  It’s possible that those six just happen to be the strains of Yersinia enterocolitica that have undergone the most pseudogenisation, but I don’t think so.

Let’s get some things straight

  • I know this is an incomplete analysis and obviously more work needs to be done
  • If I personally wanted a perfect microbial genome, I would probably use PacBio+Illumina
  • I have nothing against PacBio
  • Nanopore aren’t in this list because they’re not in the species I chose, but I am sure they would also have significant problems


As I said above, it’s not that PacBio has a problem per se, it’s that people have a problem.  Yes, many errors are correctable with Quiver, Arrow and Pilon; often multiple rounds are necessary and you still won’t catch everything.

But not everyone knows that, and it’s clear to me from the above that many people are still pumping out poor quality, uncorrected, indel-ridden PacBio genomes.

The same is true for Nanopore, I have no doubt.

Let’s stop bullshitting.  These technologies have problems.  It doesn’t mean they are bad technologies, anything but – both PacBio and Nanopore have been transformational.  There is no need for bullshit.

“With great power comes great responsibility” – in this case, the responsibility is two-fold.  One, stop contaminating public databases with sh*t assemblies; and two, stop bullshitting that this isn’t a problem for your favourite technology.  That helps no-one.


A simple test for uncorrected insertions and deletions (indels) in bacterial genomes

A friend and a colleague of mine once said about me “he’s a details man”, and it was after we had discussed the fact some of my papers consist solely in pointing out the errors other people ignore – in RNA-Seq for example, or in genome assemblies (I have another under review!).

By now, those of you familiar with my earlier work will be jumping up and shouting

“Of course!  Even as far back as 2012, he was warning us about the dangers of mis-annotated pseudogenes!  Just look at Figure 2 in his review of bacterial genome annotation!”

Well, let’s dig into that a bit more.

For the uninitiated, the vast majority of bases in most bacterial genomes fall within open-reading frames (ORFs) that are eventually translated into proteins.  Bacterial evolution, and often niche specificity, is characterised by the psedogen-isation of these ORFs – basically, many replication errors in ORFs (especially insertions and deletions) can introduce a top codon, the transcript and resulting protein are truncated, they are often no longer functional, and you have a pseudogene.  This happens when the ORFs are not under positive selection.

HOWEVER.  Sequencing errors can also introduce errors, and you end up with incorrectly annotated pseudogenes.

And OH NOES we just so happen to be flooded with data from long read technologies that have an indel problem.

Let’s assume you have a shiny new bacterial genome, sequenced on either PacBio or Nanopore, and you want to figure out if it has an indel problem.  Here is my simple and quick test:

First predict proteins using a gene finder.  Then map those proteins to UniProt (using blastp or diamond).  The ratio of the query length to the length of the top hit should be a tight and normal distribution around 1.

OK so what does this look like in practice?

Let’s look at a genome we know if OK, E coli MG1655:


On the left, we have the raw histogram, and on the right we have zoomed in so the y-axis ends at 500.  Generally speaking, this has worked I think – the vast majority of predicted proteins have a 1:1 ratio with their top hit in UniProt.

Here is Salmonella gallinarum which we know “has undergone extensive degradation through deletion and pseudogene formation”


Well, would you look at that!  There they are, all those pseudogenes appearing as predicted proteins that are shorter than their top hit.  For those of you saying “It’s not that obvious”, in the plots above, MG1655 has 157 proteins < 90% of the length of their top hit, whereas for Salmonella gallinarum the figure is 432.   By my recollection, gallinarum has over 200 annotated pseudogenes.

If you’re convinced that this actually might be working, then read on!

So here are a couple of PacBio genomes.  I genuinely chose these at random from recent Genome Announcements.  Generally speaking, all had high PacBio coverage, were assembled using Canu or HGAP, and polished using Pilon.

In no particular order:


(237 proteins < 90% of top hit)


(246 proteins < 90% of top hit)


(268 proteins < 90% of top hit)

Now, these are not terrible numbers, in fact they are quite good, but those long tails are a tiny bit of a worry, and are definitely worth checking out.  Bear in mind, if 200 pseudogenes in Salmonella gallinarum is “extensive degradation through deletion and pseudogene formation“, then 200 incorrect indel pseudogenes in your genome is a problem.  In the above, I can’t possibly tell you whether these  numbers of predicted short/interrupted ORFs are real or not because I don’t know about the biology.  However, I can say that given they were created with a technology known to have an indel problem, they are worth further investigation.

Indel sequencing errors are not the only problem, so are fragmented assemblies.  In fragmented assemblies, the contig ends also tend to be in ORFs and again, you get short/interrupted proteins.  Here is one of our MAGs from our recent rumen paper:


(310 proteins < 90% of top hit)

It’s not all bad news though, some of the MAGs are OK:


(126 proteins < 90% of top hit)

And at least other peoples’ MAGs are just as bad (or worse):


(265 proteins < 90% of top hit)

My absolute favourite way to assemble genomes is a hybrid approach, creating contigs with Illumina and then stitching together with a small amount of nanopore data.  We did this with B fragilis and the results look good:


(63 proteins < 90% of top hit)

So are there any examples of where it goes really badly hideously wrong?  Well the answer is yes.  We are assembling data from metagenomic samples using both PacBio and Nanopore.   The issue with metagenomes is that you may not have enough coverage to correct errors using the consensus approach – there simply aren’t enough bases at each position to get an accurate reflection.  We are seeing the same pattern for both PacBio and Nanopore, so I won’t name which technology, but…. well scroll down for the horror show.













Are you ready?  Scroll down….











A 3Mb contig from a Canu assembly:


(2635 proteins < 90% of top hit)

Now, the only work this has had done on it is that Canu performs read correction before assembly.  We did one round.  Sweet holy Jesus that’s bad isn’t it??

Can it be corrected?  To a degree.  Here it is after two rounds of Pilon and some technology-specific polishing:


(576 proteins < 90% of top hit).

So less bad, but still not great.  I’d say this assembly has problems, which is why we haven’t published yet.

What’s the take home message here?  Well there are a few:

  1. Errors matter.  Pay attention to them.
  2. Both long read technologies have indel problems and these probably cause frameshifts in your ORFs
  3. Polishing and consensus and Illumina correction helps but it doesn’t catch everything
  4. The problem will inevitably be worse if you have low coverage
  5. Comparing predicted proteins with their top hits in UniProt can identify where you have errors in your ORFs (or real pseudogenes!)

Code for doing this type of analysis will appear here.

© 2018 Opiniomics

Theme by Anders NorenUp ↑