Consistent probabilistic output for protein function prediction

Guillaume Obozinski, Charles Grant, Gert Lanckriet, Michael I. Jordan, William S. Noble

Genome Biology 2008, 9(Suppl 1):S6 [full text]

---

Abstract

In predicting hierarchical protein function annotations, such as terms in the Gene Ontology (GO), the simplest approach makes predictions for each term independently. However, this approach has the unfortunate consequence that the predictor may assign to a single protein a set of terms that are inconsistent with one another; for example, the predictor may assign a specific GO term to a given protein ('purine nucleotide binding') but not assign the parent term ('nucleotide binding'). Such predictions are difficult to interpret. In this work, we focus on methods for calibrating and combining independent predictions to obtain a set of probabilistic predictions that are consistent with the topology of the ontology. We call this procedure 'reconciliation'. We begin with a baseline method for predicting GO terms from a collection of data types using an ensemble of discriminative classifiers. We apply the method to a previously described benchmark data set, and we demonstrate that the resulting predictions are frequently inconsistent with the topology of the GO. We then consider 11 distinct reconciliation methods: three heuristic methods; four variants of a Bayesian network; an extension of logistic regression to the structured case; and three novel projection methods - isotonic regression and two variants of a Kullback-Leibler projection method. We evaluate each method in three different modes - per term, per protein and joint - corresponding to three types of prediction tasks. Although the principal goal of reconciliation is interpretability, it is important to assess whether interpretability comes at a cost in terms of precision and recall. Indeed, we find that many apparently reasonable reconciliation methods yield reconciled probabilities with significantly lower precision than the original, unreconciled estimates. On the other hand, we find that isotonic regression usually performs better than the underlying, unreconciled method, and almost never performs worse; isotonic regression appears to be able to use the constraints from the GO network to its advantage. An exception to this rule is the high precision regime for joint evaluation, where Kullback-Leibler projection yields the best performance.

---

• The project was undertaken in the context  of the Mousefunc gene function prediction competition organised by Lourdes Peña-Castillo, Tim Hugues and Fritz Roth, whose results are published in  A critical assessment of Mus musculus gene function prediction using integrated genomic evidence, Peña-Castillo et al., Genome Biology 2008, 9(Suppl 1):S2.
• The original data  is available on the website of the Mousefunc I competition.

Kernels

• The kernel matrices built from the original data can be downloaded here or from the following table: 

• Kernel transformations
  Name	Linear	Normalized linear	Linear2	Normalized linear2	RBF	Diffusion	Parameters
  MGI
  OMIM							σ = 1
  Inpar							σ = 1
  Biomart							σ = 1
  Inter							σ = 1
  PfamA							σ = 1
  PPI							τ ∈ {0.1, 1, 10}
  Su
  Zhang							σ = 1
  SAGE							σ = 1

Test set and Held out set

• The list of genes that we used as held-out data, for intermediate validation of our methods is here, the final test set designed by Peña-Castillo et al. is here.

Labels

• The binary labels proposed by Peña-Castillo et al. simply reflect that a protein is labelled with a given GO term if it is tagged with this GO term in the Gene Ontology (no protein is tagged as "Not having a given GO term"). 
• However the Gene Ontology is a hierarchy, and:
• A protein tagged with a very specific GO term has obviously also the more general functions of the parent and ancestral GO terms. 
• It probably also has a more specific function characterised by one of the descendant GO terms, but is not tagged with any of them because it is actually not known given current knowledge. 
• It is however less likely that it would have the functions corresponding to "sibling" GO terms or descendant thereof. This is of course not generally true and in particular there are many examples of cases were the hierarchy branches into two terms that coalesce into a more specific term. 
• Based on these consideration, we proposed a ternary labelling of the proteins with the encoding 1 for positive example of a term, -1 negative example and 0 unknown.
• Caveat: Given that the ontology is a directed acyclic graph (DAG) and not a tree, many node are simultaneously descendant of a tagged node and of one of its sibling. In that case, it seems reasonable that the choice of 0 or -1 should be resolved in favor of -1. However, given the depth-first search algorithm we used to propagate labels, some of these labels are at -1 instead of 0.  
• Our label matrices for the training and held-out set can be dowloaded  for CC, for MF and  for BP.