Achieving high accuracy in orthology inference is essential for many comparative, evolutionary and functional genomic analyses, yet the true evolutionary history of genes is generally unknown and orthologs are used for very different applications across phyla, requiring different precision-recall trade-offs. As a result, it is difficult to assess the performance of orthology inference methods. Here, we present a community effort to establish standards and an automated web-based service to facilitate orthology benchmarking. Using this service, we characterize 15 well-established inference methods and resources on a battery of 20 different benchmarks. Standardized benchmarking provides a way for users to identify the most effective methods for the problem at hand, sets a minimum requirement for new tools and resources, and guides the development of more accurate orthology inference methods.

This work was supported by Swiss National Science Foundation grant PP00P3_150654 (to C.D.), UK Biotechnology and Biological Sciences Research Council grant BB/L018241/1 (to C.D.), Spanish Ministry of Economy and Competitiveness grant BIO2012-37161 (to T.G.), Qatar National Research Fund NPRP 5-298-3-086 (to T.G.), European Research Council grant ERC-2012-StG-310325 (to T.G.), National Institutes of Health (NIH) grant R24 OD011883 (to S.E.L.), U41 HG002273 (to S.E.L. and P.D.T.), U41 HG007822 (to M.J.M. and I.X.), Swiss State Secretariat for Education, Research and Innovation (SERI) funding (to I.X. and C.D.), US National Science Foundation EAGER Award #1355632 (to K.S.) and ANR project BIP-BIP ANR-10-BINF-03-02 (to O.L.). Furthermore, A.S.d.S., J.H.-C., M.J.M., M.M. and P.B. acknowledge support from the European Molecular Biology Laboratory, M.M. acknowledges support from the Wellcome Trust (WT095908), S.E.L. acknowledges support from Lawrence Berkeley National Laboratory core funds (Office of Basic Energy Sciences and US Department of Energy Contract No. DE-AC02-05CH11231), L.J.J. acknowledges support from the Novo Nordisk Foundation (Grant No. NNF14CC0001) and L.P.P. acknowledges support from the La Caixa–CRG International Fellowship Program.