Abstract:
In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised learning has led to major advances in representation learning and statistical generation.
In the life sciences, the anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity. The resulting model, trained on sequences alone, contains information about biological properties in its representations. The learned representation space has a multi-scale organization reflecting structure from the level of biochemical properties of amino acids to remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and can be identified by linear projections.
Bio:
Rob Fergus is a Professor of Computer Science at the Courant Institute of Mathematical Sciences, New York University. He is also a Research Scientist at DeepMind New York. He received a Masters in Electrical Engineering with Professor Pietro Perona at Caltech, before completing a Ph.D. with Professor Andrew Zisserman at the University of Oxford in 2005. Before coming to NYU, he spent two years as a post-doc at MIT, working with Professor William Freeman. He has received several awards including a CVPR Best Paper Prize, a Sloan Fellowship & NSF Career Award, and the IEEE Longuet-Higgins Prize.
