A Call to Build Models like We Build Open-Source Software

A Call to Build Models like We Build Open-Source Software (via Zoom)

Abstract: Large pre-trained models have become a cornerstone of modern ML pipelines thanks to the fact that they facilitate improved performance with less labeled data on downstream tasks. However, these models are typically created by a resource-rich research group that unilaterally decides how a given model should be built, trained, and released, after which point it is left as-is until a better pre-trained model comes along to completely supplant it. In contrast, open-source development has proven that it is possible for a distributed community of contributors to work together to iteratively build complex and widely-used software. This kind of large-scale distributed collaboration is made possible through a mature set of tools including version control, continuous integration, merging, and more. In this talk, I will present a vision for building machine learning models in the way that open-source software is developed, including preliminary work from my lab on "merging" and "patching" models. I will also give some insight into the future work required to make this vision a reality.

Bio: Colin is an Assistant Professor in the Department of Computer Science at the University of North Carolina, Chapel Hill. He also spent one day a week as a Faculty Researcher at Hugging Face. Much of his recent research focuses on machine learning algorithms for learning from limited labeled data, including semi-supervised, unsupervised, and transfer learning.