A few years ago I summarized my research philosophy for a Neurips Workshop talk. My research focuses on algorithm design for machine learning with a specific emphasis on representation learning. Over the years my work has spanned several interconnected research directions, from fundamental questions about how to compare data to practical applications in autonomous driving.

Metric Learning

One of the fundamental challenges of machine learning is how to compare examples. My early work introduced Large Margin Nearest Neighbor (LMNN), which learns distance metrics by pulling similarly labeled inputs close while pushing dissimilarly labeled inputs apart. This framework popularized the triplet loss objective that is now widely used in computer vision.

When word embeddings became a thing, we introduced the Word Mover's Distance (WMD), a novel approach to measuring similarity between text documents. WMD elegantly incorporates the fact that different words can have similar meanings by casting document comparison as a transportation problem over word embeddings. We later extended this idea to contextual embeddings with BertScore, which has been widely adopted for evaluating machine translation and text generation systems.

Resource Efficient Learning

In industrial applications, all resources are limited and must be accounted for. My group has been among the first to formally integrate resource constraints into learning algorithms, treating feature extraction cost as a natural trade-off with accuracy. This work introduced feature hashing, now widely known as the "hashing trick," which allows learning tasks to operate within fixed memory budgets.

During the rise of deep learning, we extended these ideas to neural networks, showing that network parameters are highly redundant and can be compressed by orders of magnitude without significant accuracy loss. This contributed to the now-vibrant subfield of neural network compression and the ongoing discussion about over-parameterization.

Deep Network Architectures

Our work on network compression revealed surprising redundancy in deep networks, raising questions about whether this redundancy is necessary or avoidable. We introduced stochastic depth, showing that deliberately increasing redundancy can substantially improve generalization.

To understand the purpose of this redundancy, we designed DenseNet, which introduces direct skip connections between all layers of the same size. This architecture drastically improves both generalization performance and parameter efficiency, winning the CVPR 2017 best paper award and establishing itself as one of the most widely used neural network architectures.

Recognizing that neural networks are increasingly used in high-stakes medical decisions, we investigated why network probability outputs are poorly calibrated. Our work on calibration showed that temperature scaling is highly effective for reliable probability estimates, and this approach has become the standard method for network calibration.

More recently, we studied graph convolutional neural networks and discovered that much of their complexity was unnecessary. Our Simplifying Graph Convolutional Networks paper showed that a simple closed-form preprocessing step paired with logistic regression can match the performance of complex GCNs while being orders of magnitude faster.

Efficient Inference for Gaussian Processes

To make Gaussian Processes as accessible as deep learning, my students, Andrew G. Wilson, and I (but really, mostly Geoff Pleiss and Jake Gardner) developed GPyTorch, a highly modular library that leverages GPU-optimized matrix operations. This platform has become one of the most popular GP coding frameworks, with contributors from universities and companies worldwide.

Perception for Autonomous Driving

In collaboration with colleagues in Mechanical Engineering and Computer Science, we investigated whether 3D object detection for self-driving cars could be performed with passive cameras instead of expensive LiDAR sensors. Our pseudo-LiDAR approach mimics a LiDAR-like point cloud using stereo camera data, dramatically improving detection accuracy.