Course webpage: www.cs.umd.edu/...
Title: Plex: Towards Reliability using Pretrained Large Model Extensions
Abstract: A recent trend in artificial intelligence is the use of pretrained models for language and vision tasks, which have achieved extraordinary performance but also puzzling failures. Probing these models' abilities in diverse ways is therefore critical to the field. I will talk about our recent work exploring the reliability of models, where we define a reliable model as one that not only achieves strong predictive performance but also performs well consistently over many decision-making tasks involving uncertainty (e.g., selective prediction, open set recognition, calibration under shift), robust generalization (e.g., accuracy and log-likelihood on in- and out-of-distribution datasets), and adaptation (e.g., active learning, few-shot uncertainty). Plex builds on our work on scalable building blocks for probabilistic deep learning such as Gaussian process last-layer and efficient variants of deep ensembles. We show that Plex improves the state-of-the-art across reliability tasks, and simplifies the traditional protocol as it improves the out-of-the-box performance and does not require designing scores or tuning the model for each task.
References:
Plex: Towards Reliability using Pretrained Large Model Extensions arxiv.org/abs/... (blog: ai.googleblog....)
Practical Uncertainty Estimation & Out-of-Distribution Robustness in Deep Learning (NeurIPS'2020 tutorial slides)
A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness: arxiv.org/abs/...
Deep Ensembles: A Loss Landscape Perspective arxiv.org/abs/...
8 окт 2024
