Robotics Seminar

Date: September 3, 2025

Speaker: Hadas Kress-Gazit, Cornell Engineering

Title: Formal methods for robotics in the age of big data

Host: Tapomayukh Bhattacharjee

Date: September 10, 2025

Speaker: Manoj Karkee, Cornell CALS

Title:AI and Robotics in Specialty Crops

Host: Preston Culbertson

Date: September 17, 2025

Speaker:Yuchen Cui, UCLA

Title:Interactive Robot Learning from Non-Expert Teachers

Host: Kuan Fang
 

Date: September 24, 2025

Speaker:Oktay Arsalan, Field AI

Title: Challenges of Developing Autonomous Vehicles for Real-World Applications

Host: Tapomayukh Bhattacharjee

Date: October 1, 2025

Speaker:Vikash Kumar, MyoLabAI

Title: Should? And What Should Robots Learn from Humans?

Host: Tapomayukh Bhattacharjee

Date: October 8, 2025

Speaker:Jake Welde, Cornell University

Title:Geometric Methods for Efficient and Explainable Control of Underactuated Robotic Systems

Host: Preston Culbertson

Date: October 15, 2025

Speaker: Pragathi Praveena, Carnegie Mellon University

Title:Designing Robotic and AI Systems for Groups and Teams

Host: Tapomayukh Bhattacharjee

Date: October 22, 2025

Speaker:Carmelo (Carlo) Sferrazza, Berkley

Title:Humanoid robot learning

Host: Tapomayukh Bhattacharjee

Date: October 29, 2025

Speaker:Thomas Lew, TRI

Title:Uncertainty-Aware Control at the Limits

Host: Preston Culbertson

Date: November 5, 2025

Speaker:Karl Pertsch, UC Berkley & Stanford

Title:Building and Evaluating Generalist Robot Policies

Host: Tapomayukh Bhattacharjee

Date: November 12, 2025

Speaker: Aaquib Tabrez, Postdoctoral Associate, Cornell University

Title:Explainable Human-Robot Collaboration via Mental Model Alignment

Host: Tapomayukh Bhattacharjee

Date: November 19, 2025

Speaker:Jie Tan, Google

Title:Gemini Robotics: Bringing AI into the Physical World

Host: Tapomayukh Bhattacharjee

Date: December 3, 2025

Speaker:Anushri Dixit, UCLA

Title:Making robots trustworthy: Understanding risk and uncertainty for safe autonomy

Host: Tapomayukh Bhattacharjee

02.05.25: Learning Representations for Few-Shot Behavior Cloning
Speaker: Congyue Deng, Stanford
Host: Kuan Fang
Abstract: Representation learning, whether supervised or unsupervised, has been widely studied and applied in visual perception, where latent features encode high-level semantic information, capturing correspondences and correlations between different visual contents. Can similar paradigms be designed to encode actions and interactions? – this is an active question asked by the ongoing research in robot learning.
In this talk, I will present my work on learning representations for encoding, understanding, and generalizing robotic actions and interactions, with a particular focus on few-shot generalization under limited data. I will explore different approaches to robotic representation learning, discussing their connections to and distinctions from visual representation learning. Through this lens, I will also discuss the challenges and opportunities in developing efficient and generalizable robotic learning systems.

02.12.25: Perceiving the 4D World from Any Video
Speaker: Qianqian Wang, Berkeley
Host: Kuan Fang
Abstract: Perceiving the dynamic 3D world from visual inputs is essential for human interaction with the physical environment. While recent advancements in data-driven methods have significantly improved models' ability to interpret 3D scenes, much of this progress has focused on static scenes or specific categories of dynamic objects. How can we effectively model general dynamic scenes in the wild? How can we achieve online perception with human-like capabilities? In this talk, I will first discuss holistic representations for 4D scenes and then present a framework for online dense perception that continuously refines scene understanding with new observations. Finally, I will conclude with a discussion about the future opportunities and challenges in developing robust, scalable systems for perceiving and understanding dynamic 3D environments in real-world settings.

02.19.25: Learning, introspection, and anticipation for effective and reliable task planning under uncertainty: towards household robots comfortable with missing knowledge
Speaker: Gregory Stein, GMU
Host: Tapomayukh Bhattacharjee
Abstract: The next generation of service and assistive robots will need to operate under uncertainty, expected to complete tasks and perform well despite missing information about the state of the world or the future needs of itself and other agents. Many existing approaches turn to learning to overcome the challenges of planning under uncertainty, yet are often brittle or myopic, limiting their effectiveness. Our work introduces a family of model-based approaches to long-horizon planning under uncertainty that augment (rather than replaces) planning with estimates from learning, allowing for both high-performance and reliability-by-design.
In this talk, I will present a number of recent and ongoing projects that improve long-horizon navigation and task planning in uncertain home-like environments. First, I will discuss our recent developments that improve performance and reliability in unfamiliar environments—environments potentially dissimilar from any seen during training—with a technique we call "offline alt-policy replay," which enables fast and reliable deployment-time policy selection despite uncertainty. Second, I will discuss "anticipatory planning," by which our robot anticipates and avoids side effects of its actions on undetermined future tasks it may later be assigned; our approach guides the robot towards behaviors that encourage preparation and organization, improving its performance over lengthy deployments.

03.05.25: Learning for Dynamic Robot Manipulation of Deformable and Transparent Objects
Speaker: Jeffrey Ichnowski, Carnegie Mellon
Host: Kuan Fang
Abstract: Dynamics, softness, deformability, and difficult-to-detect objects will be critical for new domains in robotic manipulation. But there are complications--including unmodelled dynamic effects, infinite-dimensional state spaces of deformable objects, and missing features from perception. This talk explores learning methods based on multi-view sensing, acoustics, physics-based regularizations, and Koopman operators and proposes a novel multi-finger soft manipulator to enable new manipulation capabilities. We demonstrate how the proposed methods can recover transparent object geometry, densely track deformable object state over time, train robot systems using vision and Koopman operators, use sound to learn a model of friction for rapid non-prehensile object transportation, manipulate hair, shape pottery, and perform high-speed non-prehensile in-hand manipulation (aka pen spinning).

03.12.25: Robot Learning from Massive Human Videos
Speaker: Yue Wang, USC
Host: Kuan Fang
Abstract: Robotics has made remarkable strides, driven by advances in machine learning, optimal control, and hardware innovation. However, unlike conventional machine learning domains, data collection in robotics poses unique challenges: it is neither straightforward nor inherently scalable. To address this gap, our work focuses on leveraging in-the-wild human videos to enable learning for manipulation and agile robotics.
First, I will present "RAM," our recent effort to achieve zero-shot robotic manipulation by retrieval with foundation models. Next, I will delve into "UH-1," a project that enables humanoid control by learning from vast datasets of human motion videos.

03.19.25: Safe Learning-Enabled Robot Control
Speaker: Somil Bansal, Stanford
Host: Kuan Fang
Abstract: No other technology has probably impacted robotics in the last decade as much as machine learning and AI. However, integrating data-driven models into the control loop introduces a critical challenge: how can we ensure the safety of learning-enabled robotic systems?
In this talk, we present a systematic approach to embedding safety guarantees throughout the entire lifecycle of learning in robotics – from training to deployment to real-time adaptation. First, we discuss physics-informed machine learning techniques that efficiently learn safe control policies for a wide range of autonomous systems. Next, we introduce fast adaptation methods that enable robots to refine safety policies on the fly, leveraging raw sensory information and language-based feedback. Finally, we discuss how these safety-aware techniques can enhance data-driven approaches, such as imitation learning and sampling-based MPC, improving both their data efficiency and robustness. Throughout the talk, we will demonstrate these methods on various safety-critical autonomous systems, including autonomous aircrafts, legged robots, and drones.

03.26.25: Autonomously Learning World-Model Representations For Efficient Robot Planning
Speaker: Naman Shah, Brown
Host: Tapomayukh Bhattacharjee
Abstract: In recent years, it has been clear that planning is an essential tool for robots to achieve complex goals. However, robots often heavily rely on humans to provide "world models" that enable long-horizon planning. It is not only expensive to create such world models as it requires human experts who understand the domains as well as limitations of the robot, but these human-generated world models are often biased by human intuition and kinematic constraints. In this talk, I will present my research focusing on autonomously learning plannable world models. The talk would involve discussing approaches on task and motion planning, neuro-symbolic abstractions for motion planning, and how we can learn world models for task and motion planning.

04.09.25: Towards Open World Robot Safety
Speaker: Andrea Bajcsy, Carnegie Mellon
Host: Kuan Fang
Abstract: Robot safety is a nuanced concept. We commonly equate safety with collision-avoidance, but in complex, real-world environments (i.e., the "open world'') it can be much more: for example, a mobile manipulator should understand when it is not confident about a requested task, that areas roped off by caution tape should never be breached, and that objects should be gently pulled from clutter to prevent falling. However, designing robots that have such a nuanced safety understanding---and can reliably generate appropriate actions---is an outstanding challenge.
In this talk, I will describe my group's work on systematically uniting modern machine learning models (such as large vision-language models and latent world models) with classical formulations of safety in the control literature to generalize safe robot decision-making to increasingly open world interactions. Throughout the talk, I will present experimental instantiations of these ideas in domains like vision-based navigation and robotic manipulation.

04.16.25: Towards Generalizable Mobile Manipulation
Speaker: Saurabh Gupta, UIUC
Host: Kuan Fang
Abstract: What does it take to build mobile manipulation systems that can competently operate on previously unseen objects in previously unseen environments? In this talk, I will try to answer this question using opening of articulated objects as a mobile manipulation testbed. I will describe the design of a modular system for this task and discuss some takeaways from a large scale real world experimental study, including a somewhat surprising one: a modular system far outperforms an end-to-end imitation learner trained on a large number of real world demonstrations. If time permits, I will speculate why imitation learning failed and how we could mitigate those failures.

04.23.25: Toward Interpretable, Efficient, and Scalable Robot Perception
Speaker: Lu Gan, GaTech
Host: Kuan Fang
Abstract: Recent advances in foundation models have opened new opportunities for general-purpose, robot-agnostic perception systems. While these models offer strong generalization and robustness, they often overlook valuable robot-specific priors that can enhance both performance and interpretability. In this talk, I will present recent efforts from my group to develop interpretable, efficient, and scalable robot perception systems by leveraging the structure and experience of specific robots. First, I will present a series of works on state estimation for legged robots that exploit their kinematic structures and morphological symmetries. Next, I will introduce approaches to legged robot navigation by learning terrain traversability from robot experience. Finally, I will briefly discuss a scalable, continuous semantic mapping pipeline designed to support large-scale, multi-robot deployments.

04.30.25: Accelerating the Data Flywheel for Contact-rich Manipulation via Model-based Reasoning
Speaker: Tao Pang, Boston Dynamics AI Institute
Host: Kuan Fang
Abstract: The success of behavior cloning (BC) in robotic manipulation has largely been limited to “gripper-only” tasks, where training data can be reliably generated via end-effector teleoperation. In contrast, humans routinely leverage their entire hands and even body surfaces to perform contact-rich interactions—tasks that remain challenging for robots due to teleoperation difficulties and embodiment gaps. This talk introduces model-based planning as an effective data source for creating contact-rich robotic policies. First, we explore the structures of the complementarity-constrained optimization, which is ubiquitous in rigid body dynamics. By exploiting these structures, we can generate dexterous in-hand manipulation policies in minutes on a standard laptop using just the CPU. Notably, however, not all planning methods produce equally effective training data for BC. In the second part of the talk, we show that popular sampling-based planners can yield high-entropy demonstrations that adversely affect policy performance. To address this limitation, we propose building consistent, global planners by explicitly reasoning about optimality. We conclude by discussing how these insights pave the way for robust, contact-rich robotic behaviors, bridging the gap between purely gripper-centric tasks and human-level dexterity.

05.07.25: Bridging Simulation and Reality for Robot Dexterity
Speaker: Preston Culbertson, Cornell
Host: Kuan Fang
Abstract: While reinforcement learning has spurred breakthroughs enabling robust and dynamic robot locomotion, achieving similar success in dexterous manipulation has proven far more challenging. This talk addresses the complexities inherent in tasks requiring precise, forceful, and contact-rich interactions, such as tool use. We will begin with robust grasping, discussing our recent work building and using large-scale, diverse datasets to achieve reliable grasping with dexterous hands on physical hardware. Moving towards more dynamic tasks, we will explore how sampling-based Model Predictive Control (MPC), coupled with high-fidelity simulation, can enable sophisticated in-hand manipulation. A central theme will be the critical role, and current limitations, of simulation for fine-grained contact dynamics. To this end, we will discuss some recent work on improving dynamics modeling by training generative models with real-world data to model complex dynamics like a walking humanoid. Throughout the seminar, I will present experimental results from real-world robots, emphasizing the practical lessons learned in deploying these manipulation capabilities on physical robots.

09.05.24: Real-to-Sim-to-Real: A Scalable Data Diet for Robot Learning
Speaker: Abhishek Gupta
Host: Sanjiban Choudhury
Abstract: Robotic automation, powered by machine learning driven methods, has the potential to build systems that change the future of work, daily life and society at large by acting intelligently in human centric environments. As with most modern machine learning methods, a key component in building such a robotic system the availability of data, abundant, diverse and high quality. In domains of nature language or computer vision, data of this form has scaled passively with internet scale, since people naturally interact through the medium of language or images. In contrast, robots are hardly deployed in human-centric settings and certainly are not collecting internet scale data passively. The key question I will ask is - how can we develop a data diet for robotic learning that scales passively? In particular, I will discuss how simulation, despite being fundamentally inaccurate, can provide a scalable source of data for robotic learning. We will discuss a class of real-to-sim-to-real methods that are able to construct simulation content on the fly from cheap real-world data, enabling scalable robust robot training. In doing so, I hope to shed some light on the unique challenge that data acquisition plays in robot learning and discuss how developing truly open-world robotic learning systems will necessitate a careful consideration of data quality and quantity.

09.12.24: Learning to build an actionable, composable, and controllable digital twin
Speaker: Wei-Chiu Ma
Host: Sanjiban Choudhury
Abstract: Simulation has been the driving force behind robot development. With recent advances in computer vision and graphics, simulating sensor observations has particularly drawn wide attention across the community, since it may enable end-to-end testing of full autonomy systems. Unfortunately, existing sensor simulators, while impressive, still suffer from realism and can neither effectively model the outcomes of actions nor hallucinate counterfactual scenarios. In this talk, I will summarize our recent efforts to enable this goal.
First, I will discuss how we develop a high-fidelity closed-loop sensor simulator for self-driving vehicles. Our key insight is to build a digital twin directly from real-world data and leverage the compositional structure of the world to decompose the scene into foreground actors and background. This not only allows us to synthesize extremely high-quality sensor observations to avoid domain gap, but also facilitates better modeling of the interactions among the actors and the scene. Next, I will discuss how we can further expand the simulator to generate physically plausible sensor observations under different lighting conditions and improve the robustness of autonomous systems. Finally, I will present our recent efforts on pushing the boundaries of digital twins with generative models. I will showcase how we distill knowledge from multimodal LLMs into existing 3D systems, making them interactable, actionable, and thus suitable for physical intelligence.

09.26.24: Do We Need Social Robots? Towards Impactful and Long-term Interaction in Social Robotics and Personalized Healthcare/Medical Systems
Speaker: Chung Hyuk Park
Host: Tapomayukh Bhattacharjee
Abstract: The exponential growth of research efforts in robotics and human-robot interaction (HRI) has unveiled an increasing number of collaborative fields. One of the highly impactful areas that has shown a steep increase of interests and collaboration is the healthcare and medical domain, which has welcomed the technological advancement in assistive robotics, artificial intelligence (AI) and machine learning (ML). Assistive robotics is an expanding field of research that holds many potentials for impacting human health and quality of life, along with the advancement of AI/ML. In this presentation, I will discuss my research activities focused on four main themes related to assistive robotics and AI/ML: (1) My vision for socially assistive robots and initial approaches of SARs as embodied agents with multi-modal perception and interactions will be shared; (2) We will then delve into the realm of robotic learning, specifically targeting interactive learning and socio-emotional interactions for autistic individua; (3) I will examine contextual and mutual learning for personalized interaction, with the ultimate goal of facilitating clinical outcomes and long-term human-robot interaction; and (4) Our latest findings and new approaches will be shared, with several directions addressing AI/ML aided healthcare and telemedicine applications.  Along with my research endeavors, I will also impart the experiences and knowledge gained from cross-disciplinary studies and translational research, aimed at advancing assistive robotics for healthcare and personalized interventions.

10.03.24: The Impact of VLMs on Semantic Navigation: A Before and After View of Object Search
Speaker: Bernadette Bucher
Host: Kuan Fang
Abstract: Understanding how humans leverage semantic knowledge to navigate unfamiliar environments and decide where to explore next is pivotal for developing robots capable of human-like search behaviors. For example, when looking for a fork, a person would look near refrigerators and ovens, not beds and sofas. To perform similar reasoning, a robot needs to have and use priors about the expected semantic layout of the environment. In this talk, I will present two of my solutions to this object search problem which leverage semantic priors developed directly before (ICLR 2022) and directly after (ICRA 2024) the recent rapid improvements in LLMs and vision-language models (VLMs). I will discuss how these advances in natural language and computer vision changed our solution to this robotics problem, and I will also talk about the connection between these solutions to the object search problem and other unsolved semantic reasoning challenges in robotics.

10.10.24: Toward Flexible and Effective Human-Robot Teaming
Speaker: Mike Hagenow
Host: Tapomayukh Bhattacharjee
Abstract: Despite nearly seventy years of development, robots are not yet realizing their promise of handling the undesirable day-to-day tasks of skilled industrial workers. Recent studies indicate that today’s robots are still too inflexible and difficult to program, particularly for less structured and high-variability tasks. In this talk, I will present three recent approaches to human-robot teaming that aim to unlock new opportunities for robots. These approaches address key questions in human-robot teaming, such as how to optimize human input during teaming and how skilled workers can teach robots complex behaviors. I will conclude by discussing open problems in the area and outlining next steps toward more widespread human-robot teaming.

10.17.24: Attention and Inattention for Minimalist Robot Learners
Speaker: Dinesh Jayaraman
Host: Kuan Fang
Abstract: Industry is placing big bets on "brute forcing" robotic control through scaling data, compute, and models, but a key blind spot of such scaling methods is that they are profligate in their use of expensive resources: power, compute, time, data, etc. Towards developing more minimalist robotic control stacks, my research group studies how agents can select and attend to task-relevant information during sensing, representation, decision making, and policy learning. I will speak about my group's work on exploiting privileged sensors at training time, combining pre-training language and vision models to compute task-relevant representations, and task-relevant world model learning.

10.24.24: Scaling Robot Learning with Passively-Collected Human Data
Speaker: Danfei Xu
Host: Kuan Fang
Abstract: The foundation of modern AI is scalable knowledge transfer from humans to machines. While Computer Vision and NLP can glean such knowledge from exabytes of human-generated data on the Internet, Robot Learning still heavily relies on resource-intensive teleoperation for data collection. Can we capture real-world human interactions as effortlessly as the Internet captures the virtual world? We propose that passive human data collection is a crucial step towards this future. Just as the Internet evolved into an unintentional data repository for AI, an ideal data collection system should capture sensorimotor data from everyday human activities, without humans’ conscious participation. 

10.31.24: What Foundation Models Can and Cannot Do for Bringing Helpful Robotic Assistants into Our Lives
Speaker: Roberto Martin-Martin
Host: Tapomayukh Bhattacharjee
Abstract: The past few years have seen remarkable advancements in AI. What began with the NLP revolution has sparked new ideas across many fields, including robotics, driving the search for a "RobotGPT." But is this all we need to finally have robots assist humans in everyday tasks? What challenges have been addressed, and which remain unsolved as we look ahead? In this talk, I will discuss recent ways we have integrated Foundation Models into robotic solutions, as well as the limitations of the current FM-based paradigm—particularly in learning from few human demonstrations and in seeking information for complex manipulation tasks.

11.14.24: A Vision-Language-Action Flow Model for General Robot Control
Speaker: Quan Vuong
Host: Kuan Fang
Abstract: Robot learning has the potential to unlock flexible, general, and dexterous systems while addressing key AI challenges. However, achieving the generality needed for real-world applications faces obstacles like data, generalization, and robustness. This talk will describe the journey in building our flagship model, Pi_0 [1]. We propose a novel flow-matching architecture built on a pre-trained vision-language model to leverage Internet-scale semantic knowledge. The model is trained on diverse datasets from various dexterous robots, including single-arm, dual-arm, and mobile manipulators. We evaluate its zero-shot performance, ability to follow language instructions, and capacity to learn new skills through fine-tuning across tasks like laundry folding, table cleaning, and box assembly.

11.21.24: Generative Simulation for Embodied AI in Urban (Micro)mobility
Speaker: Bolei Zhou
Host: Kuan Fang
Abstract: Public urban spaces like streetscapes and plazas accommodate human social life in all vibrant variations. Advances in micromobility make public urban spaces no longer exclusive to humans: Food delivery bots and electric wheelchairs are sharing sidewalks with pedestrians, while robot dogs and humanoids have recently emerged in the street. Embodied AI plays a transformative role in shaping the future of urban micromobility, by assisting human operator of these mobile machines to navigate through the unpredictable sidewalks safely. In this talk, I will introduce our effort of building MetaDriverse, a simulation platform that facilitates the computer vision and autonomy research for urban mobility and micromobility. It incorporates generative AI capabilities to simulates diverse urban environments, encompassing a wide range of visual appearances, behavioral dynamics, and terrain structures. It enables the scalable training of embodied AI agents and safety evaluation before real-world deployment. Relevant projects are available at https://metadriverse.github.io/.  

12.05.24: Moving from Data Collection to Data Generation: Addressing the Need for Data in Robotics
Speaker: Ajay Mandlekar
Host: Kuan Fang
Abstract: Imitation learning from human demonstrations has emerged as a widely adopted paradigm for teaching robots manipulation skills. However, data collection for imitation learning is costly and resource-intensive, often spanning teams of human operators, fleets of robots, and months of persistent data collection effort. Instead, in this talk, I will advocate for the use of automated data generation methods and simulation platforms as a scalable alternative to fuel this need for data. I will introduce a suite of automated data generation tools that make use of robot planning methods and small sets of human demonstrations, to synthesize new demonstrations automatically. These tools are broadly applicable to a wide range of manipulation problems, including high-precision and long-horizon manipulation, and can be used to produce performant, often near-perfect agents for such tasks. The data generated in simulation can also be used to address real-world robotic manipulation, making synthetic data generation a compelling tool for imitation learning in robotics.

02.08.24: Expressive robot swarms and multi-human-swarm interaction for social applications
Speaker: Merihan Alhafnawi
Host: Tapomayukh Bhattacharjee
Abstract: Robot swarms can be used as a new and exciting medium through which people can express themselves to one another. The swarm can help facilitate social group tasks that people might face in their daily lives. A robot swarm could especially be useful due to its distributed nature that helps with scaling to many robots and many users. In her talk, Merihan will present her research that focuses on designing, building and applying expressive robot swarms that support simultaneous interactions with many people in social tasks. During her PhD, Merihan designed and built MOSAIX, a swarm of 100 robot Tiles, that was applied in group tasks such as brainstorming, decision-making and collective art. MOSAIX was used in many public events by more than 400 people. In her postdoctoral research, Merihan combines swarm robotics and architecture to create self-adaptive structures that respond to the environment and human interaction.

02.22.24: Efficient Reductions for Inverse Reinforcement Learning
Speaker: Gokul Swamy
Host: Sanjiban Choudhury
Abstract: Interactive approaches to imitation learning like inverse reinforcement learning (IRL) have become the preferred approach for problems that range from autonomous driving to mapping. Despite its impressive empirical performance, robustness to compounding errors + causal confounders, and sample efficiency, IRL comes with a strong computational burden: the requirement to repeatedly solve a reinforcement learning (RL) problem in the inner loop. If we pause and take a step back, this is rather odd: we’ve reduced the easier problem of imitation to the harder problem of RL. In this talk, we will discuss a new paradigm for IRL that leverages a more informed reduction to expert competitive RL (rather than to globally optimal RL), allowing us to provide strong guarantees at a lower computational cost. Specifically, we will present a trifecta of efficient algorithms for IRL that use information from the expert demonstrations during RL to curtail unnecessary exploration, allowing us to dramatically speed up the overall procedure, both in theory and practice.

02.29.24: Bad Robot, Good Robot - Rethinking the Agency of Our Artificial Teammates
Speaker: Reuth Mirsky
Host: Sanjiban Choudhury
Abstract: Should autonomous robots always obey instructions and comply with our expectations? Most existing research on collaborative robots and agents assumes that a “good” robot should abide by the instructions it is given and should act in a way that will not interfere with its surroundings: a fetching robot should do its work seamlessly, a navigating robot should always avoid collisions, and a guide robot should always do as it is instructed. In this talk, I will question this assumption by presenting the Guide Robot Grand Challenge and discussing the components required to design and build a service robot that can replace or surpass the functionalities of a guide dog. This challenge encompasses a variety of research problems, for each of which I will present a novel contribution: reasoning about other agents, initiating an interaction, teaching teammates, and more. Finally, I will discuss the many remaining challenges towards achieving a guide robot and how I plan to tackle these challenges.

03.07.24: On Building General-Purpose Home Robots
Speaker: Lerrel Pinto
Host: Sanjiban Choudhury
Abstract: The concept of a "generalist machine" in homes — a domestic assistant that can adapt and learn from our needs, all while remaining cost-effective — has long been a goal in robotics that has been steadily pursued for decades. In this talk, I will present our recent efforts towards building such capable home robots. First, I will discuss how large, pretrained vision-language models can induce strong priors for mobile manipulation tasks like pick-and-drop. But pretrained models can only take us so far. To scale beyond basic picking, we will need systems and algorithms to rapidly learn new skills. This requires creating new tools to collect data, improving representations of the visual world, and enabling trial-and-error learning during deployment. While much of the work presented focuses on two-fingered hands, I will briefly introduce learning approaches for multi-fingered hands which support more dexterous behaviors and rich touch sensing combined with vision. Finally, I will outline unsolved problems that were not obvious initially, which, when solved, will bring us closer to general-purpose home robots.

03.14.24: Dexterous Multimodal Robotic Tool-use: From Compliant Tool Representations to High-Resolution Tactile Perception and Control
Speaker: Nima Fazeli
Host: Sanjiban Choudhury
Abstract: Dexterous tool manipulation is a dance between tool motion, deformation, and force transmission choreographed by the robot's end-effector. Take for example the use of a spatula. How should the robot reason jointly over the tool’s geometry and forces imparted to the environment through vision and touch? In this talk, I will present two new tools in our tool-box for dexterous tool manipulation: multimodal compliant tool representations via neural implicit representations and our recent progress on tactile control with high-resolution and highly deformable tactile sensors. Our methods seek to address two fundamental challenges in object manipulation. First, the frictional interactions between these objects and their environment is governed by complex non-linear mechanics, making it challenging to model and control their behavior. Second, perception of these objects is challenging due to both self-occlusions and occlusions that occur at the contact location (e.g., when wiping a table with a sponge, the contact is occluded). We will demonstrate how implicit functions can seamlessly integrate with robotic sensing modalities to produce high-quality tool deformation and contact patches and how high-resolution tactile controllers can enable robust tool-use behavior despite the complex dynamics induced by the sensor mechanical substrate. We’ll conclude the talk by discussing future directions for dexterous tool-use.

03.21.24: Robot Foundation Model via Simulation
Speaker: Pulkit Agrawal
Host: Sanjiban Choudhury
Abstract: Unlike natural language and image processing where internet data is easily available for training foundation models, data for robot learning is unavailable. I will discuss how simulators can be used to learn complex and generalizable sensorimotor skills in a manner that reduces human effort and is easily scaled to many tasks. I will elaborate using the following case studies:

(i) a dexterous manipulation system capable of re-orienting novel objects of complex shapes and peeling vegetables.
(ii) a quadruped robot capable of fast locomotion, manipulation, and whole-body control on diverse natural terrains.
(iii) a lifelong learning robotic agent that can request and learn new rigid-object manipulation skills in a few minutes. 

Next, I will discuss some algorithmic ideas aimed at mitigating human effort in reward design, hyper-parameter tuning and enabling seamless combination of learning signals from demonstrations, rewards, and the agent's self-exploration. The resulting framework provides a way to collect high-quality data for multiple tasks that can be used to train a robotic foundation model. 

03.28.24: Trust, Robots, and Trust Repair
Speaker: Connor Esterwood
Host: Tapomayukh Bhattacharjee
Abstract: Trust is a dynamic force, evolving over time. When a trustee excels, trust flourishes; conversely, a trustee's failures can erode trust. While managing trust increases is relatively straightforward, trust declines can have enduring consequences, souring collaborative endeavors. Though trust breakdowns are inevitable they can be mitigated through verbal repairs such as apologies, denials, explanations, and promises. However, it remains unclear whether these strategies effectively restore trust in robots. This gap in understanding hinders the development of resilient robots capable of gracefully recovering from inevitable failures, thus limiting the potential of human-robot collaboration. This presentation highlights and summarizes my recent work in trying to solve the problem of trust violations and determining trust repairs in the context of human--robot interaction. 

04.11.24: Human-Centered AI for Accessible and Assistive Robotics: Towards a Disability-Centered HRI
Speaker: Elaine Short
Host: Tapomayukh Bhattacharjee
Abstract: Powered by advances in AI, especially machine learning, robots are becoming smarter and more widely used.  Robots can provide critical assistance to people in a variety of contexts, from improving the efficiency of workers to helping people with disabilities in their day-to-day lives.  However, inadequate attention to the needs of users in developing these intelligent robots results in systems that are both less effective at their core tasks and more likely to do unintended harm.  The Assistive Agent Behavior and Learning (AABL) Lab at Tufts University seeks to apply human-centered design thinking, especially disability ethics, to the design of state-of-the-art robot learning algorithms and interaction frameworks.  This talk will explore how disability-community-centered thinking can be used to inspire new directions for intelligent interactive robotics and review recent work from the AABL lab at the intersection of assistive robotics, robot learning, and human-robot interaction.  

04.25.24: Robot-Assisted Feeding: Recent Advances and Future Directions in the Personal Robotics Lab
Speaker: Taylor Kessler-Faulkner
Host: Tapomayukh Bhattacharjee
Abstract: Eating is a personal and intricate task that we perform every day. However, approximately 1.8 million people in the US alone cannot eat without assistance. The ADA (Assistive Dextrous Arm) project in the Personal Robotics Lab (PRL) at UW seeks to address this through a robot-assisted feeding system. In this presentation, I will discuss PRL's recent advances in both the human-robot interaction and autonomous robotics components of assistive feeding, as well as promising future research avenues.

05.02.24: Design and Perception of Wearable Multi-Contact Haptic Devices for Social Communication
Speaker: Cara Nunez
Host: Tapomayukh Bhattacharjee
Abstract: During social interactions, people use auditory, visual, and haptic (touch) cues to convey their thoughts, emotions, and intentions. Current technology allows humans to convey high-quality visual and auditory information but has limited ability to convey haptic expressions remotely. However, as people interact more through digital means rather than in person, it becomes important to have a way to be able to effectively communicate emotions through digital means as well. As online communication becomes more prevalent, systems that convey haptic signals could allow for improved distant socializing and empathetic remote human-human interaction.
Due to hardware constraints and limitations in our knowledge regarding human haptic perception, it is difficult to create haptic devices that completely capture the complexity of human touch. Wearable haptic devices allow users to receive haptic feedback without being tethered to a set location and while performing other tasks, but have stricter hardware constraints regarding size, weight, comfort, and power consumption. In this talk, I will present how I address these challenges through a cyclic process of (1) developing novel designs, models, and control strategies for wearable haptic devices, (2) evaluating human haptic perception using these devices, and (3) using prior results and methods to further advance design methodologies and understanding of human haptic perception