Meetup In 5 days
November
Sunday
14:00–17:00 2026 · San Francisco time
You sign up with the host. foundercal only links to it.
Your event? Manage this listing Report a change
🤖🍨 Grab a sundae and join Sundae Robotics, a private, invite-only Sunday series bringing together robotics researchers, founders, and builders working at the frontier of physical intelligence.
Sundae Robotics 12Dexterous Manipulation, Tactile Perception & Robot LearningFeatured Talk: Beyond the Gripper — Teaching Robots Dexterity Through Touch, Vision, and Play
Beyond the Gripper: Teaching Robots Dexterity Through Touch, Vision, and Play
Keynote: Akash SharmaResearch Scientist, Amazon Frontier AI & Robotics · CMU Robotics Institute PhD
Robots that impress in demos still mostly use parallel-jaw grippers, and true in-hand dexterity remains out of reach. The two standard routes each face fundamental limitations: demonstrations often force humans to act like robots, while simulation cannot faithfully model touch or real-world vision.
In this talk, Akash will present four ingredients that aim to close those gaps.
First, he will introduce PTLD, which learns tactile policies without ever simulating a tactile sensor by distilling a privileged simulation-trained teacher on real hardware.
He will then discuss INVAR, which learns visual invariance once from clean and randomized simulation pairs, allowing policies trained on clean simulation to transfer zero-shot.
Next, Akash will present CARD, which retargets human video, glove, and motion-capture demonstrations according to what the hand touches rather than simply how the hand moves.
Finally, he will discuss DSP, which turns minutes of human play into goal-conditioned dexterity through autonomous play, world models, and distillation.
Together, these works point toward a common recipe for scaling dexterous manipulation: let humans be humans, use a privileged teacher, scale with the robot's own experience, and distill what is learned into a deployable policy.
Akash Sharma is a Research Scientist at Amazon Frontier AI & Robotics (FAR), where he works with Jitendra Malik on robot hands and dexterous manipulation.
He received his PhD from the Carnegie Mellon University Robotics Institute, where he was advised by Prof. Michael Kaess. During his PhD, he was also a visiting researcher with the FAIR Robotics team at Meta, where he worked closely with Mustafa Mukadam and Francois Hogan.
Akash's research focuses on robot perception using varied sensing modalities, particularly vision and touch, to enable dexterous manipulation. He is also broadly interested in state estimation and reconstruction in dynamic environments.
His recent work explores how tactile sensing, simulation, human demonstrations, and large-scale robot experience can be combined to overcome limitations of conventional dexterous learning pipelines. His work on PTLD develops a sim-to-real approach for tactile dexterity that avoids tactile simulation by using privileged real-world sensing to distill information into deployable tactile policies.
Akash has also worked extensively on general-purpose tactile representations. Sparsh, Sparsh-skin, and Sparsh-X explore self-supervised representations across vision-based tactile sensors, distributed tactile skin, and multimodal touch, while DexterityGen investigates combining reinforcement-learned dexterous primitives with human guidance for complex manipulation.
His broader research spans multi-fingered manipulation, human-to-robot skill transfer, tactile perception, sim-to-real learning, object-centric manipulation, state estimation, and scalable data generation for robot learning.
Before his PhD, Akash received a Master's in Robotics from Carnegie Mellon University, where he worked on object-based SLAM for robot navigation. He received his bachelor's degree in Electronics and Communication Engineering from Sri Jayachamarajendra College of Engineering.
Topics
• Moving beyond parallel-jaw grippers toward true in-hand dexterity
• Vision and touch for multi-fingered robot manipulation
• Why conventional human demonstrations struggle to scale to dexterous hands
• Why tactile simulation remains a bottleneck for sim-to-real learning
• PTLD and privileged tactile latent distillation
• Learning tactile policies without simulating tactile sensors
• Using privileged sensing on real hardware to supervise deployable policies
• Sim-to-real transfer for tactile manipulation
• INVAR and learning visual invariance for zero-shot transfer
• Separating visual robustness learning from downstream policy learning
• Training policies in clean simulation while remaining robust to real-world vision
• CARD and contact-aware human-to-robot retargeting
• Retargeting demonstrations based on interaction rather than pose matching
• Learning from human video, gloves, and motion capture
• Preserving task-relevant contacts across human and robot embodiments
• DSP and learning dexterity from human play
• Autonomous robot play as a scalable source of experience
• World models for goal-conditioned dexterous manipulation
• Distilling large-scale robot experience into deployable policies
• Combining human priors, privileged teachers, and autonomous learning
Open Discussion + Q&A
• Why has multi-fingered dexterity remained so much harder to scale than parallel-jaw manipulation?
• Which aspects of touch are most important for in-hand manipulation: contact location, force, slip, pressure, or something else?
• Can we obtain the benefits of tactile simulation without actually building an accurate tactile simulator?
• What information should a privileged teacher have access to that the deployed robot should not?
• How much real-world tactile data is needed to successfully distill privileged information?
• Should visual domain randomization happen during every policy training run, or can invariance be learned once and reused?
• What kinds of visual invariances transfer across different manipulation tasks?
• Why is matching human and robot poses often the wrong objective for retargeting dexterous demonstrations?
• Is preserving object contact more important than preserving human hand trajectories?
• How can demonstrations from video, gloves, and motion capture be unified despite providing very different kinds of information?
• What can humans demonstrate naturally that is difficult to express through robot teleoperation?
• How far can a few minutes of human play be amplified through autonomous robot experience?
• What role should world models play in scaling goal-conditioned dexterous behavior?
• When is distillation preferable to deploying a larger teacher policy directly?
• Could autonomous play eventually generate most of the data needed for dexterous robot learning?
• What combination of vision, touch, demonstrations, simulation, and autonomous experience is most likely to unlock general-purpose robot dexterity?
Full text on Luma .
Investors in the room
