Representation
learning
What has a policy learned, and which inputs influence its actions?
Tharun V. Puthanveettil
Robotics R&D Engineer
My research takes inspiration from how people learn, adapt, and develop skills. I work on robot learning, multimodal models, and the systems needed to study them on physical robots. My long-term aim is to build robots that improve through experience and understand human intent with the contextual sensitivity people use to understand one another—and act or learn from that understanding.
Representation
Egocentric
policies
How can embodied demonstrations become useful robot experience?
World models
for robotics
How can predicted outcomes and inverse dynamics support action learning?
Human–robot
interaction
How can human instruction and correction help robots acquire skills?
Background
How the research
developed.
My earlier work in perception, multimodal systems, and physical robotics provides the technical basis for my current research in human-inspired robot learning.
Data, perception, and early physical systems
Work in statistical learning, computer vision, soft robotics, human-facing AI, and ROS established experience in turning observations into decisions and actions.
Rainfall · Water quality · Soft robot · My Style · Try-on · AI Cricket · ROSPerception applied to manipulation and HRI
At IISc and UMD, I worked on depth estimation, precision weeding, and an exploratory imitation-learning system using demonstrations, speech, and gesture.
Depth · Precision weeding · ROS 2 · Speech + gesture + demonstrationMultimodal autonomy and evaluation
I developed systems that combined partial evidence across sensors or viewpoints and connected perception to control, coordination, or anomaly detection.
PoseFusion · Fall detection · Auto-Platoon · TAR · Multimodal anomaly detectionRobot learning on physical systems
My current work brings these threads together in bimanual manipulation: teleoperation, multimodal data, compliant control, learned policies, digital twins, and real-world evaluation.
Deformables · Bimanual manipulation · Compliance · Learned policies · Sim-to-realQuestions guiding the research
01What representations support understanding of people, tasks, and environments?
02How can a robot adapt through experience without losing prior capability?
03How should capability be evaluated beyond task-level success?
Present · R&D engineer, AI & robotics
Research infrastructure
A shared system for studying teleoperation, data collection, learned policies, human intervention, simulation, deployment, and comparative policy evaluation on bimanual manipulation tasks.
I designed and implemented the workflow from teleoperation and synchronized collection through dataset conversion, policy training, evaluation, testing, and deployment.
The physical cell and digital twin use the same operator, recording, review, conversion, and policy interfaces, with separate hardware and physics backends. This supports controlled comparisons between simulation and real deployment.
Interventions and failed rollouts can be recorded, reviewed, converted, and returned to training. Policy evaluation is currently performed manually. I am also developing a common evaluation methodology that goes beyond aggregate success rate to compare model behavior more fairly.
The research platform
Implemented Six shared layers · physical and simulated backends · staged evaluation
Modular ROS 2 infrastructure for bimanual manipulation, integrating teleoperation, perception, simulation, data capture, training, and deployment.
Human takeover, correction capture, review, dataset conversion, retraining, manual evaluation, and controlled policy rollout.
Shared interfaces across the real cell and digital twin, with staged testing and recorded evidence for each policy version.
Current research directions
Each line of work is marked by its current stage of development and validation.
-
01
Continual learning through intervention
Implemented workflow Human takeover, correction capture, review, conversion, retraining, testing, and deployment are connected. Evaluation is manual while automated policy evaluation is being developed.
Refusal becomes the next data campaign -
02
Real → sim → real
Active experiments I use the digital twin to study transfer across sensing, timing, control, contact, and visual variation. Current experiments examine representations and evaluation methods that remain useful when simulated and physical environments do not match closely.
Backend substitution · shared owners after the bar -
03
Policy explainability and representation
Experiments completed I use attention maps, Grad-CAM, state-vector ablations, prompt comparisons, and comparisons across self-supervised and pretrained representations to study what influences policy outputs. These are diagnostic tests; they do not by themselves establish a causal explanation.
Cut one input · measure the output response -
04
Egocentric demonstration learning
In development We are developing a portable egocentric interface for recording human manipulation demonstrations. It reuses the teleoperation, synchronization, data review, and conversion infrastructure I built for the current manipulation system.
Current stateShared data infrastructure implemented; demonstration-capture hardware and egocentric policy studies in development. -
05
Video-action models and inverse dynamics
Offline evaluation completed I adapted a published video-action architecture to teleoperation data and completed offline evaluation. Physical evaluation has not yet been performed. I am continuing to study how predictive and inverse-dynamics models can support action learning and transfer.
Next validationPhysical rollouts and controlled comparison with existing policy approaches. -
06
Policy evaluation beyond success rate
Methodology in development I am evaluating leading open-source vision-language-action models across tasks and developing a common methodology for fairer comparison. The work treats aggregate success rate as incomplete and studies partial progress, failure modes, repeatability, recovery behavior, and sensitivity to control conditions.
Controlled comparisonTask completion · partial progress
Policy analysis instrument
Select an input source, then compare the nominal and controlled-perturbation views.
Input under study
Diagnostic view
Vision Which regions redirect the policy response when visual evidence changes?
Selected systems
Projects and
technical contributions.
Selected work in robot learning, perception, control, multimodal interaction, and physical deployment.
Robotic intelligence for unstructured manipulation
Research and engineering for bimanual manipulation of deformable objects—coupling multimodal perception, force-aware compliant control, human demonstrations, learned action policies, high-fidelity simulation, and real-world ROS 2 deployment.
Open case study
Physical deployment
Hybrid IK for plant-safe planning
A plant is not a rigid obstacle with a clean bounding box. This system first estimates roots, leaf spread, and bed geometry, then couples learned plant geometry with a hybrid inverse-kinematics controller so the manipulator can reach weeds while reducing unintended contact.
See the living workspaceVision extracts stems, spread areas, and task boundaries—not merely object classes.Borrow from human craftA flower-carpet-inspired planner converts irregular plant layouts into an ordered manipulation sequence.Warp before solvingThe reachable workspace is reshaped around delicate geometry before hybrid IK selects a motion.
AGGRIP · CAD concept
End-effector co-design
Precision weeding end effector
I co-designed this gripper with a mechanical engineer as part of the same research program. Vision-adaptive control positions the tool to reduce contact with non-weed plants, while the end effector combines mechanical plucking with targeted spraying or watering.
Pluck + sprayPluck + waterMinimal crop contact
Architecture designed across two studiesAction recognition from partial evidence
I defined the core architecture across two studies: pose graphs encode spatial structure through GCNs, transformers model action over time, and attention fuses incomplete evidence across camera views. The work supported privacy-preserving fall detection and multi-view recognition under occlusion—then became the architectural starting point for my multimodal anomaly-detection work.
0.82 F1 under occlusion3 views fused semanticallyGCN → Transformer spatial to temporal
Software-latched multi-robot autonomy
A safety-aware leader–follower stack combining learned detection, feature-based tracking, monocular depth, Kalman state estimation, cooperative sensing, and explicit engage–disengage transitions.
Prototype still
Pneumatic wall-climbing robot
A lightweight inspection prototype built around three single-bending pneumatic actuators—moving compliance from a control objective into the robot’s physical design.
View publication
Real UAV deployment
Track Anything Rapter · TAR
One target.
Many ways to specify it.
I co-developed a ROS 2 aerial system that accepts a click, box, image, or text query; combines foundation models for detection and segmentation; then closes the loop through visual servoing on a PX4-enabled VOXL2 M500 drone.
- 01Specifytext · image · click · box
- 02PerceiveDINO · CLIP · SAM
- 03Tracktarget pose · re-detection
- 04Actvisual servoing · PX4
Multimodal human instruction
Learning robot actions from demonstrations, speech, and gesture
Using the NatSGD dataset provided by my PhD guide, Snehesh Shrestha, I implemented a kitchen-object detector, a language-conditioned recurrent policy, and two experimental gesture pathways. The notebooks establish working model and training prototypes, while task success and the benefit of gesture remain unvalidated.
Contribution boundary: I did the notebook implementation; the NatSGD dataset and paper belong to Shrestha and his co-authors. The model builds on Stepputtis et al.'s language-conditioned imitation-learning method.
Related research contextNatSGD later formalized speech, gesture, and synchronized robot demonstrations at UMD. Image: Shrestha et al., CC BY 4.0.
Technical module · U-ASTROD
Multimodal anomaly detection
For the ASTROD research program, I architected the fusion pipeline that encoded LiDAR, odometry, and network traffic independently before combining their learned representations in a shared anomaly detector.
The published system paired modality-specific spatiotemporal encoders with feature-wise reconstruction scoring. On real robot attacks, that scoring increased multimodal detection accuracy from 72% to 98% and retained 97% accuracy with half the training data.
Research map
One program, tested through different systems.
The projects below are not separate interests. Each isolates a capability needed for robots to perceive evidence, interpret people, learn skills, and act reliably.
Multimodal representation
Preserve useful structure when views, body regions, or sensor channels disappear.
Human-specified intent
Translate different forms of instruction into a target that a physical system can act upon.
Embodied data and skill learning
Connect demonstrations, synchronized datasets, policy training, evaluation, and correction.
World and action structure
Use geometry, depth, simulation, prediction, and inverse dynamics to make action consequences testable.
Coordination and interaction
Make local observations, human correction, compliance, and shared state influence group behavior.
Evaluation and explanation
Diagnose what a policy uses, how it fails, and whether a metric reflects physical competence.
Independent products · 2025–present
Independent software projects.
Tools I build outside my robotics work to explore local-first software, technical knowledge management, and human–agent workflows.
01 / 02Use arrows, tabs, or swipe
UN
Local-first · self-hosted · open source
UltraNote Lite
“The notebook I wanted
but couldn’t find.”
A single-user workspace for daily notes, tasks, projects, journaling, and research capture. It occupies the space between heavyweight cloud tools and bare Markdown: fast enough for every thought, structured enough for long-running work, and owned entirely by the person using it.
Local firstNo cloud, account, telemetry, or data lock-in.Capture firstOne shortcut routes a task, note, journal entry, or idea.Agent readyPeople and coding agents share one durable project record.
01Capture anywhereDesktop · PWA · share sheet · bookmarklet
02Organize deliberatelyDaily · projects · research · wiki-links
03Keep the source of truthOne Node process · one owned data file
04Collaborate with agentsContext · query · search · validated updates
IC
Offline-first · LAN-hosted · open source
Intra-Chat
“The operating layer
for a physical team.”
An actively developed, self-hosted team hub for labs, workshops, and robotics crews. It keeps conversation, technical knowledge, files, equipment, and agent-generated context on the local network—without external accounts, cloud dependency, or telemetry.
Durable memoryTurn chat into manuals and project knowledge.Built for the benchTrack equipment, owners, status, and check-outs.Agent readyScoped keys let scripts and agents push knowledge.
01Work offlineOne Flask process · any browser · local network
02Preserve team memoryChat · manual · projects · knowledge base
03Run the labFiles · inventory · ownership · check-outs
04Connect local agentsScoped API keys · Ollama summaries · no egress
Project archive
Additional work.
A filterable index of research systems, prototypes, course projects, and independent software.
01
Current research platform
Robot Learning for Unstructured Manipulation ↗
Bimanual manipulation through multimodal perception, compliance, learned policies, simulation, and real-world deployment.
02
Multimodal HRI · exploratory
Learning Actions from Human Intent ↗
My language-conditioned policy and gesture-grounding experiments on the NatSGD dataset.
03
Plant–robot interaction
Vision-Guided Precision Weeding ↗
Weed perception, adaptive control, plant-safe hybrid-IK planning, and a co-designed end effector.
04
Connected autonomy
Auto-Platoon ↗
Bio-inspired leader–follower coordination using software latching.
05
Multimodal UAV
Track Anything Rapter ↗
Click, image, text, and box-conditioned tracking with foundation models and visual servoing.
06
Aerial robotics
ORB-SLAM Drone Localization ↗
Vision-based localization for aerial asset inspection and management.
07
Aerial inspection
Surface Defect Detection ↗
YOLO-based visual inspection for detecting structural surface defects.
08
Image understanding
SLIC Image Segmentation ↗
Superpixel segmentation for compact image representation.
09
Representation learning
Implicit Neural Representation ↗
Image outpainting through coordinate-conditioned neural representations.
10
3D perception
Monocular Depth Estimation ↗
Transfer learning for estimating scene depth from a single camera.
11
Applied AI
Virtual Try-On ↗
Visual apparel synthesis and personalized style evaluation.
12
Egocentric interaction
AI Cricket Coach ↗
Action recognition and immersive feedback for batting practice.
13
Product intelligence
My Style ↗
An AI-assisted shopping and personal-style application.
14
Reinforcement learning
RL Cricket Simulation ↗
A bowler agent that adapts to batting styles and field configurations.
15
Soft robotics · published
Wall-Climbing Robot ↗
A lightweight soft-robotic prototype for pipe inspection.
16
Cyber-physical security · published
Multimodal Anomaly Detection ↗
Modality-specific LiDAR, odometry, and network encoders fused for abnormal-behavior detection.
17
Local-first product
UltraNote Lite ↗
A self-hosted workspace for daily capture, research, projects, and human–agent collaboration.
18
Research poster · 2021
Event-Based Dynamic Obstacle Avoidance ↗
An exploratory outdoor-safety concept using asynchronous event streams.
19
Local-first product
Intra-Chat ↗
A LAN-hosted team hub for conversation, project knowledge, files, and equipment records.
Research record
Selected publications.
Peer-reviewed publications, preprints, and posters, with links to the available manuscripts and project material.
2026 · Chapter 03Autonomous CPS
Efficient Integration of Domain Knowledge and Data-Driven Methods for Securing Autonomous Cyber-Physical Systems
How can domain structure and learned evidence work together to secure autonomous systems?
Connects first-principles system knowledge with data-driven detection for more efficient cyber-physical security.
2025 · Chapter 02Precision farming
Vision-Based Hybrid IK Task Planning with Feedforward Neural Network for Collaborative Plant–Robot Interaction
How can vision and learned task planning make plant–robot interaction more collaborative?
Combines vision, hybrid inverse kinematics, and a feedforward network for precision-farming task execution.
2024 · Chapter 03Real-world evaluation
Multimodal Anomaly Detection for Autonomous Cyber-Physical Systems Empowering Real-World Evaluation
How should autonomous systems be evaluated when abnormal behavior appears across multiple modalities?
Architected the multimodal fusion pipeline within a team framework for anomaly detection and evaluation beyond controlled laboratory conditions.
2024 · Chapter 03Multi-agent autonomy
Auto-Platoon: Freight by Example
How can autonomous followers coordinate reliably with a leader in dynamic environments?
Introduces software latching and a layered perception–decision–control architecture for real-time multi-agent coordination.
2024 · Chapter 03Multimodal UAV
Track Anything Rapter (TAR)
Can a UAV track what a person means, regardless of how they specify it?
Combines DINO, CLIP, SAM, multimodal queries, visual servoing, and motion control on a PX4-enabled aerial platform.
2021 · Chapter 01Forecasting
A Univariate Data Analysis Approach for Rainfall Forecasting
How can lightweight time-series analysis provide useful early warning of unexpected rainfall?
Develops an efficient and implementable approach for short-term rainfall forecasting.
2019 · Chapter 01Regression
Prediction of Rainfall Using Data Mining Techniques
How do regression and statistical models compare when predicting regional rainfall intensity?
Evaluates multiple regression approaches using relative error on rainfall data from Coonoor.
2018 · Chapter 01Classification
A Comparative Study of Various Classification Techniques to Determine Water Quality
Which classical learning method best characterizes groundwater quality from measured samples?
Compares decision trees, K-nearest neighbours, and support-vector machines on groundwater data.
2018 · Chapter 01Assessment
Water Quality Assessment Using Data Mining Techniques
How can data-mining methods turn raw water measurements into an actionable quality assessment?
Applies classical learning techniques to environmental water-quality analysis.
2018 · Chapter 01Time series
Rainfall Forecasting Using Time Series Analysis
What temporal structure in historical rainfall can be converted into a practical forecast?
Uses time-series analysis to model and forecast rainfall patterns.
2017 · Chapter 01Soft robotics
Wall-Climbing Robot Using Soft Robotics
Can compliant mechanisms produce a lightweight inspection robot for difficult pipe surfaces?
Develops a soft-robotic wall-climbing prototype for inspection inside and outside pipes.
Experience
Research and engineering roles.
Present
R&D Engineer · AI & Robotics
Bimanual deformable-object manipulation, multimodal perception, force-aware compliance, demonstration-driven policies, digital twins, sim-to-real evaluation, and ROS 2 deployment.
University of Maryland
Robotics & AI Researcher
Mobile-robot navigation security, abnormal-behavior detection, action recognition, and multimodal human–robot interaction.
Indian Institute of Science
Research Assistant
Human-collaborative agricultural robotics, manipulation, and perception-driven precision weeding.
Wipro Innovations Lab
AI Team Lead
Computer vision, mobile AI, egocentric interaction, mixed reality, and rapid prototyping across emerging technologies.
Technical storytelling
Writing on robotics, AI, and their human context.
I write about the assumptions behind models and demonstrations, the engineering between a benchmark and deployment, and the human consequences of technical choices.
Human in the Loom is an independent publication for longer essays on these themes.
Research direction
Robots that can understand people and learn from them.
My long-term aim is to develop algorithms that allow robots to interpret human intent in ways that support action and learning.
My work approaches this problem through policy understanding, representation learning, world models, embodied experience, human interaction, continual learning, and rigorous evaluation.
I am particularly interested in methods that remain interpretable and testable when they move from offline benchmarks to physical systems.
I hold an M.Eng. in Robotics from the University of Maryland and bring experience across academic research, industrial R&D, and real-world robotics deployment.
Collaborate