Tharun V. Puthanveettil

Robotics Engineer · Robot Learning and Deformable Manipulation

Optimal, assigned to General Motors R&D

I study how robots can understand what people want and carry what they learn from one interaction into the next. I use physical robots and simulation to test how policies learn from demonstrations and corrections, and what information they actually use when they act.

Tharun speaking at an event

Research questions

Current work

Learning from people.
Understanding policies.

The bimanual robot and its digital twin share a platform I built to study how policies learn from human corrections and what information they use to act.

The correction loopImplemented workflow
From a failed rollout to the next policy A person takes over when the policy fails. The correction is recorded, reviewed, and added to demonstrations to train the next policy. Policy runs Human takes over Review correction Train next policy Policy fails
  1. 01

    Learning from human corrections

    Implemented workflow

    When a policy fails, a person can take control of the arm. The workflow records the correction so it can be reviewed and added to the demonstrations used to train the next policy.

    People and policies use the same arm commands, so their actions can be compared step by step. Policy evaluation is currently performed manually. Whether the corrections improve later policies without eroding earlier skills remains an open question.

  2. 02

    What information does a policy use to act?

    Simulation and physical robot studies

    In simulation, representation probes showed that the tested policies encoded links between words and objects. Activation patching showed that their action heads did not use those links.

    Bimanual robot trials measured object choice separately from pick completion. In those trials, higher task success did not necessarily mean the policy was better at choosing the object named in the instruction.

Related studies

Learning from people

Egocentric demonstrationsIn development

A portable interface is designed to capture human manipulation demonstrations using the same recording and review tools as teleoperation.

Understanding policies

What policies retain from trainingSimulation studies

Simulation tests compared teacher-student training with learned auxiliary heads to see what policies retain when privileged inputs are removed. Attention maps help diagnose behavior, but they do not by themselves establish a causal explanation.

Policy evaluation beyond success rateIn development

The goal is to compare policies on partial progress, failure modes, recovery and repeatability as well as task success.

Prediction and transfer

Predictive models and inverse dynamicsOffline evaluation completed · simulation studies

Offline tests evaluated a video-action model adapted to teleoperation data. Simulation tests compared actions inferred from video with recorded actions and commands sent to the robot. They assessed future-feature prediction, instruction grounding, and failure signals separately from task success. Physical validation is pending.

Transfer between simulation and hardwareActive experiments

Experiments use a digital twin to study how changes in sensing, timing, contact and appearance affect transfer between simulation and hardware.

Platform and evaluation tools

I designed and implemented the workflow from teleoperation and synchronized recording to dataset conversion, policy training, evaluation, and deployment. The physical robot and digital twin share the operator, data review, and policy interfaces, with separate hardware and physics backends.

The research platform

Implemented Six shared layers · physical and simulated backends · staged evaluation

The gated research platform Six shared platform layers crossed by seven promotion gates. One evidence path descends through the system and a failed authorization returns to capture as recovery data. OperatePerceiveTwinCaptureEvaluateOrchestrate multi-arm teleoperationscene statephysical-authoritativesynchronized evidencelearned policiesconfirmation-gated skills RECORDQCREVIEWFREEZECONVERTSHADOWAUTHORIZE RECOVER · RELABEL · RELEARN
Shared interfaces across physical and simulated backendsInspect a stage · policy evaluation is currently manual
Illustrative evaluation criteria
MODEL AMODEL B SAME TASK+ CONDITIONS OUTCOMEcompletion · progress BEHAVIORfailure · recovery CONTROLcompliance · repeatability
Controlled comparisonTask completion · partial progress
Illustrative policy diagnostics (not measured results)

Policy analysis instrument

Select an input source, then compare the nominal and controlled-perturbation views.

Input under study
Diagnostic view

Vision Which regions redirect the policy response when visual evidence changes?

Attention routing under input ablationFour cell fields compare the input, attention with all inputs, attention after one input is ablated, and the resulting difference. INPUT FIELD ATTENTION · ALL INPUTS ATTENTION · ONE ABLATED Δ · ROUTED ELSEWHERE
Controlled diagnostic schematic · attribution is not causationIllustrative field—not measured output

Research portfolio

Selected projects

Case studies with methods, results, and links to papers or code.

Real UAV deployment
2024 · PreprintPhysical flight trials

Track Anything Rapter

A person can use a click, box, image, or text to tell a drone what to follow. I co-developed the system, implemented the high-level controller, and designed the trajectory evaluation against motion capture.

Physical flight trials compared target and drone trajectories. Obstruction increased the distance between the aligned trajectories in the reported trials.

U-ASTROD · GameSec 2024

Multimodal anomaly detection

For the ASTROD research program, I architected the fusion pipeline that encoded LiDAR, odometry, and network traffic independently before combining their learned representations in a shared anomaly detector.

The published system paired modality-specific spatiotemporal encoders with feature-wise reconstruction scoring. It achieved up to 98% accuracy with vector-based reconstruction loss versus 72% with scalar-based loss, and 97% with half the training data.

U-ASTROD process flow with actual network, odometry, and LiDAR samples feeding modality-specific encoders and a shared anomaly detector
U-ASTROD process flow · redrawn from the published architecture
Chapter 02 · 2021–23Formative research exploration

Multimodal human instruction

Learning robot actions from demonstrations, speech, and gesture

Using the NatSGD dataset provided by Snehesh Shrestha (UMD), I implemented a kitchen-object detector, a language-conditioned recurrent policy, and two experimental gesture pathways. The notebooks establish working model and training prototypes, while task success and the benefit of gesture remain unvalidated.

Contribution boundary: I did the notebook implementation; the NatSGD dataset and paper belong to Shrestha and his co-authors. The model builds on Stepputtis et al.'s language-conditioned imitation-learning method.

NatSGD overview showing synchronized speech, gesture, visual observations, and robot demonstrations

Related research contextNatSGD later formalized speech, gesture, and synchronized robot demonstrations at UMD. Image: Shrestha et al., CC BY 4.0.

Additional projects

A filterable index of research systems, prototypes, course projects, and independent software.

01
Current research platform

Robot Learning for Unstructured Manipulation

Bimanual manipulation through multimodal perception, compliance, learned policies, simulation, and real-world deployment.

04
Connected autonomy

Auto-Platoon ↗

Bio-inspired leader–follower coordination using software latching.

05
Multimodal UAV · preprint

Track Anything Rapter ↗

Click, image, text, and box-conditioned tracking with foundation models and visual servoing.

TAR multimodal detection and tracking examples
12
Egocentric interaction

AI Cricket Coach ↗

Action recognition and immersive feedback for batting practice.

13
Product intelligence

My Style ↗

An AI-assisted shopping and personal-style application.

16
Cyber-physical security · published

Multimodal Anomaly Detection ↗

Modality-specific LiDAR, odometry, and network encoders fused for abnormal-behavior detection.

U-ASTROD multimodal fusion architecture
17
Local-first product

UltraNote Lite ↗

A self-hosted workspace for daily capture, research, projects, and human–agent collaboration.

UltraNote research workspace
19
Local-first product

Intra-Chat ↗

A LAN-hosted team hub for conversation, project knowledge, files, and equipment records.

Intra-Chat team conversation interface

Independent software projects

Independent products · 2025–present

Tools I build outside my robotics work to explore local-first software, technical knowledge management, and human–agent workflows.

01 / 02Use arrows, tabs, or swipe

UN
Local-first · self-hosted · open source

UltraNote Lite

“The notebook I wanted
but couldn’t find.”

A single-user workspace for daily notes, tasks, projects, journaling, and research capture. It occupies the space between heavyweight cloud tools and bare Markdown: fast enough for every thought, structured enough for long-running work, and owned entirely by the person using it.

Local firstNo cloud, account, telemetry, or data lock-in.Capture firstOne shortcut routes a task, note, journal entry, or idea.Agent readyPeople and coding agents share one durable project record.
01Capture anywhereDesktop · PWA · share sheet · bookmarklet
02Organize deliberatelyDaily · projects · research · wiki-links
03Keep the source of truthOne Node process · one owned data file
04Collaborate with agentsContext · query · search · validated updates

Research record

Selected publications.

Selected peer-reviewed publications and preprints. Also published as V. P. Tharun. Earlier forecasting and data-mining work remains available below.

2017 · Chapter 01Soft robotics

Wall-Climbing Robot Using Soft Robotics

L. P. Pratap, P. M. Shailendrasingh, A. Anand, V. P. Tharun

Can compliant mechanisms produce a lightweight inspection robot for difficult pipe surfaces?

Develops a soft-robotic wall-climbing prototype for inspection inside and outside pipes.

Earlier data-mining and forecasting publications
2021 · Chapter 01Forecasting

A Univariate Data Analysis Approach for Rainfall Forecasting

V. P. Tharun, P. Ramya, S. Renuga Devi

How can lightweight time-series analysis provide useful early warning of unexpected rainfall?

Develops an efficient and implementable approach for short-term rainfall forecasting.

2019 · Chapter 01Regression

Prediction of Rainfall Using Data Mining Techniques

V. P. Tharun, R. Prakash, S. R. Devi

How do regression and statistical models compare when predicting regional rainfall intensity?

Evaluates multiple regression approaches using relative error on rainfall data from Coonoor.

2018 · Chapter 01Classification

A Comparative Study of Various Classification Techniques to Determine Water Quality

R. Prakash, V. P. Tharun, S. R. Devi

Which classical learning method best characterizes groundwater quality from measured samples?

Compares decision trees, K-nearest neighbours, and support-vector machines on groundwater data.

2018 · Chapter 01Assessment

Water Quality Assessment Using Data Mining Techniques

P. Ramya, V. P. Tharun, S. Renuga Devi

How can data-mining methods turn raw water measurements into an actionable quality assessment?

Applies classical learning techniques to environmental water-quality analysis.

2018 · Chapter 01Time series

Rainfall Forecasting Using Time Series Analysis

V. P. Tharun, P. Ramya, S. Renuga Devi

What temporal structure in historical rainfall can be converted into a practical forecast?

Uses time-series analysis to model and forecast rainfall patterns.

View complete Google Scholar profile

Experience

Research and engineering roles.

01

Present

Robotics Engineer – Robot Learning and Deformable Manipulation, General Motors R&D

Optimal – assigned to General Motors R&D

Bimanual deformable-object manipulation, multimodal perception, force-aware compliance, demonstration-driven policies, digital twins, sim-to-real evaluation, and ROS 2 deployment.

02

University of Maryland

Researcher – AI and Robotics

Mobile-robot navigation security, abnormal-behavior detection, action recognition (graduate course project), and multimodal human–robot interaction.

03

Indian Institute of Science

Researcher – Collaborative Robotics and Computer Vision

Collaborative agricultural robotics, manipulation, and perception-driven precision weeding.

04

CTO Office, Wipro Digital

Project Engineer – AI Innovation

Computer vision, mobile AI, egocentric interaction, mixed reality, and rapid prototyping across emerging technologies.

Technical storytelling

Writing on robotics, AI, and their human context.

I write about the assumptions behind models and demonstrations, the engineering between a benchmark and deployment, and the human consequences of technical choices.

Human in the Loom is an independent publication for longer essays on these themes.

Human in the LoomIndependent publication ↗

Research direction

Robots that can understand people and learn from them.

My long-term aim is to develop algorithms that allow robots to interpret human intent in ways that support action and learning.

My work approaches this problem through policy understanding, representation learning, world models, embodied experience, human interaction, continual learning, and rigorous evaluation.

I am particularly interested in methods that remain interpretable and testable when they move from offline benchmarks to physical systems.

I hold an M.Eng. in Robotics from the University of Maryland and bring experience across academic research, industrial R&D, and real-world robotics deployment.

Collaborate

Research and engineering
collaboration.

Contact me