Hi, I'm Arav Bhargava

The Core Question:

What happens when the learner assumes ideal execution vs learning under its own real execution distribution?

Why this matters for robotics:

• SARSA (on-policy) updates using the action actually taken, including exploration noise — learns policies robust to imperfect execution
• Q-learning (off-policy) optimizes toward the best possible next action — can overestimate safety near constraints if it assumes perfect control
• Real robots have actuator noise, state estimation error, and latency — the agent should learn a policy robust to these factors

The Result:

• Off-policy (Q-learning) learns the shortest path near danger — assumes it will act optimally next step
• SARSA learns a safer path farther from hazards — accounts for its own exploratory behavior
• Robotics translation: SARSA ≈ conservative + safer when randomness exists; Q-learning ≈ aggressive + can be brittle near constraints

What I built:

• Implemented both SARSA and Q-learning training loops with ε-greedy exploration
• Trained on CliffWalking and Taxi environments
• Visualized learned Q-values and final policies
• Recorded side-by-side demos showing behavioral differences

Why this matters:

• In legged robotics, you constantly trade off speed, energy, stability, and wear
• Multi-objective framing makes tradeoffs explicit instead of hidden in a single reward
• The minimum speed constraint (forward ≥ 85% of max) turns this into a realistic engineering decision
• This is Pareto optimization with a deployment constraint — very close to how you'd pick a controller for field use

What I built:

• Wrapped a multi-objective environment where reward is a vector: [forward_progress, energy_cost]
• Trained PPO policies with scalarization: reward = forward − wE × energy
• Swept energy weights to generate visibly different gaits and behaviors
• Evaluated each policy on true vector objectives (speed + energy independently)
• Selected policies by measured behavior:
  • FAST = max forward progress
  • ENERGY EFFICIENT = min energy among policies meeting the minimum speed constraint

Results:

The Challenge:

Sparse reward problems are notoriously hard. You only get signal when the object lands exactly where it should — early on, the robot almost never randomly succeeds, so there's nearly no learning signal.

How HER Solves This:

• "You didn't reach the desired goal — but you did reach some state"
• HER relabels that state as if it were the goal, turning failures into useful training data
• Now the agent learns how actions cause goal changes, even from unsuccessful episodes
• This makes sparse-reward manipulation go from "hopeless" to learnable

What I built:

• Trained on FetchPickAndPlace (Gymnasium Robotics) with goal-conditioned observations: [robot_state, achieved_goal, desired_goal]
• Used Soft Actor-Critic (SAC) — stable off-policy continuous control with replay buffer
• Combined with HER for goal relabeling — necessary for sparse reward learning
• Policy outputs continuous actions controlling end-effector + gripper in MuJoCo physics

Results:

100% success rate means the policy reliably grasps and places objects to the target location across all evaluation episodes.

Robotics Applications:

This is directly applicable to: pick-and-place, bin picking, assembly primitives, warehouse manipulation, and imitation/RL bootstrapping for real robot arms.

Tech Stack:

Python • Gymnasium Robotics • Stable-Baselines3 • SAC • HER • MuJoCo

The Problem:

Football teams spend enormous manual effort tagging clips and tracking assignments during film review. FilmAI aims to reduce this work by automatically detecting players and key field context, making film review more searchable, measurable, and consistent.

What I Built:

• Player detection + tracking overlays on video (bounding boxes / IDs)
• Software interface concept for navigating clips and visualizing detections
• Realistic "product story" showing how teams would use it during review
• Fast filtering, quick annotations, and repeatable breakdown workflow

Feedback / Traction:

I shared the demo with people close to real team workflows and received positive feedback, including from the New York Jets. This validated the core premise: if the system can reliably extract structure from film and make review easier, teams care.

Why This Matters:

• Computer vision — turning video into structured data
• Human-in-the-loop workflow design — making outputs useful to real analysts
• Product prototyping — building something concrete to test with users

Even as a demo, FilmAI forced me to think like both an engineer and a product builder.

The Problem:

In most places, getting a usable prosthesis still means custom fitting, multiple appointments, and a price tag that can hit $10,000+. For amputees in low-resource settings, that combination makes access basically impossible. Even when devices are donated, comfort and fit are the difference between daily use and something that sits in a closet.

Technical Innovation:

• Patented socket architecture that adjusts on-body using real-time tightening (no casting required)
• Skin-safe silicone interface + pressure distribution features to reduce hotspots during load-bearing
• Designed for fast fitting: the goal is a <5 minute setup by minimally trained staff
• Manufacturing approach tuned for iteration: rigid CF-infused prints + flexible TPU parts where compliance matters
• Print settings and infill are intentionally varied by region to balance stiffness, comfort, and durability

Results:

• Cost: ~$40 materials (vs. $10,000+ traditional)
• Testing: Load-bearing validation + iterative comfort testing across socket revisions
• Partners: Morph Labs (Neuroprosthetics)
• Team: Led a 6-person MIT/Harvard team through prototyping and validation
• Patent: US Patent #12,245,957

The Challenge:

Flight testing at insect scale is unforgiving: small turbulence and velocity drift can swamp the signal you are trying to measure. Commercial wind tunnels that hit the right flow quality are expensive and often too large for a typical bench setup, so I built a version that’s cheap, modular, and reproducible.

What I Built:

• Bench-top vertical wind tunnel designed around flow conditioning (screens/honeycomb + contraction)
• Closed-loop airflow control with Arduino-based regulation (0.2–2.5 m/s with ~±0.02 m/s resolution)
• Compact form factor (sub-600mm height) so it can actually live next to the robot + cameras
• PIV-compatible test section for quantifying flow profile and turbulence

Results:

I published the design so other labs can replicate it; it’s now been adopted by teams at MIT and Berkeley.

Technical Implementation:

• EMG acquisition pipeline with analog bandpass filtering (20–500 Hz) to isolate usable muscle activity
• Signal conditioning chain (amplification + noise reduction) tuned for consistent classification
• KNN-based gesture classifier optimized for fast per-user calibration (quick training, stable inference)
• Tendon-driven, multi-servo actuation so the hand responds immediately and predictably

Results:

Achievements:

• VEX Robotics World Championships 2021
• 6 consecutive state qualifications
• Tournament Skills Champion
• Team Captain (Years 9, 11, 12)

Technical Focus:

• Autonomous programming built around sensor feedback and repeatability (not “perfect conditions”)
• Pneumatics integration and debugging for consistent cycle times
• CAD for full mechanisms and drivetrains, including gearing and packaging constraints
• 150+ page engineering notebooks documenting tests, failures, and iteration decisions

Design Requirements:

• Climb 30-degree ramps reliably
• Place objects into goals at heights up to 21 inches
• Weight limit: 4.87 lbs (2.21 kg)

Technical Features:

• 4WD drivetrain with gear ratios chosen for torque on ramps without stalling
• 2-bar arm designed around stiffness and manufacturability, validated with FEA before cutting metal
• Custom cast Ecoflex wheels for traction and compliance on uneven surfaces
• Machined aluminum/steel components with tight tolerances for alignment, backlash control, and durability

Technical Architecture:

• MediaPipe's 21-point hand landmark detection
• Normalized coordinate extraction for position-invariant recognition
• Random Forest classifier (200 estimators)
• Real-time inference at 30+ FPS

Implementation Pipeline:

• Data Collection: 100 images per class across 26 ASL letters
• Feature Extraction: Normalized hand landmark coordinates
• Model Training: Scikit-learn with train/test validation
• Inference: Live camera feed with prediction overlay

Tech Stack:

Python • OpenCV • MediaPipe • Scikit-learn • Random Forest

The Problem:

Manual OSHA verification is time-consuming and error-prone. Compliance failures can delay projects or compromise occupant safety.

What I Built:

• Custom Dynamo algorithms for egress path calculation
• Real-time analysis from every building point
• Seamless BIM integration within Revit
• Automated identification of safety concerns before construction

Impact:

• Eliminates manual safety verification processes
• Reduces compliance failures and project delays
• Enables proactive safety analysis during design phase

Tech Stack:

Dynamo Visual Programming • Autodesk Revit • BIM