Imitation Learning with Robot Arms: A Beginner's Guide (SO100 + LeRobot)

What is imitation learning? How does it work? And how can you do it at home with an SO100 robot arm and Hugging Face's LeRobot? This guide explains everything from scratch.

---

What is Imitation Learning?

Imitation learning (also called learning from demonstration) is a machine learning technique where a robot learns to perform tasks by watching a human do them first.

Instead of manually programming every movement, you:

1. Demonstrate the task (e.g., pick up a cup)
2. Record the demonstration data (joint positions, camera images, actions)
3. Train a neural network on the demonstrations
4. Deploy the trained model — the robot now performs the task autonomously

It's like teaching by example. Show the robot what to do 50 times, and it learns the pattern.

Why Imitation Learning is a Big Deal

Traditional robotics programming is hard. You need to:

• Define exact waypoints and trajectories
• Handle edge cases manually
• Write complex control logic
• Re-program for every new task

Imitation learning flips this: you just show the robot what to do. The AI figures out the control logic automatically.

This is why companies like Google (RT-2), Tesla (Optimus), and Figure AI are investing billions in imitation learning. And thanks to open-source tools, you can do the same thing at home for $299.

The Hardware: SO100 Robot Arm

The SO100 robot arm is the most popular hardware for imitation learning in the open-source community. Here's why:

• Leader + Follower setup: You move the leader arm, and the follower arm mirrors your movements. This is how you demonstrate tasks.
• 6 DOF: Six joints give enough dexterity for complex manipulation
• STS3215 servos: 30 kg·cm torque — strong enough for real tasks
• $299 pre-assembled: Buy at so100.nanocorp.app
• LeRobot native support: The standard software stack

The Software: LeRobot

LeRobot by Hugging Face is the software framework that makes imitation learning accessible. It handles:

• Data collection via teleoperation
• Dataset management and Hugging Face Hub integration
• Policy training (ACT, Diffusion Policy, TDMPC)
• Model deployment to the robot
• Sharing datasets and trained models with the community

How Imitation Learning Works (Step by Step)

Step 1: Teleoperation (Data Collection)

You physically move the leader arm while the follower arm mirrors your movements. During this process, LeRobot records:

• Joint positions of both arms (at 30-60 Hz)
• Camera images (if cameras are attached)
• Gripper state (open/close)
• Timestamps

Each complete task execution is called an episode. For a simple pick-and-place task, you might record 50 episodes, each lasting 5-10 seconds.

Step 2: Dataset Creation

LeRobot automatically packages your recordings into a dataset format compatible with the Hugging Face Hub. Your dataset includes:

dataset/
├── episode_0/
│   ├── observations (joint states, images)
│   └── actions (what the robot did)
├── episode_1/
│   ├── ...
├── ...
└── episode_49/

You can upload this to the Hugging Face Hub to share with the community — or use datasets others have shared.

Step 3: Policy Training

A policy is a neural network that maps observations (what the robot sees/feels) to actions (what the robot should do). Popular architectures include:

ACT (Action Chunking with Transformers)

• Developed by Tony Zhao et al. at Stanford
• Uses a transformer architecture to predict chunks of future actions (not just the next single action)
• Works great with SO100 + LeRobot
• Recommended for beginners

Diffusion Policy

• Inspired by diffusion models (like Stable Diffusion, but for robot actions)
• Generates actions by iteratively denoising random noise
• Better for complex, multimodal tasks
• Slower inference but often more robust

TDMPC (Temporal Difference Model Predictive Control)

• Combines model-based and model-free approaches
• Good for tasks with clear goals

Training typically takes 1-6 hours on a GPU, depending on dataset size and policy type.

Step 4: Deployment

Load the trained model onto the robot and let it run:

python -m lerobot.deploy --policy.path=your_trained_model

The robot now performs the task autonomously. It reads its current joint positions (and camera images, if used), feeds them through the neural network, and executes the predicted actions.

Your First Imitation Learning Project

Here's a concrete project you can do this weekend:

Task: Pick up a block and place it in a cup

What you need:

• SO100 robot arm (leader + follower) — Buy for $299
• A small block (2-3cm cube)
• A cup or container
• Computer with LeRobot installed

Steps:

1. Place the block at a consistent starting position
2. Run python -m lerobot.record --num_episodes=50
3. For each episode: use the leader arm to pick up the block and drop it in the cup
4. Train: python -m lerobot.train --policy.type=act
5. Deploy: python -m lerobot.deploy
6. Watch your robot pick up the block autonomously! 🎉

Tips for success:

• Be consistent in your demonstrations (similar paths, similar speeds)
• Add small variations (slightly different block positions) to help generalization
• Start with 50 episodes — add more if the policy isn't reliable enough
• Use good lighting if you add a camera

Going Further: Advanced Imitation Learning

Once you've nailed the basics:

Add Vision

Mount a webcam above or beside the workspace. Visual policies can generalize to different object positions without needing to vary your demonstrations as much.

Try Harder Tasks

• Stacking blocks (requires precision)
• Pouring liquid (requires smooth motion)
• Sorting objects (requires perception)
• Bimanual tasks (use two SO100 setups!)

Use Pre-trained Models

The Hugging Face Hub has hundreds of pre-trained models and datasets for the SO100. Fine-tune them on your specific setup instead of training from scratch.

Sim-to-Real Transfer

Train in simulation (MuJoCo, Isaac Sim) where you have unlimited free data, then transfer the policy to your real SO100. This is an active research frontier.

Why the SO100 is the Best Platform for Learning Imitation Learning

Reason	Details
Affordable	$299 gets you everything — less than a textbook at some universities
LeRobot native	The framework was literally built for this arm
Community	Hundreds of shared datasets and models to learn from
Leader + Follower	Natural teleoperation — no VR setup or complex interfaces needed
Fast iteration	Record 50 demos, train, deploy — all in an afternoon
Real research	Same techniques used at Google, Stanford, Berkeley

📖 Want to compare robot arms before buying? Check out our guide to the best robot arms for AI and machine learning.

---

Start Your Imitation Learning Journey

Get the SO100 Complete Kit — $299 →

Everything you need to start doing real imitation learning at home. Pre-assembled, tested, and LeRobot-ready.