From camera feedback to physical action

I built Ninibot to explore how a low-cost robotic arm can turn noisy visual observations into a complete tabletop manipulation task. A wrist-mounted camera guides an SO-101 arm through cup acquisition, staged transfer, and a marker-guided button press.

My role: System architecture, perception integration, kinematics and motion logic, manipulation state machines, ROS 2 orchestration, hardware tooling, tests, and experimental learning pipelines.

Stack: ROS 2 Jazzy · Python · OpenCV / ArUco · NumPy · YOLO · PyTorch · RViz · SO-101

Key contributions at a glance

Automated demonstration collection: the robot records visual fine-alignment episodes and randomly repositions the cup between successful cycles, removing routine manual scene resets.
Reusable hand-guided motion: a taught transition is smoothed, retargeted to new endpoints, and used by the manipulation pipeline.
Calibration-to-control integration: checkerboard calibration, URDF camera frames, and TF connect camera observations to robot motion.

These capabilities sit within a complete five-package ROS 2 system with hybrid perception, feedback-driven skills, workspace-aware transfer motion, and independently testable controllers.

See it in motion

Cup transfer demonstration · 38 seconds

Watch cup transfer on YouTube

Marker-guided button press · 24 seconds

Watch button pressing on YouTube

The challenge: making uncertainty manageable

A working manipulation system needs more than an object detector and a list of joint positions. The wrist camera moves with the arm; detections jitter or disappear; marker poses vary from frame to frame; and rotating at full reach creates a large sweep across the table.

I organized Ninibot around explicit, observable skills. Each skill has its own feedback loop, transition conditions, and failure behavior, while a task manager coordinates the complete sequence.

One task, six deliberate steps

Lock the destination. Scan the placement marker and estimate a stable target in the robot base frame before picking up the cup.
Find the cup. Search using the wrist camera and a selectable HSV, YOLO, or hybrid perception backend.
Align and approach. Use horizontal image error to steer the base and apparent cup width to adjust approach distance.
Grasp and lift. Close the gripper and establish a carry height.
Transfer and release. Retract, rotate, extend, lower, release, and move clear of the cup.
Locate and press. Transform the button marker into the base frame, refine alignment, and execute the press sequence.

Automated demonstrations with physical scene reset

I built a repeatable collection loop that connects the rule-based visual controller, episode logger, collection orchestrator, and random-placement skill. The robot searches and fine-aligns, saves the episode, grips and repositions the cup, returns, waits for settling, and starts again. Normal successful cycles do not require someone to move the cup between episodes.

Sampled placements are checked against angular/radial bounds and IK reachability. The orchestrator supports episode limits, pause/resume, reset timeouts, and configurable failure behavior. Physical collection remains supervised; automatic reset does not imply recovery from every scene or hardware failure.

The logger records vision errors, real and target joints, task states, and action deltas, with metadata and optional images. Offline training splits data by episode to prevent samples from the same episode entering both training and validation sets. Policy deployment and online retraining are outside this collection loop.

Collection orchestrator · Random-placement skill

Connecting camera measurements to robot motion

I implemented checkerboard-based intrinsic calibration to estimate the camera matrix and distortion coefficients, report RMS and per-image reprojection errors, and save a ROS-compatible calibration file.

The URDF models the camera mounting and its optical frame. Joint feedback updates the transform chain from robot base through the arm and gripper to the camera; TF converts ArUco observations into base-frame targets. Geometry tools display frame axes and marker/gripper positions so calibration and manipulation can be debugged together.

Camera mounting extrinsics are configured in the URDF and checked with these tools. This is not an automatic hand-eye calibration solver; measurement accuracy depends on physical calibration and the workcell.

Calibration and frame setup

Engineering decisions that shaped the system

1. Close the loop instead of trusting one frame

The acquisition controller smooths detections, requires multiple centered cycles, and uses separate acquire/release tolerances to avoid chattering near a threshold. A short detection loss holds the previous target; a longer loss returns the arm to search. Before grasping, the destination is statistically locked from repeated marker observations, giving the transfer skill a stable target before the scene changes. I isolated this logic from ROS I/O so recovery behavior can be tested without a camera or arm.

2. Separate object identity from the control target

HSV provides a lightweight baseline for the controlled workcell. YOLO provides semantic cup selection. Hybrid mode uses YOLO to identify the relevant region and HSV to supply the control centroid. Candidate gating, temporal continuity, reuse of the previous region of interest, and task-dependent fallbacks handle ambiguous detections and brief tracking losses. These backends share the same grasp controller, making their trade-offs easier to inspect.

3. Plan transfer around the arm's geometry

I represented cup motion in cylindrical coordinates: angle, radius, and height. The arm lifts and retracts before rotating, then extends toward the destination. This reduces the swept radius compared with rotating at full reach. Reach margins, joint limits, minimum-jerk transitions, and motor velocity/acceleration limits constrain the motion.

4. Use measured geometry for button pressing

The button target combines an ArUco pose with a calibrated marker-relative offset. Fine alignment repeats a sample–solve–move cycle using a URDF-derived numerical IK solver. The public configuration bounds Cartesian corrections to 6 mm per step, with joint limits, residual checks, timeouts, and explicit failure states.

Architecture built for inspection

The implementation is split into five ROS 2 packages. vision_controller handles cup detection, ArUco markers, and camera transforms; arm_controller provides the robot model, kinematics, motor bridge, and RViz; ninibot_interfaces defines two actions and an IK service. The top-level ninibot package composes skills and orchestration. A separate audio_controller supports a water-flow sensing experiment.

Camera observations feed the skill controllers, kinematics translates their targets into joint commands, and the motor bridge executes bounded motion. Joint feedback, status topics, and action feedback make progress and failures visible to the task manager.

Teaching motion without hard-coding its endpoints

For the transition from search to fine alignment, I record a hand-guided movement with motor torque disabled. I then resample and smooth the trajectory, preserve its endpoints, and reconstruct wrist orientation from shoulder/elbow geometry. The resulting motion template is used by the manipulation pipeline, making a complex transition easier to author and refine.

The template separates an endpoint baseline from a smoothed nonlinear residual. At runtime, the baseline adapts to the current start and desired end while retaining the useful motion shape. This provides a way to author compact transitions, although space savings from the taught path remain unquantified; smoothing alone does not establish a smaller sweep.

The repository includes a traceable example: a 19.14-second recording becomes a 6.14-second processed trajectory with 616 samples at 100 Hz, accompanied by position and joint-dynamics plots. This is a motion-development artifact, not a task-success benchmark.

Validation and current scope

The public repository documents 25 focused test modules containing 112 test functions covering geometry, state transitions, target locking, search recovery, carry planning, IK, and motor ramps. My workflow layers deterministic tests, calibration tools, RViz dry-runs, individual skill runs, and supervised hardware trials.

Ninibot is an active research prototype tuned for one tabletop workcell. The main hardware demo uses rule-based state machines. Imitation learning currently covers automated demonstration collection, offline training/evaluation, and a separate RViz policy preview. The learned policy does not control the main hardware demo, and audio water-flow sensing is not yet connected to the full task manager.

Next steps are to measure repeated task success, test lighting and object variation, add collision-aware planning and hardware-in-the-loop regression tests, and compare learned fine alignment with the existing controller.

Explore the implementation

GitHub repository · System architecture · Learning and taught trajectories

Ninibot — Vision-Guided Robotic Manipulation