Situation

I developed Ninibot as a personal robotics project to explore how a low-cost robotic arm could perform a complete vision-guided manipulation task.

Although the task appears straightforward—find a cup, pick it up, place it at a target, and press a button—the physical system introduced uncertainty at every stage. Camera detections could jitter or temporarily disappear, ArUco marker poses varied between frames, and rotating the arm while fully extended created a large swept area. A simple sequence of memorized joint positions would not adapt reliably to these variations.

I therefore wanted to build more than a scripted demonstration. The system needed to be modular, observable, testable, and capable of recovering from common perception and motion failures.

Task

My task was to design and implement the complete robotics pipeline, including:

Cup and marker perception.
Visual alignment and grasp control.
Kinematics and constrained motion generation.
Cup transfer and release.
Marker-guided button pressing.
ROS 2 communication and task orchestration.
Motor control, calibration, diagnostics, and testing.
Safe dry-run visualization before physical execution.

Action

I organized the project into five ROS 2 packages with clear responsibilities for arm control, computer vision, audio experiments, interface definitions, and top-level task execution. This architecture allowed each capability to be developed and tested independently rather than combining the entire system inside one monolithic node.

For cup detection, I implemented three interchangeable perception backends: HSV color segmentation, YOLO object detection, and a hybrid method. The hybrid approach uses YOLO to identify the correct object while HSV provides a stable geometric center for feedback control.

I converted cup acquisition into a closed-loop state machine instead of commanding the robot from a single camera frame. The controller smooths detection results, requires several stable observations before advancing, and uses separate acquire and release thresholds to prevent rapid switching near a boundary. A short detection loss holds the previous target, while a longer loss returns the arm to its search behavior.

Before grasping the cup, Ninibot scans and locks the destination marker using multiple observations. The system rejects an unstable target and stores the averaged position only after the marker measurements become sufficiently consistent. This prevents one noisy frame from defining the entire transfer motion.

I represented transfer motion in robot-centered cylindrical coordinates, making the motion easier to relate to the tabletop workspace. The arm lifts the cup, retracts toward the base, rotates, extends toward the destination, lowers, and releases. Retracting before rotation reduces the arm’s swept radius compared with rotating at full extension. Joint limits, reach margins, velocity limits, and acceleration limits are applied before commands are sent to the hardware.

For the button-pressing task, I used ArUco marker geometry rather than memorized joint positions. The system repeatedly observes the marker, calculates the gripper’s positional error, solves for a bounded correction, and moves incrementally until it reaches the pressing position.

I also developed a retargetable kinesthetic-teaching workflow. Instead of replaying a recorded joint trajectory exactly, the system separates the demonstrated motion shape from its original start and end positions. This allows the movement to be adapted to new endpoints while preserving useful characteristics of the original demonstration.

To reduce the cost and risk of hardware iteration, I created an RViz dry-run workflow and separated individual skills into independent launch configurations. I also isolated much of the decision logic from ROS communication so it could be tested as deterministic Python.

Result

Ninibot successfully implements the complete task sequence:

Scan and lock the placement marker.
Search for and visually align with a cup.
Grasp and lift the cup.
Retract, rotate, extend, and release it at the destination.
Find and press a second marked target.

The repository contains five ROS 2 packages, three selectable perception backends, two ROS actions, one inverse-kinematics service, and 25 focused test modules containing 112 test functions. These tests cover controller transitions, target locking, search recovery, kinematics, transfer planning, marker tracking, and constrained motion.

The result is an active research prototype with a modular foundation for continued development. Experimental extensions—including structured imitation learning, automated episode collection, and audio-based water-flow detection—remain separated from the primary rule-based hardware controller until they can be validated more thoroughly.

What I Learned

Ninibot strengthened my understanding of complete robotic-system integration, from perception and coordinate transforms to motion planning, motor commands, testing, and physical debugging.

Most importantly, I learned that reliable robotics depends on explicitly designing for uncertainty and failure. Smoothing, recovery behavior, bounded corrections, staged motion, modular architecture, and dry-run validation were just as important as the core detection and kinematics algorithms.

Ninibot — Vision-Guided Robotic Manipulation