Full Website found here

Overview

My team designed a robotic system which implements visual tracking to mirror a user’s hand and finger movements on a Sawyer arm and custom humanoid hand end effector. I designed the custom robotic hand to be integrated with a robotic arm and custom electronics, assisted with PCB design and assembly, as well as code integration.

Background

Humanoid robotics is a growing field of research and startups which leverage the adaptability of the human body for a wide variety of applications. Given the large corpus of data concerning human activities (videos, images, live sensing, etc.), robots can solve human-level tasks faster through imitation of humans. However, there’s a hole in the data concerning the force applied by human actions (Eg: how much force is needed to push a button? What is the compensation by the arm when using a certain tool?). Our goal is to augment the existing tools to map human motion with a measurement device for applied force to an object in the grasp of a human-like hand for use in pose estimation (and eventually extend to imitation learning more broadly).

Goals

Our initial ambitious goal was to create a glove worn by a human user with strain gauges and pressure sensors on the fingertips. The hand movements in space, finger movements, and forces would then be replicated on a custom robotic hand on a Sawyer robot. The force sensing goal was removed from the scope of the project due to integration time constraints. The force sensing electronics and software worked, but integration onto the custom hand was not feasible in our timeline.

Mechanical Design

Hand Base and Palm

The main structure of the hand is manufactured using 3D-printed PLA. Holes route tension lines through the hand and to the phalanxes. All critical hardware and electronics for the hand, including the PCB and servos are mounted to this part using bolts and spacers. This part is designed to be structurally strong and mount securely to the Sawyer. The hand must be able to hold up to repeated testing without requiring extensive remanufacturing and reassembly. The fingers are mounted to secure slots next to angled faces to facilitate proper bending angle of each finger. Holes in the back line up with the PCB holes for servo wire routing.

Phalanxes and Tension Lines

Each finger is made up of 3 different sized phalanxes with tension line routing holes. Similar to the hand base, each phalanx has a 2 mounting slots for joints and angled faces for the ideal bending angle of each phalanx. When the tension lines are correctly routed through the phalanxes, shortening the length of the tension line through the phalanxes causes a motion similar to that of a human finger in flexion. The tension lines are securely tied to the final phalanx of each finger to keep tension.

Finger joints

Made of TPU (85A Shore Hardness) for the optimal balance of flexibility and elasticity and finger flexion behavior. Geometry clearances to securely fit inside of phalanx slots without any slipping.

Micro Servos

Five SG90 micro servos actuate the fingers using the tension line system. The tension lines are secured to the end of servo horns mounted to each micro servo output. 9G servo motors are used to meet the torque requirement for basic hand movements while being small enough to fit inside of the hand palm.

Custom PCB

Mounted on the back of the hand for servo power distribution, sensing, microcontroller, and communication. Regulates 12V power input to 5V, 1.5A output to the servos. Unfortunately, due to a small mistake in the circuit design, the microcontroller was unusable, so the circuit ended up only being used for power regulation. Sensing worked, but time limitations unfortunately meant it wasn't implemented in time. The PCB is mounted to the hand using bolts and spacers to ensure secure, stable mounting and leaving room for solder and wires underneath the board.

Design Limitations

While simplifications made to the hand reduces complexity and increases reliability, it prevents a fully accurate representation of a human hand's range of motion. This limits the robotic hand’s applicability in tasks requiring nuanced thumb movements or fine motor skills, such as pinching or precision grips.

Although the hand can perform basic tasks, such as gripping a solo cup, it struggles with more intricate or variable object shapes and sizes. In real-world engineering applications, this would limit its utility in environments requiring fine manipulation.

Results

The mechanical hand worked well, successfully actuating each finger to their desired flexion. The robot can consistently follow the user's hand position and mimic its rotation while estimating the user's joint angles.

Conclusion/Discussion

The robotic hand successfully performed basic gripping tasks, such as holding a solo cup or objects with similar large diameters. However, the limited servo torque and DOF in the current robotic hand meant that gripping smaller and heavier objects was impossible.

The robot effectively mirrors user's movements onto the 3D plane, but there's also latency in movement to new positions. On occasions, with an outwardly simple movement to a new position, the Inverse Kinematics solution would lead the Sawyer Arm to rotate its linkages and take its time to reach the new position.

Future Improvements

The next stage of the project is to integrate a single camera system capable of both hand orientation and location. We plan to also implement depth detection for mirroring movements in a 3D space, instead of a 2D plane. To accomplish that, however, we'd need to use a different camera, possibly with depth but with better resolution.

We also aim to integrate pressure sensors and strain gauges into the robotic hand's fingers to improve robotic force control, gather gripping data applicable to research and accurately imitate the user's force in grabbing objects. This addition will allow the robotic hand to adjust its gripping force, allowing for effective manipulation of fragile objects (such as an egg).

We would like to optimize the Inverse Kinematics solver and improve the camera, so that the lag between detecting movement and moving the arm can be fixed while possibly fixing the issue of certain poses taking extremely long routes to reach.

Finally, increasing the degrees of freedom in the fingers and thumb can improve future robotic hands. By adding additional actuation points and types of joints, the robotic hand can achieve more human motion. The improved dexterity can mean improvements to handling and gripping tasks.

Final Thoughts

If given further time and resources, the project could become a highly functional robotic hand applicable in assistive technologies, research, or manufacturing. The project was originally inspired by Kind Humanoid Robotics, for precision applications of mimicking and robotic hand movements and grasp functionalities. Combining the hardware and software upgrades would significantly improve its precision, adaptability, and real-world applications.

Robot Humanoid Hand Mirroring