Pickup and Handover Object to Person Using ArUco Markers

Hi all!

We recently created a demo showcasing Stretch autonomously picking up a hat off the floor and handing it over to someone. We call the system “SIRA” or socially interactive robot assistant, and built it on top of FUNMAP, ArUco marker detection, and the autonomy demos.

I am currently working at the INSPIRE Lab at the Spaulding Rehabilitation Hospital in Cambridge. Our lab is motivated to increase functional independence for those with high level spinal cord injuries. We have recently teamed with the Travis Roy Foundation and started a center for independence for those living with paralysis. We believe autonomous robots, like Hello Robot’s Stretch model, could be a key part of regaining independence and increasing quality of life. For example, someone who is paralyzed may have trouble picking things up from the ground, or they may rely on somebody else to accomplish this task for them. However, SIRA is able to pick an object up from the ground and hand it over to the person, eliminating the need for a caregiver and giving them a sense of independence they had previously lost.

This demo was built using FUNMAP, and the ArUco marker detection. SIRA uses two ArUco Markers, one on the object to be picked up (in our case the hat) and one on the person that the object is meant to be handed to (in our case Julia). SIRA localizes to its environment, then looks for the hat ArUco marker. Once this is found, SIRA may pick the hat up from the ground. Then SIRA looks for the other marker showing where the person is. SIRA is then able to locate the handover position, and hand the hat over to the person wearing the ArUco marker. For further instructions to try the demo out yourself, refer to the the README on the github page for SIRA (linked below).

I would like to thank Binit Shah for all of his help getting this demo up and running. Additionally, I would like to thank the rest of the Hello Robot team for their continued support throughout this project.

See it in action on YouTube: SIRA Hat Pickup Demo - YouTube

The code is open source on Github: GitHub - jcgangemi1/sira

For further instructions on how to use, refer to the README: sira/README.md at main · jcgangemi1/sira (github.com)


Julia Gangemi


Exciting to see, thanks for sharing your code as well with the community!

1 Like