Does anyone know of previous instances of mounting a screen/tablet on the stretch, and interfacing through that? Not just having the tablet be mounted on the stretch - I know there’s already a tool for that. I more mean connect and interface with the tablet through ROS[2].
My goal for a demo is to have the Stretch be able to assist students in various web tasks around the lab, such as creating an account with the lab, showing different resources available, or just diagrams in general.
I have most of the demo set up - entity detection + aruco tags for navigating, verbal commands and prompts, and docking and starting routine automated. Just need to nail the human interaction piece.
Hey @byrnete, I know of two projects that might be relevant:
@Atharva_Pusalkar and his capstone team at CMU mounted a tablet on Stretch’s head and developed a video conferencing app for it. He may be able to speak to how the video was chauffeured off the robot and to the tablet.
@hello-vinitha created Stretch Web Teleop, which has elements that are useful for creating a web interface, such as a React based GUI library and a frontend ROS2 bridge that can talk with ROS2 topics/services/actions through WebRTC and rosbridge. These modules aren’t well documented at the moment however. Also worth noting that @vynguyen91, @hello-lamsey, @hello-amal, and @hello-vinitha are exploring video conferencing interfaces on Stretch’s tablet tool, and the ergonomics of how a tablet should be presented to a user.
Thanks for the help! I knew of Stretch Web Teleop, but hadn’t know it had a frontend ROS2 bridge, I’ll definitely look into that further as that sounds perfect.
The ergonomics topic also sounds like it’d be a big help too, so I’ll see what I can find there too.
Perhaps this forum post, highlighting new features in Stretch Web Teleop, could be helpful? It also talks about features for a tablet mount, and we expect to release a more detailed guide on mounting a tablet shortly.