I saw that the Stretch repo has a few demo open-source deep learning models in it.
I was curious to know how can we add more models (own or other open source models) so that they work with Stretch?
Hi @rishi, there’s a couple tools we’re using to work with deep learning models on Stretch. We use OpenCV’s DNN module to load the models in the perception models repo and to perform inference on sensor input. We’ve created some scaffolding and visualization tools to work with these models, available in the stretch_deep_perception ROS package. This allows us to use ROS to pass the sensor data (e.g. images from Stretch’s head) to the deep perception models.
You can use this strategy with other deep learning ecosystems to work with deep learning models on Stretch. For example, you can find models on websites like ModelZoo, and create a PyTorch, Keras, etc. ROS wrapper in Python.
Also, keep in mind that the ROS ecosystem often has deep learning models wrapped and ready to use. For example, if you wanted to use Mask R-CNN, you could search for ROS packages on Github and come across many.