Thoughts on RT-2

Any thoughts on this from deepmind?

How difficult would it be to get this running on a stretch?

1 Like

It’s really impressive work! I’d love to see more language or language + behavior models used to make planning for/interacting with Stretch better. With RT-1 and SayCan, they published checkpoints, their code, and their dataset. I imagine these would serve as a good starting point if you wanted to reproduce their work. AFAIK, they haven’t released the same with RT-2 yet.

@cpaxton has done a bit of this, so he might have some advice on how to get started. He’s one of the authors of the HomeRobot, a library that makes creating/running agents on Stretch easier.

1 Like

RT-1/2 are predicting end effector positions so in principle it could work, I think the big obstacle will be whether there is enough Stretch data to fully adapt to running it on the robot.

Would be interesting to look into though!