I recently started using Stretch 2 with the home-robot library. I’m testing out their navigation policies on carpeted surfaces at our university. Stretch 2 works smoothly, but the base gets stuck occasionally. For example, when we execute a turn left/right by 30 degrees motion on stretch, it remains in place for a long time (sometimes several minutes) before responding. During that period, the base makes a lot of noise. I’m presuming this is from one of the wheel motors. @cpaxton has also encountered this issue based on my conversations with him.
Resetting the robot helps. I also found that slightly tilting it and letting go quickly resolves the issue. Is there a better solution to this issue?
Hi Santhosh, thanks for the post to the forum. I haven’t seen this issue personally but my first guess is that the stepper controller may be current limited for some reason (possibly due to the carpet).
To understand what’s going on, let’s try out the new Trace function:
(This function is still fairly new so let us know if you encounter any issues).
First enable the Trace per the Knowledge Base instructions. Run your code and when the issue occurs, go ahead and kill your process that has instantiated the Robot class. This will stop the trace logging. Next, go ahead and zip up the data in the trace directory and send it to us at support@hello-robot.com. We can sift through it to better understand what may be going on.
Thanks for the suggestion. I’ll enable the trace and send logs when I encounter this issue next. To be 100% clear, is this the file you’re suggesting I modify?
@aedsinger - I was not able to reproduce the several minutes-long delay. But here are some steps where it takes 4-5x longer to execute the action (25-30 secs instead of 5 secs). I’ve attached all the logs I collected in the past 1 hour of testing (see here).
In the trace we can see places where the motor position is reporting as zero when it likely shouldn’t. In order to dig in further we need to trace additional data. Can you run the code below and send the trace back? Ctrl-C out of the program when the issue occurs so we can ensure it is captured in the trace. Also please modify the script as needed such that it replicates the issue.
I’ve not been able to reproduce this issue using the above script (or even a modified version which includes 0.25m forward + backward translation). I wonder if this could be related to the home-robot library that I’m using. How can I add this tracing code to my existing experiment script to collect data?
The home-robot library seems to communicate with Stretch through ROS (my guess based on this launch file), so you could add the tracing code directly to the node that runs Stretch’s ROS drivers: stretch_driver. By default, this node lives at:
Thanks for your patience and help @bshah and @aedsinger. I’ve not been able to reproduce this issue over the past couple of days. I guess this is both good and bad. In case this issue reoccurs, I’ll share the latest logs with you. Thanks.
This issue might have reoccured. I didn’t notice any sounds from the base this time, but the robot took too long to execute actions (60 secs for action 1 — 10x longer than usual, and it didn’t respond for over 2 minutes for action 2 — I terminated the program at this point). Here are the ros launch messages:
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 103.236783 above maximum frequency of 80.00 Hz. Motor commands dropped: 39
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 82.690377 above maximum frequency of 80.00 Hz. Motor commands dropped: 41
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 88.158228 above maximum frequency of 80.00 Hz. Motor commands dropped: 42
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 82.011302 above maximum frequency of 80.00 Hz. Motor commands dropped: 44
[INFO] [1680739632.382521]: /stretch_driver: Changed to mode = navigation
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 90.624951 above maximum frequency of 80.00 Hz. Motor commands dropped: 47
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 99.379315 above maximum frequency of 80.00 Hz. Motor commands dropped: 48
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 80.560541 above maximum frequency of 80.00 Hz. Motor commands dropped: 51
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 86.242217 above maximum frequency of 80.00 Hz. Motor commands dropped: 55
The trace messages are attached here. The most recent ones are the relevant logs. I hope this was it.
Thanks, @aedsinger ! I’ll send this when I start working on the robot today. Btw, the issue was resolved after turning off the robot, recharging it for a few hours, and turning it on again. Should i run the diagnostics only when the issue reoccurs, or can I run it even under normal operation conditions?
Thank you for the trace and diagnostics data. It appears that the issue may be related to the Pimu trigger_motor_sync issue that we are debugging on this thread.
One thing you can try is disabling the motor sync functionality temporarily. This will cause the wheels to move slightly out of synchronization (and therefore introduce odometery errors). However, if you no longer see the issues then it will confirm that this is indeed the root cause.
To turn it off, try adding to your ~/stretch_user/stretch_re2_xxx/stretch_user_params.yaml:
Also, we’d love to recreate your issue on our robots (using the home_robot stack I believe). If you have a minute can you connect with us directly at support@hello-robot.com so we can correspond on how to do this.
Thanks @aedsinger . Setting the above did not completely resolve the issue. The robot sometimes takes forever to execute a single action. I’ll get in touch at support@hello-robot.com to reproduce this issue.