Strange sounds from wheel motors and slow action responses

[srama2512]

Hi,

I recently started using Stretch 2 with the home-robot library. I’m testing out their navigation policies on carpeted surfaces at our university. Stretch 2 works smoothly, but the base gets stuck occasionally. For example, when we execute a turn left/right by 30 degrees motion on stretch, it remains in place for a long time (sometimes several minutes) before responding. During that period, the base makes a lot of noise. I’m presuming this is from one of the wheel motors. @cpaxton has also encountered this issue based on my conversations with him.

Resetting the robot helps. I also found that slightly tilting it and letting go quickly resolves the issue. Is there a better solution to this issue?

[aedsinger]

Hi Santhosh, thanks for the post to the forum. I haven’t seen this issue personally but my first guess is that the stepper controller may be current limited for some reason (possibly due to the carpet).

To understand what’s going on, let’s try out the new Trace function:

Using the Robot Trace tool to debug Stretch Body

Starting with Stretch Body 0.4.26 and Stretch Factory v0.3.15 we have introduced a new debugging tool for Stretch Body called Robot Trace. Note: This tool is recommended for advanced users only The Robot Trace tool periodically logs Stretch Body status data to disk in the background. This data can then be inspected and plotted offline. The Trace tool can be useful to capture and analyze events when debugging transient issues and behavior with the robot. For example, if the robot powers down…

(This function is still fairly new so let us know if you encounter any issues).

First enable the Trace per the Knowledge Base instructions. Run your code and when the issue occurs, go ahead and kill your process that has instantiated the Robot class. This will stop the trace logging. Next, go ahead and zip up the data in the trace directory and send it to us at support@hello-robot.com. We can sift through it to better understand what may be going on.

Thanks!
–Aaron

[srama2512]

Hi @aedsinger ,

Thanks for the suggestion. I’ll enable the trace and send logs when I encounter this issue next. To be 100% clear, is this the file you’re suggesting I modify?

/home/srama/stretch_user/stretch-re2-2044/stretch_user_params.yaml

[aedsinger]

Yes, that is the one!

[srama2512]

@aedsinger - I was not able to reproduce the several minutes-long delay. But here are some steps where it takes 4-5x longer to execute the action (25-30 secs instead of 5 secs). I’ve attached all the logs I collected in the past 1 hour of testing (see here).

[aedsinger]

Thanks. We’ll dig in and get back to you asap.

[aedsinger]

In the trace we can see places where the motor position is reporting as zero when it likely shouldn’t. In order to dig in further we need to trace additional data. Can you run the code below and send the trace back? Ctrl-C out of the program when the issue occurs so we can ensure it is captured in the trace. Also please modify the script as needed such that it replicates the issue.

#!/usr/bin/env python3

import stretch_body.robot
from stretch_body.hello_utils import *
import time



def base_trace_cb(robot, data):
    data['base.right_wheel.mode'] = robot.base.right_wheel.status['mode']
    data['base.right_wheel.current'] = robot.base.right_wheel.status['current']
    data['base.right_wheel.err'] = robot.base.right_wheel.status['err']
    data['base.right_wheel.diag'] = robot.base.right_wheel.status['diag']
    data['base.right_wheel.runstop_on'] = robot.base.right_wheel.status['runstop_on']
    data['base.right_wheel.in_guarded_event'] = robot.base.right_wheel.status['in_guarded_event']
    data['base.right_wheel.in_safety_event'] = robot.base.right_wheel.status['in_safety_event']
    data['base.right_wheel.is_moving'] = robot.base.right_wheel.status['is_moving']
    data['base.right_wheel.is_mg_moving'] = robot.base.right_wheel.status['is_mg_moving']
    data['base.left_wheel.mode'] = robot.base.left_wheel.status['mode']
    data['base.left_wheel.current'] = robot.base.left_wheel.status['current']
    data['base.left_wheel.err'] = robot.base.left_wheel.status['err']
    data['base.left_wheel.diag'] = robot.base.left_wheel.status['diag']
    data['base.left_wheel.runstop_on'] = robot.base.left_wheel.status['runstop_on']
    data['base.left_wheel.in_guarded_event'] = robot.base.left_wheel.status['in_guarded_event']
    data['base.left_wheel.in_safety_event'] = robot.base.left_wheel.status['in_safety_event']
    data['base.left_wheel.is_moving'] = robot.base.left_wheel.status['is_moving']
    data['base.left_wheel.is_mg_moving'] = robot.base.left_wheel.status['is_mg_moving']
    return data

robot = stretch_body.robot.Robot()
robot.trace.add_trace_callback(base_trace_cb)
robot.startup()

for i in range(100):
    print('Itr',i)
    robot.base.rotate_by(x_r=deg_to_rad(30.0))
    robot.push_command()
    time.sleep(3.0)
    robot.base.rotate_by(x_r=deg_to_rad(-30.0))
    robot.push_command()
    time.sleep(3.0)
robot.stop()

[srama2512]

Thanks @aedsinger . Working on replicating this issue. I will share the trace asap.

[srama2512]

I’ve not been able to reproduce this issue using the above script (or even a modified version which includes 0.25m forward + backward translation). I wonder if this could be related to the home-robot library that I’m using. How can I add this tracing code to my existing experiment script to collect data?

[bshah]

The home-robot library seems to communicate with Stretch through ROS (my guess based on this launch file), so you could add the tracing code directly to the node that runs Stretch’s ROS drivers: stretch_driver. By default, this node lives at:

/home/hello-robot/catkin_ws/src/stretch_ros/stretch_core/nodes/stretch_driver

Here’s what the diff looked like for me after adding the tracing code to stretch_driver:

diff --git a/stretch_core/nodes/stretch_driver b/stretch_core/nodes/stretch_driver
index 14bdc26..e70c002 100755
--- a/stretch_core/nodes/stretch_driver
+++ b/stretch_core/nodes/stretch_driver
@@ -32,6 +32,28 @@ from joint_trajectory_server import JointTrajectoryAction
 from stretch_diagnostics import StretchDiagnostics
 
 
+def base_trace_cb(robot, data):
+    data['base.right_wheel.mode'] = robot.base.right_wheel.status['mode']
+    data['base.right_wheel.current'] = robot.base.right_wheel.status['current']
+    data['base.right_wheel.err'] = robot.base.right_wheel.status['err']
+    data['base.right_wheel.diag'] = robot.base.right_wheel.status['diag']
+    data['base.right_wheel.runstop_on'] = robot.base.right_wheel.status['runstop_on']
+    data['base.right_wheel.in_guarded_event'] = robot.base.right_wheel.status['in_guarded_event']
+    data['base.right_wheel.in_safety_event'] = robot.base.right_wheel.status['in_safety_event']
+    data['base.right_wheel.is_moving'] = robot.base.right_wheel.status['is_moving']
+    data['base.right_wheel.is_mg_moving'] = robot.base.right_wheel.status['is_mg_moving']
+    data['base.left_wheel.mode'] = robot.base.left_wheel.status['mode']
+    data['base.left_wheel.current'] = robot.base.left_wheel.status['current']
+    data['base.left_wheel.err'] = robot.base.left_wheel.status['err']
+    data['base.left_wheel.diag'] = robot.base.left_wheel.status['diag']
+    data['base.left_wheel.runstop_on'] = robot.base.left_wheel.status['runstop_on']
+    data['base.left_wheel.in_guarded_event'] = robot.base.left_wheel.status['in_guarded_event']
+    data['base.left_wheel.in_safety_event'] = robot.base.left_wheel.status['in_safety_event']
+    data['base.left_wheel.is_moving'] = robot.base.left_wheel.status['is_moving']
+    data['base.left_wheel.is_mg_moving'] = robot.base.left_wheel.status['is_mg_moving']
+    return data
+
+
 class StretchDriverNode:
 
     def __init__(self):
@@ -410,6 +432,7 @@ class StretchDriverNode:
             rospy.signal_shutdown('Found old stretch_body version.')
 
         self.robot = rb.Robot()
+        self.robot.trace.add_trace_callback(base_trace_cb)
         self.robot.startup()
 
         mode = rospy.get_param('~mode', "position")

[srama2512]

Thanks for your patience and help @bshah and @aedsinger. I’ve not been able to reproduce this issue over the past couple of days. I guess this is both good and bad. In case this issue reoccurs, I’ll share the latest logs with you. Thanks.

[aedsinger]

Ok, do let us know if it reoccurs. We’d love to squash this bug!

[srama2512]

This issue might have reoccured. I didn’t notice any sounds from the base this time, but the robot took too long to execute actions (60 secs for action 1 — 10x longer than usual, and it didn’t respond for over 2 minutes for action 2 — I terminated the program at this point). Here are the ros launch messages:

Warning: Rate of calls to Pimu:trigger_motor_sync rate of 103.236783 above maximum frequency of 80.00 Hz. Motor commands dropped: 39
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 82.690377 above maximum frequency of 80.00 Hz. Motor commands dropped: 41
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 88.158228 above maximum frequency of 80.00 Hz. Motor commands dropped: 42
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 82.011302 above maximum frequency of 80.00 Hz. Motor commands dropped: 44
[INFO] [1680739632.382521]: /stretch_driver: Changed to mode = navigation
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 90.624951 above maximum frequency of 80.00 Hz. Motor commands dropped: 47
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 99.379315 above maximum frequency of 80.00 Hz. Motor commands dropped: 48
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 80.560541 above maximum frequency of 80.00 Hz. Motor commands dropped: 51
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 86.242217 above maximum frequency of 80.00 Hz. Motor commands dropped: 55

The trace messages are attached here. The most recent ones are the relevant logs. I hope this was it.

[aedsinger]

Thanks for the update. The issue may be related to the ‘Pimu:trigger_motor_sync’ errors (an issue we are working with another user right now).

Can you run the Stretch Diagnostics with --simple and send us the zip as described in this Knowledge Base post:

Using the Stretch Diagnostics tool to debug an issue

We have introduced a new tool called Stretch Diagnostics to assist when debugging an issue. The tool runs a suite of tests to check for correct device configurations, sensor values, software package versions, etc. It isn’t yet comprehensive but can serve as a first step when working with Hello Robot support to diagnose an issue. To install the tool run: pip3 install hello-robot-stretch-diagnostics The stretch_diagnostic_check.py tool has numerous options. To start, we recommend running: str…

[srama2512]

Thanks, @aedsinger ! I’ll send this when I start working on the robot today. Btw, the issue was resolved after turning off the robot, recharging it for a few hours, and turning it on again. Should i run the diagnostics only when the issue reoccurs, or can I run it even under normal operation conditions?

[aedsinger]

Just run the diagnostics anytime. It provides us with basic system configuration information.

Let me know if the issue reoccurs and if it is indeed correlated with battery charge (which is unexpected!).

Thanks!

[srama2512]

Thanks for the guidance @aedsinger . Here are the log files as instructed in the above post.

[aedsinger]

Thank you for the trace and diagnostics data. It appears that the issue may be related to the Pimu trigger_motor_sync issue that we are debugging on this thread.

One thing you can try is disabling the motor sync functionality temporarily. This will cause the wheels to move slightly out of synchronization (and therefore introduce odometery errors). However, if you no longer see the issues then it will confirm that this is indeed the root cause.

To turn it off, try adding to your ~/stretch_user/stretch_re2_xxx/stretch_user_params.yaml:

hello-motor-right-wheel:
  gains:
    enable_sync_mode: 0
hello-motor-left-wheel:
  gains:
    enable_sync_mode: 0

We’ll keep you posted when we resolve the underlying Pimu trigger_motor_sync issue.

[aedsinger]

Also, we’d love to recreate your issue on our robots (using the home_robot stack I believe). If you have a minute can you connect with us directly at support@hello-robot.com so we can correspond on how to do this.

Thanks!

[srama2512]

Thanks @aedsinger . Setting the above did not completely resolve the issue. The robot sometimes takes forever to execute a single action. I’ll get in touch at support@hello-robot.com to reproduce this issue.