I am consistently getting the following warning and was wondering what it means and if it needs to be fixed:
“Warning: Rate of calls to Pimu:trigger_motor_sync above maximum frequency of 20.00 Hz. Motor commands dropped: 1”.
The number of motor commands dropped increases over time. I’m confused why I’m getting this warning as I am only pushing commands to the robot at 10 Hz.
Hi @akhilp, good question! First, here’s an explanation of the warning. The power + imu (pimu) board in the robot issues motor sync pulses to each joint’s motor controller, which keeps the individual joints moving synchronously with each other. Therefore, a sync pulse is sent out for each robot.push_command() call from Stretch Body. The Pimu throttles the rate at which sync pulses can be issued to 20hz, so the extra pulses aren’t sent and a warning is issued every time a user script calls robot.push_command() faster than this rate.
In your case, it sounds like something is going wrong since you’re pushing commands at 10hz and still seeing the warning. I can look into the issue for you if you can add Python code that serves as a minimal working example of the issue. Let me know if you have any questions.
I am also seeing this issue using the stretch driver in navigation mode.
Thanks @nwright, I’ll look into stretch driver’s nav mode and see if I can replicate the issue.
Hello, I am having a similar issue Warning: Rate of calls to Pimu:trigger_motor_sync above maximum frequency of 80.00 Hz. Motor commands dropped: 866” when I run anything (including a calibration)
What should I do to fix this?
I believe this is a bug on our end (presuming you aren’t actually sending motor commands faster than 80Hz). We’ll figure out a fix ASAP and let you know. Can you tell us a bit more:
REx_firmware_updater.py --current?More soon!
I have a Stretch 2 model and I am commanding through Stretch ROS. This is the firmware I am running:
Hello. We have released Stretch Body v0.4.15 which should fix this issue. Please install it with:
pip3 install -U --no-cache-dir hello-robot-stretch-body
Details on the fix can be found with the following PR.
Let us know how it goes. Thanks!
This fixed the issue! Thank you so much!!
Hello!
I am also experiencing this issue even after upgrading hello-robot-stretch-body to 0.4.30.
I am using a Stretch RE1 and I’m commanding it by having stretch_driver (from stretch_ros) in navigation mode while publishing to /cmd_vel
Even without publishing anything I am constantly getting the error mentioned in this thread. Publishing to /cmd_vel at 15Hz yields the exact same situation.
Here’s an example of the warnings I’ve been getting
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 45.096164 above maximum frequency of 20.00 Hz. Motor commands dropped: 78
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 49.509591 above maximum frequency of 20.00 Hz. Motor commands dropped: 134
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 48.065641 above maximum frequency of 20.00 Hz. Motor commands dropped: 201
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 111.089734 above maximum frequency of 20.00 Hz. Motor commands dropped: 270
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 36.877865 above maximum frequency of 20.00 Hz. Motor commands dropped: 338
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 32.484773 above maximum frequency of 20.00 Hz. Motor commands dropped: 413
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 50.301668 above maximum frequency of 20.00 Hz. Motor commands dropped: 478
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 40.280270 above maximum frequency of 20.00 Hz. Motor commands dropped: 549
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 25.266587 above maximum frequency of 20.00 Hz. Motor commands dropped: 623
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 94.430151 above maximum frequency of 20.00 Hz. Motor commands dropped: 688
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 30.720525 above maximum frequency of 20.00 Hz. Motor commands dropped: 744
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 52.240083 above maximum frequency of 20.00 Hz. Motor commands dropped: 814
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 143.630710 above maximum frequency of 20.00 Hz. Motor commands dropped: 886
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 77.267358 above maximum frequency of 20.00 Hz. Motor commands dropped: 958
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 20.760901 above maximum frequency of 20.00 Hz. Motor commands dropped: 1018
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 102.894880 above maximum frequency of 20.00 Hz. Motor commands dropped: 1068
Warning: Rate of calls to Pimu:trigger_motor_sync rate of 22.935987 above maximum frequency of 20.00 Hz. Motor commands dropped: 1122
Any help is appreciated, thanks!
Hi Austin, we’ll dig in on this today and get back to you shortly.
Until then, can you run stretch_diagnostic_check.py --simple and then stretch_diagnostic_check.py --zip and then email support the zip file?
More details on Stretch Diagnostics are here.
Thanks!
We have a draft PR that should fix your issue:
We still have more testing to do before pushing it to stretch_ros master. Until then, feel free to try it out:
cd ~/catkin_ws/src/stretch_ros
git checkout bugfix/push_command_sync
Let us know how it goes. We’ll post back here when the fix is on master.
Thanks for the prompt fix. However, I am still getting the same warnings after checking out bugfix/push_command_sync and rebuilding stretch_ros. For reference I am on stretch-body 0.4.30
Also, is there a way to have the main loop run faster than 15hz? According to the params in stretch_body the maximum pimu update rate for is 20hz for Stretch RE1 and 80hz for Stretch 2. It would be ideal if it was possible to perform velocity control at higher frequencies, since that is what we are mainly using for navigation.
That’s surprising. Do the errors persist if the robot is not in navigation mode?
No, in position mode I don’t see the warnings at all.
Thanks. We’ve been able to replicate the issue on one of our RE1 robots (but not RE2) – so we’re working on a fix.
With respect to the control rates: In theory the hardware can support higher control rates. Currently the system design limits the rates to about 15hz. We’re looking at ways to optimize the system design to allow navigation velocity control that approaches the theoretical limits. We’ll report back any progress on the Forum.
I just checked out and tested the draft PR on my Stretch 2, can confirm that the warnings no longer shows up there. The warnings are only persistent on my Stretch RE1.
Thanks for looking into this, and also for the update on higher control rates!
Also ideally what control rates would you like to command the base velocity at?
Higher is always better :D, but given how the maximum velocity of the robot is 30cm/sec and that it would be ideal to have sub-centimeter accuracy for navigation, it would be nice to be able to control it at at least 30hz.
Of course, I know that there are limits on the acceleration, and in fact we also have our own implementation of a trapezoidal velocity profile for velocity control, but 30hz would be a good rough estimate of an upper bound where gains would be minimal once we go above that.
Thanks!