Hi all,
We have found that a recent software update for the RealSense cameras has caused Stretch’s head camera, a RealSense D435i, to have degraded performance. This update causes the point clouds returned from the RealSense ROS drivers to be sparse or entirely empty. Sensor readings from the RealSense Python library, pyrealsense2, are not affected.
To mitigate the issue, all newly shipped robots have a temporary fix that rolls back the RealSense software to an older version. We are investigating the issue further and will publish an update to this post once we’ve found a permanent fix. Until then, we recommend users continue using APT to update packages (e.g. using sudo apt upgrade), but use the fix described below to pin RealSense packages to an older version.
For Stretch users who already have the latest software update, this post explains how to identify whether your robot’s head camera is experiencing degraded performance and how to apply the temporary fix.
Identifying degraded performance
From ROS Melodic or Noetic, while running any of the ROS launch files that depend on the d435i_high_accuracy launch file (e.g. roslaunch stretch_calibration collect_head_calibration_data.launch), look for any of the following warnings/errors:
[ WARN] [1629233696.134929301]: Param '/camera/stereo_module/auto_exposure_limit' has value 0 that is not in range [1, 165000]. Removing this parameter from dynamic reconfigure options.
[ WARN] [1629233696.146681406]: Param '/camera/stereo_module/auto_gain_limit' has value 0 that is not in range [16, 248]. Removing this parameter from dynamic reconfigure options.
[ WARN] [1629233706.492064090]: Hardware Notification:Depth stream start failure,1.62923e+12,Error,Hardware Error
[ WARN] [1629234766.232692638]: frame's time domain is HARDWARE_CLOCK. Timestamps may reset periodically.
WARNING: There are too few points from the depth image for plane fitting, so only using the RGB ArUco estimate. number of points = 0
[ERROR] [1629240042.350686635]: Exception thrown while processing service call: get_xu(id=2) failed! Last Error: Resource temporarily unavailable
Traceback (most recent call last):
File "/home/hello-robot/catkin_ws/src/stretch_ros/stretch_core/nodes/d435i_configure", line 79, in <module>
node.main()
File "/home/hello-robot/catkin_ws/src/stretch_ros/stretch_core/nodes/d435i_configure", line 63, in main
self.turn_on_high_accuracy_mode()
File "/home/hello-robot/catkin_ws/src/stretch_ros/stretch_core/nodes/d435i_configure", line 27, in turn_on_high_accuracy_mode
self.parameter_client.update_configuration({'visual_preset' : self.locked_mode_name})
File "/opt/ros/melodic/lib/python2.7/dist-packages/dynamic_reconfigure/client.py", line 224, in update_configuration
raise DynamicReconfigureCallbackException('service call failed')
dynamic_reconfigure.DynamicReconfigureCallbackException: service call failed
If any, some, or all of these warnings/errors appear, then proceed with applying the fix described below.
Applying temporary downgrade
The downgrade is available as a bash script that can be run from the terminal. First, clone the Stretch Install repo:
cd ~
git clone https://github.com/hello-robot/stretch_install.git
Next, run the “realsense_downgrade_patch.sh” script:
cd ~/stretch_install
./realsense_downgrade_patch.sh
Finally, restart the robot using hard restart procedure and then use the RealSense viewer to update firmware (the GUI will pop up an option to install the latest firmware):
realsense-viewer
When running ROS launch files that rely on the RealSense, you should no longer see any errors. Stay tuned for updates!