The Realsense D435i is the RGB-D camera found in the Stretch’s head pan-tilt module and it functions as the primary perception sensor for the robot. When the Realsense D435i camera is connected to a Linux computer via USB, the operating system will recognize it as a USB device. The camera uses a standard USB video class (UVC) driver which will create video device files (i.e. /dev/video*) that can be accessed by any applications easily on the computer to receive video frames. An application can use a software library called Librealsense, which provides a high-level interface for working with the camera’s data. For example, the realsense-ros nodes use the Librealsense library to access the realsense camera’s various image streams (RGB, Depth, IR) and create a ROS interface.
On a few occasions, some users might have faced some stability issues with the realsense.
These issues include:
- Camera getting disconnected or lost from the USB bus frequently while running or starting ROS applications. This can be rectified by a power cycle of the robot to get back the camera.
- Realsense ROS node errors out with
xioctl(...)exceptions - Multiple warning messages thrown out from the realsense drivers (e.g.
Hardware Notification: Depth stream start failure, etc)- Generally warning messages can be ignored safely unless and until the camera stream isn’t available
It’s not uncommon to experience stability issues with Realsense cameras at certain times, which may be caused by a variety of factors such as Linux Kernel level resource constraints, incompatible drivers and in rare events it could be a bad USB cables or the camera itself. Identifying the exact cause of a stability issue is challenging. However, we have compiled some methods that you can try to improve the stability of your Realsense camera in Stretch robots.
Using the RealsenseViewer to check the camera stream and perform firmware upgrades
Realsense Viewer is a GUI-based software tool from Intel that allows you to test Realsense devices, view color and depth streams, view device information and perform firmware updates. In the past, we have seen realsense dropout issues getting solved by just doing a recommended firmware update from the RealsenseViewer GUI tool.
Open Realsense Viewer by executing:
realsense-viewer
And on the top right, a prompt would pop up asking you to install the latest firmware if one is available, as shown below.
Increasing the USB Kernel memory
By default, the kernel memory allocated for USB usage on Linux systems is usually small (e.g. 16 MB on a 64-bit Ubuntu system). However, for machine vision applications with high data transfer rates as it serves as a temporary buffer for image data transfer. Increasing the USB kernel memory to 1GB or more could potentially address realsense dropout issues.
You can increase the USB kernel memory temporarily at runtime using the command:
sudo sh -c 'echo 1024 > /sys/module/usbcore/parameters/usbfs_memory_mb'
You can refer to this source for more information.
Practice to decrease System resource usage
In the past, we have seen the realsense stability issues occurring frequently while running resource-intensive tasks. There are some general practices you can try to follow to decrease the System resource usage such as:
- Use the robot Untethered using SSH
- Avoid Remote Desktop Servers
- Reduce the Display Resolution
Reinstall Librealsense2 SDK and rebuild realsense-ros packages
Remove all the existing Librealsense2 packages:
dpkg -l | grep "realsense" | cut -d " " -f 3 | xargs sudo dpkg --purge
Reinstall Librealsense2 Packages:
sudo apt install librealsense2 librealsense2-dkms librealsense2-udev-rules librealsense2-utils librealsense2-dev librealsense2-dbg
Rebuild RealSense Ros packages already existing in the path ~/catkin_ws/src/realsense-ros:
cd ~/catkin_ws
rm -rf build/realsense-ros
catkin_make --pkg realsense2_camera --force-cmake
catkin_make --pkg realsense2_description --force-cmake
Build Librealsense2 SDK from source code
You can build and install the Librealsense2 SDK using the official Installation documentation. And you would have to again rebuild the realsense-ros packages again as mentioned in the above method. This is a more time-consuming process with multiple steps.
Downgrading Librealsense2 SDK and Camera Firmware to a specific version (ROS1 users only)
Another solution you can try is downgrading the Librealsense SDK to v2.50.0 and D435i firmware to v13.00.50 which was suggested in a realsense forum post that these specific version packages might provide increased stability while using realsense-ros ROS1 version.
Note: These downgrades might be incompatible with the realsense ROS2 drivers.
We created shell scripts that you can execute to perform these version-specific downgrades with ease.
Downgrade Librealsense2 to v2.50.0 and rebuild realsense-ros ROS1 packages using librealsense_2_50_install.sh:
wget https://gist.githubusercontent.com/hello-fazil/196201e2cede9e4240bc4f5d4e65a088/raw/533bab5974548f662217c50b81b27ac19e1c3f69/librealsense_2_50_install.sh
sh librealsense_2_50_install.sh
Downgrade D435i firmware to v13.00.50 using rs_fw_13050.sh:
wget https://gist.githubusercontent.com/hello-fazil/fb8a4063203d50214ff4e7556560fa21/raw/4e135235f952c7acb127299662dfe3050f762502/rs_fw_13050.sh
sh rs_fw_13050.sh
Support from Realsense
You could also find a lot of helpful resources and issues discussed in the D400 Series - Intel Realsense Help Center community forum. Additionally, we also recommend going through the official Realsense ROS and ROS2 wrapper documentation to understand the realsense camera’s ROS interface in detail.
We hope all the above-mentioned details find helpful for our users, and we are committed to providing more deterministic debugs and troubleshooting methods for Realsense in the near future.