Lidar_node process has died

Ask
1

Hi everyone,

Once I quit and restart navigation with a previously created map, in both cases of Funmap and stretch_navigation, the launch is not successful. The terminal outputs the following error messages (stretch_navigation):

[ERROR] [1687772255.077072615]: Error, operation time out. RESULT_OPERATION_TIMEOUT! *
[lidar_node-7] process has died [pid 10882, exit code 255, cmd /opt/ros/noetic/lib/rplidar_ros/rplidarNode __name:=lidar_node __log:=/home/hello-robot/.ros/log/11ce94c8-1405-11ee-aef5-2d69b595ef22/lidar_node-7.log].
log file: /home/hello-robot/.ros/log/11ce94c8-1405-11ee-aef5-2d69b595ef22/lidar_node-7.log

[ WARN] [1687772256.213964382]: Timed out waiting for transform from base_link to map to become available before running costmap, tf error: canTransform: target_frame map does not exist… canTransform returned after 0.101011 timeout was 0.1.

The error message for launching Funmap is similar. To solve this I have tried stretch_rp_lidar_jog.py --reset without success. Currently the only way to solve this is a trivial reboot, so I am wondering why this is happening and how I can fix it.

I am working on a Stretch RE2 in ROS 1 and everything is up to date.

Thanks in advance,
Noor

2

Hi @roboor, based on the [lidar_node-7] process has died error, it looks like the lidar isn’t streaming data in ROS. Would you run the following four commands to verify that the lidar is able to stream data via the Python API?

$ stretch_rp_lidar_jog.py --reset
For use with S T R E T C H (R) RESEARCH EDITION from Hello Robot Inc.
---------------------------------------------------------------------

[INFO] [rplidar]: Reseting the sensor
None
[INFO] [rplidar]: Stoping motor

$ stretch_rp_lidar_jog.py --motor_on
For use with S T R E T C H (R) RESEARCH EDITION from Hello Robot Inc.
---------------------------------------------------------------------

[INFO] [rplidar]: Starting motor

$ stretch_rp_lidar_jog.py --health
For use with S T R E T C H (R) RESEARCH EDITION from Hello Robot Inc.
---------------------------------------------------------------------

[INFO] [rplidar]: Asking for health
('Good', 0)
[INFO] [rplidar]: Stoping motor

$ stretch_rp_lidar_jog.py --range
For use with S T R E T C H (R) RESEARCH EDITION from Hello Robot Inc.
---------------------------------------------------------------------

[INFO] [rplidar]: Starting motor
[INFO] [rplidar]: Asking for health
[INFO] [rplidar]: starting scan process in normal mode
0: Got 76 measurments
Scan [(11, 157.671875, 1114.5), (15, 162.125, 1103.25), (15, 170.171875, 231.5), (15, 167.53125, 1511.25), (15, 168.953125, 1514.0), (15, 184.984375, 1553.75), (15, 186.453125, 1574.0), (15, 187.875, 1584.5), (15, 193.015625, 847.0), (15, 202.515625, 621.5), (15, 209.8125, 607.0), (15, 211.328125, 619.75), (15, 212.6875, 634.75), (15, 215.515625, 667.25), (15, 216.953125, 662.25), (15, 218.46875, 646.25), (15, 219.9375, 631.0), (15, 221.484375, 617.25), (15, 223.015625, 604.25), (15, 224.46875, 591.75), (15, 234.359375, 147.5), (15, 234.78125, 145.0), (15, 236.234375, 142.5), (15, 237.6875, 140.0), (15, 239.15625, 138.0), (15, 255.859375, 472.75), (15, 257.4375, 470.75), (15, 259.015625, 470.0), (15, 260.359375, 469.25), (15, 261.875, 469.0), (15, 263.296875, 469.25), (15, 264.765625, 469.5), (15, 266.109375, 470.0), (15, 267.6875, 470.75), (15, 269.25, 472.25), (15, 270.453125, 473.75), (15, 271.9375, 475.5), (15, 273.359375, 478.0), (15, 275.0, 480.5), (15, 276.515625, 483.25), (15, 277.765625, 486.75), (15, 279.3125, 490.5), (15, 280.59375, 494.25), (15, 282.203125, 499.0), (15, 283.34375, 503.5), (15, 285.0, 508.25), (15, 286.28125, 513.75), (15, 287.859375, 519.5), (15, 289.21875, 526.25), (15, 290.765625, 532.75), (15, 291.921875, 540.0), (15, 293.515625, 548.25), (15, 295.0, 556.25), (15, 296.25, 566.0), (15, 297.71875, 576.0), (15, 299.140625, 586.0), (15, 300.484375, 597.0), (15, 302.046875, 609.5), (15, 303.328125, 622.25), (15, 304.875, 622.0), (15, 306.359375, 623.25), (15, 307.765625, 630.75), (15, 310.984375, 575.5), (15, 313.921875, 547.25), (15, 315.609375, 539.25), (15, 316.890625, 543.0), (15, 319.484375, 575.75), (15, 322.734375, 526.0), (15, 324.1875, 525.25), (15, 328.140625, 669.0), (15, 329.453125, 683.75), (15, 336.3125, 882.5), (15, 344.671875, 1223.75), (15, 346.125, 1219.0), (15, 347.53125, 1217.75), (15, 348.984375, 1218.0)]
[INFO] [rplidar]: Stoping motor

If you don’t see any measurements, let’s collect a dmesg log using REx_dmesg_monitor.py. Run this CLI is a separate terminal before running the experiment. When the Lidar fails, return to this terminal to save the log.

1 Like
3

Hi @bshah, I ran these commands, and somehow could not encounter the described issues anymore. If the lidar is causing problems again, I will run the commands again, to find out more.

However, these are the results from running these commands at the time when the lidar and the nav launching worked perfectly fine:

$ stretch_rp_lidar_jog.py --reset
For use with S T R E T C H (R) RESEARCH EDITION from Hello Robot Inc.
---------------------------------------------------------------------

[INFO] [rplidar]: Reseting the sensor
None
[INFO] [rplidar]: Stoping motor

$ stretch_rp_lidar_jog.py --motor_on
For use with S T R E T C H (R) RESEARCH EDITION from Hello Robot Inc.
---------------------------------------------------------------------

[INFO] [rplidar]: Starting motor

$ stretch_rp_lidar_jog.py --health
For use with S T R E T C H (R) RESEARCH EDITION from Hello Robot Inc.
---------------------------------------------------------------------

[INFO] [rplidar]: Asking for health
('Good', 0)
[INFO] [rplidar]: Stoping motor

$ stretch_rp_lidar_jog.py --range
For use with S T R E T C H (R) RESEARCH EDITION from Hello Robot Inc.
---------------------------------------------------------------------

[INFO] [rplidar]: Starting motor
[INFO] [rplidar]: Asking for health
[INFO] [rplidar]: starting scan process in normal mode
0: Got 56 measurments
Scan [(15, 72.625, 6441.75), (15, 73.96875, 6404.75), (15, 75.28125, 6347.5), (15, 76.609375, 6315.0), (15, 77.9375, 6290.0), (15, 79.25, 6254.5), (10, 84.390625, 11549.5), (15, 88.359375, 10740.5), (15, 89.671875, 10762.0), (15, 90.984375, 10820.5), (15, 92.3125, 10953.5), (14, 152.015625, 732.0), (9, 153.46875, 720.75), (15, 156.109375, 711.75), (15, 187.9375, 657.0), (15, 233.453125, 145.0), (15, 234.828125, 143.0), (15, 236.140625, 140.75), (15, 237.46875, 138.75), (15, 239.359375, 136.75), (15, 241.859375, 134.0), (15, 259.640625, 506.25), (15, 261.109375, 502.0), (15, 262.328125, 500.0), (15, 263.953125, 498.0), (15, 264.96875, 496.0), (15, 266.578125, 495.25), (15, 267.609375, 494.5), (15, 269.21875, 493.0), (15, 270.5625, 492.75), (15, 271.671875, 494.75), (15, 278.609375, 2076.0), (1, 279.921875, 2066.5), (15, 281.234375, 2030.0), (4, 282.546875, 2030.0), (15, 283.859375, 2034.25), (9, 287.8125, 2037.25), (15, 294.03125, 4264.75), (15, 299.9375, 1581.25), (15, 305.578125, 1167.5), (15, 306.875, 1161.5), (15, 308.203125, 1173.75), (15, 312.515625, 5007.5), (15, 316.515625, 4468.5), (15, 317.828125, 4414.25), (15, 321.90625, 3118.5), (15, 323.28125, 3074.75), (15, 324.625, 3012.0), (15, 325.921875, 2973.75), (15, 339.078125, 3481.0), (15, 341.890625, 2582.25), (15, 343.21875, 2566.5), (15, 344.515625, 2551.25), (15, 345.84375, 2538.0), (15, 347.21875, 2526.75), (15, 348.5625, 2505.5)]
[INFO] [rplidar]: Stoping motor
1 Like
4

So now after quite some while of working in ROS2, I switched to ROS1 and after some tests, I stumbled again over the same issue where the rplidar does not reset itself:

Summary 
hello-robot@stretch-re2-2029:~$ stretch_rp_lidar_jog.py --reset
For use with S T R E T C H (R) RESEARCH EDITION from Hello Robot Inc.
---------------------------------------------------------------------

[INFO] [rplidar]: Reseting the sensor
None
[INFO] [rplidar]: Stoping motor

hello-robot@stretch-re2-2029:~$ roslaunch stretch_navigation mapping.launch 
... logging to /home/hello-robot/.ros/log/0803bf94-4e53-11ee-ba9c-fd8d52a91004/roslaunch-stretch-re2-2029-24694.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://stretch-re2-2029:34783/

SUMMARY
========

PARAMETERS
 * /aggregator_node/analyzers/hardware/analyzers/Arm/find_and_remove_prefix: ['stretch_driver:...
 * /aggregator_node/analyzers/hardware/analyzers/Arm/path: Arm
 * /aggregator_node/analyzers/hardware/analyzers/Arm/type: diagnostic_aggreg...
 * /aggregator_node/analyzers/hardware/analyzers/Base/find_and_remove_prefix: ['stretch_driver:...
 * /aggregator_node/analyzers/hardware/analyzers/Base/path: Base
 * /aggregator_node/analyzers/hardware/analyzers/Base/type: diagnostic_aggreg...
 * /aggregator_node/analyzers/hardware/analyzers/EndOfArm/find_and_remove_prefix: ['stretch_driver:...
 * /aggregator_node/analyzers/hardware/analyzers/EndOfArm/path: EndOfArm
 * /aggregator_node/analyzers/hardware/analyzers/EndOfArm/type: diagnostic_aggreg...
 * /aggregator_node/analyzers/hardware/analyzers/Head/find_and_remove_prefix: ['stretch_driver:...
 * /aggregator_node/analyzers/hardware/analyzers/Head/path: Head
 * /aggregator_node/analyzers/hardware/analyzers/Head/type: diagnostic_aggreg...
 * /aggregator_node/analyzers/hardware/analyzers/Lift/find_and_remove_prefix: ['stretch_driver:...
 * /aggregator_node/analyzers/hardware/analyzers/Lift/path: Lift
 * /aggregator_node/analyzers/hardware/analyzers/Lift/type: diagnostic_aggreg...
 * /aggregator_node/analyzers/hardware/analyzers/Pimu/find_and_remove_prefix: ['stretch_driver:...
 * /aggregator_node/analyzers/hardware/analyzers/Pimu/path: Pimu
 * /aggregator_node/analyzers/hardware/analyzers/Pimu/type: diagnostic_aggreg...
 * /aggregator_node/analyzers/hardware/analyzers/Wacc/find_and_remove_prefix: ['stretch_driver:...
 * /aggregator_node/analyzers/hardware/analyzers/Wacc/path: Wacc
 * /aggregator_node/analyzers/hardware/analyzers/Wacc/type: diagnostic_aggreg...
 * /aggregator_node/analyzers/hardware/path: Hardware
 * /aggregator_node/analyzers/hardware/type: diagnostic_aggreg...
 * /aggregator_node/path: Stretch
 * /aggregator_node/type: diagnostic_aggreg...
 * /joint_state_publisher/rate: 15.0
 * /joint_state_publisher/source_list: ['/stretch/joint_...
 * /laser_filter/scan_filter_chain: [{'name': 'shadow...
 * /lidar_node/angle_compensate: True
 * /lidar_node/frame_id: laser
 * /lidar_node/inverted: False
 * /lidar_node/scan_mode: Boost
 * /lidar_node/serial_baudrate: 115200
 * /lidar_node/serial_port: /dev/hello-lrf
 * /robot_description: <?xml version="1....
 * /robot_state_publisher/publish_frequency: 15.0
 * /rosdistro: noetic
 * /rosversion: 1.16.0
 * /stretch_driver/broadcast_odom_tf: True
 * /stretch_driver/controller_calibration_file: /home/hello-robot...
 * /stretch_driver/mode: navigation
 * /stretch_driver/rate: 15.0
 * /stretch_driver/timeout: 0.5
 * /teleop_twist_keyboard/speed: 0.04
 * /teleop_twist_keyboard/turn: 0.1

NODES
  /
    aggregator_node (diagnostic_aggregator/aggregator_node)
    gmapping_record_map (gmapping/slam_gmapping)
    joint_state_publisher (joint_state_publisher/joint_state_publisher)
    laser_filter (laser_filters/scan_to_scan_filter_chain)
    lidar_node (rplidar_ros/rplidarNode)
    robot_state_publisher (robot_state_publisher/robot_state_publisher)
    rviz (rviz/rviz)
    stretch_driver (stretch_core/stretch_driver)
    teleop_twist_keyboard (teleop_twist_keyboard/teleop_twist_keyboard.py)

auto-starting new master
process[master]: started with pid [24709]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 0803bf94-4e53-11ee-ba9c-fd8d52a91004
process[rosout-1]: started with pid [24726]
started core service [/rosout]
process[joint_state_publisher-2]: started with pid [24733]
process[robot_state_publisher-3]: started with pid [24734]
process[stretch_driver-4]: started with pid [24735]
process[aggregator_node-5]: started with pid [24736]
process[lidar_node-6]: started with pid [24738]
[ WARN] [1694182902.836843482]: The root link base_link has an inertia specified in the URDF, but KDL does not support a root link with an inertia.  As a workaround, you can add an extra dummy link to your URDF.
process[laser_filter-7]: started with pid [24743]
process[teleop_twist_keyboard-8]: started with pid [24748]
process[gmapping_record_map-9]: started with pid [24754]
[ INFO] [1694182902.879239265]: RPLIDAR running on ROS package rplidar_ros, SDK Version:2.0.0
process[rviz-10]: started with pid [24758]
[INFO] [rospy.topics]: topicmanager initialized
[INFO] [1694182904.081867]: For use with S T R E T C H (TM) RESEARCH EDITION from Hello Robot Inc.
[INFO] [1694182904.082965]: /stretch_driver started
[INFO] [1694182904.613675]: mode = navigation
[INFO] [1694182904.614839]: /stretch_driver: Changed to mode = navigation
[INFO] [1694182904.616636]: broadcast_odom_tf = True
[INFO] [1694182904.620213]: Loading controller calibration parameters for the head from YAML file named /home/hello-robot/catkin_ws/src/stretch_ros/stretch_core/config/controller_calibration_head.yaml
[INFO] [1694182904.625821]: controller parameters loaded = {'arm_retracted_offset': 0.0025550286250485095, 'pan_angle_offset': -0.012313715472842945, 'pan_looked_left_offset': -0.0012904399890229065, 'tilt_angle_backlash_transition': -0.4, 'tilt_angle_offset': 0.015868523909082964, 'tilt_looking_up_offset': -0.027955338296670372}
[INFO] [1694182904.634759]: head_tilt_calibrated_offset_rad in degrees = 0.9091994470928927
[INFO] [1694182904.639562]: head_pan_calibrated_offset_rad in degrees = -0.7055239267188396
[INFO] [1694182904.642274]: head_pan_calibrated_looked_left_offset_rad in degrees = -0.07393676508592083
[INFO] [1694182904.644588]: head_tilt_backlash_transition_angle_rad in degrees = -22.918311805232932
[INFO] [1694182904.645893]: head_tilt_calibrated_looking_up_offset_rad in degrees = -1.601722899259652
[INFO] [1694182904.647128]: arm_calibrated_retracted_offset_m in meters = 0.0025550286250485095
[INFO] [1694182904.685632]: /stretch_driver rate = 15.0 Hz
[INFO] [1694182904.687069]: /stretch_driver timeout = 0.5 s
[INFO] [1694182904.690840]: /stretch_driver use_fake_mechaduinos = False
[INFO] [1694182904.693612]: /stretch_driver base_frame_id = base_link
[INFO] [1694182904.696899]: /stretch_driver odom_frame_id = odom

Reading from the keyboard  and Publishing to Twist!
---------------------------
Moving around:
   u    i    o
   j    k    l
   m    ,    .

For Holonomic mode (strafing), hold down the shift key:
---------------------------
   U    I    O
   J    K    L
   M    <    >

t : up (+z)
b : down (-z)

anything else : stop

q/z : increase/decrease max speeds by 10%
w/x : increase/decrease only linear speed by 10%
e/c : increase/decrease only angular speed by 10%

CTRL-C to quit

currently:	speed 0.04	turn 0.1 
[ERROR] [1694182906.927481753]: Error, operation time out. RESULT_OPERATION_TIMEOUT! 
     [lidar_node-6] process has died [pid 24738, exit code 255, cmd /opt/ros/noetic/lib/rplidar_ros/rplidarNode __name:=lidar_node __log:=/home/hello-robot/.ros/log/0803bf94-4e53-11ee-ba9c-fd8d52a91004/lidar_node-6.log].
                                                            log file: /home/hello-robot/.ros/log/0803bf94-4e53-11ee-ba9c-fd8d52a91004/lidar_node-6*.log

A reboot always fixes the issue. Just wanted to share if anyone else encounters this as well or to bring it to the developers’ attention. I think the problem may also lie with the rplidar itself, since I have experienced this in other robot setups before, too.

5

I have also noticed this same error. I haven’t updated my robot’s (RE2) software recently, but I did recently begin using the stretch_scan_matcher. It seems to randomly fail after restarting the rplidar.launch process. I noticed that the lidar also does not stop spinning after I kill the launch file; not sure if it is related. Is there a process that I could kill from the terminal to manually stop the lidar?

6

The stretch_rp_lidar_jog.py --motor_off command does this. The lidar actually spins by default and stops when a signal is sent. In the past, the rplidar ROS package (maintained by Slamtec) sent the signal when rplidar.launch ended, but in a recent code change, it doesn’t do this anymore.

@roboor and @lamsey, thanks for sharing info about these issues. I don’t know much about the Slamtec drivers, but I can try to help here. @roboor, in your case, launching rplidar.launch multiple times will cause a Error, operation time out. RESULT_OPERATION_TIMEOUT! until you power cycle the robot, right? @lamsey, in your case, stretch_scan_matcher.launch stops publishing the odom → base_link TF after relaunching rplidar.launch, right?

1 Like
7

Yes, but as said, the error is quite rare and unpredictable so far.

8

Sorry for delay, I had to wait for the lidar error to happen again. Running the stop motor command didn’t fix the rplidar.launch error. After the error, I checked my /tf tree, and there’s still a transform from odom to base_link. However, nothing is being broadcast on /pose2D from the scan matcher.

1 Like
9

Gotcha, thanks. I’ll work on reproducing these issues here and let you guys know if I can.