Stretch AI throws error with Traceback

Ask
1

Hello hello robot team,

I followed the instructions for the Stretch AI setup using a GPU machine and docker image on the Stretch 3. After completing this and seeing the images streaming to the GPU machine, I attempt to try the ‘language-directed pick and place’. I tried the llm version that allow text entry in the terminal. After entering the text, I got an error with Traceback. There there anything specific I should try to debug this error? It seems to happen during a rotate in place operation. See error output below:

You: Place the tennis ball in the cardboard box

Passing generation_config together with generation-related arguments=({‘max_new_tokens’}) is deprecated and will be removed in future versions. Please pass either a generation_config object OR all generation parameters explicitly, but not both.
Both max_new_tokens (=4096) and max_length(=20) seem to have been set. max_new_tokens will take precedence. Please refer to the documentation for more information. ( Generation )
[stretch.agent.task.pickup.pickup_executor] Resetting agent…
[WARNING] Resetting the robot’s spatial memory. Everything it knows will go away!
[stretch.agent.task.pickup.pickup_executor] Command: 0 say “I am picking up the tennis ball and placing it in the cardboard box.”
[stretch.agent.task.pickup.pickup_executor] Saying: “I am picking up the tennis ball and placing it in the cardboard box.”
[stretch.agent.task.pickup.pickup_executor] Command: 1 pickup tennis ball
[stretch.agent.task.pickup.pickup_executor] [Pickup task] Pickup: tennis ball
[stretch.agent.task.pickup.pickup_executor] 2 place cardboard box
[stretch.agent.task.pickup.pickup_executor] [Pickup task] Place: cardboard box
[stretch.agent.task.pickup.pickup_executor] [Pickup task] Pickup: tennis ball Place: cardboard box
Warning in search_for_tennis ball_on_floor: Overwriting target object class from None to tennis ball.
Warning in search_for_cardboard box: Overwriting target object class from None to cardboard box.
place_object_on_receptacle
Trying go to navigation mode: will switch to navigation mode.
[TASK INFO] Starting operation go to navigation mode
Running go to navigation mode: Switching to navigation mode.
go to navigation mode: Robot is in navigation mode: True
[TASK INFO] Operation complete.
!!! go to navigation mode !!!: Robot is in navigation mode.
[TASK INFO] Operation go to navigation mode successful.
[TASK INFO] Transitioning to rotate_in_place
Trying rotate_in_place: Rotating for 8 steps.
[TASK INFO] Starting operation rotate_in_place
Running rotate_in_place: rotating for 8 steps.
[stretch.agent.robot_agent] Rotate in place
==================== Waiting for 16 at goal ====================
Traceback (most recent call last):
File “/home/greg/miniforge3/envs/stretch_ai/lib/python3.10/runpy.py”, line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File “/home/greg/miniforge3/envs/stretch_ai/lib/python3.10/runpy.py”, line 86, in _run_code
exec(code, run_globals)
File “/home/greg/stretch_ai/src/stretch/app/ai_pickup.py”, line 227, in
main()
File “/home/greg/miniforge3/envs/stretch_ai/lib/python3.10/site-packages/click/core.py”, line 1485, in call
return self.main(*args, **kwargs)
File “/home/greg/miniforge3/envs/stretch_ai/lib/python3.10/site-packages/click/core.py”, line 1406, in main
rv = self.invoke(ctx)
File “/home/greg/miniforge3/envs/stretch_ai/lib/python3.10/site-packages/click/core.py”, line 1269, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/greg/miniforge3/envs/stretch_ai/lib/python3.10/site-packages/click/core.py”, line 824, in invoke
return callback(*args, **kwargs)
File “/home/greg/stretch_ai/src/stretch/app/ai_pickup.py”, line 217, in main
ok = executor(llm_response)
File “/home/greg/stretch_ai/src/stretch/agent/task/pickup/pickup_executor.py”, line 287, in call
self._pickup(target_object, target_receptacle)
File “/home/greg/stretch_ai/src/stretch/agent/task/pickup/pickup_executor.py”, line 104, in _pickup
task.run()
File “/home/greg/stretch_ai/src/stretch/core/task.py”, line 214, in run
self.current_operation.run()
File “/home/greg/stretch_ai/src/stretch/agent/operations/rotate_in_place.py”, line 23, in run
self.agent.rotate_in_place(
File “/home/greg/stretch_ai/src/stretch/agent/robot_agent.py”, line 545, in rotate_in_place
self.update()
File “/home/greg/stretch_ai/src/stretch/agent/robot_agent.py”, line 884, in update
self.get_voxel_map().add_obs(obs)
File “/home/greg/stretch_ai/src/stretch/mapping/voxel/voxel.py”, line 1452, in locked_method
return method(*args, **kwargs)
File “/home/greg/stretch_ai/src/stretch/mapping/voxel/voxel.py”, line 391, in add_obs
self.add(
File “/home/greg/stretch_ai/src/stretch/mapping/voxel/voxel.py”, line 571, in add
self.instances.process_instances_for_env(
File “/home/greg/stretch_ai/src/stretch/mapping/instance/instance_map.py”, line 762, in process_instances_for_env
embedding = self.encoder.encode_image(cropped_image).to(cropped_image.device)
File “/home/greg/stretch_ai/src/stretch/perception/encoders/siglip_encoder.py”, line 86, in encode_image
image_features /= image_features.norm(dim=-1, keepdim=True)
AttributeError: ‘BaseModelOutputWithPooling’ object has no attribute ‘norm’

2

Hi @gregory_sin,

Thanks for flagging this and for sharing the detailed logs, that’s very helpful.

We’re currently reviewing this internally to better understand what might be causing the error during the rotate_in_place step. We’ll get back to you as soon as we have more information or suggested debugging steps.

Thanks for your patience in the meantime.

Best,
Jason

3

Hi @gregory_sin ,

We’ve been working on a fix and created a branch with the updates. On your GPU machine, could you please try the following:

pip install "transformers>=4.50.0,<5"

# Navigate to your stretch_ai repo (adjust path if different)
cd ~/stretch_ai
git fetch
git checkout fix/transformers-trimesh-compat

cd ~/stretch_ai/src
pip install -e .

Then rerun:

python -m stretch.app.ai_pickup --use_llm

Let us know if this resolves the issue for you, or if anything else comes up.

Best,
Jason

4

Hello @Jason_Koubi ,

Thank you. That fix is working well! I will let you know if any other issue comes up.

1 Like
5

Hi @gregory_sin,

That’s great to hear, glad the fix is working well on your end.

Please don’t hesitate to reach out if anything else comes up.

Best,
Jason

1 Like