So, in this case, would I use the stretch_ai_0.3.3 folder? And do I need to install the libraries separately?
Hi @Mallak_Alqaisi,
Thanks for sharing this and congrats for running the Pick and Place demo.
The message:
Success detection not implemented
does not indicate a failure. The task completed successfully, it just means Stretch does not currently perform an automatic verification step after placing the object.
If the object was placed correctly, everything worked as expected.
Let us know if you noticed any unexpected behavior.
Best,
Jason
Things are going well so far, but I have a question… Can I make changes to the code to create a new task for the robot? And can you help me with another demonstration or tasks that will help me understand the robot better?
Hi @Mallak_Alqaisi ,
Yes, you can create new tasks. We have official documentation that walks through how to add a new LLM task here.
Please let us know how it goes for you.
Best,
Jason
Should I start by creating a new task, or should I first go through the Dynamem demo or collectiondata from demonstration?
Because I find creating a new task a bit difficult, can you help me get started?
Hi @Mallak_Alqaisi,
The How to Add a New LLM Task guide is definitely the right reference, but I wouldn’t recommend trying to absorb all of it at once.
Here’s a simpler way to approach it:
- First, run:
python -m stretch.app.ai_pickup --use_llm --use_voice
Just observe how the full pipeline works.
- Then focus only on these 3 pieces in the guide:
hand_over_task.py→ how a task is structuredpickup_executor.py→ how tasks are calledpickup_prompt.py→ how the LLM is taught new actions
Ignore the rest for now (like creating new operations or apps).
A great first step is to slightly modify the handover task (change a message or add one SpeakOperation). That will help you understand the flow without starting from zero.
Best,
Jason
Hi Jason,
I need to run these commands every time before run python -m < > command right ?
ON ROBOT YOU HAVE TO run :
1- Homed robot with: stretch_robot_home.py
2- Run the Robot’s Script BY RUN THIS COMMAND: ./scripts/run_stretch_ai_ros2_bridge_server.sh
THEN YOU HAVE TO RETURN TO PC START DEMO:
1- RUN THIS COMMAND:mamba activate stretch_ai_0.3.3
2- RUN THIS COMMAND: ./scripts/set_robot_ip.sh 192.168.1.2
When i’m running this command i got these screens then the robot said: sorry i can’t understand that when i said for example “ pick the green ball”
I run these commands to open the codes :
1- code src/stretch/agent/task/pickup/pickup_executor.py
2- code src/stretch/agent/task/pickup/hand_over_task.py
3- code src/stretch/agent/task/pickup/pickup_prompt.py
Then i need to create copy of it to make edits.
Hi @Mallak_Alqaisi,
You don’t need to run all of those commands every time, only once per session (unless you reboot something).
On the robot:
stretch_robot_home.py→ required after each robot boot./scripts/run_stretch_ai_ros2_bridge_server.sh→ start once per session and keep it running
On the PC:
mamba activate stretch_ai_0.3.3→ required in each new terminal./scripts/set_robot_ip.sh 192.168.1.2→ once per session (unless the IP changes)
After that, you can run:
python -m stretch.app.<app_name>
as long as you’re in a terminal where the stretch_ai_0.3.3 environment is activated.
If you want to run a different app, just press Ctrl+C to stop the current one, then start the new app in the same terminal (or activate the environment again in a new one).
Best,
Jason
Hi @Mallak_Alqaisi,
From the logs, the app is starting correctly and it is recording audio, but the speech-to-text step is returning an empty transcript (I heard: is blank). Because of that, the pickup executor receives no commands and prints No commands to execute!, which is why the robot says it can’t understand.
This usually means the microphone input on the PC isn’t being captured correctly.
Could you please try the following on the same PC where you’re running --use_voice?
1) Play back the recorded audio from the last attempt
Right after a failed voice command, run:
ls -lh /tmp/tmp*.wav | sort -k6,7
This will output the recorded voice files. Then, from this output, try playing one of the audio files by running:
aplay /tmp/<name_of_audio_file>
Where <name_of_audio_file> should be starting with tmp and ending in .wav
(Use the most recent file name shown.)
Please confirm whether you can clearly hear your voice in that file.
If you cannot hear you voice, then you can proceed with the next step 2).
2) Quick microphone test
arecord -f cd -d 5 /tmp/mic_test.wav
aplay /tmp/mic_test.wav
Speak during the 5 seconds. If playback is silent or very faint, the system is likely using the wrong input device.
3) List detected audio devices
python3 -c "import sounddevice as sd; print(sd.query_devices())"
python3 -c "import sounddevice as sd; print('default:', sd.default.device)"
Please send me the outputs from these commands.
Once I see the outputs, I’ll guide you to the next step.
Best,
Jason
Hello Jason,
When i run this: ls -lh /tmp/tmp*.wav | sort -k6,7 i got cannot access ‘/tmp/tmp*.wav’: No such file or directory as shown in the image below:
Hi @Mallak_Alqaisi,
Could you please confirm that a microphone is physically connected to the GPU computer?
First, check which devices Ubuntu sees by running:
pactl list sources short
arecord -l
You should see a USB or analog input device listed.
Then test the microphone by recording a short clip:
arecord test.wav
# speak for a few seconds, then press Ctrl+C
aplay test.wav
If you can hear your voice, the microphone is working correctly.
Please let me know the results.
Thank you!
Hi @Mallak_Alqaisi,
From the output, the system only detects the internal motherboard audio device , and no external microphone.
Could you please confirm whether a microphone is physically connected to the GPU computer (either USB or the 3.5 mm mic jack)?
Thank you!
Jason
I don’t have a microphone physically connected to the GPU PC
Hi @Mallak_Alqaisi,
Thanks for confirming. That explains the issue, the voice interface requires a microphone connected to the GPU computer in order to capture speech.
Please connect either a USB microphone or a 3.5 mm microphone to the GPU PC, and then try running the program again.
Once the microphone is connected, the system should be able to record and process voice commands.
Thank you!