AI Passport Voice Client

Run Qwen Voice Bean on an AI Passport (ESP32-C3) card to use qwen-audio-agent over the LAN. The card captures speech, plays replies, and displays character animation; the Gateway on the computer handles realtime conversation, tools, and optional backend Agent tasks.

Demo

Turn on sound. Due to hardware limitations, the reference device currently supports half-duplex mode only. Microphone upload pauses during reply playback; automatic voice interruption is not supported.

Core features

• Hardware microphone, speaker, buttons, and screen replace a desktop or browser UI.
• Gateway Client Protocol (GCP) messages travel over the LAN; the device relay splits audio into smaller chunks for the card.
• The normal Gateway retains its voice frontend, tools, and backend Agent configuration.

Connection setup

The card does not connect directly to the Gateway. It connects to the device relay over the LAN, which connects to the local Gateway on the same computer.

Component	Responsibility
Qwen Voice Bean firmware	Wi-Fi setup, capture/playback, half-duplex control, buttons, and animation on the AI Passport card.
Device relay (device-relay.mjs)	Listen on the computer's LAN port 3101, validate device tokens, split audio into small chunks, and forward GCP messages.
qwen-audio-agent Gateway	Listen on 127.0.0.1:18888 on the same computer; handle realtime conversation, tools, and optional backend tasks.

The device relay and Gateway run as two separate processes on the same computer, handling card transport and conversation/task execution respectively.

Run the example

Configure the ordinary Gateway using the quickstart. From a repository checkout, install dependencies and start it on a loopback port:

npm ci
node cli/bin/qwenaudio.mjs gateway run --url http://127.0.0.1:18888

In another terminal, copy and edit the relay configuration:

cp examples/ai-passport/.env.example examples/ai-passport/.env.local

Set DEVICE_ACCESS_TOKEN to a private token of at least 24 characters, set DEVICE_HOST=0.0.0.0 explicitly for trusted-LAN access, and leave GATEWAY_URL=http://127.0.0.1:18888 and DEVICE_PORT=3101 for this setup. The device token is not a model API key.

npm run example:ai-passport

Install the firmware from the hardware community, then configure the card with the computer's LAN IP and the matching device token. The reference firmware uses ws://COMPUTER_LAN_IP:3101/api/realtime. Disconnect other conversations for the same Gateway user before connecting it.

Keep the computer awake and allow port 3101 through its firewall. This example uses plaintext WebSocket on a trusted LAN only. Keep token validation enabled and do not expose the relay to the internet.

Half-duplex use and limitations

Short-press confirm to enable the microphone, speak, and wait for the reply to finish before speaking again. Press down for a manual interruption. Playback pauses microphone upload; the current device has no AEC or automatic voice interruption.

Source and acknowledgements

• Example and protocol checklist: full setup instructions, audio transport, and tests.
• External firmware: hardware drivers, interaction, and UI.
• Tao Liu implemented the firmware, hardware interaction, character UI, and device relay; Li Xu maintains the framework-side integration and documentation. The FoloToy community hosts firmware distribution.