Extending qwen-audio-agent
This section is for developers integrating clients, voice services, Backend Agents, or knowledge systems through existing extension interfaces. To configure built-in capabilities, start with Configuration and Quickstart.
The core logical components are the Frontend Agent, Orchestration Runtime, and Backend Agent. The Gateway exposes runtime capabilities as a service; clients own I/O and environment interaction. Extend the relevant model provider, tool, BackendPort, or client protocol without moving business logic into transport. See the Architecture Overview.
Frontend Tools: MCP, OpenAPI, Profiles
Add chatbot tools without touching the voice path: connect MCP servers through the frontend MCP client, expose selected REST operations from an OpenAPI 3.x document, or bundle persona plus tool configuration as a versioned Frontend Profile.
→ Frontend MCP client · Frontend OpenAPI adapter · Frontend Profiles
Voice Frontend: Custom Realtime Provider
Swap the realtime speech model for another cloud service or your own stack by implementing the provider contract and registering it in the provider registry.
Knowledge: Retrieval Provider
The runtime connects knowledge modules through a small Provider interface. Use the built-in basic library, or connect the knowledge system you already operate.
→ Knowledge Retrieval Provider
The LightRAG integration example shows how to connect a complete, independently deployed knowledge system while leaving its models, indexes, and data under LightRAG's control.
Backend: Connect a New Agent
Four paths put a backend behind the protocol-neutral BackendPort: the zero-code generic ACP entry, a remote A2A agent, a custom adapter via the Backend Adapter SDK, or a first-class backend with one-click install.
→ Connecting a New Backend · Backend Adapter SDK · A2A Backend Adapter
Persona and Memory
The default assistant name, personality, and expression style live in ASSISTANT.md; output voice is configured separately. The default Markdown provider stores user preferences and durable facts in USER.md / MEMORY.md; configure the optional VoiceMem connector, or replace the provider with another memory engine, without changing the voice runtime.
→ Assistant Profile and User Preferences · Memory Provider
VoiceMem installation and configuration example: examples/voicemem.
Client: Build Your Own
The Gateway speaks typed events over a single WebSocket. Build a custom client against the client protocol, or embed the assistant into a host page through the stability contract — the same channel the desktop orb, TUI, and WebUI use.
The AI Passport example connects Qwen Voice Bean to the Gateway through a LAN relay. It currently supports half-duplex only; firmware and audio drivers are maintained in the external project.
→ Gateway Client Protocol · Gateway Contract
Desktop Appearance
The desktop orb renders replaceable pet skins: a pet.json manifest plus a spritesheet.