Voice Agents

What are Voice Agents?#

Voice agents combine speech-to-text, LLM reasoning, and conversation memory into a single interactive loop. A user speaks, the agent transcribes the speech, generates an intelligent response using an LLM, and maintains context across the conversation. This enables hands-free, voice-driven interactions for government services.

Architecture#

The voice agent pipeline executes in a single request:

1. Audio input — User sends an audio recording (file upload or microphone capture)
2. Transcription — Whisper converts speech to text
3. LLM reasoning — The transcribed text is sent to an LLM with the system prompt and conversation history
4. Response — The LLM's text response is returned along with the original transcription

The LLM selection depends on the detected language: llama-3.3-70b-versatile for English, allam-2-7b for Arabic content.

Using the Voice Chat Endpoint#

Send audio and receive an AI response in one step:

curl -X POST http://localhost:8003/api/v1/voice/chat \
  -F "file=@question.mp3" \
  -G -d "language=ar" \
  -G -d "conversation_id=conv-001"

Response:

{
  "transcription": "ما هي الخدمات المتاحة في مركز الخدمة؟",
  "response": "يوفر مركز الخدمة عدة خدمات تشمل...",
  "language": "ar",
  "conversation_id": "conv-001",
  "duration": 3.2
}

Conversation Memory#

Pass a conversation_id to maintain context across multiple voice interactions. The agent retains the full conversation history for the session, enabling multi-turn dialogues:

# First turn
curl -X POST http://localhost:8003/api/v1/voice/chat \
  -F "file=@turn1.mp3"
# Returns conversation_id: "conv-001"

# Second turn — continues the conversation
curl -X POST http://localhost:8003/api/v1/voice/chat \
  -F "file=@turn2.mp3" \
  -G -d "conversation_id=conv-001"

Custom System Prompts#

Override the default agent behavior with a custom system prompt:

curl -X POST http://localhost:8003/api/v1/voice/chat \
  -F "file=@question.mp3" \
  -G -d "system_prompt=You are a UAE visa services assistant. Answer only in Arabic."

WebSocket Streaming with JWT Auth#

For real-time voice conversations, the WebSocket endpoint provides low-latency streaming transcription with JWT-based authentication.

const token = "eyJhbGciOiJIUzI1NiIs...";
const ws = new WebSocket(
  `ws://localhost:8003/api/v1/transcribe/stream?token=${token}`
);

ws.onopen = () => {
  console.log("Connected — streaming audio");
};

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.status === "completed") {
    // Display transcription, send to LLM, play response
    handleTranscription(msg.text, msg.language);
  }
};

Government Use Cases#

Automated Citizen Inquiry Line#

Deploy a voice agent as the first point of contact for government service hotlines. Citizens call in, speak their question in Arabic, and receive an immediate spoken response — reducing wait times and freeing human agents for complex cases.

Document Intake by Voice#

Citizens describe their request verbally. The voice agent transcribes, classifies the intent, and initiates the appropriate workflow — whether that is a permit application, document request, or service inquiry.

Meeting Transcription Bot#

Integrate a voice agent into virtual meeting platforms to provide real-time transcription and summarization. The agent joins the call, transcribes all participants, and generates meeting minutes when the call ends.

Configuration#