Streaming

Overview#

Anar Chat delivers LLM responses as a stream of tokens via Server-Sent Events (SSE). Instead of waiting for the full response to generate, the client receives each token as it is produced, enabling real-time display of the assistant's reply. This is the default and only response mode for the chat completions endpoint.

SSE Protocol#

The chat completions endpoint returns a text/event-stream response. Each frame is a JSON object prefixed with data: and terminated by a double newline:

data: {"token": "The", "conversation_id": "abc-123"}

data: {"token": " UAE", "conversation_id": "abc-123"}

data: {"token": " National", "conversation_id": "abc-123"}

data: {"token": " AI", "conversation_id": "abc-123"}

...

data: {"done": true, "conversation_id": "abc-123"}

Frame Types#

Token frame -- carries a single token from the LLM output:

{"token": "string", "conversation_id": "string"}

Done frame -- signals the end of the stream. Sent after the full response has been persisted to the conversation history:

{"done": true, "conversation_id": "string"}

The conversation_id is present in every frame. If the request created a new conversation (no conversation_id in the request), the first frame contains the newly generated ID that the client should use for subsequent messages.

Sending a Streaming Request#

curl -N http://localhost:8001/api/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Summarize the UAE AI strategy",
    "knowledge_base_id": "kb-001"
  }'

The -N flag disables curl's output buffering, which is required to see tokens as they arrive.

Request Body#

Model Selection#

When model is specified, Chat resolves the provider automatically:

If model is omitted, Chat uses the default model for the configured LLM_PROVIDER.

Client Integration#

Python#

import httpx

with httpx.stream(
    "POST",
    "http://localhost:8001/api/v1/chat/completions",
    headers={"Authorization": f"Bearer {token}"},
    json={"message": "What is Abu Dhabi's digital strategy?"},
) as response:
    for line in response.iter_lines():
        if line.startswith("data: "):
            import json
            data = json.loads(line[6:])
            if data.get("done"):
                print("\n[Stream complete]")
                break
            print(data["token"], end="", flush=True)

JavaScript (EventSource)#

const response = await fetch("http://localhost:8001/api/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${token}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    message: "What is Abu Dhabi's digital strategy?",
    knowledge_base_id: "kb-001",
  }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
let conversationId = null;

while (true) {
  const { value, done } = await reader.read();
  if (done) break;

  const text = decoder.decode(value);
  for (const line of text.split("\n")) {
    if (line.startsWith("data: ")) {
      const data = JSON.parse(line.slice(6));
      conversationId = data.conversation_id;
      if (data.done) {
        console.log("\n[Stream complete]");
      } else {
        process.stdout.write(data.token);
      }
    }
  }
}

JavaScript (EventSource API)#

For browser environments, you can use the native EventSource API. However, since the chat endpoint requires a POST request with a body, you need a library like eventsource-parser or a fetch-based approach as shown above. The native EventSource API only supports GET requests.

How Streaming Works Internally#

When a chat completion request arrives:

1. Conversation resolution -- creates a new conversation or loads an existing one
2. Message persistence -- saves the user's message to the database
3. History retrieval -- loads the last 10 messages from the conversation for context
4. RAG retrieval -- if a knowledge base is specified, retrieves relevant chunks and appends them to the system prompt
5. LLM streaming -- opens a streaming connection to the LLM provider (Groq, Azure, or sovereign cloud) via the OpenAI SDK's stream=True parameter
6. Token forwarding -- each token from the LLM is wrapped in an SSE frame and sent to the client
7. Response persistence -- after the stream completes, the full response is saved to the conversation history
8. Done signal -- the final SSE frame signals completion

The full response is accumulated in memory during streaming and written to the database in a single operation after the stream ends. The conversation's updated_at timestamp is refreshed on each completion.

Multi-Provider Streaming#

All three LLM providers support streaming through the OpenAI SDK's unified interface:

• Groq -- streams via https://api.groq.com/openai/v1 with a custom User-Agent header (required by Groq's Cloudflare protection)
• Azure UAE North -- streams via the Azure OpenAI endpoint with deployment-specific routing
• Sovereign cloud -- streams via an OpenAI-compatible API at the configured base URL

When Anar Gateway is configured (GATEWAY_URL), all streaming requests route through Gateway regardless of the target provider. Gateway preserves the SSE stream transparently.

Conversation Memory#

Chat maintains a sliding window of the last 10 messages per conversation. Each streaming request includes this history in the LLM prompt, giving the model context from the ongoing conversation. Both user messages and assistant responses are persisted, so the conversation history survives server restarts.

Conversation Export#

Completed conversations can be exported in JSON or Markdown format:

# Export as JSON
curl -s "http://localhost:8001/api/v1/chat/conversations/abc-123/export?format=json" \
  -H "Authorization: Bearer $TOKEN" \
  -o conversation.json

# Export as Markdown
curl -s "http://localhost:8001/api/v1/chat/conversations/abc-123/export?format=markdown" \
  -H "Authorization: Bearer $TOKEN" \
  -o conversation.md

The Markdown export includes timestamps, role labels, and source citations for each message.

Error Handling#

If the LLM provider returns an error or the connection drops mid-stream, the SSE connection closes without a done frame. Clients should handle this by:

• Detecting when the stream closes without a done frame
• Displaying what was received so far
• Offering a retry option to the user

The partially streamed response is not persisted to the conversation history -- only complete responses are saved.