RAG Pipeline

Overview#

Anar Chat implements a retrieval-augmented generation (RAG) pipeline that grounds LLM responses in your organization's documents. When a user sends a message tied to a knowledge base, Chat retrieves the most relevant document chunks via vector similarity search, injects them as context into the LLM prompt, and instructs the model to cite its sources.

The pipeline has four stages: text extraction, chunking, embedding, and retrieval. An optional fifth stage -- LightRAG graph-enhanced retrieval -- adds entity and relationship traversal for complex multi-hop queries.

Text Extraction#

When a document is uploaded, Chat extracts raw text based on the file type:

Extraction runs in a thread pool (asyncio.to_thread) to avoid blocking the async event loop during CPU-intensive PDF parsing.

Chunking#

Extracted text is split into overlapping chunks before embedding. Chat uses two chunking strategies, selected automatically based on detected language.

English Chunking#

Character-based sliding window with configurable size and overlap:

The chunker slides a window of chunk_size characters across the text, advancing by chunk_size - overlap on each step. This ensures that context around chunk boundaries is preserved in adjacent chunks.

Arabic Chunking#

Arabic text uses sentence-aware chunking that respects natural sentence boundaries (Arabic question marks, periods, exclamation points, and newlines):

The Arabic chunker splits text into sentences first, then groups sentences into chunks that stay under the character limit. When a chunk boundary is reached, the last two sentences carry over into the next chunk as overlap, maintaining coherence across chunk boundaries.

Language Detection#

Chat detects the language of each document by counting Arabic vs. Latin characters:

• >70% Arabic -- Arabic chunking
• <30% Arabic -- English chunking
• 30-70% (mixed) -- Arabic chunking (preserves sentence boundaries for bilingual content)

Arabic Normalization#

Before chunking, Arabic text goes through normalization to improve retrieval consistency:

• Diacritics (tashkeel) removal
• Alef variant unification (hamza forms collapse to bare alef)
• Teh marbuta to heh
• Alef maqsura to yeh (word-final)
• Tatweel (kashida) stripping
• Zero-width character removal

This normalization ensures that the same word spelled with or without diacritics maps to the same embedding, significantly improving recall for Arabic queries.

Embedding Generation#

Chunks are embedded using Azure text-embedding-3-large with 3072 dimensions. Embedding generation supports batching (50 chunks per batch) for efficient processing of large documents.

The embedding provider resolution follows this order:

1. Anar Gateway -- if GATEWAY_URL is set, embeddings route through Gateway
2. Azure OpenAI -- if AZURE_OPENAI_API_KEY is set, uses Azure UAE North directly
3. Sovereign cloud -- fallback via OpenAI-compatible API

Vector Storage (pgvector)#

Embeddings are stored in PostgreSQL using the pgvector extension. Each chunk is stored as a row in the document_chunks table with its embedding vector, content text, and metadata (document ID, filename, chunk index, language, knowledge base ID).

Retrieval uses pgvector's cosine distance operator to find the nearest vectors:

SELECT content, metadata_, embedding <=> query_embedding AS distance
FROM document_chunks
WHERE knowledge_base_id = :kb_id
ORDER BY distance
LIMIT :top_k

Cosine distance scores are converted to similarity scores (1 - distance) in the response, where higher values indicate stronger matches.

Retrieval#

When a user sends a chat message with a knowledge_base_id, the retrieval pipeline:

1. Detects the query language and applies Arabic normalization if needed
2. Generates an embedding for the normalized query
3. Searches pgvector for the top-K most similar chunks (default: 5)
4. Formats the retrieved chunks into a context block with source citations
5. Appends the context to the system prompt

The formatted context looks like:

Here are relevant excerpts from the knowledge base:

[Source 1: uae-ai-strategy.pdf]
The UAE National AI Strategy 2031 aims to position the UAE...

[Source 2: digital-government.pdf]
Abu Dhabi Digital Authority launched the AI adoption framework...

The LLM system prompt instructs the model to cite sources when using knowledge base information.

LightRAG Graph-Enhanced Retrieval#

When enabled, LightRAG builds a knowledge graph over ingested documents by extracting entities and relationships using an LLM. At query time, it performs hybrid retrieval that combines:

• Graph traversal -- finds entities and relationships relevant to the query
• Vector search -- standard pgvector cosine similarity (always active)

The LightRAG context is prepended to the vector search results, giving the LLM both structural knowledge (entity relationships) and raw text passages.

How It Works#

During document ingestion, LightRAG:

1. Chunks the document into token-sized segments (default: 1200 tokens)
2. Sends each chunk to the LLM to extract entities and relationships
3. Builds a graph structure linking entities across chunks and documents
4. Stores the graph in a per-knowledge-base working directory

During retrieval, LightRAG:

1. Queries the knowledge graph for relevant entities and their relationships
2. Returns a structured context string with entity descriptions and connections
3. This context is merged with pgvector results before being sent to the LLM

Retrieval Modes#

The LIGHTRAG_RETRIEVAL_MODE setting controls how LightRAG queries the graph:

Configuration#

Graceful Degradation#

LightRAG failures never break the RAG pipeline. If graph construction fails during ingestion, the vector store still has the chunks. If graph retrieval fails at query time, Chat falls back to vector-only search. Errors are logged but never surfaced to the user.

Docs-to-Chat Ingest Pipeline#

Anar Docs can push pre-processed document chunks directly into Chat's RAG store via the POST /api/v1/rag/ingest endpoint. This enables a pipeline where Docs handles OCR and extraction (including Arabic OCR via EasyOCR/Tesseract fallback chains), then sends structured chunks to Chat for embedding and storage.

import httpx

chunks = [
    {"text": "Extracted paragraph one...", "index": 0, "metadata": {"page": 1}},
    {"text": "Extracted paragraph two...", "index": 1, "metadata": {"page": 1}},
]

response = httpx.post(
    "http://localhost:8001/api/v1/rag/ingest",
    headers={"Authorization": f"Bearer {token}"},
    json={
        "knowledge_base_id": "kb-001",
        "document_id": "doc-from-ocr-001",
        "filename": "scanned-arabic-doc.pdf",
        "chunks": chunks,
        "language": "ar",
    },
)

This endpoint accepts pre-chunked text with metadata, generates embeddings, and stores everything in pgvector. It skips the text extraction and chunking stages since Docs has already handled those.