Knowledge Bases

Overview#

Knowledge bases are the organizational unit for documents in Anar Chat. Each knowledge base is an isolated collection of documents with its own vector index in pgvector. When a user sends a chat message with a knowledge_base_id, the RAG pipeline retrieves context exclusively from that knowledge base.

This design lets you create separate knowledge bases per department, project, or domain -- for example, "UAE AI Policy", "HR Regulations", or "Procurement Guidelines" -- and scope conversations to the right document set.

Creating a Knowledge Base#

Only users with the admin or manager role can create knowledge bases.

curl -s http://localhost:8001/api/v1/knowledge-bases \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "UAE AI Policy",
    "description": "National AI strategy and governance documents"
  }'

Response:

{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "name": "UAE AI Policy",
  "description": "National AI strategy and governance documents",
  "document_count": 0,
  "owner_id": 1,
  "created_at": "2026-02-15T10:30:00Z"
}

The document_count starts at zero and increments as documents are uploaded.

Listing Knowledge Bases#

All authenticated users can list knowledge bases. Admins see all knowledge bases across the system. Managers and users see only knowledge bases they own.

curl -s http://localhost:8001/api/v1/knowledge-bases \
  -H "Authorization: Bearer $TOKEN"

Supports pagination with skip and limit query parameters (default: skip=0, limit=50, max limit=100).

Uploading Documents#

Upload PDF, DOCX, or TXT files to a knowledge base. The upload endpoint accepts multipart/form-data and triggers the full RAG ingestion pipeline: text extraction, language detection, chunking, embedding generation, and pgvector storage.

curl -s http://localhost:8001/api/v1/documents/upload \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@uae-ai-strategy.pdf" \
  -F "knowledge_base_id=a1b2c3d4-e5f6-7890-abcd-ef1234567890"

Response:

{
  "id": "doc-001",
  "filename": "uae-ai-strategy.pdf",
  "content_type": "application/pdf",
  "knowledge_base_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "chunk_count": 42,
  "created_at": "2026-02-15T10:35:00Z"
}

The chunk_count tells you how many vector chunks were created from the document. A 20-page PDF typically produces 40-80 chunks at the default 1000-character chunk size.

Supported File Types#

What Happens During Upload#

1. Validation -- verifies the knowledge base exists, the user has permission, and the file type is allowed
2. Text extraction -- extracts raw text from the file (runs in a thread pool for PDFs and DOCX)
3. Language detection -- counts Arabic vs. Latin characters to classify the document
4. Arabic normalization -- if Arabic or mixed, normalizes diacritics, alef variants, and tatweel
5. Chunking -- splits text into overlapping chunks (sentence-aware for Arabic, character-based for English)
6. Embedding -- generates 3072-dim embeddings in batches of 50 chunks
7. Storage -- writes chunk embeddings and metadata to pgvector
8. LightRAG indexing -- if enabled, builds graph entities and relationships (async, non-blocking on failure)

Listing Documents#

View documents in a knowledge base or across all knowledge bases.

# All documents in a specific knowledge base
curl -s "http://localhost:8001/api/v1/documents?knowledge_base_id=a1b2c3d4-e5f6-7890-abcd-ef1234567890" \
  -H "Authorization: Bearer $TOKEN"

# All documents across all knowledge bases
curl -s http://localhost:8001/api/v1/documents \
  -H "Authorization: Bearer $TOKEN"

Supports pagination with skip and limit query parameters.

Deleting Documents#

Delete a document and its vector chunks from the knowledge base.

curl -s -X DELETE http://localhost:8001/api/v1/documents/doc-001 \
  -H "Authorization: Bearer $TOKEN"

This removes both the document record from PostgreSQL and all associated chunk embeddings from pgvector. Only admins and managers can delete documents.

Chatting with a Knowledge Base#

To use a knowledge base in a conversation, pass the knowledge_base_id when sending a chat message:

curl -N http://localhost:8001/api/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "What are the key pillars of the UAE AI strategy?",
    "knowledge_base_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
  }'

The RAG pipeline retrieves the top-K most relevant chunks from the specified knowledge base and injects them into the LLM's system prompt. The model is instructed to cite sources, so responses reference the original document filenames.

Once a conversation is created with a knowledge base, subsequent messages in the same conversation automatically use that knowledge base even if knowledge_base_id is omitted.

Docs-to-Chat Pipeline#

For documents that require advanced processing (scanned PDFs, Arabic OCR, complex layouts), Anar Docs can extract and chunk the text, then push the results directly into Chat:

import httpx

response = httpx.post(
    "http://localhost:8001/api/v1/rag/ingest",
    headers={"Authorization": f"Bearer {token}"},
    json={
        "knowledge_base_id": "kb-001",
        "document_id": "doc-from-docs-001",
        "filename": "scanned-contract.pdf",
        "chunks": [
            {"text": "Article 1: Definitions...", "index": 0, "metadata": {"page": 1}},
            {"text": "Article 2: Scope...", "index": 1, "metadata": {"page": 2}},
        ],
        "language": "ar",
    },
)

This endpoint skips extraction and chunking (Docs already handled those) and goes directly to embedding generation and pgvector storage. Each chunk can carry arbitrary metadata that gets stored alongside the embedding.

pgvector Storage Details#

Chat stores vector embeddings directly in PostgreSQL using the pgvector extension. Each knowledge base shares the same document_chunks table, scoped by the knowledge_base_id column.

The table schema includes:

Retrieval queries use pgvector's cosine distance operator (<=>) with an ORDER BY and LIMIT to return the top-K nearest chunks for a given query embedding.