Ingestion and indexing

POST /api/ingest accepts a PDF and streams progress as Server-Sent Events (SSE).

Expand diagram

Document ingestion

Vector indexing and graph extraction are separate stages. A graph-stage failure can leave the document searchable through its PostgreSQL chunks.

Use the zoom controls, drag to pan, or pinch on a touch screen. Press Escape to close.

Processing stagesCopy Anchor Link

1. Extract textCopy Anchor Link

The backend uses pypdf to extract text. Scanned PDFs without a text layer are not OCRed: they produce no searchable chunks.

2. Split into overlapping chunksCopy Anchor Link

chunk_text defaults to a chunk size of 300 tokens and an overlap of 75 tokens. The overlap carries context across adjacent boundaries, helping when a relevant fact would otherwise be split between chunks.

This small chunk size is an intentional choice for the retrieval and vector-inspection workflows. Earlier planning notes that mention 800 / 100 do not describe the current implementation.

3. Embed and storeCopy Anchor Link

The configured embedding provider creates a 768-dimensional vector for each chunk. PostgreSQL's document_chunks table stores the vector with its document name, chunk index, text, and row ID. Inserts use PostgreSQL COPY for bulk loading.

The current local setup uses LM Studio with nomic-embed-text-v1.5; Gemini embeddings are also supported. Ingest and delete invalidate the process-level corpus and index caches so later searches see the changed corpus.

4. Extract graph dataCopy Anchor Link

The graph stage asks the configured model to extract entities and relationships. Neo4j stores entity mentions and source chunk IDs. This stage is separately reported and time-bounded. It can be partial for large documents or fail while the vector index remains searchable.

Progress eventsCopy Anchor Link

The stream emits named stage events for vector and graph work. Events can include a stage label and completed/total counts. Keep-alive comments are sent while slower model calls run so the client can distinguish work from a broken connection.

Delete a documentCopy Anchor Link

DELETE /api/documents/:name removes the document chunks and associated graph provenance, then invalidates retriever caches. Clients should URL-encode the document name in the path.

Current constraintsCopy Anchor Link

• No OCR fallback exists for image-only PDFs.
• Graph extraction is capped and time-bounded; a graph-stage failure does not roll back vector chunks.
• An entity missing source_chunk_ids cannot produce a document-grounded graph citation.
• Upload and delete are guarded only when DEMO_TOKEN is configured.