Snippet Search

Summary

Every document in a library (both manually-written private libraries and org-subscribed public sources) is automatically broken into snippets — prose sections and fenced code blocks — and indexed for fast, ranked retrieval. Snippet search lets an agent find the specific passage or code sample it needs across a whole corpus, instead of reading full documents one at a time.

There are two snippet kinds:

Snippets are retrieved with hybrid ranking: a semantic (vector) arm finds conceptually related content even without matching words, and a full-text arm finds exact keyword/identifier matches. Results from both arms are fused with Reciprocal Rank Fusion (RRF), plus a boost when the query's extracted code identifiers overlap a snippet's own identifiers.

Indexing

Snippets are produced and embedded by a background sweep (SnippetIndexingHostedService), not synchronously on write. When a document's content changes, its ProcessingStatus resets to Pending; the sweep ticks every 30 seconds (AppSettings:SnippetIndexing:SweepIntervalSeconds) and drains the entire pending backlog each tick — a large backfill runs at full throughput, not one batch per tick.

Each tick runs two stages:

1. Parse and compose: pending documents are parsed, split into section/code snippets, and reconciled against previously stored rows by content hash. An unchanged snippet keeps its existing embedding; changed or new snippets get a fresh row with a null embedding, ready to be picked up by the next stage.
2. Embed: snippets with a missing or stale embedding are claimed in batches and sent to the embedding provider. Snippets are always full-text searchable immediately after step 1 — the vector arm only lights up once step 2 completes for that row.

A document that fails to parse is marked Failed and does not stall the rest of the sweep.

Degraded mode

If the embedding provider is unavailable or times out (query path only — the sweep retries internally), search silently falls back to full-text-only ranking rather than failing the request. Callers can detect this via the degraded flag on results.

Re-embedding on model change

Each snippet's embedding_model column is stamped with the model + dimension count used to embed it (e.g. qwen3-embedding-8b@768). Changing AppSettings:Llm:Embeddings:Model or :Dimensions makes every existing row's stamp stale, so the sweep automatically re-claims and re-embeds the whole corpus with no separate migration or backfill script — the claim query matches on embedding_model IS DISTINCT FROM {current model}.

Forcing a full reindex

The sweep only reprocesses a document when its ProcessingStatus is Pending/Stale, or a stale Indexing row (crash recovery). A document whose content hasn't changed since it was last indexed never becomes Pending on its own — which matters whenever the parsing or composition logic itself changes (e.g. a SnippetIdentifierExtractor tuning pass), since that kind of change isn't reflected in any stored row's content hash.

To force every document back through parse → compose → (re-)embed, reset processing_status directly:

UPDATE zeeq.docs_library_documents SET processing_status = 'Pending', updated_at = now();
UPDATE zeeq.docs_public_documents SET processing_status = 'Pending', updated_at = now();

Scope to one library or one public source instead of the whole corpus by adding a WHERE:

UPDATE zeeq.docs_library_documents
SET processing_status = 'Pending', updated_at = now()
WHERE organization_id = '<org_id>' AND library_id = '<library_id>';

UPDATE zeeq.docs_public_documents
SET processing_status = 'Pending', updated_at = now()
WHERE public_source_id = '<public_source_id>';

The sweep picks these up on its next tick (SweepIntervalSeconds, 30s by default) — no restart needed. Reconciliation still matches composed snippets against stored rows by (Kind, ContentHash, Ordinal): since content is unchanged, every row matches and keeps its existing embedding (no wasted provider calls), but EmbeddingPayload and Identifiers are unconditionally refreshed on a match — the same self-healing fix in PostgresLibraryDocumentSnippetStore/PostgresPublicDocumentSnippetStore that stops a parsing/extraction logic change from being silently invisible on already-indexed content.

MCP Tools

Two MCP tools expose snippet search to agents, split by kind so an agent can target the retrieval mode it actually needs:

Both tools share the same required/optional parameters:

Results are formatted as markdown, grouped by source document, with a pointer to read_document_by_path for the full document. A degraded-mode search appends a single note line stating that semantic ranking was unavailable.

Prefer search_sections/search_code_snippets over reading whole documents when only a focused passage is needed — they are the finer-grained sibling of search_documents, which ranks whole documents rather than snippets within them.

HTTP API

GET /api/v1/orgs/{orgId}/libraries/{name}/snippets/search — org and library are both route segments, matching every other library/document endpoint's convention.

An unknown library name returns 404; a missing/invalid kind or query returns 400.

Each result row includes the same fields the MCP tools use to rank internally, so this endpoint doubles as a tuning workbench — it exists to let a person inspect and tune ranking, not just to serve results:

{
  "documentPath": "guides/getting-started.md",
  "documentTitle": "Getting Started",
  "headingPath": "Guide > Install > Linux",
  "header": "Install",
  "language": null,
  "content": "...",
  "score": 0.031,
  "vectorRank": 1,
  "textRank": 3,
  "identifierMatch": false,
  "degraded": false
}

vectorRank/textRank are 1-based ranks within each arm; 0 means the row was not a hit in that arm at all (not literal rank zero). score is the fused RRF score, higher first.

The existing library "Test search" panel in the web UI (src/web/src/views/libraries/DocumentSearchPanel.vue) exposes this endpoint through a three-mode toggle (Documents / Sections / Code) alongside the existing document search, rendering the score components so ranking behavior can be inspected visually.

Configuration

AppSettings:SnippetIndexing

AppSettings:Llm:Embeddings

Local development

dotnet user-secrets set "AppSettings:Llm:Embeddings:ApiKey" "<fireworks-key>"

Same Fireworks account key used for the chat-completion Fast tier authorizes both — one account key, two separately-configured settings slices. Without this key set (or with Enabled: false), the sweep still parses and full-text-indexes every document; only the vector arm is inactive, and search runs degraded.

Production

The embeddings API key ships as a Secret Manager secret (AppSettings__Llm__Embeddings__ApiKey), mounted as an environment variable on both the web service and worker pool — see Google Cloud Configuration.