Repository Content Ingest

Summary

Repository content ingest clones a GitHub repository, walks its Markdown files, and upserts each one as a document. It runs against two kinds of source:

• Public sources — one repository shared across every organization. Documents land in a single global table, and any library can subscribe to a subset of them.
• Private sources — a repository owned by one organization's library. Documents land in that library's own table, exactly like hand-authored documents.

Both kinds go through the same pipeline: acquire a workspace (a git clone), walk the files that match the configured filter, parse and hash each one, and upsert it with move detection. The only thing that differs is which database table the documents land in.

High-Level Flow

Trigger (manual HTTP call, or the scheduler on a timer)
  -> publish a "sync requested" message
  -> a message handler picks it up
  -> the handler asks a dispatcher to run the job
  -> the dispatcher acquires a workspace (git clone)
  -> the runner walks the workspace and upserts documents
  -> the dispatcher disposes the workspace (deletes the clone)

Every step after "publish a message" happens outside the original HTTP request. Triggering a sync returns immediately with a token; the actual clone and upsert work happens on a message queue worker.

Key Pieces

Public vs. Private Sources

Triggering a Sync

A sync can start two ways:

1. Manual trigger — an HTTP request to one of the endpoints above. Rate-limited (5 triggers per hour per source/library by default) and idempotent (a 409 if a sync is already queued or running).
2. Scheduled — IngestSchedulerHostedService ticks on a timer (5 minutes by default), atomically claims every source/library whose next_sync_at has passed, and publishes a sync request for each one. After a run finishes, next_sync_at is set to roughly one hour later with ±20% random jitter, so a fleet of sources doesn't all wake up in lockstep.

Both paths publish the same kind of message and go through the same handler and runner — a scheduled sync and a manual one are indistinguishable once they're queued.

The Clone: Two Layers of Filtering

Getting from "clone the whole repo" down to "only the Markdown files that matter" happens in two separate steps, at two different levels:

1. Git-level narrowing (coarse, extension-only). The clone itself uses a sparse checkout restricted to three extensions — *.md, *.mdc, *.mdx — combined with a partial clone (--filter=blob:none) and shallow history (--depth=1). This means git never downloads file content for anything that isn't a Markdown file, and never downloads more than the latest commit. For a large repository this is the difference between fetching a handful of megabytes and fetching the entire repository history.

This step is deliberately not directory-aware. It doesn't know or care about your include/exclude filters — it only narrows by extension.

2. In-process filtering (fine-grained, directory-aware). Once the clone lands on disk, RepositoryIngestRunner walks every Markdown file that survived step 1 and asks IngestFileFilter.IsIncluded whether it's actually in scope for this sync's configured filter. This is where include globs, exclude globs, and directory targeting apply.

Why split it this way? Translating an arbitrary include/exclude glob into git's own sparse-checkout pattern language (which has its own dialect, with cone and non-cone modes) is a second glob dialect to get right, for a bandwidth-only benefit. Worst case with the current design: a narrowly-filtered sync's clone briefly materializes some Markdown files on disk that the in-process filter then correctly excludes from the database — it never under-materializes what correctness requires. IngestFileFilter is the single source of truth for what actually gets ingested.

Filtering

Syntax

A filter is two lists of glob patterns: IncludeGlobs and ExcludeGlobs. The glob dialect is intentionally small — just two wildcards:

There's no special ** syntax. Because * already crosses directory boundaries, docs/* and docs/** would behave identically if ** existed — so it doesn't.

Matching is case-insensitive and always against the full repository-relative path, normalized to forward slashes with no leading slash (a path like .agents/plans/notes.md, not /.agents/plans/notes.md).

The Two Narrowing Rules That Always Apply

Regardless of any configured filter:

1. Only .md, .mdc, and .mdx files are ever ingested.
2. An empty include list means "everything" (subject to rule 1) — it is not the same as "include nothing."

Include and Exclude, Together

Worked Examples

These are real results from dry-running the ingest pipeline against public repositories — see Dry-Run Diagnostics below.

That last example is worth slowing down on, because of one file: .agents/skills/hosting-integration-authoring/resources/dashboard-ux.md. Its filename starts with d — but it was still included. The exclude glob .agents/skills/d* matches on the full path starting right after .agents/skills/, so it only excludes directories that themselves start with d. hosting-integration-authoring doesn't start with d, so nothing under it is excluded, no matter what the individual filenames look like.

Move Detection

Every file upsert resolves to exactly one of four outcomes, decided in this order:

If two or more existing documents share the same content hash as an incoming file with no path match (for example, two files that happen to be byte-for-byte identical), the ambiguous case falls through to Added rather than guessing which one moved. An exact path match always takes priority over a hash match, so this ambiguity only ever affects genuine moves where the destination content also happens to collide with something else in the same source.

After every file in a sync has been processed with zero failures, a deletion sweep removes any document in that source/library that wasn't touched by this run's stamp — it's no longer present upstream. If even one file failed to process, the sweep is skipped entirely for that run, so a partial failure never causes correct documents to be deleted because the run couldn't prove they were actually gone.

Dry-Run Diagnostics

Because a real sync writes to the database, testing filter behavior against a real repository used to mean either accepting real writes or writing a one-off script. Zeeq.Platform.Ingest.Diagnostics provides a permanent way to run the real pipeline — real clone, real parsing, real reads against existing data for accurate move detection — with all writes only logged, never persisted.

This is meant to be run through the zeeq-dotnet-repl skill against a live local server, not called from application code. It's how the worked examples above were produced: a real clone of a real public repository, filtered, walked, and logged, with nothing landing in Postgres afterward.

The Library UI

The Libraries view's create/edit slideover (LibraryFormSlideover.vue) is how an organization actually sets up repository-sourced libraries, rather than calling the endpoints above directly.

Create. An "Import from GitHub" switch reveals a source picker with two tabs: Public (a free-text GitHub URL — creates or reuses a docs_public_sources row keyed by normalized repo URL) and Private (a picker restricted to repositories already configured for the organization's GitHub App installation, since a private clone needs an installation token). Both tabs expose the same include/exclude filter text areas as the manual-library form. Submitting immediately follows up with a call to the same trigger-ingest endpoint the "Sync now" button uses, so the first sync queues without a separate user action — a 409/429 from that follow-up call (already in flight or rate-limited) is swallowed into an informational toast rather than failing the create.

Edit. Once a library exists, the slideover becomes tabbed:

A source-backed library's repo URL and source kind are permanently fixed at creation. There is no "change the URL" affordance anywhere in the edit form — the intended path to point a library at a different repository is to delete it and create a new one, which also avoids having to reconcile move/rename detection across two different upstream repositories.

The document editor (md-editor-v3) is read-only for any document whose origin resolves to "remote" — which covers every document that came from a sync, both public- and private-sourced. Hand-authored documents in a plain (non-source-backed) library remain fully editable.

Known Gaps

• Public-source union filters (planned, not built). A public source's effective filter should be the union of every subscribing library's filter. Today it only uses the source's own defaults.
• Public visibility re-verification (planned, not built). A public source that goes private upstream should be detected and quarantined before its content is ever confused with a repository Zeeq no longer has permission to read. Not implemented yet.
• Bounded concurrency is coarse. Each source kind has its own message channel with a fixed number of concurrent workers (noOfPerformers, a compile-time constant matching IngestSettings.MaxConcurrentPublic/MaxConcurrentPrivate's defaults). If those settings are ever tuned without a matching code change, the two will drift.
• A rare failure can strand a source at running. If the final write that resets a source back to idle fails, nothing else resets it — the scheduler only ever claims idle sources. Documented as an accepted, low-probability gap rather than fixed, since there's no real production traffic yet to justify the extra machinery.