Document search over a corpus walk-through: indexing pipeline, permissions and reindexing

Goal

Search over a corpus that lives in a system of record, with results that respect permissions and an index that never lags the source by more than a stated bound.

Prerequisites

A definition of a document (id, title, body, metadata, owner, visibility, updated_at), the languages involved, and a freshness bound the product accepts.

Steps

1. Constraints: the source stays authoritative and the index is derived and rebuildable; a user must never see a snippet of a document they cannot open; per-keystroke querying is a product decision, not a default.
2. Components: an indexer fed by change events (an outbox) or by polling updated_at; an index store, starting with the database's own full-text index and moving to a dedicated engine when ranking features or scale demand it; a query service that parses input, applies permission filters, ranks and highlights; a reconciler that compares source and index.
3. Data model: search_doc(doc_id, version, text fields, acl_keys[], updated_at, indexed_at); the ACL keys (owner id, group ids, public) are indexed fields so filtering happens inside the engine rather than after ranking. In PostgreSQL the documentation states that GIN indexes are the preferred text search index type, holding an entry per lexeme with a compressed list of locations.
4. Index management: version the index (docs_v3) and switch an alias after a full rebuild; keep the indexer idempotent (upsert by doc_id, ignore versions older than the indexed one); handle deletes as explicit events, never by absence.
5. Ranking: start with the engine's default relevance plus a title weight and a recency term; log query, result ids and click position so a later ranking change can be judged against the log.
6. Failure modes: missed change events (the reconciler compares counts and maximum updated_at per range and reindexes the difference); permission changes not propagated (treat ACL edits as document updates); very long or wildcard-heavy queries (cap length, disallow leading wildcards); reindexing under load (throttle, read from a replica); snippets exposing fields the visible document hides (index only what may be shown).
7. Measure: index lag (source updated_at minus indexed_at), query latency, zero-result rate, share of queries with a click in the top results, drift found by the reconciler, full reindex duration.
8. Not first: semantic or vector search, synonyms and spelling correction, personalised ranking, autocomplete, cross-language search.

Expected result

A document edited in the source appears within the freshness bound, disappears when deleted or hidden, and a rebuild from scratch is a routine operation rather than an incident.

Limits and test basis

Proposed design, no measurements. Relevance quality is not addressed beyond logging enough to evaluate it later; the freshness bound is a product input, not a property of the design.