End-to-end pipeline.

One feed entry, followed from the moment it is fetched to the moment it appears as a citation under an answer.

The whole path

Fig. Six stages, fetch to answer. Provenance is attached at the first stage and travels with the record.

1 · Fetch

The ingest-scheduler worker holds one interval per connector. When a connector fires it opens an ingestion_runs row first, then fetches upstream — so a crash mid-fetch still leaves a record that the attempt happened.

Cadence is per connector, not global — KEV moves on a different rhythm from NVD.
The run row carries started_at, a status, and the record counts it wrote.
A failed fetch leaves the previous records in place. It does not blank the source.

2 · Normalise

Each connector maps its upstream payload onto the STIX 2.1 shapes in packages/domain. This is where an upstream's idea of “malicious” becomes a typed verdict and a confidence number rather than a vendor-specific string.

Normalising at the edge is what makes the rest of the system source-agnostic: nothing downstream needs to know which feed a record came from in order to work with it — only to report it.

3 · Dedupe and link

The enrichment stage resolves the record against what is already stored. The same IP arriving from three feeds becomes one indicators row with three evidence rows, not three indicators.

Canonicalise the value — casing, trailing dots, URL normalisation.
Upsert the entity; append an evidence row per contributing source.
Link into the CTI graph: edges to actors, malware, campaigns, techniques.
Backfill an embedding so the record is reachable by semantic search.

Why evidence is a separate table

Because the verdict is derived. Keeping each source's claim as its own row means a verdict can be recomputed, disagreement between sources stays visible, and an answer can cite the specific claim rather than the conclusion.

4 · Store

Entities land in the canonical tables; documents and their chunks land in the knowledge base; relationships land in the graph. Anything tenant-scoped is written with the tenant's id and is thereafter reachable only under that tenant's row-level security context.

See data model for the table-by-table breakdown.

5 · Retrieve

A question is classified, a retriever mix is planned, and the retrievers run — lexical over full text, semantic over pgvector, graph over edges. Their results are fused into one ranked context.

what a retrieval trace looks like on the wire

"trace": {
  "retrievers": [
    { "name": "lexical",  "hits": 2, "latency_ms":  8 },
    { "name": "semantic", "hits": 1, "latency_ms": 14 }
  ]
}

The trace ships in the response. An unexpected answer is therefore diagnosable — you can see which strategy contributed, how much, and how long it took.

6 · Answer

The answer is generated from the retrieved context, and each claim carries a citation back to the retained document behind it — source, fetch time, confidence, TLP marking.

evidence, abbreviated

"evidence": [
  {
    "source": "abusech/urlhaus",
    "source_url": "https://urlhaus.abuse.ch/url/3000001/",
    "fetched_at": "2026-05-13T08:21:11Z",
    "confidence": 0.8,
    "tlp": "TLP:CLEAR",
    "snippet": "URLhaus: malware_download (status=online) — added 2025-12-01."
  }
]

What the pipeline does not do yet

Known gaps

Scheduled advisory sends are not dispatched. An advisory can be moved to queued with a scheduled_send_at, and the UI will show that time, but no worker consumes it. Nothing sends until someone presses send.