The Ingestion Flow

When a document enters a workspace, it moves through a pipeline that extracts its text, detects sensitive data, and produces a redacted rendition — leaving a full audit trail of what was found and what was done about it. This page describes that flow in general terms; the glossary defines each term precisely.

The pipeline is driven by two kinds of plan, and keeping them apart is the key to understanding everything else:

• An analysis plan decides what is detected — which categories are looked for, which detectors run, and the minimum confidence worth recording. Analysis plans are defined estate-wide.
• A transformation plan decides what happens to what was detected — which categories are redacted or substituted, at what sensitivity and confidence thresholds, and in what style. Transformation plans are owned by the workspace.

How content arrives

Ingestion starts in one of three ways:

1. Discovery. When a source is imported into a workspace (or refreshed), Enprivacy enumerates its contents — mailbox messages, bucket objects, drive files — and creates an item for each new or changed entry. Refreshes are normally incremental, picking up only what changed since last time; a full sync additionally reconciles items whose originals have been removed. Sources can also refresh on a schedule.
2. Upload. A file uploaded directly through a source link becomes an item immediately.
3. Manual re-analysis. The New analysis action on any item re-runs the pipeline on demand, with an analysis plan of your choice — useful after a plan has been improved.

Every new or changed item is then processed as a workflow run, visible (and retryable) under the workspace’s Workflows page.

The pipeline

flowchart TD
    A["Source imported, refreshed,<br/>or file uploaded"] --> B["Item created"]
    B --> C["<b>Extract</b><br/>text pulled from the file<br/>(OCR for PDFs, scans, office files)"]
    C --> D["<b>Detect</b><br/>the analysis plan's detectors<br/>each scan the text independently"]
    D --> E["<b>Analyse</b><br/>detections merged into one analysis<br/>with a marker per sensitive span"]
    E --> F["<b>Resolve entities</b><br/>people and organisations matched<br/>to known identities"]
    F --> G["<b>Transform</b><br/>the transformation plan chooses what to<br/>redact and substitutes each span"]
    G --> H["<b>Render</b><br/>redacted document written<br/>in the original format"]
    G --> I["<b>Embed</b><br/>redacted content indexed<br/>for search and chat context"]
    H --> J["Artefact stored in the<br/>workspace's redacted repository"]

Extract

The item’s text is pulled out of its file. Plain text passes straight through; PDFs, scans, and office documents go through the OCR service, which converts them to clean text. Emails contribute their message text. Everything downstream works on this extracted text.

Detect and analyse

The analysis plan’s detectors — pattern matchers, gazetteers, machine-learning models, LLM-based detectors — each scan the text independently. Their raw findings are then merged into a single analysis: overlapping detections are reconciled, confidence from multiple detectors is combined, and each resulting sensitive span becomes a marker recording its category and confidence. Findings below the plan’s confidence floor, or in categories outside the plan’s scope, are set aside (but kept for audit).

Detectors are isolated from one another — one failing doesn’t sink the run. If every detector fails, the analysis is recorded as impaired rather than silently empty.

Resolve entities

Markers for people and organisations are matched against the estate’s known entities, so the same person is recognised as the same identity wherever they appear — across documents, sources, and workspaces.

Transform

The workspace’s transformation plan now decides what to do about each marker. It selects which categories to redact and applies its own sensitivity and confidence thresholds; where several markers cover the same span, the strongest one wins so each span is handled exactly once. Every selected span is replaced with a substitution — either a round-trippable token (the same value maps to the same token wherever it appears in the workspace) or a human-readable numbered tag. The result records the full before-and-after picture: what was exposed, and what remains.

Render and embed

Finally the pipeline produces the outputs:

• A redacted rendition of the document in its original format — the artefact you see on the item’s Artefacts tab — stored in the workspace’s redacted repository.
• Embeddings of the redacted content, which power search and give chats their knowledge of the workspace.

Note

Emails are analysed and transformed like everything else, but no redacted file is rendered for them — they get an analysis, a transformation preview, and embeddings, without an artefact on the Artefacts tab.

An item’s page presents this whole history: the original content, its analyses, its transformations, and its artefacts, with the Topology graph showing how each record derives from the one before.

Databases

Structured data follows the same idea with a database-shaped pipeline. Analysis happens in two tiers: a fast heuristic pass over the schema itself, then a deep pass over sampled rows using the very same marker machinery as documents.

flowchart TD
    A["Host imported into<br/>the workspace"] --> B["<b>Discover</b><br/>schemas and tables enumerated"]
    B --> C["<b>Classify</b><br/>columns heuristically flagged<br/>from names and types"]
    C --> D["<b>Sample</b><br/>rows collected from a table<br/>(live sample or uploaded file)"]
    D --> E["<b>Analyse</b><br/>each sampled record scanned with<br/>the same detectors as documents"]
    E --> F["<b>Roll up</b><br/>findings aggregated per column —<br/>where sensitive data actually lives"]

The result is a column-level map of sensitive data: each column shows what categories were found in its sampled values, at what confidence, feeding the same risk reporting as content.

Where the results surface

• Risk tiers. Every item and source carries a sensitivity score mapped to a risk tier (from none to critical) — these drive the ranked source cards on the Content overview and the striped high-risk rows in tables.
• Residual risk. The workspace’s Residual risk report pools every marker into a before-and-after view: what detection found, versus what still survives the workspace’s transformation plans.
• Topology. The Topology graph shows the lineage of every record in the pipeline — which analysis came from which item, which artefact from which transformation.
• Chats. The same detect–analyse–transform machinery runs on every chat message before it reaches an AI model — see Chats and the AI Assistant.