pii-masking-pipeline-builder

Overview

Authoring a masking pipeline requires three classifications per field (regulatory regime, operator, reversibility) and one global decision (pipeline ordering + audit hooks). This workflow produces a deployable YAML spec that downstream tools execute:

• presidio-pii-detection runs the detector.
• Faker substitution operators (references/faker-masking-operators.md) / synthea-healthcare-data supply substitute values.
• The verification pass (below) audits the output for leaks.

When to use

• Promoting a production-data snapshot to a staging environment.
• Building a recurring refresh pipeline that masks nightly.
• Establishing a per-table contract that PR-reviewers can audit.

Step 1 - Inventory the source

Enumerate every column / field in the source dataset. For each, record:

Column	Type	Sample value	Cardinality	Cross-table join?
users.email	string	alice@acme.com	high	yes (joins events)
users.ssn	string	123-45-6789	high	no
users.dob	date	1985-03-14	medium	no
users.zip	string	02139	low	no
users.country	string	US	very low	no

A schema introspector can produce the first columns; cardinality and join graph need a quick analytical pass.

Step 2 - Classify each field

Look up each column in the cross-regime catalog (references/pii-categories.md) and record which regulatory regime(s) apply. Include linkable fields explicitly (NIST 800-122 §2.2).

Column	GDPR	CPRA SPI	NIST	HIPAA	Risk
users.email	✓	-	✓	✓ #6	direct
users.ssn	✓	✓	✓	✓ #7	direct, high-sensitivity
users.dob	linkable	-	linkable	✓ #3	linkable
users.zip	linkable	-	linkable	✓ #2 (sub-state)	linkable
users.country	-	-	-	-	non-PII

Any field marked direct OR linkable enters the masking scope. A field marked only "linkable" still gets masked because it identifies in combination with others (Sweeney 87% rule, see references/pii-categories.md).

Step 3 - Pick an operator per field

Match each field to a technique in the techniques catalog (references/masking-techniques.md). Decision tree:

1. Must round-trip for authorised consumer? (e.g., payments) → Tokenisation (vault) or FPE.
2. Must join across tables? → Deterministic substitution or salted hash with consistent salt per source value.
3. Continuous numeric needing analytics? → Number variance.
4. Categorical demographic for analytics? → Generalisation + l-diversity.
5. Free text potentially containing PII? → Presidio detect → replace / redact.
6. No analytical or operational use? → Nulling / redaction.

Column	Operator	Rationale	Reversible?
users.email	Faker substitution (deterministic via hash-seed)	Joins across tables; need referential integrity	Yes (via salt vault)
users.ssn	Tokenisation (vault)	Strict regulator scope; round-trip needed for auth	Yes (via vault)
users.dob	Generalisation to year	Analytics needs age bracket, not exact DOB	No
users.zip	Truncation to first 3 digits	HIPAA Safe Harbor #2 rule (>20k pop only)	No
users.country	Pass-through	Not PII	n/a

Step 4 - Pseudonymisation vs anonymisation gate

For each masked field, mark whether the result remains personal data under GDPR Art. 4(5):

• Reversible techniques (deterministic substitution, tokenisation, encryption, salted hashing with retained salt) = pseudonymised → output is still personal data → masking pipeline output is still in GDPR scope.
• Irreversible techniques (random substitution, generalisation, nulling, redaction) = anonymised → potentially out of GDPR scope (subject to Recital 26 reasonable-likelihood test).

Document the gate decision per dataset:

output_classification: pseudonymised  # GDPR scope retained
gdpr_lawful_basis: Article 6(1)(f) legitimate interests
retention: 90 days
access_control: only-dev-environment-team

vs.

output_classification: anonymised
gdpr_lawful_basis: out-of-scope per Recital 26
retention: indefinite
access_control: open

The author cannot claim "anonymised" if any reversible technique is in the pipeline.

Step 5 - Compose the pipeline

A standard order:

1. Schema-aware mask - apply per-column operators from Step 3 (deterministic, fast, no NER needed).
2. Free-text detect + mask - for any string column wider than ~50 characters, run presidio-pii-detection to catch embedded PII (e.g., a user-typed comment that contains an email).
3. Audit hook - sample N rows of output and run the verification pass (below) before declaring the run complete.
4. Manifest - emit a per-run manifest recording: pipeline version, source snapshot ID, row count in / out, operator versions, salt vault key version.

Step 6 - Emit the YAML spec

Recommended shape - consumable by a generic pipeline runner:

pipeline:
  name: users-staging-refresh
  source:
    type: postgres
    connection: $PROD_RO_DSN
    schema: public
    table: users
  classification:
    output: pseudonymised
    regimes: [gdpr, cpra, hipaa]
  fields:
    - column: email
      operator: deterministic_substitution
      provider: faker
      provider_method: internet.email
      seed_strategy: hash(salt + value)
      salt_ref: vault://masking/users.email
    - column: ssn
      operator: tokenisation
      vault: vault://masking/users.ssn
    - column: dob
      operator: generalisation
      params:
        granularity: year
    - column: zip
      operator: truncation
      params:
        keep_chars: 3
        from: start
    - column: country
      operator: passthrough
  free_text_columns:
    - notes
    - support_message
  free_text_detector:
    type: presidio
    language: en
    score_threshold: 0.45
    entities: [PERSON, EMAIL_ADDRESS, PHONE_NUMBER, US_SSN, CREDIT_CARD, IP_ADDRESS]
    on_detect: replace
  audit:
    sample_rows: 100
    fail_on_critic_block: true
  output:
    type: postgres
    connection: $STAGING_RW_DSN
    schema: public
    table: users
  manifest:
    write_to: s3://masking-manifests/${run_id}.json

Step 7 - Worked example

A SaaS app refreshes its staging from prod nightly. Source has 4M users with 22 columns, 3 of which are free-text. Synthesised spec:

pipeline:
  name: prod-to-staging-nightly
  source: { type: postgres, table: users }
  classification: { output: pseudonymised, regimes: [gdpr, cpra] }
  fields:
    - { column: user_id, operator: passthrough }                  # internal opaque ID
    - { column: email, operator: deterministic_substitution,
        provider: faker, provider_method: internet.email,
        seed_strategy: hash(salt + value), salt_ref: vault://prod/email }
    - { column: full_name, operator: substitution,
        provider: faker, provider_method: name }
    - { column: phone, operator: substitution,
        provider: faker, provider_method: phone_number }
    - { column: address_line1, operator: substitution,
        provider: faker, provider_method: address }
    - { column: country, operator: passthrough }
    - { column: language, operator: passthrough }
    - { column: created_at, operator: passthrough }
    - { column: last_login_at, operator: passthrough }
    - { column: signup_ip, operator: encryption,
        params: { algo: fpe-ff1 }, key_ref: vault://prod/ip-fpe }
    - { column: notes, operator: free_text_mask }
  free_text_detector:
    type: presidio
    language: en
    score_threshold: 0.5
    on_detect: replace
  audit: { sample_rows: 100, fail_on_critic_block: true }

Pipeline classification: pseudonymised (email is deterministic, IP is FPE-encrypted with key retained). The user explicitly accepts that this output remains in GDPR scope.

Verification pass - auditing the masked output

Every pipeline run ends with an adversarial leak audit that re-detects PII in the masked output and challenges the pipeline's "clean" claim. Audit checklist:

1. Sample the output. N rows (default 1000) uniformly; 10 000+ for high-risk datasets: shuf -n 1000 masked-users.csv > sample.csv.
2. Detect. Re-run presidio-pii-detection against the sample with the strictest entity set and a lower score_threshold than the pipeline used during masking (e.g. 0.4 vs 0.5+) - the audit should catch hits the pipeline filtered out as low-confidence. Scan every column, including declared-non-PII passthrough columns.
3. Cross-reference. For each hit: was the column in the spec, and what operator ran? A hit in an unclassified or passthrough column is a leak unless the column is genuinely non-PII per references/pii-categories.md. Was the operator appropriate per references/masking-techniques.md? Did it silently fail (literal "NULL" string still detected)?
4. Classify by regime. Map each leak to its regulator(s) via the cross-jurisdiction table; a leak counts against every regime listing it.
5. Verdict.

BLOCK if any hit is:
  - A CPRA SPI / GDPR Art. 9 / HIPAA Safe Harbor identifier
  - A direct identifier in a column where the pipeline declared
    "anonymised" output
  - A hit in a column the pipeline didn't classify

PASS-WITH-CAVEATS if:
  - Only linkable (not direct) leaks remain
  - The pipeline output is declared "pseudonymised" (GDPR scope
    retained, so linkable hits are tolerable when access-controlled)

PASS if:
  - Zero hits, OR
  - Only false-positive hits that the analyst flags as
    Presidio-noise (e.g., a fake-shaped string that's actually a
    UUID)

The audit refuses to mark a run "pass" if any CPRA SPI / GDPR Art. 9 / HIPAA Safe Harbor identifier appears unmasked, if the spec lacks a manifest (no provenance = no audit trail), or on a "we'll fix it next time" promise - leaks block the promotion. Findings are suppressed only via an explicit per-row waiver. Re-audit on every pipeline-spec change; detection is heuristic (Presidio's recogniser ceiling), so custom PatternRecognizers cover in-house ID formats and full-dataset scans replace sampling for comprehensive audits.

Anti-patterns

Anti-pattern	Why it fails	Fix
Per-column operator without referential check	Joins break after masking	Group columns that share keys; apply deterministic operators consistently
Free-text columns skipped	Embedded PII (user-typed emails) leaks	Always run Presidio on any string column > ~50 chars
Claiming "anonymised" when any reversible op is in the pipeline	False GDPR compliance claim	Audit the pipeline; pseudonymised if any operator is reversible
No audit step	Operator failure or recogniser drift goes unnoticed	Always sample output and run the verification pass
Salt vault key shared across pipelines	Salt-rotation breaks every downstream pipeline at once	Per-pipeline salt; rotate independently
No manifest	Cannot reproduce a past run; auditors can't trace lineage	Always emit manifest with version IDs
Pipeline runs on prod-write connection	Risk of writing masked data back over prod	Strict source = read-only DSN; output = staging-write DSN

Limitations

• No automated regime mapping. The author must classify each field against the regimes (Step 2) - the tool doesn't infer it.
• Pipeline runners vary. This skill emits a generic YAML; the team needs a runner (custom Python / dbt / Spark job / commercial tool) to execute it.
• Free-text detection is heuristic. False positives + negatives are real (see presidio-pii-detection limitations).
• Doesn't cover application-layer PII generation. A pipeline masks data at rest; the application might still write fresh PII to logs at runtime - pair with log-masking middleware.

References

• Cross-regime PII catalog (GDPR / CCPA-CPRA / NIST / HIPAA citations): references/pii-categories.md, with full per-regime enumerations in references/regime-catalogs.md.
• Masking-operator catalog (ISO/IEC 20889 + Presidio citations): references/masking-techniques.md, with privacy models in references/privacy-models.md.
• Faker as a masking substitution operator (seed-exposure matrix): references/faker-masking-operators.md. Fixture-style fake data built from nothing lives in the qa-test-data plugin's faker-data skill.
• Composes: presidio-pii-detection, synthea-healthcare-data.
• GDPR Art. 4(5) pseudonymisation - gdpr-info.eu/art-4-gdpr/.
• NIST SP 800-188 privacy models - csrc.nist.gov/pubs/sp/800/188/final.