2026-07-09-dedup-content-hash-and-purge

Decision

Rebuild the document-dedup foundation on two invariants:

1. Tier 1 is byte-identical content, hashed from the raw upload bytes. content_hash = sha256(buffer). Not derived from storagePath or docId. Two independent user sessions that upload the same PDF from disk now collide correctly.
2. Tier 2 is per-form logical identity via documentInstanceKey. Persisted on the doc row as instance_key (nullable — envelope forms like 26AS/AIS/TIS/MF CAS/ITR-V return null and match on (form_type, ay) alone). Broker CG uses broker_name; Form 16 uses employer TAN; bank statements use account_no | period_start | period_end; etc.

And add the onDocumentPurged Firestore trigger — the missing counterpart to onDocumentConfirmed. It fires on delete or on a transition from confirmed to one of superseded / rejected_owner_denied / rejected_transfer_denied / rejected_duplicate, and:

• batch-deletes ledger_entries where document_id == docId
• batch-deletes portfolio_positions where document_id == docId
• calls recomputeAfterConfirm for the AY

Why

The 2026-07-06 fresh audit surfaced three related correctness holes that together made the "same document uploaded twice does not double-count" property false:

• Fingerprint was path-hashed, not byte-hashed — every fresh upload got a new storagePath and therefore a new fingerprint, making Tier 1 dedup near-dead code.
• documentInstanceKey was defined but never called — Tier 2 dedup used (form_type, ay, account_id, period_start, period_end) with account_id often null for brokers, causing cross-broker CG uploads to silently supersede each other.
• No purge trigger existed — a stale deleteDocument comment claimed a "Phase 5 onDocumentDeleted trigger" that never landed. Superseded and deleted docs stranded their ledger_entries in place, and tax-review-engine reads ledger by AY with no join to documents.status. Every re-upload of a confirmed doc double-counted its ledger contributions on the tax review.

Impact

Schema changes on documents/{docId}:

• fingerprint: z.string() removed (no legacy — this branch has no customer data).
• content_hash: z.string().nullable() added — full 64-char SHA-256 of the raw bytes. Nullable only for Free-tier view_only docs whose bytes are downloaded later by the backfill worker.
• instance_key: z.string().nullable() added — result of documentInstanceKey(formType, fields).

Runtime changes:

• ingest-vault-document.ts — hashes the downloaded buffer, computes instance_key from the parsed fields, passes both to findDedupLinks, and persists them on the doc row.
• parser/run-and-persist.ts — after LLM fallback resolves the form_type, re-stamps content_hash and instance_key so rescan / backfill agree with initial ingest.
• _lib/dedup.ts — rewritten. Tier 1 queries where("content_hash", "==", …). Tier 2 queries where("form_type", …).where("ay", …) .where("instance_key", …) — Firestore null-equality collapses envelope forms.
• New triggers/firestore/on-document-purged.ts and handlers/purge-document-fanout.ts.
• New composite index in firebase/firestore.indexes.json for the Tier-2 dedup query.

The deleteDocument callable's stale "Phase 5 trigger" comment is now accurate.

Status

Active.

Sources

• Fresh-perspective audit of the document lifecycle, 2026-07-09 (branch production-readiness-audit-v2).
• Related decisions: 2026-07-06-v1-launch-blockers (B3 wired dedup into ingest but on the broken fingerprint semantics; this decision fixes the semantics), 2026-07-04-temporal-anchor-per-category (feeds the Tier-2 ay axis).
• Related concepts / entities: upload-only, money-in-paise, pillar-tax, pillar-expense, pillar-portfolio.