Real bank-narration corruption + missing broker-holdings ingestion — three real bugs found auditing self.ritesh's live data

Decision

User reported three flaws after browsing their own real data (self.ritesh@chitragupt.ai, not a demo persona): Expense savings looked wrong, transactions looked wrong, and Portfolio showed none of their real holdings. All three traced to real, fixable bugs — not the arithmetic itself, which was already internally consistent.

1. Merchant name corruption (_categorise-narration.ts) — extractMerchant()'s UPI regex grabbed the literal DR/CR debit/credit marker as the merchant name instead of the real payee three fields later (UPI/DR/<ref>/<payee>/<bank>/<vpa>/...). Affected 1,478 of 2,496 real expense rows (59%) for this user alone. Fixed with a targeted regex (UPI\/(?:DR|CR)\/\d+\/([^/]+)\/) that skips the marker and reference number to the real payee field.

2. Bank-statement PDF narration truncation (packages/parser/src/bank-statement/rows/sbi.ts, Format C2/C3) — pdfjs flattens a whole PDF page into one line with no row/column separators; SBI's Ref No./Cheque No. column glyphs land AFTER the Debit/Credit/Balance amounts in that flattened order instead of before them. The row extractors only used that trailing span to recover the row's year and then discarded the rest — silently truncating narration to just the boilerplate prefix ("TO TRANSFER- TRANSFER TO", "by debit card-") for every affected row, losing the payee/location entirely. Fixed by folding the (year-stripped) trailing span back into the narration. Confirmed via apps/functions/scripts/rescanDocument-equivalent reprocessing that this is idempotent — re-running extraction against the same source PDF regenerates correct data, no migration script needed. Repaired self.ritesh's 13 confirmed bank-statement documents this way (0 data loss — transaction counts identical before/after, only narration text improved).

3. Missing broker-holdings ingestion (new broker-holdings-statement doc type) — Chitragupt V1 had no document type that captures a broker's CURRENT equity/MF holdings with real market value. broker-trade-log is metadata-only (no money fields — trade_count only, by design, V2 scope). broker-cg is realised P&L, not live positions. A real Zerodha "Holdings" export (2026-04-zerodha-holdings.xlsx, self.ritesh's actual upload) was misclassified onto broker-cg via a stopgap classifier pattern whose own comment said "V2 will split into a broker-holdings form_type" — so the user's real ~₹25L equity+MF portfolio (500+ real trades, real capital gains) was completely invisible on the Portfolio page; only PPF/EPF/ELSS showed.

Built the deferred V2 feature: new broker-holdings-statement form type (schema, classifier, parser for Zerodha Console's Combined-sheet export), and a NEW multi-instrument portfolio-position mapper (mapToPortfolioPositions) — every other form type collapses to one synthetic position per statement; this one emits one real portfolio_holdings/portfolio_positions row per instrument (keyed by ISIN), the schema's originally-intended shape that no ingestion path had ever actually used. Reclassified and reprocessed self.ritesh's real document: Portfolio NAV went from ₹14.05L (PPF+EPF+ELSS only) to the real ₹39.05L (adding ₹16.86L direct equity + ₹7.88L equity MF + ₹0.29L debt MF across 75 real instruments).

Bonus fix found during verification: computeAndPersistPortfolioReview's Firestore write used {merge: true} on a FULLY-recomputed review document. Firestore merges nested map fields (by_sector_paise, by_asset_class_paise) key-by-key rather than replacing them wholesale, so stale sector/asset-class keys from a prior computation silently accumulated forever instead of reflecting current truth — caught only because reprocessing with a corrected sector-mapping left BOTH the old raw broker sector strings ("BUILDING MATERIALS") and the new canonical enum values ("materials") in the same map, failing schema validation. Fixed to a full overwrite — a value that IS the entire current truth, not a delta, must never be merge-written.

Why

User's own words: "I still see many flaws" — the demo-persona wireframe-alignment work earlier this session hadn't touched the pillar's data-quality against REAL uploaded documents. This audit is the real-corpus counterpart to that session's synthetic-data verification.

Impact

• apps/functions/src/triggers/firestore/ledger-mappers/_categorise-narration.ts — extractMerchant() fix.
• packages/parser/src/bank-statement/rows/sbi.ts — Format C2/C3 trailing-fragment fix; new sbi.test.ts.
• packages/shared/src/schemas/document.ts, document-types-registry.ts, form-type-category.ts — new broker-holdings-statement form type (schema, registry row, category map).
• packages/parser/src/broker-holdings-statement/ (new) — parser, temporal, schema, tests.
• packages/parser/src/classify.ts — new classifier entry; removed from broker-cg's stopgap pattern.
• packages/parser/src/dispatcher.ts — wired both switches.
• apps/functions/src/portfolio/portfolio-position-mapper.ts — new mapToPortfolioPositions (plural), normaliseSector() (broker free-text → canonical PortfolioSectorSchema).
• apps/functions/src/triggers/firestore/handlers/persist-confirmed-document.ts — branches to the plural mapper for the new form type.
• apps/functions/src/portfolio/portfolio-review-engine.ts — removed {merge: true} on the review-doc write (real bug, not specific to this feature).
• apps/mobile/src/services/documents.ts — FORM_TYPE_LABELS row (exhaustive map).
• pillar-portfolio, inbox-document-types — new capability documented.
• Not addressed: broker-trade-log/broker-ledger-statement/broker-dividend-log still carry no per-trade line items (V2 scope, per their own schema comments) — the new holdings statement type is a parallel, independent ingestion path, not a fix to those.
• Production note: any real user who already confirmed a bank-statement PDF before this fix has the same narration corruption baked into their ledger_entries. This session repaired one user's data by hand (script-driven rescanDocument-equivalent reprocessing); a production backfill job would need the same treatment at scale — not built this session.

Status

Active

Sources

• Real document: ~/git/personal/documents/docs/self_ritesh/investments/zerodha/2026-04-zerodha-holdings.xlsx
• apps/functions/src/triggers/firestore/ledger-mappers/_categorise-narration.ts
• packages/parser/src/bank-statement/rows/sbi.ts
• packages/parser/src/broker-holdings-statement/
• apps/functions/src/portfolio/portfolio-position-mapper.ts