yarn db:self round 3 — cross-document income double-counting, still real after two prior fix rounds

Decision

User reported /expense's "Where your money went each month vs what you earned" chart still showed implausible monthly income (peaks of ₹8.4L–₹8.9L in single months) and /portfolio still didn't match .context/designs/web/portfolio/portfolio.html, despite same-day fixes in 3eaeab41 (salary triple-count) and c66015b3 (bank-statement dedup, portfolio chart summing bug). A fresh, from-scratch yarn db:seed && yarn db:self (434 real docs, self.ritesh, FY2018-19 → FY2026-27) traced the residual distortion to bugs those two commits did not cover, fixed them, and re-verified with two more full wipe-and-reseed cycles.

Real bugs found and fixed this session

1. Expense/Tax: Torc/Randstad consulting income double-counted — once as an invoice, once as the wire-transfer evidence for that same invoice. professional-receipt.ts (invoice → professional_receipt_income) and foreign-remittance.ts (payout evidence → foreign_salary) are independent mappers with no cross-doc awareness. Real corpus proof: May 2025's ritesh-torc-may-2025.pdf invoice (₹3,31,500.09) is matched to the paise by two payout-evidence uploads dated 2025-05-07 + 2025-05-21 (₹1,64,958.68 + ₹1,66,541.51 = ₹3,31,500.19) — same underlying payment, two documents, two ledger rows. This pattern repeated across ~7 of 12 months in AY2026-27. Fixed with a new cross-document reconciliation pass, reconcileProfessionalReceiptsAgainstRemittances in persist-confirmed-document.ts: for each professional_receipt_income row, look at foreign_salary/foreign_dividend rows in the 35 days before its invoice date (same-AY, subset-sum matched within 3% tolerance — not a flat month-sum, so an unrelated remittance in the same window doesn't block a match), and demote the invoice to entry_class: "other" when a match is found — bank evidence is preferred as the cash-basis source, the invoice is accrual paperwork for the same event. Also added the missing half of the fix: Tax's income-summary.ts bucket-routing loop previously ignored entry_class entirely (routed purely by entry_type), so an entry_class: "other" demotion — the same mechanism already used for non-interest bank credits since 3eaeab41 — silently did NOT exclude a row from Tax's income summary the way its own code comment claimed ("excluded from every pillar KPI"). Added an explicit entry_class === "other" skip before bucket routing. AY2026-27 income fell from a peak single-month ₹8.9L to a ~₹3.2–5.3L range consistent with the prior years' trend.
2. Inbox/Expense/Tax: a document with zero extracted identity signal skipped owner-validation entirely and silently auto-attributed to the uploader. run-and-persist.ts only calls validateDocumentOwnership when identityExtracted is non-empty OR the form is a self-doc type — a bank-interest-cert whose OCR found no name at all (identity_extracted: {match_confidence: 0}, no name field) fell through both conditions and landed straight on unreviewed (auto-owned), never reaching pending_identity_confirmation. Real corpus proof: fy2025-26-rakesh-federal-interest-balance-cert.pdf (the father's bank cert, per the earlier c66015b3 real-data audit's family-member finding) auto-confirmed under self.ritesh and rolled ₹12,528 of someone else's interest income into his Expense/Tax totals — the exact failure mode c66015b3's name-mismatch guard was built to catch, just reached through a different gap (missing name, not mismatched name). Fixed at the root: added isAccountHolderForm to packages/shared/src/schemas/document-types-registry.ts, derived from each form type's existing identity_type (bank_savings, loan_home, loan_education, loan_ev — not employer, where the same field holds an employee code, and not institutional counterparties like broker/mf_amc). run-and-persist.ts now also runs validateDocumentOwnership for these form types even on empty extraction; the matcher already degrades an empty identity to no_match (verified — no code change needed there), which correctly routes to pending_identity_confirmation instead of silent trust. This is a production pipeline fix, not seed-script-only — it protects every real user, not just the demo corpus. Also hardened seed-self-user.mjs's existing name-mismatch guard to fail closed (leave at pending_identity_confirmation) when identity_extracted.name is missing entirely, matching how a cautious real user would treat "Confirm this is yours?" with nothing to go on.
3. Expense: a ₹5L cheque transfer to a family member's account counted as personal consumption. _categorise-narration.ts's transfer-detection regex is deliberately narrow (its own comment: matching generic TFR/UPI tokens would zero out most real expense rows — verified live, WDL TFR alone appears 1,284 times in the corpus including plain merchant spend like Swiggy). But "CHQ XFER WD CHEQUE TRANSFER TO 0020087229644 OF Mrs. CHAYA SHANTILAL F …" — a cheque explicitly naming a beneficiary account number and holder — is a narration shape no merchant/POS transaction ever produces. Added a narrow rule matching TRANSFER TO <account-number> OF specifically (verified: exactly 1 match in the whole corpus, so no risk of over-matching). AY2026-27 Expense fell from ₹38.91L to ₹33.91L; Saving rose from 14.5% to 25.2% of income.

Verified

Three full yarn db:seed && yarn db:self wipe-and-reseed cycles (each ~4–7 min against the emulator) after each fix, cross-checked directly against ledger_entries/expense_reviews/portfolio_reviews Firestore docs and the live /expense and /portfolio pages in-browser:

• AY2026-27 monthly income now ranges ₹3.2L–₹5.3L (previously spiked to ₹8.4L–₹8.9L); annual total ₹45.36L.
• fy2025-26-rakesh-federal-interest-balance-cert.pdf now correctly sits at pending_identity_confirmation.
• The Chaya Shantilal cheque row now carries entry_class: "transfer" / expense_category: "transfers".
• yarn typecheck, yarn lint (0 errors, pre-existing warnings only), and both shared (196 tests) and chitragupt-functions (546 tests) suites pass.

Found, NOT fixed — needs a human decision or is out of scope for this session

• Two real Torc invoice PDFs (ritesh-torc-march-2026.pdf, ritesh-torc-april-2026.pdf) print DATE: 31-March-2025 on the invoice itself, a full year off from their filename/folder — confirmed by reading the raw PDF text directly (.context/raw-equivalent source under ~/git/personal/documents/docs/self_ritesh/, immutable). This is a genuine data-entry error in the source document (very likely a copy-pasted invoice template where the year field was never updated), not a parser bug — the AY-deriving logic (deriveAyFromDate off the printed invoice date, a deliberate design choice from an earlier fix) correctly reflects what the document says. Both invoices land in AY2025-26 instead of their intended AY2026-27/2027-28, inflating that AY's March by ~₹8.68L. Not code-fixable without either editing an immutable raw source or inventing an unjustified override of a real printed date — flagged for the user's awareness, not fixed.
• /portfolio's Portfolio-vs-Nifty chart's AY2024-25 bar reads as disproportionately tall. Traced to a real, single-source realised_short_term_paise: ₹4,24,969.47 STCG figure from that AY's sole confirmed broker-cg document (no duplicate document found — checked directly) plotted on the chart's own small-scale yBar axis (by design, matches the wireframe's dual-axis Chart.js config in portfolio-review.html). This is consistent with the already-documented V1 scope gap ("direct equity/demat holdings have no path into Portfolio positions" — capital gains booked via broker-cg are structurally disconnected from the 3-holding EPF/PPF/ELSS position list shown elsewhere on the page) rather than a new bug. Not changed.
• Residual interest/dividend cross-doc double-count risk (AIS vs bank-interest-cert, AIS vs broker-dividend-log/mf-cas) remains an open, previously-documented V2 gap — small paise-scale amounts in this corpus (~₹27k/year), left as-is per the existing broker-dividend-log.ts code comment. Fixed 2026-08-10, see 2026-08-10-ais-vs-evidence-doc-income-dedup.

Impact

• pillar-expense, pillar-portfolio, pillar-tax — no canonical value changed; only implementation bugs against those already-canonical definitions.
• New shared export: isAccountHolderForm in packages/shared/src/schemas/document-types-registry.ts.
• income-summary.ts now honours entry_class: "other" as a cross-pillar "audit-only, exclude everywhere" signal, not just an Expense-pillar one — this generalizes the precedent 3eaeab41 set for bank_credit to any future mapper that needs the same demotion.
• persist-confirmed-document.ts gained a second cross-document reconciliation pass (alongside the existing dedupAgainstExistingLedger span dedup from c66015b3) — both run on every confirm, in either upload order.
• run-and-persist.ts's owner-validation gate is now form-type-driven via the shared registry rather than solely extraction-driven — closes the "nothing extracted → silently trusted" gap for bank/loan documents for all users, not just this demo.
• Extends 2026-08-03-dev-self-validation-round-2 and 2026-07-25-dev-self-validation-findings; does not supersede either.

Status

Active.

Sources

• This chat session, 2026-08-06. Direct Firestore inspection (ledger_entries, expense_reviews, portfolio_reviews, documents) against three successive full yarn db:seed && yarn db:self reseeds, live-browser verification of /expense and /portfolio, and direct pdftotext reads of the two raw Torc invoice PDFs.