dev:self validation — pipeline is mostly correct, four real bugs fixed

Decision

yarn dev:self (422 real docs) is now the canonical end-to-end validator for the ingest → parse → confirm → materialize → recompute pipeline. Diagnostic reads MUST use the real schema field names — misreads produced false "pillar broken" reports twice in the same session.

What actually happens with 422 real docs

Layer	State after seed
Ingest	422/422 uploaded, 100% bucketed on confirm
Parse	229 confirmed, 137 stuck at parse_pending, 45 rejected_format, 9 superseded, 2 pending_password
Ledger	5,337 entries, 100% with bucket populated, 0 zero-paise
Expense pillar	6/8 AYs populated (total_paise, monthly_series, top_merchants, by_category_paise)
Tax pillar	6/8 AYs populated (income_summary, regime_old/new.tax_paise, recommended: "new")
Capital gains	5 AYs with STCG/LTCG totals from broker-cg docs
Portfolio	3 positions (ELSS/EPF/PPF), top_holdings renders

Real bugs found + fixed this session

1. Tax dirty=true for AY 2019-20 / 2020-21 forever — engine early-returned on unverified AYs but never cleared the run doc's dirty flag. Fixed in apps/functions/src/tax/tax-review-engine.ts — now stamps dirty:false + skipped_reason:"unverified_ay_v1".
2. 137 docs stuck parse_pending after Anthropic 400 — credit_balance_too_low was classified as retryable api_error and burned all retries against a permanent billing state. Fixed in apps/functions/src/parser/llm-fallback.ts — new insufficient_credits reason, kept OUT of retryable set.
3. JPG/PNG rejected wholesale (45 docs) — parser had no image path. Fixed by adding image_jpeg/image_png DocumentSource variants and routing bytes through Claude vision content block in llm-fallback.ts. Web client still PDF/XLSX/ZIP-only — this fix only unlocks server-side ingest (seed, admin, future CA-portal).
4. False "portfolio empty" — initial diagnostic used non-existent field names (asset_class, scheme_name, units, nav_paise on portfolio_positions; months, total_spend on expense_reviews; gross_total_income_paise, recommended_regime on tax_reviews). Actual field names: holding_id + document_id + cost_basis_paise + current_value_paise + ytd_return_pct + ay, monthly_series + total_paise, income_summary.salary_paise/bank_interest_paise/…, recommended. NOT a bug — was diagnostic error.

Real bugs found — deferred (logged as follow-up)

• Portfolio position overwrite — positions keyed by ${form_type}__${uidPrefix}, so 4 PPF passbook uploads collapse to 1 surviving row. Rekey by (form_type, statement_period) — Task #6.
• ELSS current_value_paise = 0 — parser extracts invested amount, not current NAV. Task #7.

Correct field-name cheat sheet for future diagnostics

Firestore path	Fields that DO exist
users/{uid}/portfolio_positions/{id}	holding_id, document_id, cost_basis_paise, current_value_paise, ytd_return_pct, ay, updated_at
users/{uid}/portfolio_holdings/{id}	asset_class, display_name, sector, updated_at
users/{uid}/expense_reviews/{ay}	total_paise, monthly_series, top_merchants, by_category_paise, save_rate_pct, twelve_month_avg_paise, txn_count
users/{uid}/tax_reviews/{ay}	recommended, income_summary.{salary_paise, bank_interest_paise, stcg_111a_paise, ltcg_112a_paise, tds_form16_paise, …}, regime_old/new.{tax_paise, flags}
users/{uid}/capital_gains/{ay}	realised_short_term_paise, realised_long_term_paise, realised_debt_mf_* (lines: [] is V1 by design)

When to invoke

Any time a pillar looks empty, first re-run yarn dev:self against a fresh emulator, then read Firestore with the exact field names above before opening a "pillar broken" fix task.

Impact

• Diagnostic misreads reduced (schema-agnostic sanity table above).
• parse_pending no longer eternal on permanent Anthropic errors.
• Image docs no longer wholesale-rejected on the server path.
• tax_review_runs state now truthful.

Links

• upload-only
• four-pillars
• tax-recompute-engine
• .context/ops/runbooks/upload-pipeline.html — visual journey diagram of the same pipeline