Skip to content

Work / AskCal / Wiki / Concepts

accuracy-is-measured

Conceptverified 2026-10-30

Accuracy is the product, so it is measured

packages/eval is the Phase 1 harness that answers whether asking beats guessing. A change to the estimator or a provider is not done until pnpm eval has run. It needs a key, or every row fails and the verdict reads FAIL on zero data:

GEMINI_API_KEY="$(keys get askcal GEMINI_API_KEY)" pnpm eval

Every prompt lives in packages/shared/src/prompts.ts

Nothing may re-declare one. The harness imports the same strings the Cloud Function sends, so pnpm eval measures the estimator that ships. The eval also validates against scanResultSchema, the real wire contract, for the same reason.

It did not always. packages/eval carried its own copies, and the two had drifted: the harness still asked for confidence: "low" after production moved to seen | guessed | unknown, and still carried a total must equal the sum rule production had dropped because it derives the total from the rows. Its numbers therefore moved with the model and with run-to-run variance rather than with the text under test.

See accuracy-findings.