Work / AskCal / Wiki / Concepts
accuracy-is-measured
Conceptverified 2026-10-30
Accuracy is the product, so it is measured
packages/eval is the Phase 1 harness that answers whether asking beats
guessing. A change to the estimator or a provider is not done until pnpm eval
has run. It needs a key, or every row fails and the verdict reads FAIL on zero
data:
GEMINI_API_KEY="$(keys get askcal GEMINI_API_KEY)" pnpm eval
Every prompt lives in packages/shared/src/prompts.ts
Nothing may re-declare one. The harness imports the same strings the Cloud
Function sends, so pnpm eval measures the estimator that ships. The eval also
validates against scanResultSchema, the real wire contract, for the same reason.
It did not always. packages/eval carried its own copies, and the two had
drifted: the harness still asked for confidence: "low" after production moved
to seen | guessed | unknown, and still carried a total must equal the sum
rule production had dropped because it derives the total from the rows. Its
numbers therefore moved with the model and with run-to-run variance rather than
with the text under test.
See accuracy-findings.