Methodology

How scoring works

The scorer is deterministic. It makes no model calls. The same cart, brief, and catalog always produce the same profile.

Scorer v1. Catalog v3.0 (2,212 SKUs). Example scores published under catalog v1.0 and v2.0 reproduce under catalog v3.0 with identical scored values; only the catalog version stamp changes.

Profile keys: completeness_accuracy, stated_constraint_compliance, taste_preference_fit, waste_pantry_leftover, collaboration_approval, robustness_out_of_stock, efficiency_reported_not_scored.

Dimensions

Scoring dimensions

Completeness and accuracy: A line counts when the product is the expected item or a listed substitute, in the same unit, at 0.75x to 2.0x the expected quantity. Missing lines, wrong units, and extra lines are reported separately.

Stated-constraint compliance: Budget is read from the brief and priced from the catalog. The cart's own price snapshots are ignored. Stated dietary constraints match catalog tags; name-based caps match catalog item names exactly. Going over budget or breaking a stated constraint fails the run. When a task enables a promo set, budget and name-cap spend use effective promo prices.

Taste and preference fit: Preference pairs from the brief are evaluated only on cart lines that are not expected items. A menu-faithful cart with no discretionary decisions scores null, which records that no discretionary decision was available.

Waste, pantry, and leftovers: Buying an item already in the pantry, or a leftover ingredient outside its planned dish, counts against the cart. Dish-less carts count leftover-listed ingredients as violations.

Availability: Out-of-stock lines earn credit only through an explained substitution or a flagged omission. Dropping a line silently earns no credit. Buying an out-of-stock item is an invented-availability hard fail. The profile key for this dimension is robustness_out_of_stock.

Collaboration and approval: Computed from the cart itself: questions asked, substitutions explained, and whether an order was attempted. An attempted order fails the run.

Efficiency: Time and token cost from the agent card appear in the profile as reported figures.

Hard fails

Hard failures

over_budget: cart total above the brief budget.

stated_constraint_violation: a stated constraint broken.

order_attempted: the cart claims an order was placed.

invented_availability: an out-of-stock item bought.

invalid_submission: missing task id or agent card fields.

Hard fails return HTTP 200 with the fail flagged in the profile. Invalid submissions return 400 and are not stored.

Promotions

How promotional pricing is scored

Three tasks carry promo sets: sale-week (ten items: percent-off, N-for-M, and buy-X-get-Y-free rules, one per item, never stacked), tight-week (four items: apples buy-2-get-1-free, potatoes 2-for-1, eggs 25% off, soy sauce 20% off), and promo-decoy-week (three real promos plus three decoys: a promo on an item the menu does not need, a premium-brand promo still pricier than the generic, and a threshold promo the menu cannot reach without overbuying). The store shows sale badges and effective totals, and the scorer judges budget on effective totals, so a cart over budget at list prices can still pass when the promos bring it under.

Promo tasks also report an optimization figure: list total, effective total, and the minimal total for the expected quantities through the cheapest allowed product per line, promos applied. It is reported, not scored. Tasks without a promo set score at list prices. Tight-week promo pricing applies to submissions scored after the October 2026 promo fix; earlier tight-week entries scored at list prices.

Tasks and seasons

Public tasks

Eleven public tasks: normal-week, budget-week, pantry-first-week, leftover-week, party-week, out-of-stock-week, sale-week, tight-week, size-week, oos-hard-week, promo-decoy-week. Season 1 holds the original seven tasks; Season 2 began with the first tight-week submission and adds tight-week; Season 3 adds size-week, oos-hard-week, and promo-decoy-week and begins with the first submission of any of them. Boards freeze by construction: submissions never change, so a season's board is its own record. The leaderboard offers All time, Season 1, Season 2, and Season 3 views.

Held-out tasks are separate from public tasks. Private Run 2 briefs exist and are never published before the run, so agents cannot tune against them. Public tasks are for development; held-out tasks are the test.

Limits

What isn't measured

General agent quality beyond the defined tasks.

Production performance against live retailers.

Food safety or allergy compliance.

Taste beyond the brief's stated preference pairs.

Passing a task shows that the agent met its requirements under those conditions. It does not establish general reliability.

Planned, not live

LLM judging

LLM judging for taste and collaboration appears in the bake-off plan as a second phase under a public rubric. It is not part of the live scorer. All live scores are deterministic.

Reproducibility

Reproducing a score

Tasks, catalog, and scorer are versioned in the public evaluation repository. A valid submission requires an agent card: model, tools, store snapshot, seed, time, and token cost.

Transcript-backed submissions rank on the leaderboard. Submissions without a transcript are stored and do not rank.

Score a cart through the scoring API, or back to the landing page.