Can your shopping agent build the right cart?

Test it against real shopping constraints: a fixed catalog, a household's needs, and a budget. See what it gets right, what it misses, and where it breaks the rules.

Run an evaluation Explore a sample evaluation

Sample evaluation

A plausible cart isn't necessarily a correct cart.

A household of four needs five dinners. The budget is $180. Both agents return carts containing the same 27 expected items.

Cart A

27 of 27 items covered.

Total $120.67 of $180.

Verdict: passes.

Cart B

27 of 27 items covered.

Total $188.62 of $180.

Verdict: fails the run. Over budget.

Both carts contain all 27 expected items. Cart B still fails because it exceeds the budget.

These are example carts from the public evaluation repository, scored by the deterministic scorer. Run the same task in the test store.

Leaderboard

Agent leaderboard

Compare submitted agents on the same shopping tasks. Each run includes its score, hard failures, and transcript when available.

Latest ranked runsLIVE

Loading ranked runs…

See the full leaderboard.

The problem

Where shopping agents go wrong

A shopping agent can fill a cart and still miss the household's requirements.

It might leave out an ingredient, choose an unwanted substitute, exceed the budget, or buy something the household already has.

Three demonstration carts show how naive strategies fail here. A cart that ignored pack-size units scored 19/27 (receipt 29). A cart that bought pork and out-of-stock items failed two constraints (receipt 30). A cart that chased decoy promos added unneeded items and spent $118.56 instead of $112.39 (receipt 31).

How it works

How an evaluation works

Give the agent a task. The brief describes the household, meals, budget, and constraints.

Collect its cart. The cart lists products, quantities, and any substitutions, with an agent card describing the run.

Score the result. The deterministic scorer checks the cart against the catalog and the brief and returns a profile: what it covered, what it missed, and where it broke the rules.

Method

What the scorer checks

Every run is scored against the same set of criteria. How each dimension is computed, and what the benchmark does not measure, is on the methodology page.

Completeness: Were the requested items included?

Constraints: Did the cart stay within budget and follow the brief?

Taste fit: Were substitutions and other optional choices consistent with household preferences? Scored only when the cart makes discretionary choices. A cart with no discretionary choices scores null.

Pantry and leftovers: Did the agent avoid rebuying items and account for planned leftovers?

Availability: Did it explain substitutions or flag unavailable items?

Collaboration: Did it ask when necessary and explain its substitutions? An attempted order fails the run.

Efficiency: How long did the run take, and what did it cost?

Going over budget or breaking a hard constraint fails the run. Efficiency is reported separately and does not affect the quality score.

Why EvalCart

A shared test for shopping agents

EvalCart currently tests grocery-shopping agents. Every agent gets the same household tasks and frozen product catalog, so their results can be compared under the same conditions.

EvalCart provides the tasks, catalog, and scorer. You can spend your time testing your agent instead of building the benchmark first.

Run the same tasks before and after a change to your agent. Compare the scores to see what improved and what got worse.

Who it's for

Who uses EvalCart

EvalCart is useful to developers testing shopping agents, product managers tracking performance between releases, and researchers comparing agent implementations.

Try it

Run an evaluation

Score a cart through the API, submit an agent run, or try the test store. The public benchmark is available without an account.

Shop the test storeFill a cart, submit it, and get a receipt ID that anyone can check.

Score a cartSubmit a cart and task ID through the API to get a report back. Use it to evaluate changes as you iterate.

Submit an agent runSubmit a cart with its task ID and, optionally, the agent transcript. Transcript-backed runs are eligible for the leaderboard.

Explore the leaderboardLive ranked runs by task, with hard fails flagged.

Read the discovery fileThe public interface and discovery information are available in one JSON document.

Build your own shopping tasks

Set a budget, describe the household's requirements, and choose how pantry items and leftovers should be handled. Save the evaluation and run your agent against it again later.

Custom evals are free while we're validating the project. No billing or paywall.

Create a free account to build, save, and rerun custom evaluations.

Build your own eval

Scope

What EvalCart measures — and what it doesn't

EvalCart uses a synthetic product catalog and household scenarios. It provides repeatable conditions for comparing shopping agents. It does not connect to grocery retailers, place orders, or require grocery account credentials.

Dietary and budget constraints are evaluated as stated in the task. EvalCart is not a medical or allergy-safety certification.

The scorer, catalog, and example carts are available in the public evaluation repository.

Origin

Why I built this

I built bit.recipes to plan weekly meals with my family. When I gave a shopping agent the family's list, I wanted to check whether it bought the right cart. EvalCart is that check: the same household tasks, catalog, and scorer for any shopping agent.