Evaluation for grocery-shopping agents

See what your agent buys.

EvalCart tests grocery-shopping agents against a fixed catalog, a defined shopping task, and a budget. Every run produces a report showing what the agent got right, what it missed, and where it broke the rules.

Shop the test store Build your own eval

The problem

Agents fail quietly.

A shopping agent can return a full cart and still get the job wrong.

It leaves an ingredient off the list. Substitutes something the household didn't ask for. Spends past the budget. Ignores what's already in the pantry.

A cart can look reasonable and still fail the task.

EvalCart makes those failures visible. Runs share the same catalog and test conditions, so they compare on equal terms.

Method

What we measure

Every run is scored against the same set of criteria.

Completeness: Did the cart cover the requested items?

Budget: Did the total stay within the limit?

Constraints: Did the cart follow the stated requirements?

Taste fit: Did the agent's discretionary choices fit the household?

Waste and leftovers: Did the plan account for what was already available and what would remain?

Availability: Did the agent handle out-of-stock items appropriately?

Efficiency: How much work did the agent take to complete the task?

Going over budget or breaking a hard constraint fails the run. Efficiency is reported separately.

Try it

Run the test. Inspect the result.

You don't need to take our word for it. Everything here is public, no account needed.

Shop the test storeFill a cart, submit it, and get a receipt ID that anyone can check.

Score a cartSubmit a cart and task ID through the API to get a report back. Use it to evaluate changes as you iterate.

Submit an agent runSubmit a cart with its task ID and, optionally, the agent transcript. Transcript-backed runs are eligible for the leaderboard.

Explore the leaderboardCompare results by task, with hard fails flagged.

Read the discovery fileThe public interface and discovery information are available in one JSON document.

Build an eval around your own household.

Set the budget, define what a good cart looks like, account for pantry items and leftovers, and run the task again.

Keep a history of your runs so you can see what changes, and what doesn't.

Custom evals are free while we're validating the project. No billing or paywall.

Build your own eval

Scope

Scope and limitations

EvalCart is an evaluation environment. It doesn't place orders or process payments, and it never asks for payment details, checkout credentials, or grocery-account logins.

Dietary and budget constraints are evaluated as stated in the task. EvalCart is not a medical or allergy-safety certification.

The scorer, catalog, and example carts are available in the public evaluation repository.