Skip to main content

Compare agent setups with benchmark mode

Benchmark mode tests whether another agent setup would do your project's work better. It picks a share of the project's merged coder tasks and runs each one again, hidden, on a different configuration. The re-run is a shadow: its pull request never merges. Jaah then judges the two side by side, and each judgment is a duel that counts toward that setup's rating.

Who can do this: anyone with edit access to the project. The Benchmark tab shows only while benchmark mode is on for your organization. No plan includes it: Jaah's operators switch it on per organization. To ask for it, contact support.

Turn it on for a project​

  1. In the sidebar, select Directory, then the Projects tab, and select the project's name. Then select its Benchmark tab.
  2. Tick Benchmark mode.
  3. In Sample rate, type the share of merged coder tasks to re-run, from 0% to 100%.
  4. Check Budget. It shows this month's benchmark spend against its cap, such as $4.10 of $40.00 this month. The cap is a share of the month's agent spend, set by the project's Experiment budget; select that link to change it in Settings › General.
  5. Select Save. To undo your changes before saving, select Discard.

Once the spend reaches the cap, Budget reached shows beside it, and nothing new is sampled until next month. Duels already running still finish. With no agent spend yet this month, Budget reads No agent spend yet this month: nothing sampled.

Read the duels​

  1. Open a merged coder task the sample picked, then its Details tab, then Outcome.
  2. Read Benchmark duel: Origin config and Shadow config name the two setups, Shadow PR links the shadow's pull request, and Result says which one won, Tie, or Pending while it is still judged.
  3. For the totals across many tasks, an Owner or Admin opens Reports and the Agents tab. With agent performance on your plan, group Performance by Configuration: Duel rating rates each configuration on the duels it was in, with No duels where none were decided.

To move an agent to the setup that wins, see Move an agent to its best-performing setup.

What a task's result holds is in Read a task's result, and the rest of the command center in Read the command center. Every field of the tab is in Projects, and how projects fit together in How projects work.