Compare agent setups side by side
Jaah records how each agent setup's finished work turned out: what merged and stayed, what needed rework, what it cost. Use Compare when you are choosing between setups, such as before you change a project's default model. It puts two to four setups side by side and counts only the kinds of work all of them ran, so one setup that only had easy tasks does not look better than it is.
Who can do this: your organization's Owners and Admins, with agent performance on your plan. Without it, the Quality tab does not show Compare or Ranking.
- In the sidebar, select Reports. The command center opens.
- To compare setups on one project only, pick it under Project in the bar at the top. Leave it on All to compare across your organization.
- Select the Quality tab and scroll down to Compare.
- In Configurations, pick the setups to compare. You can pick up to four. Until you pick two, the panel reads Pick 2 to 4 configurations.
- To narrow the work counted, pick a Period: Last 30 days, Last 90 days, Last year or All time, which it starts on. To count only work in one programming language, type it in Language and press Enter. Leave it empty for any language.
- Read the Compared on: line. It names the kinds of work every picked setup ran, as the agent and the task size, such as Coder · S. Every number below comes from that work only. If it reads No shared strata, the setups never ran the same kind of work: pick setups that share an agent.
- Read the table, one column per setup and one row per measure: Success, First-pass, Defects, Rework, Time to merge, Cost and Security. The range in brackets is the 95% interval: when two setups' ranges overlap, the gap between them may be chance. Hover a measure's name for what it counts.
- Check the line under each setup's name. It shows how many outcomes the setup has, or Insufficient data with the count when it has too few to judge. Give that setup more time before you act on it.
- To see every setup ranked rather than a few picked, scroll to Ranking below. Filter it by Agent, Period and Min outcomes (20, 30, 50 or 100). Setups are listed best first, and a setup with fewer outcomes than Min outcomes is left out.
When one setup comes out ahead, move the agent to it as in Move an agent to its best-performing setup. To gather more evidence first, run the two setups on the same tasks with benchmark mode. The rest of the Quality tab is in Read the command center.