Respond to a regression
When a production release makes a project's error rate or latency climb past its limits, Jaah records a Regression incident on the project. This guide starts from that incident, checks the numbers behind it on the release's Metrics tab, finds the pull request that caused it and follows the task that fixes or reverts it. To set what counts as a regression, or have Jaah file reverts on its own, see Revert a release that breaks production.
Before you start: Production QA is a plan feature. While your plan does not include it, Incidents is left out of the side menu and opening the page shows Not on your plan. Regressions are measured only for a project whose releases are connected and whose monitoring reports to Jaah: see Turn on Production QA for a project.
Who can do this: anyone who can view the project. Confirm and Dismiss on a suspect show only to people with permission to edit the project.
- Select Incidents in the side menu. Set Type to Regression, State to Open, and pick the Project.
- Select the incident's title. It opens on Overview. Read First seen, when Jaah noticed the climb, and Count and Last seen.
- Select the number under Jaah release to open the release whose window breached the limit, the one that caused the regression, then select its Metrics tab.
- Read the Error rate and p95 latency sections. Compare Baseline, before the release, with After. Change gives the difference, and Breached after it marks the metric that passed the project's limit. Window is how many minutes were measured and, when known, how many events were counted. A Note says anything else worth knowing about the measurement.
- Select the Overview tab of the release. PRs lists each pull request it shipped, and Other commits any commit no known pull request carries. One of these is most likely the cause.
- Go back to the incident and select the Suspects tab. Jaah ranks the pull requests it suspects, each with High confidence, Medium confidence or Low confidence. Confirm the one that caused the regression and Dismiss the others, as in Find the pull request behind an incident.
- Select the Tasks tab. A task Jaah filed sits under its role, Triage, Fix or Revert; select its number to open it. No tasks filed means none was, and a line under it says why a revert was not filed when the project asks for one. Start a task for the fix yourself if you need to.
- Once the fix or revert is merged and released, check the incident again. It reads Resolved once a later release's window measures that metric within the project's limits, and its History tab shows when. Until a later release is measured, it stays open. The page refreshes every 15 seconds.
Related
- Revert a release that breaks production: set what counts as a regression and how reverts are filed.
- Check that a fix resolved an incident: confirm the fix is credited to its task.
- Release runs: every field on a release's Metrics tab.
- Incidents: every tab of an incident's page.