Skip to main content

Respond to a merge lane slowdown

Jaah times every stage of a project's merge lane gate on each run. When one stage's last five passed runs take, in the middle, at least 1.3 times as long as the twenty passed runs before them, it records a Lane slowdown incident for that stage. A slower gate means every pull request waits longer to merge, so it is worth finding the change that caused it. This guide takes you from the incident to that change and on to the incident resolving.

Before you start: Production QA is a plan feature. While your plan does not include it, Incidents is left out of the side menu and opening the page shows Not on your plan. Only projects that use the merge lane can have this incident.

Who can do this: anyone who can view the project sees its incidents and its lane in Audit.

  1. Select Incidents in the side menu. Set Type to Lane slowdown and State to Open. Pick the Project too if you look after more than one.
  2. Read the incident's title. It names the stage, the middle time of its last five runs and what it was before, for example Merge-lane stage tests slowed down: median 540s over the last 5 runs, was 300s. Its Severity is Error. Select the title to open it.
  3. On Overview, read First seen: the time Jaah noticed the change, so look at what merged shortly before it. Count and Last seen keep their first values until it reopens. A lane slowdown has no environment, release or stack, so those fields read Not sent.
  4. Select the Tasks tab. When the project's Production QA switch is on and its Regressions and slowdowns automation is enabled, Jaah files one Fix task. Select it: its description names the stage, both times and the pull requests that went through the lane since the slowdown began. No tasks filed means the project doesn't act on incidents yet; see Turn on Production QA for a project if you want it to.
  5. In the sidebar, select Audit, then the Merge lanes tab, and pick the project in Project. Select Runs. Each run shows its Merge and Wall times and, under Gates, how many gates ran, newest first; use Next to page back to where they started to grow.
  6. Select History and open the task of a pull request that merged around First seen. On its page, select Details, then Summary, and scroll to Merge-lane stages. Each gate run that tested the task's pull request lists every stage with its time; Failed and Flaked mark a stage that failed or passed only on a retry. Compare the slow stage there with an older task's to find the pull request that made it slower.
  7. Fix the cause: start the Fix task, or a task of your own, to speed the stage back up or undo the change that slowed it.
  8. Wait for the next lane runs. Once each of the stage's last five passed runs is back within a tenth of its old time, the incident turns Resolved on its own and its History tab shows Resolved. If the stage slows down again later, the same incident shows Reopened instead of a new row.

You can't close the incident by hand: it resolves only when the stage is fast again.