Skip to main content

Respond to a failed health check

After each successful production release, Jaah runs the project's probes: its health URLs and smoke commands. When one fails, Jaah records a Health incident named Probe failed: and the probe's name. The same probe failing again after a later release adds to that incident's Count instead of adding a new one. This guide starts from that incident and ends when a passing probe resolves it. To set probes up, see Check production after every release.

Before you start: Production QA is a plan feature. While your plan does not include it, Incidents is left out of the sidebar and opening the page shows Not on your plan. Probes run only for a project whose releases are connected and whose Production QA switch is on: see Turn on Production QA for a project.

Who can do this: anyone who can view the project. Changing the probes needs the permission to edit the project.

  1. Select Incidents in the sidebar.

  2. Set Type to Health and pick the Project. Set State to Open to hide the probes that already pass again.

    The Incidents filters with Type set to Health: Project, Severity, State and Type

  3. Select the incident's title. It opens on Overview. Read:

    • Environment: the environment the release went to.
    • Count and Last seen: how many probe runs failed, and when the latest did.
    • Commit: the commit that release shipped. Point at it for the full hash.
  4. Select the number under Jaah release to open the release, then select its Probes tab. Under Results, the failed probe reads Failed, then HTTP or Command, its HTTP status or exit code, and how long it took.

  5. Open Output under the failed probe to read what it printed. For a health URL, compare the status with the one you expected. For a smoke command, the last lines it printed usually say what broke.

  6. Decide where the fault is:

    • Production is broken. Back on the incident, select the Suspects tab to see the pull requests most likely behind it, as in Find the pull request behind an incident.
    • The probe is wrong, such as a URL that moved. Fix it on the project's Production QA tab, under Probes, and select Save.
  7. Select the Tasks tab. With the project's Health breach automation in place, Jaah files a fix task and links it here. No tasks filed means none was; start a task for the fix yourself.

  8. Once the fix is merged, release again. See Release to production.

  9. Check the incident. The next probe run where this probe passes marks it Resolved, and the History tab shows when. If you removed the probe instead, the next probe run resolves its incident too. The page updates on its own.

A release's Probes tab also lists, under Incidents, each health incident on that release as Failing or Passing. When the probe run did nothing, Run says why, such as Skipped: Production QA off or Skipped: a later release superseded it.