Skip to main content

Find out why a node paused for its disk

A node whose disk fills up stops taking new tasks until it has room again, and cleans up after itself on its own. Its Disk history shows each pause, each cleanup and what it freed, so you can tell a passing spike from a disk that is simply too small.

Who can do this: anyone who can view nodes. Growing a disk needs the org admin role, on a node Jaah launched in your cloud.

  1. In the sidebar, select Directory, then the Machines tab. A paused node shows a line under its row such as Paused: disk 92% used. Since and the time it paused, or Paused: disk full when the node gave no current reading.
  2. Select the node's name. Its details open on the Overview tab, with the same Paused: disk chip beside its name. Hover or tap the chip to see when the pause started.
  3. On the Metrics card, find the Disk chart. The line under it reads New work paused since and a time while the pause lasts, or New work: running once it is over. The marks under the chart show each cleanup, pause and resume; hover or tap one to read it, such as Cleanup freed 7.2 GB or New work paused at 92% used, with the time. Pick 7d in the range list to see the whole week.
  4. Below Metrics, read the Disk card. It lists what the node did about its disk, newest first:
    • New work paused at a percentage, and New work resumed at a percentage once it had room again.
    • A cleanup, such as Scheduled cleanup, Cleanup at 81% used → 63% or Cleanup at warning level, with freed and the amount, and under it the kinds of files it removed, such as CI images, Build cache or Stopped containers.
    • Runs of cleanups that found nothing, folded into one line such as 12 cleanups and nothing to free.
  5. Select Details on a cleanup to see each runner's space before and after, any Removed runner users, and the Git mirrors and Plugin folder sizes the cleanup measured but kept. Hide details closes it. On a node that shares its host's disk, a row says on host and the host's name when the cleanup ran there.
  6. Decide from what you read:
    • The node paused once and resumed soon after a cleanup that freed a lot: nothing to do.
    • It pauses again and again, or cleanups keep reporting nothing to free: the disk is too small for its work. On a node Jaah launched in your cloud, select Grow disk in the cloud's section, such as AWS machine, as in Change a cloud node's machine type or disk. On a machine you run yourself, add disk the way you usually do.
    • It pauses only when many tasks run at once: lower Max running tasks so fewer share the disk. See Limit how many tasks a node runs.

While a node is paused, its running tasks carry on; only new ones wait. When it has room again it resumes by itself, and the Disk card shows New work resumed.