Find what is overloading a node
A node that is short of memory, CPU or disk takes no new tasks, and the tasks already running there slow down or die. Its Overview tab names the problem and the work behind it: which sessions use the most, which runs died in the last week, and what the node did about its disk. Each of these sections shows only when it has something to show, so a healthy node has none of them.
Who can do this: anyone who can view nodes. Dead runs needs a role that can view sessions, and its pane logs a role that can download transcripts. Changing Max running tasks needs the permission to edit nodes.
- In the sidebar, select Directory, then the Machines tab. A red or yellow bar under Load (C CPU, M memory, D disk) marks a node running short.
- Select the node's name. Its details open on the Overview tab.
- Read the Findings at the top, such as Memory critical, CPU critical or Process OOM-killed. Each says what to do and when it was first and last seen, and links the tasks it involves, each with its sessions.
- Under Sessions, read what runs on the node now, heaviest first: each task's CPU, RAM and Procs (processes). A node sharing its machine with others lists their sessions too, with each one's Node. Select a task to open it.
- Under Dead runs, read the runs that died on the node in the last 7 days, newest first, each with its end reason. Select Pane log on a row to read the last thing the run printed; a row that has none says No pane log.
- Under Disk, read the node's cleanups and any pause of new work, such as Paused: disk full.
- Under Findings, Earlier (24h) lists problems from the last day that have since cleared. A problem that keeps coming back points to a node that is too small for its load.
- Act on what you found:
- Too many tasks at once: select Limit running tasks beside the finding. It opens Properties at Max running tasks; set a lower number and select Save. See Limit how many tasks a node runs.
- One task using most of it: open that task and stop it, or let it finish before more start. To stop new work while it does, drain the node.
- A cloud machine that is simply too small: resize it.
The node also guards itself: at 90% memory or more, it takes no new tasks until it has stayed under 90% for 10 minutes.
Related
- Read a node's status — the list, the Overview fields and the Metrics charts.
- See what ran on a node — every session the node ran, from Audit.
- Nodes — every field on the Overview tab.