Skip to main content

A node is offline or won't register

Start on the node's row: in the sidebar, select Directory, then the Machines tab. What each mark and field means is in Read a node's status.

The node never joins​

Symptom: the row reads Waiting for your machine with Install command not run yet… under it, a new node stays Deploying and then turns Stalled, or the node's page says Never connected — the machine hasn’t joined yet.

Cause: the install command wasn't run on the machine, failed part-way, or its token expired. A command is good for 24 hours.

Fix:

  1. Open the row's ⋯ menu and select New install command (or Re-issue on a node that has a row of its own), and confirm. The old command stops working. Requires the org admin role.
  2. Select Copy command, and paste the whole command into a terminal on the machine.
  3. Run it as an administrator (on Linux, the command runs as root through sudo), and let it finish: it can take several minutes.
  4. If it fails, read its last lines. A download or connection error means the machine can't reach the internet over HTTPS: open outbound port 443 in its firewall or proxy, then run the command again.
  5. Watch the row: it moves through Installing on your machine to the node's own row with a green dot.

The node went offline​

Symptom: a node that worked shows Offline, a grey dot, and an old time in the clock column.

Cause: the machine is off, asleep or disconnected, or the Jaah service on it stopped.

Fix:

  1. Check that the machine is on and online. Turn off sleep for a machine that should run tasks unattended.
  2. On a Linux machine, check the service with systemctl status jaahd. If it isn't running, start it with sudo systemctl start jaahd, and read journalctl -u jaahd for the reason it stopped.
  3. If the log says the token was refused, the token was re-issued elsewhere: select Re-enrol on the node's page, confirm Re-enrol this node?, and run the new command on the machine.
  4. Once the node reports again, the clock column shows now and its tasks start again.

Tasks that were running there when it went away each use up an attempt, and run again once a slot is free.

The node is online but runs nothing​

Symptom: a green dot, but tasks for your project stay Queued.

Cause: one of three things: the node isn't Enabled (or is Draining), your project's node groups leave it out, or no model account is signed in there.

Fix:

  1. Check Enabled. On a Draining node, select Cancel drain; on a disabled one, select Enable. See Drain or disable a node.
  2. Open your project and check Node groups, in its Settings tab, section Nodes. All runs on every node; with groups, add the node to one of them. See Run a project on some nodes only.
  3. On the Overview tab, read Accounts. None, not logged in or no account placed here means no AI account can run there: sign one in as in Sign a model account in again.
  4. Read anything named at the top of Overview, and hover an issue for its remedy.

The node is full or slow​

Symptom: the Load bars are red, or every slot under S/R/A is in use.

Cause: the node runs as many tasks as it has slots, or its CPU, memory or disk is full.

Fix: wait for running tasks to finish, free disk space on the machine, or add another node. A project on All uses it at once; with node groups, add it to one of the project's groups.