# Slurm reads

`slurm_read` answers what the console shows for a Slurm cluster: the
status page (node states, GPU allocation, the live queue, GPU hours,
shared `/home` usage), one job's page, that job's place in the queue,
and its log. The tool lives on the [MCP server](https://docs.nationalcompute.com/api/mcp.md). It is read only
and single phase. No `confirm_token` is involved. Nothing on this
surface submits, cancels or changes a job: `sbatch`, `scancel` and every
other scheduler command stay on the login node, where your agent runs
them over ssh with the connection facts `grid_whoami` carries (see
[Connection facts](#connection-facts)).

```text
grid_whoami ──▶ the cluster's name, kind slurm, ssh facts
     │
     ▼
slurm_read status ──▶ nodes, queue, GPU hours, /home
slurm_read job    ──▶ one job's record, queue context, GPU metrics
slurm_read queue  ──▶ position and the jobs ahead
slurm_read log    ──▶ stdout tail, or a grep over the whole file
     │
     ▼
metrics_read node ──▶ a Slurm node's GPU and host telemetry
```

Every read is scoped to your organization. A cluster name that is not
one of your organization's Slurm clusters answers `404 not-found`,
whether it belongs to someone else, is a Kubernetes or VM cluster, or
never existed.

## Arguments

| Argument | Values | Applies to |
|---|---|---|
| `cluster` | the cluster name (`grid_whoami` lists them) | required on every call |
| `action` | `status`, `job`, `queue`, `log` | required on every call |
| `jobid` | a Slurm job id: digits, optionally `_<task>` | required for `job`, `queue`, `log` |
| `source` | `live` (default) or `mirror` | `status`; `mirror` answers from the platform's last snapshot without dialing the cluster's island |
| `tail_lines` | 1 to 1000; defaults to 200 | `log`; the lines kept from the end |
| `query` | a fixed string, matched case insensitively | `log`; greps the whole file instead of tailing |
| `file` | an index into the answer's `files` | `log`; a per task log next to the batch stdout; the default is the batch stdout |
| `incarnation` | the `coverage.job.incarnation` of a `job` answer | `log` on a [centrally read island](#centrally-read-islands) |

Each argument belongs to the actions its row names. An argument present
on another action refuses `422 bad-request`; nothing is silently
ignored. `file` and `tail_lines` must be whole numbers. A blank `query`
tails.

## action=status

```json
{
  "age": 3,
  "error": null,
  "nodes": {
    "node-1": {"state": "mixed", "flags": [], "gpus_alloc": 4, "gpus_total": 8,
               "jobs": [{"id": "4242", "user": "ada", "gpus": 4}]},
    "node-2": {"state": "drained", "flags": ["drain"], "gpus_alloc": 0,
               "gpus_total": 8, "jobs": []}
  },
  "queue": [{"id": "4243", "name": "train", "user": "ada", "state": "pending",
             "gpus": 8, "reason": "Resources", "time": "0:00"}],
  "usage": {"month": "2026-09", "users": {"ada": 12.5}, "error": null},
  "storage": {"fs": {"used": 10995116277760, "total": 109951162777600,
                     "free": 98956046500000},
              "dirs": {"ada": 7516192768}, "error": null}
}
```

| Field | What it carries |
|---|---|
| `age` | the age of the scheduler snapshot, seconds |
| `error` | the scheduler feed's error, or `null` |
| `nodes` | one entry per compute node, keyed by the name `sinfo` shows: `state`, `flags`, `gpus_alloc`, `gpus_total`, and `jobs` (the jobs holding GPUs on it) |
| `queue` | `squeue`, live: `id`, `name`, `user`, `state`, `gpus`, and `reason` while pending or `time` while running |
| `usage` | GPU hours this month per user from the scheduler's accounting, with `month`; `error` when accounting did not answer |
| `storage` | the shared `/home` export: `fs` (`used`, `total`, `free`, bytes), `dirs` (bytes per member of this cluster), `error` when the scan did not run |
| `mirror` | `true` when `source=mirror` answered |

`usage` is the scheduler's accounting. The
[billing records](https://docs.nationalcompute.com/api/billing.md) are the authority on what a job cost.
`storage.dirs` lists your organization's members only.

### Node states

A node takes no new jobs when its `state` is one of `down`, `drained`,
`draining`, `error`, `fail`, `failing`, `unknown`, `invalid`, `inval`,
or when `flags` carries `drain`, `not_responding`, `invalid_reg` or
`fail`. The console paints those red. `idle` means free, `mixed` and
`allocated` mean busy, `completing` means a job is finishing on it.
Every other word is Slurm's own state, verbatim.

## action=job

```json
{
  "job": {
    "id": "4242", "name": "train", "user": "ada", "state": "pending",
    "reason": "Resources", "nodes": 2, "nodelist": "", "gpus": 16, "cpus": 448,
    "mem": "1500G", "time_limit": "1-00:00:00", "elapsed": "00:00:00",
    "submitted": "2026-09-29T08:10:00", "started": "", "est_start": "2026-09-29T09:00:00",
    "ended": "", "workdir": "/home/ada/train", "stdout": "/home/ada/train/slurm-4242.out",
    "exit_code": "0:0", "partition": "main", "account": "ada", "qos": "normal",
    "gpu_alloc": {}, "source": "scontrol"
  },
  "queue": {"position": 2, "total": 5, "ahead": [{"id": "4240", "reason": "Priority"}],
            "ahead_total": 1},
  "metrics": {"gpus": [], "series": null, "error": null}
}
```

`job` is the scheduler's record, `scontrol` while the job is live and
`sacct` once accounting holds it (`source` says which). `started` is
empty until the job runs; `est_start` carries the scheduler's estimate
while it waits; `ended` is empty until it ends. `queue` is present only
while `state` is `pending` and is `null` otherwise. `metrics` is the
node scope GPU telemetry over the job's window; its `error` reads
`unavailable` when the metrics store did not answer, never a reason
with internals. A job accounting no longer knows answers
`404 not-found`.

## action=queue

```json
{"jobid": "4242", "state": "pending",
 "queue": {"position": 2, "total": 5, "ahead": [{"id": "4240", "reason": "Priority"}],
           "ahead_total": 1}}
```

`position` counts from 1 among pending jobs in scheduler order; `total`
is the pending count; `ahead` lists the nearest jobs ahead (at most 5)
with their wait reasons; `ahead_total` is the real count ahead. `queue`
is `null` when the job is not pending. This action reads no metrics.

## action=log

```json
{
  "path": "/home/ada/train/slurm-4242.out",
  "lines": ["step 1200 loss 2.31", "step 1210 loss 2.29"],
  "truncated": true,
  "clipped": true,
  "files": ["log-train_4242_0.out", "log-train_4242_1.out"],
  "selected": null,
  "note": null
}
```

The tail reads the batch stdout off the shared `/home` export. `lines`
carries the last `tail_lines` lines; `truncated` is `true` when the
file held more than the platform's own tail window (1000 lines or 512
KiB); `clipped` is `true` when `tail_lines` dropped lines from that
window. `files` lists the per task logs written next to the stdout by
wrapper stacks (`srun` redirections named `*_<jobid>_<rank>.out`), and
`file` selects one by index; `selected` echoes it. `note` explains an
empty or missing log ("log file is empty so far", a log on node local
storage).

With `query` the answer is a grep over the whole file: `lines` are the
matching lines (the last 500 matches at most), `lnos` their line
numbers, `matches` the count of lines returned, `filtered` reads
`true`. When `tail_lines` dropped matches, `clipped` reads `true` and
`total_matches` keeps the file wide count. A `query` longer than 200
characters is cut to 200.

## Centrally read islands

On an island whose reads the platform serves from its own record instead
of dialing the island, every answer carries `coverage`: per source,
`availability` (`complete`, `partial`, `unavailable`), `freshness`,
`observed_at`, and a `reason` when something is missing. The status
answer is then the platform's inventory (`inventory.nodes` with `state`,
`gpus_total`, `gpus_alloc`, `cpus_alloc`, plus `running_count` and
`queue_count`); `jobs`, `queue`, `usage` and `storage` read `null` with
their reasons. The job answer carries `coverage.job.incarnation`, which
`log` needs as `incarnation`. A log answer there carries `first_record`
and `last_record`, the record numbers the returned lines span. A record
that is not complete refuses with `503 station-unavailable` carrying
`code` (the read model's refusal, `central_read_unavailable`), `reason`
(the coverage's own reason, for example `job_incarnation_required` or
`central_store_disabled`) and `coverage`. The message says retry only
when the reason is one a later call can clear.

## Refusals

| Code | Status | Meaning |
|---|---|---|
| `bad-request` | 422 | an unknown `action` or `source`, a missing `jobid`, a malformed job id, an argument outside its action, a fractional `file` or `tail_lines`, `file` outside 0 to 127, `incarnation` on an island that is not centrally read |
| `not-found` | 404 | no Slurm cluster of that name in your organization; a job accounting does not know; per job monitoring not enabled for the cluster (`job`, `queue`, `log`) |
| `cluster-not-ready` | 409 | the cluster is still being built or is not ready; retry |
| `station-unavailable` | 503 | the cluster's island did not answer, or a centrally kept record is not complete (`code`, `reason` and `coverage` say why) |

## Connection facts

`grid_whoami` lists every cluster your organization holds. A `slurm`
entry adds the login endpoint the console's Connect step shows:

| Field | What it carries |
|---|---|
| `ssh_host` | the login address |
| `ssh_port` | the ssh port |
| `ssh_user` | the shared login account, or `null` when the login is your own OIDC identity |
| `ssh_auth` | `key` (keys the platform registered for your organization) or `opk` (a browser sign in through the setup script) |

All four read `null` while the endpoint is unresolved. Kubernetes and VM
entries carry none of these keys. `GET /api/whoami` by
[org API token](https://docs.nationalcompute.com/authentication.md#discovering-what-a-token-can-address)
answers the same body.

## Telemetry

[`metrics_read`](https://docs.nationalcompute.com/api/metrics.md) accepts a Slurm cluster as `cluster` for
`action=node` and `action=flags` with `node`; the node name is the key
`status.nodes` uses. Workload targets (`kind`, `name`) are Kubernetes
only and refuse `422 bad-request` on a Slurm cluster. When your
organization holds a Kubernetes cluster and a Slurm cluster of the same
name, `metrics_read` reads the Kubernetes one.