# National Compute Docs

> Documentation for National Compute: burst GPU capacity rented through a market. You declare how many GPUs you want and the most you'll pay per GPU-hour; the market grants whole nodes and bills the clearing price, never above your ceiling.

Every page of https://docs.nationalcompute.com/ in site order, each with its address under its heading. The index with one line per page is [llms.txt](https://docs.nationalcompute.com/llms.txt).

# National Compute documentation

Source: https://docs.nationalcompute.com/ (this markdown: https://docs.nationalcompute.com/index.md)

National Compute is burst GPU capacity: nodes join your cluster in real
time when you need them and leave when you don't. You declare how many
GPUs you want and the most you'll pay per GPU-hour; supply is allocated
by price, so you hold capacity while the market clears at or under your
ceiling and are billed what the auction assesses — never above your
ceiling.

[Public Marshall](https://docs.nationalcompute.com/public-marshall.md) offers model chat without sign-in,
with a sponsored allowance per browser visitor. Sign in for saved
sessions, shells and persistent files.

There are three ways to consume capacity:

- **VM clusters** — declare a GPU count and a price ceiling; granted
  nodes are yours over ssh until they are reclaimed or you scale
  down. See the [VM capacity API](https://docs.nationalcompute.com/api/vm-capacity.md).
- **Kubernetes clusters** — a dedicated cluster whose GPU jobs create
  the demand by themselves, each job one request priced in whole nodes
  on the market; the only knob is your price ceiling.
  See [Kubernetes clusters](https://docs.nationalcompute.com/kubernetes.md) and the
  [limit price API](https://docs.nationalcompute.com/api/k8s-bid.md).
- **Public Research** — one whole node at a time for a researcher at
  a public research organization, requested from Marshall and used
  over ssh at a flat rate, outside the auction. See
  [Public Research](https://docs.nationalcompute.com/public-research.md).

## Start here

1. [Burst capacity](https://docs.nationalcompute.com/market.md) — how capacity is allocated: the
   auction, protection windows, and pricing.
2. [Authentication](https://docs.nationalcompute.com/authentication.md) — mint an org API token and make
   your first call.
3. [Billing](https://docs.nationalcompute.com/billing.md) — what the meter measures, and what is never
   billed.
4. [API reference](https://docs.nationalcompute.com/api/index.md) — conventions, endpoints, errors, and
   the machine-readable OpenAPI contracts.

Questions about base load capacity, preemption, pricing, or the access model?
The [FAQ](https://docs.nationalcompute.com/faq.md) covers the ones we hear most.

## For agents

If an AI agent drives your capacity, point it at
[nationalcompute.com/FIRST-JOB.md](https://nationalcompute.com/FIRST-JOB.md)
to run your first job, and at
[nationalcompute.com/llms.txt](https://nationalcompute.com/llms.txt) and
[nationalcompute.com/AGENTS.md](https://nationalcompute.com/AGENTS.md)
for orientation — pages served unauthenticated for exactly that purpose.
The console's Getting Started page carries the prompt to paste. The
OpenAPI contracts are unauthenticated too, so an agent can discover the
wire before it holds a token:

- [`/api/vm/openapi.json`](https://nationalcompute.com/api/vm/openapi.json)
  — the VM capacity contract
- [`/api/k8s/openapi.json`](https://nationalcompute.com/api/k8s/openapi.json)
  — the Kubernetes limit price contract
- [`/api/billing/openapi.json`](https://nationalcompute.com/api/billing/openapi.json)
  — the billing records contract

An agent that speaks MCP (Model Context Protocol) takes the surface as
tools instead of routes. The hosted [MCP server](https://docs.nationalcompute.com/api/mcp.md) at
`https://nationalcompute.com/mcp` carries market data, capacity, limit
prices, billing, hardware metrics and server-side Kubernetes access
behind one OAuth sign-in, with no token to mint. Claude Code, Codex and
Cursor add it with one line of configuration.

The starter recipes FIRST-JOB.md walks through are a catalog at
[`access.nationalcompute.com/first-job/`](https://access.nationalcompute.com/first-job/):
each recipe's cluster kind, category, description, default file and
vendor variants, as JSON, next to the manifests themselves.

A program reads these pages at `nationalcompute.com/docs/<page>`, no
token needed, the same paths as this site — or as markdown here: a
page's URL minus the trailing slash, plus `.md` (`/market/` becomes
`/market.md`), is its source with every link made absolute. Every HTML
page names that form in a `<link rel="alternate" type="text/markdown">`.
[llms.txt](https://docs.nationalcompute.com/llms.txt) indexes every page
that way, with a one-line description each;
[llms-full.txt](https://docs.nationalcompute.com/llms-full.txt) is the
whole site in one file; [sitemap.xml](https://docs.nationalcompute.com/sitemap.xml)
lists the pages.

The specs are the authority whenever these pages and a spec disagree.

---

# Public Marshall

Source: https://docs.nationalcompute.com/public-marshall/ (this markdown: https://docs.nationalcompute.com/public-marshall.md)

Public Marshall is model chat without sign-in, with up to **$100 of free,
sponsored model usage per browser visitor**. Use it to plan experiments,
compare approaches, write code and ask questions about models and compute.

Open [Public Marshall](https://console.nationalcompute.com/public/marshall)
and send a message. Replies appear as they are generated. Public Marshall
uses the production default model; it has no shell, files, uploads, cluster
access or tools that perform actions on your behalf.

## Conversation lifetime

The conversation stays in the page's memory. **Refreshing or closing the
page clears it; there is no saved conversation to recover.** **New
experiment** also clears the current conversation. Use **Stop** to interrupt
an active reply before starting a new experiment. Neither action resets the
usage allowance.

[Sign in](https://docs.nationalcompute.com/sign-in.md) to save Marshall sessions, connect a shell and work
with persistent files. Signing in does not automatically save or import
the public conversation you already wrote.

## Visitor allowance

The $100 allowance lasts for the browser visitor's lifetime; it does not
reset daily or monthly. A separate secure cookie identifies the visitor
and preserves the remaining allowance across page refreshes. The cookie
does not contain conversation history. Keep cookies enabled to retain
access to that allowance. The visitor is a browser identity, not a
verified person or an account; the allowance is not synchronized across
browsers.

The cookie lasts for 365 days and is renewed when the page reconnects.
Deleting it or letting it expire loses access to that visitor's remaining
allowance; signing in cannot recover it.

Before each reply, the service reserves enough allowance to cover its
maximum model usage. Confirmed usage replaces that reservation when the
reply finishes. An interrupted reply with unknown usage keeps its
reservation until the cost is confirmed. Waiting or refreshing does not
release an uncertain reservation.

A new reply is refused when the remaining allowance cannot cover its
reservation, even if some allowance remains. The offer is therefore
**up to $100**, not a guarantee that every remaining fraction can fund a
reply. Sign in to continue using Marshall after the public allowance is
exhausted. The allowance covers public model chat, not compute capacity,
cash withdrawals or a transferable account credit.

## Data retained

Public Marshall does not store prompts, replies or conversation identifiers
in its session or usage records. It retains the visitor credential's hash,
allowance reservations, token counts, model usage and costs. Request limits
use a hashed client address; these records do not contain the chat text.

Public chat does not enter the organization-level
[Marshall data-sharing program](https://docs.nationalcompute.com/marshall-data-sharing.md). Signed-in
workspace sessions follow that program's settings.

## Request limits

These are the current public chat limits. Browsers using the same client
address share its request limits.

| Limit | Value |
|---|---|
| Visitor allowance | $100 lifetime model usage |
| New visitor grants | 5 per hour per client address |
| Chat requests | 10 per minute per client address |
| Active replies | 1 per visitor; 32 across Public Marshall |
| Conversation request | 64 KiB, including at most 64 text messages |
| Reply output | At most 4,096 tokens |
| Model response stream | 512 KiB, including usage metadata |
| Model reply timeout | 120 seconds |
| Execution slot after an unresolved request | 180 seconds; the cost reservation remains |

An incomplete reply stays visible until the page is cleared, but is excluded
from the next message's model context. The app does not automatically retry
a submitted message: a second submission can incur more model usage.
Use **Stop** to interrupt a reply. Stopping does not erase usage already
incurred or release a reservation whose cost is still unknown.

## Refusals and interrupted replies

| Code | Meaning and next step |
|---|---|
| `allowance-exhausted` | The allowance cannot cover another reply; sign in to continue |
| `rate-limited` | A request or active-reply limit was reached; wait and try again |
| `visitor-required` | Reopen the public page with cookies enabled; an invalid or revoked visitor cookie cannot restore its allowance |
| `message-too-large` | The conversation request exceeds the size limit; shorten it or start a new conversation |
| `invalid-request` | The request is malformed or the message limit was reached; send a shorter text conversation from the public page |
| `request-already-used` | That message identifier was already submitted; it is not dispatched again |
| `request-cancelled` | The request was cancelled before model dispatch |
| `reply-interrupted` | The reply did not complete; if usage is unconfirmed, its reservation remains until reconciled |
| `model-temporarily-unavailable` | A bounded model request is unavailable; try again later |
| `public-marshall-unavailable` | The service cannot confirm admission or accounting; try again later |
| `origin-refused` or `proxy-proof-refused` | The request did not come through an accepted public-page connection; reopen the public page |

---

# Sign in and create an account

Source: https://docs.nationalcompute.com/sign-in/ (this markdown: https://docs.nationalcompute.com/sign-in.md)

One door at [nationalcompute.com](https://nationalcompute.com): **Log in
with Google**, or an email address that receives a sign-in link. Members
who were given a password use **I have a password** on the same page.
Programmatic access is described in [Authentication](https://docs.nationalcompute.com/authentication.md).
[Public Marshall](https://docs.nationalcompute.com/public-marshall.md) provides model chat without sign-in;
saved sessions, shells and persistent files require sign-in.

## The email link

Typing your address and choosing **Continue with email** sends one mail.
The link in it signs you in on the browser that opens it and expires 30
minutes after it was sent; a link found later does nothing. Anyone who
opens the link before then is signed in as you, so do not forward it. An
account with an authenticator app or a passkey is asked for it after the
link, as on a password sign-in.

A session started from an email link lasts 30 days on that browser.
Google and password sessions last 7 days. Signing out ends either at once.

## Who can create an account

What happens when you type an address depends on the address:

| Address | What happens |
|---|---|
| One the platform already knows (a member, an invitee, a Google account that signed in before) | You receive a sign-in link |
| A new one at an institution with a `.edu`, `.mil` or `.gov` domain | A confirmation mail; opening its link creates your account and your public research organization |
| A new one that an existing organization's [rules](https://docs.nationalcompute.com/api/organization.md#members-and-invitations) admit | A confirmation mail; opening its link creates your account and you join that organization |
| Any other new address | The waitlist: the address is recorded and you are emailed when a spot opens ([below](#the-waitlist)) |

A new account cannot sign in until the link in its confirmation mail is
opened; opening it signs you in. If the mail never arrived, type the same
address on the sign-in page again and a new confirmation is sent. Some
email providers delay or block inbound mail: if nothing has arrived after
a few minutes, email support@nationalcompute.com.

## Public research organizations

An account created for a `.edu`, `.mil` or `.gov` address comes with its
own organization, named `<you>-<institution>` from the address
(`jdoe@cs.stanford.edu` becomes `jdoe-stanford`; a `-2` suffix resolves a
collision). It is an organization like any other: you are its owner and
can add members, set [join rules](https://docs.nationalcompute.com/api/organization.md#members-and-invitations),
and buy credits. Public-research organizations cannot earn the
[feedback reward](https://docs.nationalcompute.com/api/mcp.md#feedback-rewards), even after buying credits;
their feedback is still recorded. The organization includes $100 of
credits to get started (a deposit of source `airdrop` on the
[ledger](https://docs.nationalcompute.com/api/billing.md#the-ledger)) and no clusters; its members request whole GPU
nodes, one at a time, from Marshall through [Public Research](https://docs.nationalcompute.com/public-research.md).
[`GET /api/whoami`](https://docs.nationalcompute.com/authentication.md#discovering-what-a-token-can-address)
and the MCP tool `grid_whoami` report it as `org_kind: public_research`;
every other organization reads `standard`.

A join rule for a whole `.edu`, `.mil` or `.gov` institution
(`stanford.edu`) is refused with `400`; a department or lab domain
(`lab.cs.stanford.edu`) or an individual address is accepted.

## Data sharing

Organizations created for `.edu` addresses take part in the
[Marshall data-sharing program](https://docs.nationalcompute.com/marshall-data-sharing.md#tiers) at tier 2
(analytics, transcripts and training artifacts) as a condition of the
public research program. The setting is managed by National Compute: the
organization's data-sharing page shows it as
[placed](https://docs.nationalcompute.com/marshall-data-sharing.md#changing-the-tier), and changes go
through National Compute. Organizations created for `.gov` and `.mil`
addresses start at tier 0 (usage analytics only) and may opt in from their
data-sharing page.

## The waitlist

Early access is available in waves for public users. An address the door
cannot admit today is placed on the waitlist: the address and the time are
recorded, nothing else, and the page says so. When a spot opens you receive
an email ("National Compute sign-ups are open") — come back and sign in with
the same address; your account and your own organization are then created
exactly as for a `.edu`, `.mil` or `.gov` address, including the welcome
credits. If you already had an account, the email says your organization is
ready and you only sign in. Signing in again before then does not move you
up the list.

Signing in with Google from an address outside the program creates your
account but no organization. The console then shows the waitlist page
instead: your address is recorded, and the moment a spot opens your
organization is created and the page you are on opens it — no need to sign
in again, and no need to wait for the email. The page also links to the
public console, which you can explore while you wait. The first time an
organization opens, Marshall may say "Setting up your organization" for up
to a minute while your workspace is placed and started; it continues by
itself.

Sign-ups may also be paused for a while (the sign-in page keeps working for
existing accounts). A `.edu`, `.mil` or `.gov` address that arrives during a
pause is queued first in line and emailed when sign-ups reopen. Signing in
with Google during a pause still creates your account; your organization
follows when sign-ups reopen, and the waitlist page opens it.

Questions, or something to tell us about what you want to run? Write to
[support@nationalcompute.com](mailto:support@nationalcompute.com);
organizations outside the public research program are set up by
arrangement.

---

# Burst capacity

Source: https://docs.nationalcompute.com/market/ (this markdown: https://docs.nationalcompute.com/market.md)

You are talking to a market. You state what you want and the most you'll
pay, and the market decides what you hold, tick by tick. The one way to
book capacity outright is [base load capacity](#base-load-capacity): a
fixed block of nodes for a fixed term at a fixed rate, outside the
auction.

## Declare, don't reserve

For a **VM cluster**, you declare two numbers with
[`PUT /api/vm/capacity`](https://docs.nationalcompute.com/api/vm-capacity.md):

- `max_gpus` — how many GPUs you want. Capacity is granted as whole
  nodes, so this must be a multiple of the `gpus_per_node` value the API
  returns (off-grid values are refused, never rounded).
- `max_price_per_gpu_hour` — the most you'll pay per GPU-hour, in USD.

For a **Kubernetes cluster** there is even less to say: your GPU jobs
create the demand by themselves. Each job is one request priced as a
gang in whole nodes on the market.
[`PUT /api/k8s/bid`](https://docs.nationalcompute.com/api/k8s-bid.md) sets your price ceiling. See
[Kubernetes clusters](https://docs.nationalcompute.com/kubernetes.md) for how that plays out.

## The market clears

Capacity is auctioned in short ticks (roughly every 10 seconds). Each
tick re-runs the auction over all demand in your market:

- While the market clears **at or under your ceiling**, nodes are
  granted toward your declared capacity.
- When it clears **above your ceiling**, nodes are reclaimed.
- You are billed **what the auction assesses, never above your
  ceiling** — usually less, though during a
  [protection window](#minimum-duration-protection) the assessed rate
  can be as high as your ceiling. There is no single posted price:
  rates are set per cluster (see below). See [Billing](https://docs.nationalcompute.com/billing.md).

Because grants follow the market, holding capacity is not guaranteed
indefinitely: a ceiling under the going rate leaves declared capacity
pending and can shed nodes you hold. Treat node-local data as ephemeral
— a reclaimed or swapped node is destroyed.

## Inside the auction

The mechanism is a repeated sealed-bid **combinatorial auction**:

- **Winner determination maximizes total value.** Demand competes in
  whole-node units; multi-node groups are atomic — granted whole or
  not at all, never split.
- **Prices are second-price in nature.** A winner pays for the demand
  it displaces, not its own bid (a VCG-style payment with a core
  adjustment), and your own cluster's losing demand never sets your
  price. Bidding your true ceiling is the honest, safe strategy —
  overbidding can't lower your rate, and you never pay above your
  declared ceiling. The corollary: raising your ceiling also raises
  the cap on what you can be assessed for nodes you *already* hold, so
  displaced demand your old ceiling was hiding re-prices immediately.
  Pick your true maximum once rather than probing upward in small
  steps — several small raises cost strictly more than one honest one.
- **Prices are per cluster, not uniform.** Different clusters can be
  assessed different rates in the same tick — which is why the
  [market feed](https://docs.nationalcompute.com/api/market-feed.md) publishes statistics (a
  volume-weighted mean and the min/max spread) rather than a single
  quote.
- **Arrival order is only a tie-break.** There is no first-come,
  first-served queue: price decides, and at equal bids the older
  demand wins.
- **A bid can be too low to ever clear.** The market refuses a
  declare that could never win capacity (`bid-too-low`), and demand
  priced that low simply waits. What wins is published, never posted:
  read the [market feed](https://docs.nationalcompute.com/api/market-feed.md) and the price-to-win
  ladder, and bid against the going rate.

## Minimum-duration protection

A freshly granted node carries a **protection window** (currently one
hour for a VM node, two hours for a Kubernetes node), counted from the
moment the node is usable: while it is open, no competing bid can take
the node — the grant is a property right, not a standing auction entry.
In exchange, during the window you can be assessed up to (but never
above) your limit price: on a VM cluster the bid that won the grant, locked for
the window even if you lower your ceiling; on Kubernetes the live limit
price (below) — a raise lifts the cap, and a cut below the price the job
held when its gang started voids the window. When the window lapses,
the node competes normally again.

For Kubernetes clusters the window belongs to the **node grant**, not
to the job:

- **The first hour is a minimum hold.** A granted node stays in your
  cluster — and [bills](https://docs.nationalcompute.com/billing.md) — for at least one hour, even if
  the job that asked for it finishes sooner; idle release starts only
  once that hour has run, and after it charges accrue in minimum
  increments of five minutes. The hold is deliberate on both sides:
  it gives you a stable window to work in — a job that crashes
  restarts on capacity you still hold instead of re-entering the
  auction — and it keeps sub-hour churn from misusing capacity others
  are waiting for.
- **Jobs come and go under it.** A job that finishes inside the window
  leaves the window on the node, which holds for a five-minute idle
  grace billed at that job's rate — provided the market would still
  clear that rate: the grace only holds capacity at a price that
  clears, so a node whose job's bid could never clear is released
  the moment it goes idle. A job that takes the node within the grace
  inherits the remainder and never extends it. Once the minimum hold
  has run, a node left idle past the grace is released, window and all.
- **The limit price stays live.** You may change a job's limit price
  at any time; an increase is always fine, and a decrease below the
  price the job held when its gang started (`Provisioned=True`) voids
  the window — while the limit price sits below that price, the job's
  nodes are preemptible at once. Raising it never resets the window.
- **A lost node is owed a replacement.** If a protected node stops
  responding to your cluster (its kubelet goes dark), your cluster is
  owed a replacement node for the remainder of the window (wall-clock —
  provisioning time counts against it). The right ends early only when
  the market reclaims the node, when the node is released after sitting
  idle past the grace, or when an operator moves it out of your
  cluster.
- **A node we take out of service is replaced, not billed.** When our
  health checks cordon one of your nodes, the market treats it as
  undelivered from that moment: billing for it stops, your job reads
  short by one node and the market grants a replacement that inherits
  the remaining window at the locked rate. The faulty node stays in your
  cluster, cordoned, while we investigate; your pods on it are evicted
  after a short grace so they reschedule onto the replacement. Once our
  checks clear it, the node returns to your cluster as idle capacity and
  is released after the idle grace unless a job takes it. A node you
  cordon yourself is your own decision: it keeps billing and earns no
  replacement.
- **Growth is unprotected.** A gang is granted whole, so a job is never
  half-protected; a new job, or a `Deployment`'s added replica,
  competes at your live ceiling until its own grant lands, and its
  window starts there.
- **A base load block being delivered outranks the window.** When another
  account's [base load block](#base-load-capacity) is owed nodes and the island
  has no free ones, the market takes idle nodes on the cheapest standing
  bids first, then busy ones — inside an open window too. Your pods get the standard
  [notice](https://docs.nationalcompute.com/kubernetes.md) and the job re-enters the auction as pending
  demand. The reservable inventory is capped per island precisely so
  this stays rare.

## Base Load Capacity

**Base load capacity** is the committed product beside the market: a
fixed block of GPU nodes, yours alone for a fixed term at a fixed rate.
No bidding and no preemption. A block bills for the full term whether
you use the nodes or not. It carries the
[Base Load Capacity SLA](https://docs.nationalcompute.com/sla.md). The console's Burst capacity page
shows it in the **Base Load** section, above the **Preemptible** section
that holds the auction.

- **What you get.** Your cluster holds at least the block's node count
  for the whole term. If a node in the block fails, the market delivers a
  replacement ahead of every bid on the island; time spent provisioning
  it counts against the SLA, not against your term.
- **Node specs.** On the island that offers base load today, each node
  has 240 vCPUs (AMD EPYC, Turin), 8× AMD MI355X 288 GB OAM GPUs, 3 TB
  RAM, 8× 3.84 TB NVMe and a 3200 Gbps scale-out network. All nodes are
  fully interconnected with each other and with the on-site scalable
  NFS, and preemptible burst nodes share that fabric with
  your base load GPUs and storage. The console lists the same lines
  under **Interconnected
  GPU Node Specs** in the **Base Load** section.
- **Your jobs and the block.** Submit jobs exactly as on any
  Kubernetes cluster; nothing in a manifest names the block. On
  each auction tick the market counts your cluster's jobs against the
  block's GPUs: jobs already running come first, then waiting jobs by
  [priority class](https://docs.nationalcompute.com/kubernetes.md#priority-orders-only-your-own-queue),
  oldest first within a class. A job that fits the free base load capacity is placed on
  the block's nodes without bidding and without a balance, within a couple
  of minutes of submission. A job that does not fit never holds up
  smaller jobs behind it; they are placed first, and the base load
  capacity left over still counts toward the larger job, which bids on
  the market only for the rest. When base load capacity frees, a job
  that was waiting on the market is placed on it, and a job already
  running on a market node is folded into the block where it
  stands, with no restart, so its market billing stops.
- **Small jobs on the block.** Where the platform has enabled packing
  onto the block for your cluster, a job smaller than one node is
  placed on the block by GPU count. Several small jobs share one block
  node. A small job that does not fit right now waits for GPUs of the
  block to free. It never bids on the market. Whole node jobs take an
  empty block node first. A small job opens an empty block node when
  no whole node job is waiting for it. After a block node has sat
  empty for the idle grace, a small job that waited that long takes
  it. A job whose pods are each smaller than one node and together
  exceed one node is not placed. It waits with the reason
  `ReservedTooSmall` until you resubmit it with whole node pods or
  with fewer pods. The market never moves a running small job to
  another block node. Without a live block a small job waits with the
  reason `ReservedBusy` until a block starts.
- **Burst beyond it.** Demand above the block's node count is ordinary
  preemptible burst capacity at your live ceiling. A job smaller than
  one node is never burst demand. Nothing else
  changes: your jobs create the demand, and Kubernetes decides which
  pods yield when a market node is reclaimed. The two-hour balance rule
  applies only to the part of a job that runs beyond the block.
- **Buying.** Any member of your organization buys a block on the console's Burst
  capacity page: the **Buy Base Load Capacity** card in the **Base
  Load** section offers fixed sizes (in GPUs, always whole nodes) for
  the posted terms, with each term's rate per GPU-hour. Pick one cell
  and the card shows the whole block's price; **Buy** completes the
  purchase. A size and term with no posted rate reads **sold out**; a
  size the island cannot fit right now reads **sold out until** the
  date enough earlier blocks lapse; a size larger than the island's
  reservable inventory reads **not available on this island**.
  When your credit balance covers the block's whole price, nothing more
  is needed. Otherwise buying needs a security deposit on your credit
  balance, the block's price up to $10,000, and you wire the remaining
  amount by the next business day. The block bills across its term.
  A block starts immediately and appears under **Your Base Load
  Capacity**. A cluster
  holds one live block at a time; a second purchase for the same cluster
  is refused (`one-per-cluster`) until the first ends, and the page
  folds the buying card behind one line while a block is live. On an
  island where buying on the page is not open, the card says so and your
  account team arranges the block.
- **Changing it.** Changes are made with your account team. A change to
  the rate or the size ends the current block at that moment and
  starts a new one with the new figures, so billing is exact on both
  sides; the end date and notes can be changed in place.
- **Billing.** The block's rate is charged to your credit balance
  across the term, in the same five-minute cadence as everything else
  on [Billing](https://docs.nationalcompute.com/billing.md), for every node in the block — idle or busy,
  delivered or being replaced. Work inside the block is never
  metered by the market, and the block's nodes never appear in the
  market's clearing rates.
- **At the end of the term.** There is no automatic renewal. At the
  end date the block's nodes stay in your cluster as ordinary market
  members and step down under the standard protection window from that
  moment, locked at your declared price — or, when you have not
  declared one or yours is too low to ever clear, at the lowest rate
  the market clears. A node running your
  work keeps running until the window closes and bills the locked
  price. An idle node returns to the market after the usual idle grace;
  the step down carries no minimum hold, because the term you paid for
  is over. When the window closes the node competes at your live
  ceiling on each auction tick, and a ceiling too low to ever clear
  releases it. A job smaller than one node does not step down with its
  node. At the end date it is requeued at once as a new request. That
  request waits with the reason `ReservedBusy` until your cluster
  holds a live block again. It never moves to a market node. Arrange a
  renewal with your account team before the end date if you need
  continuity.
- **Ending early.** A live block keeps running when your balance goes
  negative, while the billing hold reclaims your other market capacity
  as usual. When the balance has stayed at or below zero for 24 hours,
  the block is cancelled at that moment: its nodes return to the market
  through the standing hold, billing for the block stops, and you can
  buy again once the balance is restored. The market's own capacity,
  protection windows and pricing are untouched by blocks you do not
  hold.

## A bid that can never clear

A declare too low to ever clear the market is refused outright
(`bid-too-low`): pending demand costs nothing, but a bid the market
can never meet means nothing will ever arrive, so the write refuses it
where you can see why. A Kubernetes cluster with no limit price
effectively bids $0 — its jobs create demand, but nothing is granted
until you set a real ceiling.

Market conditions can also move past a **standing** bid (the
write-time check never re-runs). The capacity and limit price reads
then report `bid_too_low: true`, and inside a Kubernetes cluster the
verdict lands on the job itself as a `BidTooLow` event. What to bid is
read off the market's own numbers — the
[market feed](https://docs.nationalcompute.com/api/market-feed.md) and the price-to-win ladder — never
off a posted price. See
[how the bid prices the market](https://docs.nationalcompute.com/api/k8s-bid.md#how-the-bid-prices-the-market)
for the Kubernetes mechanics.

## Prices come back to you

The [market price feed](https://docs.nationalcompute.com/api/market-feed.md) publishes what capacity
actually clears at, tick by tick and as a long-run series, so you can
tune your ceiling against the going rate instead of guessing.

## Scaling down is always free

Lowering `max_gpus` needs no market's permission and always works —
declare `0` to release everything. The market gates only
what you *acquire*, never what you give back. On scale-down you can
nominate which nodes go first (the `release` list on the capacity
declaration), and released capacity simply stops billing — there is
nothing to resell or wind down.

---

# Kubernetes clusters

Source: https://docs.nationalcompute.com/kubernetes/ (this markdown: https://docs.nationalcompute.com/kubernetes.md)

A Kubernetes tenancy is a **dedicated cluster**: you hold the
kubeconfig, you deploy your own workloads, and the platform moves GPU
nodes in and out of your cluster as your jobs demand them. Nobody runs
your containers for you — the cluster is yours; the platform only decides
how many nodes it has.

Cluster nodes run the Kubernetes agent directly on the metal — there is
no hypervisor layer in the Kubernetes offering.

## Connecting

The kubeconfig is a public client and the cluster's certificate. It
carries no secret. Fetch it with your org API token:

```sh
curl -fsSL -H "Authorization: Bearer $NC_TOKEN" \
  "https://nationalcompute.com/api/k8s/cluster/kubeconfig?cluster=<cluster>" \
  -o ~/.kube/<cluster>.yaml
```

The token names your organization, so the route resolves `<cluster>`
inside it. Another organization's cluster can never answer, whatever its
name. The first `kubectl` call prints a sign in link. A person opens
the link once and approves. After that `kubectl` refreshes its own sign
in while it is in use. A day without a `kubectl` call needs one more
approval: the next call prints a fresh link and waits on it. A
`kubectl` call that seems hung is showing a sign in link — agents run
`kubectl` where its output stays readable while the command runs, and
relay the link to a person.

The setup script installs `kubectl` and the sign in plugin when they are
missing and merges a `<cluster>` context into `~/.kube/config`:

```sh
curl -fsSL https://access.nationalcompute.com/setup.sh | sh -s -- you@example.com <cluster>
```

The script reads the token from `NC_TOKEN` or from
`~/.config/nationalcompute/token` and fetches the kubeconfig by token
when it finds one. Without a token it asks the public onboarding path by
name. That path refuses a name that more than one cluster carries with
`409 ambiguous-cluster` and points at the token route.

The script never waits on the sign in: run without a terminal, it
prints the link and exits while sign-in polls in the background.
Approval lands the token on its own, and
`kubectl --context <cluster> get nodes` then answers at once. A fresh
link in its place means the code expired; the new link replaces the
old.

Workloads and automation authenticate to the cluster with ServiceAccount
tokens rather than a sign in, and every cluster is a public OIDC issuer
for them — see [Authentication](https://docs.nationalcompute.com/authentication.md#kubernetes-service-account-tokens).

## Demand comes from your jobs

There is nothing to declare except your price. The market derives your
demand from the jobs you submit, through Kueue — the queue installed in
your cluster that holds a job until the market grants its nodes:

- **A GPU job is one request for whole nodes.** `kubectl apply` a
  `Job`, `JobSet` or `MPIJob` whose pods request full-node GPU
  multiples; Kueue creates it suspended and files one request for the
  whole gang — every pod the job needs, priced together. When your limit price
  clears, the nodes join and every pod starts in the same moment, on
  exactly those nodes. Nothing else is needed: no queue name, no
  `suspend: true`, and the toleration for the GPU worker taint is
  injected on admission.
- **A job smaller than one node runs on your base load block.** On a
  cluster where the platform has enabled packing onto the
  [base load block](https://docs.nationalcompute.com/market.md#base-load-capacity), a pod may request
  fewer GPUs than one node. Kueue files one request for it. The market
  seats it on a block node beside your other small jobs. It never
  bids. It never runs on a market node. Without a live block it waits.
- **A waiting job has no pods.** `kubectl --context <cluster> get pods` is empty until the
  whole gang is granted; a gang is fully waiting or fully running, never
  "1 of 2". The exceptions are divisible: a `Deployment` replica or a
  bare GPU pod is its own request and exists as a pod held
  `SchedulingGated` until granted.
- **Idle nodes return to the pool.** Once a node's one-hour
  [minimum hold](https://docs.nationalcompute.com/market.md#minimum-duration-protection) has run, a node
  whose job has ended sheds after an idle grace of five minutes, billed
  at that job's rate, unless another job takes it first — the grace
  covers between-job gaps, so batchy workloads don't churn nodes.
- **CPU-only work is outside the market.** A job requesting no GPUs
  runs at once on the cluster's CPU worker (below) and neither bids nor
  holds a GPU node. It passes through the queue without a market check,
  so it can show `Suspended` for a moment.

The result is that capacity — and [billing](https://docs.nationalcompute.com/billing.md) — tracks what
you actually run: when the queue drains, nodes shed and the meter
stops, with no action from you.

### Reading the queue

The queue is read on the Job, never on pods. `kubectl --context <cluster> get job` shows
the Job `Suspended`; `kubectl --context <cluster> get workloads` lists its Workload (Kueue's
queue entry for the job, `<kind>-<jobname>-<hash>`) and whether it is
admitted; `kubectl describe job` carries the market's verdicts as
Events — one per change of reason, then `NodeGranted` per granted
node. The same verdict is the `Provisioned=False` condition of the
ProvisioningRequest — the object Kueue files to ask the market for the
gang's nodes, `<workload>-market-<attempt>` — read with
`kubectl describe provisioningrequest`, and is copied into the
Workload's admission check; once granted, the condition reads
`Provisioning` (`k of N granted nodes ready`) until every node is ready
and it turns `Provisioned=True`. The reason is empty until the market's
first read, one tick after the request appears; the
[limit price API](https://docs.nationalcompute.com/api/k8s-bid.md#queue-events) tables every reason. The
console's Workloads page lists the queued job before it has pods, with
the same reason, and marks a running gang **gang scheduled**.

## The cluster CPU worker

Newly provisioned clusters come with one small CPU-only worker node in
addition to the control plane. It is a plain schedulable node with the
`cpu` role in `kubectl --context <cluster> get nodes`. It carries no GPUs and no taints, so
untolerating pods (queues, controllers, data prep, small services)
land there by default while GPU jobs wait in the queue.

The CPU worker sits outside the market: it never bids, never sheds on
idleness, and is never reclaimed when the market clears above your
ceiling. It is created with the cluster and removed with the cluster.
It cannot be resized or multiplied; for CPU capacity beyond it, run
CPU work on GPU nodes you already hold or contact support.

## The limit price { #the-bid }

[`PUT /api/k8s/bid`](https://docs.nationalcompute.com/api/k8s-bid.md) sets the most you'll pay per
GPU-hour — your limit price. Each request bids its gang's GPU total ×
your limit price, in whole nodes, so a request's exposure is its node
count × `gpus_per_node` × your ceiling. A request smaller than one
node bids nothing. It waits for GPUs of your base load block. A job
carries its own limit price in the label
`nationalcompute.com/limit-price: "<price>"` (USD per
GPU-hour, as a quoted string), which replaces the cluster's for that job; `"0"` is a real
zero limit price — the job waits with the reason `BidTooLow` and never
falls back to the cluster's price. The limit price is read live at every tick for the
request's whole life, waiting or granted: you may change a job's limit
price at any time. An increase is always fine. A decrease below the
price the job held when its gang started (`Provisioned=True`) voids its
protection window ([below](#when-the-market-reclaims-a-node)).

With **no limit price set, your jobs bid $0**, which never clears:
the demand is visible, but nothing is granted until you declare a real
ceiling. A limit price too low to ever clear the market is refused at
write time (`bid-too-low`); a standing one the market moved past reads
`bid_too_low: true` on the limit price read, and its requests wait
with the reason `BidTooLow` — they never fail — and clear at the first
tick after you raise the price. See
[how the bid prices the market](https://docs.nationalcompute.com/api/k8s-bid.md#how-the-bid-prices-the-market)
for the arithmetic.

## The max cluster size

Every cluster has a max cluster size in GPUs — set by the platform,
read-only to you (an edit of the ClusterQueue, or of any other Kueue
object the platform owns, is refused with `StationOwned`), shown on the
cluster's console page and in-cluster as the quota of Kueue's
ClusterQueue `market` (`kubectl --context <cluster> get clusterqueue market`). While the
GPUs of your running and requested jobs stay under it, every new job
files its request at once. Beyond it, jobs wait in your own queue and
file their requests as room frees: the Workload's `QuotaReserved`
condition names the quota as the cause (`kubectl describe workload`),
and the console's Workloads page lists the job waiting, with no request
filed yet. A job larger than the limit on its own never reaches the
market; ask support to raise it.

## Priority orders only your own queue

Priority orders a job against your other jobs, and only while the max
cluster size binds: set the label `kueue.x-k8s.io/priority-class` to
`low`, `normal` or `high` on a suspended Job — a job without the label
ranks below `low` — and the higher class files its request first when
room frees. Every job under the limit reaches the market at once, where
price decides — priority never buys market position over another tenant
and never evicts a running job. On a cluster with a
[base load block](https://docs.nationalcompute.com/market.md#base-load-capacity) the same order decides who takes the
GPUs of the block as they free: the higher class first, oldest first within a
class — so a serving Deployment marked `high` reclaims its node in the block
after a rollout ahead of waiting batch work.

## When the site has no capacity

A shortage is not a pricing problem. When the site runs out of
capacity of the class your jobs request, no bid clears it: the request
stays in the queue, the console's Workloads page shows the verdict as
the reason **capacity unavailable**, and the
[market feed](https://docs.nationalcompute.com/api/market-feed.md)'s ticks read `null` while nothing
clears. Raising your limit price does not start the job sooner. The request's
own reason does not distinguish a shortage — it reads `PendingSupply`,
`Outbid` or `Pending` as usual; the console's Workloads page is where
the shortage verdict shows.

There is nothing to do. Pending demand costs nothing. Leave the limit
price where it is; the platform checks for returning capacity on its own, and
the job starts when capacity returns, with no action from you.

## Launch admission

Submitting a GPU workload is checked at admission on market-managed
clusters — a violation is refused at `kubectl apply` with the exact
numbers in the error, never left silently queued:

- **Full nodes on the market** (`JobSizeTooSmall`). Every pod template
  of a `Job`, `JobSet` or `MPIJob` must request a multiple of
  `gpus_per_node` GPUs. The same rule binds a bare pod. A multi node
  job runs as several full node pods. Templates requesting no GPUs (an
  MPI launcher, for one) are exempt. The market assigns whole nodes
  for strong isolation. A partial node GPU pod is refused rather than
  priced. The exception is a cluster with packing onto the base load
  block enabled ([above](#demand-comes-from-your-jobs)). There a pod
  may request fewer GPUs than one node. Such a pod runs on the block
  only. A pod larger than one node must still request a whole number
  of nodes (`JobSizeNotNodeAligned`).
- **Minimum balance** (`BalanceTooLow`). The org's credit balance must
  cover **two hours** (currently) of the job at your limit price — 2 × limit price ×
  the job's GPU total, counted across `parallelism` (or replicas). The
  denial names the required and current balance. The rule reduces
  churn — a job that takes nodes must be fundable at its own limit price for
  more than minutes. The denial binds at apply only — a job already in
  the queue is never refused later; a shortfall while it waits pends
  the request instead (below).

The market runs the same two-hour balance check again at every tick
while the request waits, in case the situation has changed: a request
the org's balance can no longer fund stays in the queue with the reason
`BalanceTooLow`, whose message names the balance available and what the
request costs ([queue events](https://docs.nationalcompute.com/api/k8s-bid.md#queue-events)); a top-up
clears it at the next tick and nothing fails.

The platform can switch packing onto the block off for a cluster. From
that moment the full node rule binds again at admission: a new pod
smaller than one node is refused with `JobSizeTooSmall`. A Deployment
update and a Job retry create new pods and are refused the same way.
Pods already running are never touched. Resubmit the work as whole
node pods until packing is switched on again.

## When the market reclaims a node

If the market clears above your ceiling (or you withdraw the limit price), a
gang's nodes are reclaimed: the whole job is suspended, its pods are
deleted together, and the job requeues as a new request at your current
limit price — it resumes when granted again, on whatever nodes the
market grants then. A reclaim is never a failed job: the Workload goes
back to pending, and with the spec below in place it does not consume
the Job's `backoffLimit`. On the console's Workloads page the requeued
job's row shows the current episode — queued since the requeue, running
since the last grant — and its Run History the episodes and the totals.

A reclaim caused by a **higher bid from another tenant** against your
standing limit price comes with **one minute's warning**. A reclaim
caused by your own account — your limit price becomes too low to ever
clear, you withdraw or lower it, or your credit balance runs out — is
**immediate**, even when another tenant takes the node: no advance
warning, the drain starts at once, and pods get the same 60-second
eviction grace described below. At notice time the request
gains a `PreemptionNotice=True` condition (reason `MarketPreemption`)
naming the deadline, the node is cordoned and tainted, a
`NodePreempting` event is posted on the node and on each of its pods,
every pod gets a `marketplace.nationalcompute.com/preempt-at`
annotation carrying the deadline (readable in-pod through a
downward-API volume — annotation files update live), and the pods are
evicted through the Eviction API with the window as their grace period
— SIGTERM at notice is your checkpoint signal, and `preempt-at` the
deadline to checkpoint by. PodDisruptionBudgets are honored inside the
window, never past it. At the deadline the whole job is suspended and
requeued: the market revokes the request (it reads `Failed=True`,
reason `MarketRevoked`), and Kueue evicts the Workload, deletes
whatever pods remain and files the new request. Set
`terminationGracePeriodSeconds: 60` on your pods so their own grace
matches the window. A reclaim withdrawn mid-notice uncordons the node,
flips the condition to `PreemptionNotice=False` (reason `Rescinded`)
and posts a `PreemptionRescinded` event on the node.

A granted node that leaves your cluster after the request turns
`Provisioned=True` ends the request the same way: the request reads
`Failed=True` with reason `MarketRevoked` and the job requeues under a
fresh request at your current limit price.

## When we take a node out of service

A node our health checks cordon is replaced, not reclaimed. The market
stops counting it toward your grant the moment the cordon lands: billing
for it stops, your gang reads short by one node, and a replacement is
granted that inherits the remaining protection window at the locked
rate. The faulty node stays in your cluster, cordoned and tainted, while
we investigate; the pods on it are evicted with a 60-second grace so the
job reschedules them onto the replacement once it joins — the
replacement carries your request's node label, so the pinned pods place
themselves. Nothing changes on the request: it stays `Provisioned=True`
throughout. In the console the node's tile reads **Node Fix In
Progress** and the request's row shows **Replacement Pending** until the
replacement joins. The replacement claim belongs to your cluster, not to
one job: whichever of your pending requests the market seats first takes
it, at the rate your cluster locked for the node that failed, and that
request's row says "protected at the cluster's locked rate" when that
rate is above its own bid. When our checks clear the node it is
uncordoned and returns as idle capacity under the usual idle grace. Cordoning a node yourself
does none of this: an unmarked cordon is read as your own scheduling
choice, the node keeps billing, and no replacement is owed.

Design for reclaims the way you would for any preemptible capacity:

- Checkpoint long-running training; make jobs resumable.
- Treat node-local disk as ephemeral; keep durable state on shared or
  external storage.
- Watch the [market feed](https://docs.nationalcompute.com/api/market-feed.md) and keep your ceiling
  above the going rate for work that shouldn't be interrupted.

Without the spec below, every evicted pod counts against the Job's
`backoffLimit` (default 6) and the Job creates replacement pods while
the evicted ones are still terminating — a long-running Job could fail
from reclaims alone. Exempt reclaims from the retry budget with
`podReplacementPolicy: Failed` (no replacement pod while an evicted pod
is terminating) plus `podFailurePolicy` Ignore rules for the
`DisruptionTarget` condition the eviction sets and for your SIGTERM
exit code (143 for `sh` and any process that dies on the signal;
`podFailurePolicy` is valid only with `restartPolicy: Never`):

```yaml
spec:
  podReplacementPolicy: Failed
  podFailurePolicy:
    rules:
      - action: Ignore
        onPodConditions:
          - type: DisruptionTarget
            status: "True"
      - action: Ignore
        onExitCodes:
          operator: In
          values: [143]
  template:
    spec:
      restartPolicy: Never
      terminationGracePeriodSeconds: 60
```

A granted Kubernetes node carries the same minimum-duration protection
window as a VM node, anchored at grant — counted from the moment the
node is usable; currently two hours on Kubernetes against one on VM
clusters ([minimum-duration protection](https://docs.nationalcompute.com/market.md#minimum-duration-protection)).
Its first hour is a minimum hold: the node stays in your cluster and
bills for at least one hour even if the job finishes sooner — which also
means a job that crashes restarts on the nodes it already had. The
window belongs to the node grant, not to the job, so it survives job
turnover on the node — and a node that stops responding to the cluster
is owed a replacement for the remainder of its window (a node an
operator moves out is not). The window does not lock the price: while a
job's limit price sits below the price it held when its gang started
(`Provisioned=True`), its nodes carry no window and are preemptible at
once; raising the price never resets or voids the window.

## Exposing services: `type: LoadBalancer`

Serving needs a stable public endpoint that survives node churn. Create
a standard Kubernetes Service with `type: LoadBalancer` and the
platform provisions an external TCP load balancer for it:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: serve
spec:
  type: LoadBalancer
  selector:
    app: serve
  ports:
    - port: 443
      targetPort: 8443
```

Within a couple of minutes the Service's `status.loadBalancer.ingress`
carries a **static public IP**. The console shows the same endpoint on
the cluster's Workloads page beside the Service. The IP is stable for
the life of the Service. It holds through node churn as the market
moves nodes in and out of your cluster. Deleting the Service, or
changing its `type` away from `LoadBalancer`, releases it.

What to know:

- **TCP only.** UDP ports on a Service are skipped. A Warning Event on
  the Service says so.
- **Every Ready worker backs the endpoint** at the Service's NodePort.
  Keep `allocateLoadBalancerNodePorts` enabled (the default).
  `externalTrafficPolicy: Local` works as expected: health checks
  route around nodes with no serving pod.
- **Your pods see the client's real source IP.** The path performs no
  source NAT. IP allowlists and client keyed rate limits work inside
  your workload.
- **Refusals are Events.** When the external IP stays `<pending>`, run
  `kubectl --context <cluster> describe service <name>`. The Event names the reason:
  unsupported protocol, no allocated NodePort, no Ready workers, or a
  limit.
- **Limits exist.** Each cluster has a load balancer cap. The platform
  also has a shared pool limit. The Event names which one you hit; ask
  support to raise it.
- A Service that sets `spec.loadBalancerClass` belongs to whatever
  controller you run for that class. The platform leaves it alone.

Load balancer provisioning is enabled per region. Where it is not yet
enabled, the Service stays `<pending>`; ask support.

## The shared volume

Clusters provisioned with shared storage come with one **2 TiB shared
volume**, mounted at the same path on every node (the control plane
tells you where: `/mnt/shared` today). Clusters created before
September 2026 carry 1 TiB. The Storage page shows each volume's size. Use it from pods through a
`hostPath` volume at that path. It is the place for datasets,
checkpoints and outputs that must outlive any one node. The disk on a
node is ephemeral and goes with the node when the market reclaims it.

The volume is created with the cluster and attached to every node that
joins, GPU nodes and the CPU worker alike. Where storage billing is
enabled for your site, the volume is billed on the bytes it holds and
not on its size ([billing](https://docs.nationalcompute.com/billing.md)); your Storage
page shows the current charge when that is the case. When you delete
the cluster the volume is preserved by default, so your data survives
the teardown. A preserved volume stays yours until you delete it.

### Seeing and deleting your volume

The console's **Storage** page lists every shared volume your
organization owns. Each volume shows its label (`<cluster>-shared`),
the storage name under it when the two differ, the cluster it is
attached to (or "preserved, no cluster" once that cluster is gone),
the bytes it holds, and its current charge where storage billing is
enabled for your site.

To delete a volume, use **Delete volume** on its row. The delete is
irreversible: it destroys the volume and every file on it, and the
platform keeps no copy. You confirm by typing the volume's storage name
back exactly, the name shown under its label. Where storage billing is
enabled, billing for the volume stops once the delete is accepted.

Two rules apply:

- **An attached volume cannot be deleted.** Delete the cluster first.
  The volume detaches with the cluster and is preserved by default, so
  it then appears on the Storage page as "preserved, no cluster" and
  can be deleted from there.
- **Deleting a cluster preserves its volume unless you opt in.** The
  cluster delete offers a checkbox, "also delete my storage volume",
  unchecked by default. Leave it unchecked and the volume and its data
  survive (and stay billable where storage billing is enabled). Tick
  it and the volume is destroyed with the cluster. The platform still
  refuses to destroy a volume while any node holds it; in that case the
  volume is preserved and can be deleted later.

Listing your volumes is available to automation through the
[Kubernetes API](https://docs.nationalcompute.com/api/k8s-bid.md#shared-storage-volumes). Deleting one
is a console action only: the API refuses a token with
`session-required`, whatever its scope. A limit price is reversible. Data
destruction is not, so the irreversible delete keeps a human in the
loop with the typed confirmation.

## What you'll never see on this surface

No node IPs, no slots, no release verbs — node membership is entirely
platform-managed. Your controls are the limit price, your job specs (GPU
requests, limit price, priority), and your own cluster's scheduling
configuration.

---

# Slurm clusters

Source: https://docs.nationalcompute.com/slurm/ (this markdown: https://docs.nationalcompute.com/slurm.md)

A Slurm tenancy is a **dedicated cluster**: a login node you ssh into,
a controller triplet the platform runs, and GPU compute nodes the
platform moves in and out of your cluster. You submit jobs with the
standard Slurm commands (`srun`, `sbatch`, `squeue`, `sacct`); nobody
runs them for you. Slurm 26.05 with the enroot and pyxis container
plugins is installed on every node.

## Connecting

Every Slurm cluster has one login account, `tenant`, with root
privileges through `sudo`. It is a member of the GPU device groups
(`video`, `render`) on every node, so GPU tools and runtimes work
without `sudo`. The account accepts the SSH public keys the platform
team registered for your organization; there is no password and no
sign-in flow.

```sh
ssh tenant@<login-address>
```

The login address is on your cluster's page in the console under
Connect. To add or remove a key, send the platform team the public key
line (`ssh-ed25519 AAAA…`); the change reaches the login node within a
minute of being registered. Keys you append to `~/.ssh/authorized_keys`
yourself work on the compute nodes as well, because the home directory
is shared across the cluster (see [Storage](#storage)) — the login's
managed set is the one the platform keeps.

The setup script writes an `ssh <cluster>` alias for the account when
the cluster uses key login:

```sh
curl -fsSL https://access.nationalcompute.com/setup.sh | sh -s -- you@example.com <cluster>
```

## Jobs

Jobs run in the `main` partition. Request GPUs with `--gres`; each
requested GPU brings a default CPU allocation sized for collective
libraries, so a whole-node job needs no CPU flags:

```sh
srun -N1 --gres=gpu:8 hostname
sbatch --nodes=4 --ntasks-per-node=8 --gres=gpu:8 train.sbatch
```

| limit | value |
|---|---|
| maximum wall time (`--time`) | none — a job runs until it finishes or reaches the `--time` it asked for |
| default wall time when `--time` is absent | unlimited |
| default CPUs per requested GPU | 28 (currently; sized so a `gpu:8` job holds the node) |
| GPUs per compute node | 8 |

A job that requests fewer GPUs than a node has shares the node with
other jobs; `ROCR_VISIBLE_DEVICES` inside the job lists the GPUs it
holds, and the others are not visible to it. The whole GPU-node memory
is allocatable to a full-node job.

Accounting is per cluster: every job is charged to the cluster's single
account, and `sacct` shows it with `gres/gpu` in `AllocTRES`. There are
no per-user limits, quotas or fair-share weights.

## Job hooks

Every compute node runs the executables in two directories around each
job, as root, with Slurm's job environment (`SLURM_JOB_ID`,
`SLURM_JOB_USER`, `SLURM_JOB_GPUS`):

| directory | when | time limit per hook |
|---|---|---|
| `/etc/slurm/prolog.d/` | before the job's first step, after the platform's health gate has passed | 20 s |
| `/etc/slurm/epilog.d/` | after the job's last step, including cancelled and timed-out jobs | 60 s |

Hooks run in name order. A hook that exits non-zero or reaches its time
limit is logged to the node's syslog (tags `slurm-prolog` and
`slurm-epilog`) and otherwise ignored: it cannot fail the job and cannot
drain the node. Files without the executable bit are skipped.

The directories are yours; write to them with `sudo` on each node. They
are per-node state: a node that joins your cluster arrives with both
directories empty, and a node that leaves is re-imaged. Keep the hook
sources on the shared volume and install them onto new nodes.

Per-job GPU accounting is the intended use. Every GPU node runs AMD's
ROCm Data Center daemon, `rdcd`, unauthenticated on `localhost:28051`,
so every `rdci` call takes `-u --host localhost:28051` after its subcommand and needs no certificates. The daemon is read-only: telemetry
and job statistics, no power, clock or reset controls. A start hook
records against a GPU group you created with `rdci group -c <name> -u --host localhost:28051`, and a
stop hook reports:

```sh
# /etc/slurm/prolog.d/50-rdc-stats
rdci stats -s "$SLURM_JOB_ID" -g <group id> -u --host localhost:28051

# /etc/slurm/epilog.d/50-rdc-stats
rdci stats -j "$SLURM_JOB_ID" -u --host localhost:28051 >> "/mnt/shared/jobstats/$SLURM_JOB_ID.txt"
rdci stats -x "$SLURM_JOB_ID" -u --host localhost:28051
```

## Containers

Every compute node runs jobs inside a container image when you pass
`--container-image`; nothing is installed on the cluster for that. The
GPUs and the fabric devices of the allocation are visible inside the
container.

```sh
srun -N1 --gres=gpu:1 --container-image=rocm/pytorch:latest python -c "import torch; print(torch.cuda.is_available())"
```

| flag | effect |
|---|---|
| `--container-image=<registry>#<repo>:<tag>` or a `.sqsh` path | image to run (registry `#` separates host and repository) |
| `--container-name=<name>` | keep the unpacked image on the node between jobs; the first start of a 20 GB image takes minutes, later starts seconds |
| `--container-mounts=/mnt/shared:/mnt/shared,/scratch:/scratch` | bind host paths into the container; only your home directory is mounted by default, the shared volume and the node-local scratch are not |
| `--container-writable --container-save=<path>.sqsh` | build an image inside a job and save it to the shared volume for later jobs |
| `--mpi=pmix` | the launcher for MPI and collective programs across nodes |

The first pull of an image is per node. Store images you reuse on the
shared volume and reference the `.sqsh` file.

## Storage

| path | scope | what it is |
|---|---|---|
| `/home` | login and every compute node | your home directory, on the cluster's shared volume; the same files everywhere |
| `/mnt/shared` | login and every compute node | the shared volume itself: datasets, images, checkpoints; sized at cluster creation |
| `/scratch` | one compute node | node-local NVMe scratch, writable by any user; not shared, wiped when the node leaves your cluster; inside a container only with `--container-mounts=/scratch:/scratch` |

Deleting the cluster preserves the shared volume and every byte on it;
a new cluster cannot take the same name while the preserved volume
exists, so ask the platform team to reattach or destroy it. Node-local
scratch is destroyed with the node.

## Controllers and failover

Three controllers run the scheduler; one is primary and two are
backups, with the scheduler state on the shared volume. If the primary
stops, a backup takes over within about 2.5 minutes. Running jobs keep
running through the takeover; `squeue` and new submissions wait until
the backup is in control. When the original controller returns it
resumes control within a minute, again without touching running jobs.

## Nodes

The platform adds and removes compute nodes; you see them in `sinfo`
under their assigned names. A node under maintenance is drained first,
so running jobs on it finish, or reach the `--time` they asked for, before
it leaves. There is no API for adding nodes to a Slurm cluster — ask the
platform team.

## Monitoring

Your cluster's page in the console has a Monitoring tab with one
dashboard per view, each scoped to the selected cluster and refreshed
every minute. The data comes from the scheduler itself and from an
exporter on every compute node.

| dashboard | what it shows |
|---|---|
| GPU Nodes | per node: GPU utilization, memory, power and temperature; host CPU, memory, disk and network |
| CPU-Only Nodes | the same host view for nodes without GPUs; present only when the cluster has them |
| Node Network | public ingress and egress volume and network errors per node |

The dashboards cover the nodes, not the scheduler: queue depth, job
states and drain reasons come from `squeue`, `sinfo -R` and `sacct` on
the login node, and `sacct` is the record of finished jobs, including the
GPUs each one held. An agent reads the same node states, the queue, a
job's record and its log without a login through the MCP server's
[`slurm_read`](https://docs.nationalcompute.com/api/slurm.md), and a node's telemetry through
[`metrics_read`](https://docs.nationalcompute.com/api/metrics.md). There are no alerts and no notifications; a pending
job with idle GPUs is something you notice, not something the platform
tells you.

---

# Public Research

Source: https://docs.nationalcompute.com/public-research/ (this markdown: https://docs.nationalcompute.com/public-research.md)

Public Research is one whole GPU node at a time for an individual
researcher, requested from Marshall and used over `ssh`: no cluster to
set up, no limit price to pick, no auction to wait on. A member of a
[public research organization](https://docs.nationalcompute.com/sign-in.md#public-research-organizations)
asks Marshall for a node, waits in a first-come, first-served queue,
holds the node for a lease, and releases it; the meter runs from the
moment the node is theirs to the moment it is not. The organization an
eligible `.edu`, `.mil` or `.gov` address receives at sign-in is one;
National Compute sets the kind for any other organization by arrangement.

## The node

A node is yours whole: every GPU in it, its CPUs, memory and local NVMe,
with nobody else on it. Nodes come in two shapes, 8 GPUs each: 8× AMD
MI355X 288 GB or 8× NVIDIA B300 288 GB. Marshall's quote states the GPU
model, the GPU count and the memory per GPU of each node on offer, and
the quote is the authority whenever this page and the quote differ.

The local NVMe is scratch. Releasing a node, or losing it to a preempt,
rebuilds it in place and destroys every byte on it; there is no undo and
no recovery. Copy results to your workspace disk, or to storage of your
own, before you release; a public research organization has no shared
folder. Marshall says so before the first request and again as the lease
nears its end.

What a node does not have: a second node to pair with (no multi-node
jobs, no RDMA or scale-out fabric between pool nodes) and no notebook
server (`ssh` is the only way in).

## Asking Marshall

Marshall is the one door: there is no console form, no API route and no
MCP tool for a request, and only a member of a public research
organization is offered one. Marshall quotes before it asks: the hourly
figure for the node, the worst case for one full lease (lease length ×
GPUs × rate), the balance a request needs, and whether your job fits one
node. A job that needs more than one node belongs on the
[market](https://docs.nationalcompute.com/market.md), not here. Every request ends in a confirmation
card, Full Access or not, because a request starts a meter.

With two node shapes open, Marshall lists both offers, each with its
rate per GPU-hour, the worst case for one full lease and the current
queue, and marks one **Recommended** from your job's software stack: a
CUDA stack points to the NVIDIA B300, a ROCm stack to the AMD MI355X;
when both fit, the cheaper offer, then the shorter queue. The mark is a
mark, not a choice: you pick either row, and the pick is the
confirmation. A request must name one offer; one that names none is
refused ("site required") with the offers listed, and one naming an
offer that is not open is refused with what is open. While only one
shape is open, Marshall quotes it alone and asks as before.

A request needs a balance that covers the first hour of a lease at the
quoted rate (GPUs × rate), not the whole lease; past that hour the meter
runs against your balance like any other charge. A public research
organization starts with $100 of credits (the `airdrop` deposit on its
ledger); once those are spent a request is refused until credits are
bought on the console's Billing page
([card or wire](https://docs.nationalcompute.com/billing.md#buying-credits)); Marshall names the amount
it needs. A request is also refused while a
[billing hold](https://docs.nationalcompute.com/api/billing.md#hold-state) is active.

## The queue

Requests are served first come, first served, one node per request. A
request is `queued` from the moment it is accepted; Marshall reports
your place in line first and then an estimated wait. The estimate takes
the nodes in the pool, the requests ahead of you and the time left on
the leases running now, and assumes every lease runs its full length; a
release before that moves every estimate behind it earlier, so it is an
estimate and never a promise. The line itself moves only when a lease
ends or an idle node is added. When a node is free for you,
handover takes seconds: the platform deposits Marshall's public key on
the node and hands back its address and host key. Nothing is installed
and nothing is rebuilt on the way in.

| State | Meaning |
|---|---|
| `held` | accepted, waiting behind another request of yours; enters the queue at the back when that one ends |
| `queued` | in line; `position` is your place |
| `provisioning` | a node is being handed over; seconds |
| `active` | the node is yours and the meter is running |
| `ended` | the lease is over; `end_reason` is `released`, `preempted`, `lost`, `balance` (the organization's balance reached zero) or `admin` |

You hold one node at a time (currently one queued or active request per
user). A request from a second Marshall session while a first is live is
accepted as `held`, outside the line, and joins the back of the queue
when the live request ends; a second request from a session that is
already waiting is refused, and held requests are capped (currently
three per user). The console's sessions rail shows which session is
waiting and which one has a node, and the console's Public Research page
shows the same per GPU type: a slot for each type that holds the session
using a node of that type, with the sessions waiting for one listed
beneath it, each with its place in line and estimated wait; a click on a
session opens it. Each type's heading shows how
many GPUs the pool currently holds for it, and the console's Grid entry
shows the total across types. An empty slot offers a button naming the
GPU type, "Get MI355X with Marshall" or "Get B300 with Marshall": one
click opens a new Marshall session that
requests a node of that type, walks you through how the pool works and,
once the node is yours, opens a shell on it and shows its GPUs, with the
usual confirmation before anything is requested. Under the button the
slot shows the estimated wait a request made now would get, before you
have asked for anything.

## The lease

A lease is 4 hours (currently) and is preempted only when someone is
waiting. While nobody is queued the lease runs past 4 hours and keeps
billing until you release. Once another request is queued and your lease
has passed its length, a preempt notice is set: 10 minutes (currently)
to checkpoint, then the node is taken and rebuilt. Marshall wakes you 30
and 10 minutes before the lease length is reached and the moment a
notice is set, with the time left and a reminder to copy results off the
node.

Release when you are done: the end is stamped before the rebuild starts,
so the recycle costs you nothing. Marshall never releases a node on its
own: when a job finishes it says the meter is still running and asks you;
releasing, like requesting, goes through a confirmation card. A node the platform loses (a failed
node, a failed handover or rebuild) ends the lease at the earliest
evidence of the failure, and billing with it. A lease also ends the
moment the organization's balance reaches zero: the node is taken back,
the meter stops and `end_reason` reads `balance`; add credits before
requesting again. National Compute can end a lease; `end_reason` then
reads `admin`.

To keep working past a lease, ask Marshall for another node from the
session that holds it: the new request is `held`, joins the back of the
queue when the lease ends, and lands on a freshly rebuilt node. Nothing
carries over from the old node.

## Working on the node

Marshall reaches the node with plain `ssh` on its public address, port
22, as the login user the request names. The one key is minted by
Marshall on your workspace disk and reused for every request; the node's
host key arrives with the node and is pinned before Marshall announces
it, so the first connection is verified rather than trusted on first
use. A shell pane in your workspace carries the same key.

Long jobs detach on the node (`nohup`, `tmux`): Marshall's own command
runs are time-capped and end with the session. Copy results back with
`scp` or `rsync` before release.

The first job on a node is the platform's nanoGPT starter, started on
the node with one line and detached from the ssh session:

    ssh <ssh_user>@<public_ip> 'curl -fsSL https://access.nationalcompute.com/first-job/nanogpt-node.sh | sh'

It trains in the stock PyTorch container for the node's GPUs (the node
runs it with `docker`), writes its log to `~/first-job/nanogpt.out`
there, and ends on its own after about an hour; the node keeps billing
until you release it. The
[starter recipe catalog](https://access.nationalcompute.com/first-job/)
lists it with `cluster_kind` `public_research`, and
[FIRST-JOB.md](https://nationalcompute.com/FIRST-JOB.md) walks an agent
through request, connect, train, copy back and release. Your own
container runs the same way with `docker run` on the node. The Serve and
Post-train recipes are Kubernetes manifests and do not apply here.

Wiping your workspace disk mid-lease takes the key with it; the lease
keeps running, and the way back in is release and a new request.

## Billing

A lease bills per GPU-hour, for every GPU in the node, at the rate shown
in your Marshall quote and on the console's Billing page under Rates
(label "Public Research · <GPU model> GPU", unit GPU; one line per node
shape), from the moment the node is yours to the moment the lease ends:
release, preempt, loss or an end by National Compute alike. The rate is
fixed at grant for the whole lease; a change applies to leases granted
afterwards and never reprices a running one. Waiting costs nothing, and
the rebuild after a lease costs nothing.

Charges land on your organization's credit ledger about every five
minutes, in the same credit unit as every other charge, as rows of
[source `public_research`](https://docs.nationalcompute.com/api/billing.md#the-ledger) whose entries name
the node, the GPUs, the rate and the window. The Billing page shows them
under the **Public Research** chip; the
[billing summary](https://docs.nationalcompute.com/api/billing.md#balance-and-totals) carries the month's
GPU-hours and charged amount as `public_research`. A free rate still
counts GPU-hours and writes no charge.

A request is admitted only with a balance covering the first hour of a
lease, and a [billing hold](https://docs.nationalcompute.com/api/billing.md#hold-state) refuses new
requests. A balance that reaches zero ends a running lease at once: the
node is taken back and the meter stops (`end_reason` `balance`), so a
lease never runs far past what the balance covers. When nobody is
waiting, a lease runs and bills past its length with no other cap.
Release when the job is done.

---

# Billing

Source: https://docs.nationalcompute.com/billing/ (this markdown: https://docs.nationalcompute.com/billing.md)

You are billed **the clearing price for capacity you hold** —
never above your declared ceiling, and usually below it (during a
[minimum-duration protection window](https://docs.nationalcompute.com/market.md#minimum-duration-protection)
the assessed rate can be as high as your ceiling). Billing rides prepaid
credits managed on the console's **Billing** page, where self-serve
purchases are available.

## Buying credits

Credits are prepaid at **$1 = 1 credit**, topped up from the Billing
page in one of two ways:

- **Credit card ($50,000 per 7 days)** — self-serve, through a
  Stripe-hosted checkout (card details never touch the console). Your
  organization's card payments may total $50,000 inside any rolling 7
  days. Every card payment counts, one time top ups and auto reload
  charges alike. The Billing page shows what a card can still carry
  right now. The cap frees up as payments older than 7 days age out.
- **Wire transfer** — for payments above $50,000 and for anything
  beyond the 7 day card cap. The receiving-bank
  and beneficiary details and your organization's wire reference are on
  the console's Billing page under **Wire transfer**, and on the billing
  API's summary as [`wire`](https://docs.nationalcompute.com/api/billing.md#wire-transfer) for an agent.
  You **must**
  include the reference in the wire's memo field — it identifies your
  organization and ensures your account is credited promptly. Email
  [billing@nationalcompute.com](mailto:billing@nationalcompute.com)
  when you send a wire so we can keep an eye out for it.

## What the meter measures

For burst capacity the market's own record is the bill: every auction
tick records the rate assessed to each winning unit, and charges are
accounted at those market-run boundaries — when a node arrives, when
its price changes, when it leaves. A [base load block](https://docs.nationalcompute.com/market.md#base-load-capacity)
is the exception and has its own section below.

- **VM clusters** — a node bills from grant completion to the start of
  reclaim. You pay while you hold it, whether or not jobs are running;
  holding is under your control (`max_gpus`, `release`, swap).
- **Kubernetes clusters** — a node bills while it is a joined member
  serving your cluster, with a minimum of one hour per granted node:
  the [minimum hold](https://docs.nationalcompute.com/market.md#minimum-duration-protection) keeps the
  node yours for its first hour even if its job finishes sooner. After
  the first hour, charges accrue in minimum increments of five
  minutes, and because [nodes join when jobs need them and shed when
  idle](https://docs.nationalcompute.com/kubernetes.md), billing tracks actual use without any action
  from you.

## Base Load Capacity

A [base load block](https://docs.nationalcompute.com/market.md#base-load-capacity) bills its **fixed rate for
every node in the block across the whole term**, idle or busy,
delivered or being replaced — that is what the fixed rate buys. Charges
land on your credit balance in the same five-minute cadence as burst
capacity; each charge names the block it belongs to. The block's
nodes never appear in the market's clearing rates, and burst nodes you
hold beyond the block bill as ordinary burst capacity. Billing
stops at the end of the term, or at the moment a block is
cancelled by a billing hold reaching enforcement.

## Marshall model usage

Marshall's model calls are metered per member and charged to your
organization's credits from the first call, in the same credit
unit as capacity, about every five minutes, as one ledger row per
organization naming the members and the amount each used. The charge
counts toward your [monthly spend limit](https://docs.nationalcompute.com/api/billing.md#balance-and-totals)
like every other charge.

When your organization's balance reaches zero (or a billing hold is
active), Marshall is paused for every member. Adding credits resumes
Marshall within a minute. Marshall is never paused while your balance
is positive, so the balance can run slightly below zero on the last few
calls.

The ledger reads these rows as source `inference`; the console's
Billing page shows them under the **Marshall** chip and as the
"Marshall model usage" series on the usage chart. The
[billing summary](https://docs.nationalcompute.com/api/billing.md#balance-and-totals) carries the
organization's charged model usage as `inference`.

## Public Research leases

A [Public Research](https://docs.nationalcompute.com/public-research.md) node bills per GPU-hour, for
every GPU in the node, at the flat rate in force when the lease was
granted, from the moment the node is yours to the moment the lease ends:
released, preempted, lost, ended at zero balance or ended by National
Compute alike. The rate
is on the Marshall quote and on the Billing page under Rates; a change
to it applies to leases granted afterwards and never reprices a running
one. Waiting in the queue and the rebuild after a lease cost nothing. A
request is admitted only with a balance covering the first hour of a
lease; a balance that reaches zero ends a running lease, and a billing
hold refuses new requests.

The ledger reads these rows as source `public_research`; the console's
Billing page shows them under the **Public Research** chip, and the
[billing summary](https://docs.nationalcompute.com/api/billing.md#balance-and-totals) carries the
month's GPU-hours and charged amount as `public_research`.

## What is never billed

- **Ticks that clear at $0.** An uncontended market can clear at a
  real $0 rate — those hours cost nothing.
- **Gaps in the market record.** If the platform's market loop has an
  outage, the gap is unbilled — the platform's loss, never yours.
- **Pending capacity.** Declared demand that hasn't been granted costs
  nothing; you pay only from the moment a node is yours.

## Reading your records

Every charge above is a ledger row your organization can read with its
API token — per node, per price window, with burst, base load and
storage told apart — through the
[billing records API](https://docs.nationalcompute.com/api/billing.md). The console's Billing page
renders the same rows.

## Predicting your spend

Read your rate instead of estimating it. `GET /api/billing/balance` on
the [billing records API](https://docs.nationalcompute.com/api/billing.md#balance-and-burn-rate) returns
your balance with the hourly rate the meter last assessed across your
organization, folded by kind, cluster and unit price, so jobs bidding
different ceilings show as separate lines, and
`GET /api/billing/balance/history` shows how the balance has moved. The
figure is 5 to 10 minutes behind the market. Your worst case is still
arithmetic: held GPUs × your declared ceiling. The
[market feed](https://docs.nationalcompute.com/api/market-feed.md) publishes what capacity really clears
at, per tick and as a long-run series, and every capacity read echoes
your own declared ceiling.

## Invoices

An organization that can pay by card can also issue itself an invoice
for prepaid credits and pay it by wire through its accounts payable
team. The platform issues, records and serves the document; nobody at
National Compute is in the loop. Request one on the portal's billing
page, under the wire transfer rail, or ask Marshall for one.

The bill-to block is the organization's **billing profile**: a name,
one to six address lines and an optional email, set once by any member
and printed on every invoice issued afterwards. Editing the profile
never rewrites an issued document.

**The invoice id is the wire memo.** Put it in the transfer's reference
or memo field. Credits land when the wire does; issuing an invoice moves
no credits. Terms are due on receipt.

The billing page lists the organization's invoices with a download for
each. An open invoice can be cancelled there (or through Marshall): it
stops being payable and no credits move. A paid invoice is never
cancelled, and a cancelled invoice is never paid.

| Limit | Value |
| --- | --- |
| Amount | any positive dollar figure, at most two decimals |
| Open invoices per organization | 10 (currently); pay or cancel one before issuing another |
| Invoices per organization | 1,000 (currently), every status counted |
| Organizations on a reserved arrangement | refused; finance invoices them under the contract |

Issuing refuses with one of these codes; the portal and Marshall name
the fix in the same answer.

| Code | Status | Meaning |
| --- | --- | --- |
| `bad-request` | 422 | a field is malformed; `field` names it |
| `profile-missing` | 409 | set the billing profile first |
| `self-serve-off` | 409 | the organization cannot pay by card (a reserved arrangement, or card purchases off) |
| `open-limit` | 409 | ten invoices are open; `open_limit` and `open_count` carry the numbers |
| `invoice-limit` | 409 | the organization holds its maximum; `invoice_limit` carries the number |
| `not-open` | 409 | a cancel of an invoice that is paid, void or already cancelled; `invoice_status` says which |

With an org API token the invoices are readable, not issuable: there is
no issue verb on the token API.

| Route | Answers |
| --- | --- |
| `GET /api/billing/invoices` | `{profile, invoices, invoice_count, invoice_limit, open_count, open_limit, self_serve}` |
| `GET /api/billing/invoices/{id}.pdf` | the document, `application/pdf` |

Each invoice carries `id`, `amount_usd`, `status` (`issued`, `paid`,
`void` or `cancelled`), `date`, `due_date`, `bill_to` as printed,
`wire_memo` (the id), `cancellable` and `pdf_path`. Another organization's id, an unknown id and a
malformed id are the same `404`. Organizations without the feature read
`404` on both routes.

---

# Base Load Capacity SLA

Source: https://docs.nationalcompute.com/sla/ (this markdown: https://docs.nationalcompute.com/sla.md)

Base load capacity carries an uptime commitment of **99% per GPU
node**, backed by the [credit schedule](#credit-schedule) below: up to
a 100% credit of a node's charges for the measurement period.

A **base load block** is the committed product: a fixed block of GPU
nodes, yours alone for a fixed term at a fixed rate. No bidding and no
preemption. It bills for the full term whether used or not. Demand you
declare beyond the block competes in the [market](https://docs.nationalcompute.com/market.md) as
ordinary preemptible burst capacity.

## Scope

The commitment and its credits apply to **base load capacity only**:
GPU nodes inside a block, for the block's term. Two products are
explicitly outside it:

- **Preemptible burst capacity** — everything on
  [Burst capacity](https://docs.nationalcompute.com/market.md), including nodes an account holds
  beyond its base load block. No uptime credit schedule attaches: a
  node that fails stops [billing](https://docs.nationalcompute.com/billing.md) at the failure, and a
  [protected](https://docs.nationalcompute.com/market.md#minimum-duration-protection) cluster is owed
  a replacement node, but uptime credits are a base load term.
- **Legacy long-term reservations** — capacity sold under earlier
  long-term agreements is governed by those agreements, not by this
  page.

## Uptime measurement

Uptime is measured **for the base load block**, in full-minute
increments, over the shorter of a calendar month or the block's
term — month by month for a longer term, the whole term for a
shorter one. The block's uptime is the share of the block's node-minutes
during which a usable GPU node was serving the block; a
block of eight nodes with one node missing for an hour has lost
one node-hour of its eight.

**Downtime runs from the moment a node fails until a replacement GPU
node is made available** to the block: time spent provisioning
the replacement counts against uptime, not just the failure itself. The
credit tier below is set by the block's uptime and applies to the
block's base load charges for the period; a per-node breakdown is
available from your account team on request.

## Credit schedule

A node whose uptime falls below the commitment earns a credit against
that node's base load charges for the measurement period:

| Uptime per GPU node | Credit |
| :-- | :-- |
| ≥ 99.0% | none — the commitment is met |
| 95.0% – < 99.0% | 10% |
| 90.0% – < 95.0% | 25% |
| 85.0% – < 90.0% | 50% |
| < 85.0% | 100% |

A node below 85% uptime is fully credited — the period costs nothing
for that node.

---

# Authentication

Source: https://docs.nationalcompute.com/authentication/ (this markdown: https://docs.nationalcompute.com/authentication.md)

Two credentials exist, for two shapes of caller:

- **Org API tokens** — the REST APIs' credential: minted on the console,
  acts for the organization. Everything below describes them.
- **OAuth sign-in** — the [MCP server](https://docs.nationalcompute.com/api/mcp.md)'s credential: you
  sign in as yourself in a browser, your agent acts as *you*, and access
  follows your org membership (removing a member cuts their agent's
  access within seconds). No token to store.

Every REST API endpoint authenticates with an **org API token**, sent as
a Bearer header:

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/vm/capacity
```

## Minting a token

Tokens are minted on the console's **API Keys** page — [sign in](https://docs.nationalcompute.com/sign-in.md)
at [nationalcompute.com](https://nationalcompute.com), then **API → Keys**
in the sidebar. Any member of your organization can mint one, and the
secret is shown once, at mint. A token belongs to your organization and
**acts for it on every API**: reading and declaring capacity, setting a limit price,
swapping nodes, SSH keys, the price feed, and the
[billing records](https://docs.nationalcompute.com/api/billing.md). There is one grant, not a set of
permissions — the `scopes` a token echoes on `/api/whoami` are recorded
at mint and describe that one grant; no route narrows a token by them.

At mint a token is bound to every cluster or to one — that is the
[binding](#token-cluster-resolution) the API resolves below. Tokens are
revoked on the same page, instantly. Treat them like passwords: anyone
holding the token can act as your organization.

A token stops working in four ways. It expires at the `expires_at` set
at mint, at most a year out. A member revokes it on the API Keys page.
The token revokes itself with
[`DELETE /api/org/tokens/self`](https://docs.nationalcompute.com/api/organization.md#revoking-your-own-token).
The platform revokes it when the member who minted it leaves the
organization, or when the organization is retired. The last case writes
the revoke into the token's [activity trail](https://docs.nationalcompute.com/api/organization.md#activity)
under the actor `system:offboard`, so the remaining members see why it
closed. Every case answers `401 unauthorized` on the next call.

## Token ↔ cluster resolution

A token may be **bound to one cluster** or **org-wide**:

- A cluster-bound token addresses its own cluster; no cluster name is
  needed (passing one that matches is fine — only a contradicting one
  refuses). A token bound to a VM cluster is refused on the Kubernetes
  surfaces and vice versa.
- An org-wide token resolves automatically when your org holds exactly
  one cluster of the relevant kind. If it holds several, pass
  `cluster` (query parameter on GETs, body field on writes) or you get
  a `409 ambiguous-cluster`.

## Discovering what a token can address

`GET /api/whoami` answers the bootstrap questions with the token
itself — no cluster name required:

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/whoami
```

```json
{
  "org_id": "8f2c…",
  "org_display": "acme",
  "org_kind": "standard",
  "scopes": ["capacity:read", "capacity:write"],
  "cluster": {"name": "acme-inference", "kind": "k8s",
              "gpu_vendor": "…", "gpu_model": "…",
              "market": true, "gpus_per_node": 8,
              "shared_volume": true, "cpu_worker": true},
  "clusters": [
    {"name": "acme-inference", "kind": "k8s",
     "gpu_vendor": "…", "gpu_model": "…",
     "market": true, "gpus_per_node": 8,
     "shared_volume": true, "cpu_worker": true},
    {"name": "acme-train", "kind": "vm",
     "gpu_vendor": "…", "gpu_model": "…",
     "market": true, "gpus_per_node": 8,
     "shared_volume": false, "cpu_worker": false}
  ]
}
```

`org_display` is your organization's display name, the segment in
console URLs. `org_kind` is `standard`, or `public_research` for an
organization created at [sign-in](https://docs.nationalcompute.com/sign-in.md#public-research-organizations)
for a `.edu`, `.mil` or `.gov` address. `cluster` is the token's binding (`null` for an org-wide
token); `clusters` is every cluster in your org, both kinds,
name-sorted. Each entry names the GPU class the cluster trades as
`gpu_model` (`null` while the platform has not set one) and its maker
as `gpu_vendor`. An agent then knows what it prices before its first
limit price or declaration. Each entry also carries the facts the
console gates its pages on: `market` (`true` = the cluster trades on
the market, so a limit price or a capacity declaration applies;
`false` = a fixed cluster the platform sizes, with no limit price to
set; `null` = not known yet, the cluster record has not landed),
`gpus_per_node` (the node to GPU conversion), `shared_volume`
(`true` = a shared filesystem is bound to the cluster; `null` = the
storage record could not be read) and `cpu_worker` (the cluster
carries a CPU only node). A `slurm` entry adds its login endpoint as `ssh_host`, `ssh_port`,
`ssh_user` and `ssh_auth`; the [Slurm reads](https://docs.nationalcompute.com/api/slurm.md#connection-facts)
page states what each carries and when they read `null`. Any token may call it. Agents should call this **first** and only
ask a human to choose when several clusters of the right kind exist —
and a `409 ambiguous-cluster` refusal lists the candidate `clusters`
too ([error reference](https://docs.nationalcompute.com/api/errors.md)).

## Kubernetes service account tokens

A Kubernetes cluster issues its own credentials, separate from org API
tokens: **ServiceAccount tokens**. An org API token acts for your
organization on the platform API; a ServiceAccount token acts as one
account inside one cluster, with exactly the RBAC you bind to it. Use
ServiceAccount tokens for automation that talks to the cluster API and
for workloads that prove their identity to outside services.

### Minting one

A signed-in `kubectl` (see [Connecting](https://docs.nationalcompute.com/kubernetes.md#connecting)) mints
a token for any ServiceAccount, with the lifetime and, when a relying
party asks for one, the audience of your choice:

```sh
kubectl -n <namespace> create serviceaccount <name>
kubectl -n <namespace> create token <name> --duration=24h [--audience=<audience>]
```

The token is printed once and is not stored anywhere; mint another when
it expires. Its subject is `system:serviceaccount:<namespace>:<name>`.
Inside a pod the same account's token is already mounted as a projected
volume and refreshed by the kubelet, so a workload never mints one.

Tokens from `create token` expire at the duration you ask for. Deleting
the ServiceAccount invalidates every token minted for it at once. A
token that must not expire is a legacy `kubernetes.io/service-account-token`
Secret you create yourself; nothing rotates it, so prefer a short
duration and a re-mint.

### Using one against the cluster

The cluster API accepts a ServiceAccount token as a Bearer credential
from anywhere the API is reachable. Three inputs are needed, all of
them in the kubeconfig you downloaded: the server URL, the cluster CA
certificate, and the token. No sign in, no kubeconfig plugin:

```sh
kubectl --server=https://<api-host>:<port> --certificate-authority=ca.pem \
        --token="$TOKEN" -n <namespace> get pods
```

What the account may do is the RBAC you bind. A fresh ServiceAccount can
do nothing; a namespaced read-only binding is the usual first grant:

```sh
kubectl -n <namespace> create rolebinding <name>-view \
  --clusterrole=view --serviceaccount=<namespace>:<name>
```

Anything outside the binding is refused with `403 Forbidden`, and a
token that is expired, revoked, or malformed is refused with
`401 Unauthorized`.

## The cluster as an OIDC issuer

Every cluster signs its ServiceAccount tokens as an OpenID Connect
issuer that outside services can verify. The issuer is

```
https://nationalcompute.com/oidc/k8s/<cluster-id>
```

where `<cluster-id>` is the cluster's permanent id, not its name. Read
it from the cluster itself:

```sh
kubectl get --raw /.well-known/openid-configuration | jq -r .issuer
```

The issuer serves the two documents an OIDC relying party fetches,
anonymously and over public TLS: `/.well-known/openid-configuration`
and `/openid/v1/jwks`. The key set is the cluster's public signing keys
and nothing else; no cluster name, address, or membership is exposed.

A service that accepts OIDC federation (Tailscale, AWS, GCP, Vault, and
the like) is configured with the issuer URL, the token's subject and,
where it asks for one, an audience. Mint the token with that audience
(`--audience`), present it, and the service verifies it against the
issuer without any shared secret. For Tailscale's workload identity
federation, for example, the trust credential takes a custom issuer set
to the URL above, the subject `system:serviceaccount:<namespace>:<name>`
and an empty audience; the credential it returns is short-lived and
scoped to what you granted.

Signing keys rotate. A rotation reaches the public key set within about
15 minutes (currently a 10 minute refresh plus a 5 minute cache), and
both the old and the new key are published while tokens signed by
either are alive. Tokens minted before a cluster gained its public
issuer stay valid: the cluster accepts its former in-cluster issuer
alongside the public one. Nothing about `kubectl` access changes.

## Auth errors

| Status | Meaning |
|---|---|
| `401` | missing, malformed, expired, or revoked token, or a token whose organization was retired |
| `403` | `session-required`: a token on a console-only action (deleting a preserved volume, changing billing settings) — every token is refused there |
| `404` | no cluster of the right kind matches the token, or another organization's `{org}` in the path |

Errors are plain JSON — see [API conventions](https://docs.nationalcompute.com/api/index.md).

## What needs no auth

The discovery surfaces are deliberately unauthenticated: the OpenAPI
contracts (`/api/vm/openapi.json`, `/api/k8s/openapi.json`,
`/api/market/openapi.json`, `/api/billing/openapi.json`), the agent
orientation pages (`/llms.txt`, `/AGENTS.md`) and these docs — so a
client can learn the wire before it holds a valid token. A program reads
the docs under the apex too, `nationalcompute.com/docs/` + the page path
(`/docs/authentication/` is this page), no token needed; a token that is
sent must be valid, as on every route.

---

# Trust Center

Source: https://docs.nationalcompute.com/trust/ (this markdown: https://docs.nationalcompute.com/trust.md)

Tenant separation rests on two mechanisms: every tenancy holds
**whole nodes** in a dedicated cluster, and a node that leaves a
tenancy is **sanitized** before it can serve another. This page
states what each mechanism does, what evidence it leaves, and —
because a security page is only useful when it is honest — what the
platform does not claim.

## Tenant isolation

The market assigns whole nodes — a host serves one organization at a
time, never two ([Burst capacity](https://docs.nationalcompute.com/market.md)).

Every tenancy is a **dedicated cluster** with its own control plane
and its own credentials: Kubernetes tenants hold the kubeconfig and
cluster-admin on their own cluster ([Kubernetes
clusters](https://docs.nationalcompute.com/kubernetes.md)), Slurm tenants get their own login node,
controllers, and scheduler ([Slurm clusters](https://docs.nationalcompute.com/slurm.md)), and a VM
capacity grant hands you the node over SSH, reachable by the keys
your organization registered ([VM capacity](https://docs.nationalcompute.com/api/vm-capacity.md)).
For GPU capacity there are no shared multi-tenant namespaces and no
shared schedulers — the tenant boundary is the cluster boundary.
Marshall workspaces hold no GPUs: each runs as its own pod on a
shared workspace plane outside the tenant clusters, separated from
every other workspace by network policy and identity.

Each GPU host runs a default-deny inbound firewall rendered from the
platform's record of cluster membership and re-applied on every node
move. A node holding no tenancy either answers to platform
management alone or no longer exists — released and reclaimed nodes
are destroyed ([Burst capacity](https://docs.nationalcompute.com/market.md)).

## Node sanitization

A node that leaves a tenancy is sanitized before it can join
another. Every site declares its mechanism — **wipe in place** or
**recycle** — and a site that declares neither cannot move nodes
between tenants at all: the system fails closed rather than moving
an unsanitized node.

**Wipe in place.** GPU memory (HBM) is overwritten with zeros — the
GPU driver does not zero freed device memory, so without this step
the next tenant could allocate memory and read the previous tenant's
residual weights, activations, or cache. Local scratch devices are
discarded at the block layer, sampled reads of the raw device are
verified to return zeros, and the filesystem is recreated empty.
There is deliberately no file-by-file deletion fallback: a device
that cannot prove zeroed blocks aborts the move, and the node stays
out of service.

**Recycle.** The node is destroyed and replaced outright rather than
wiped; the replacement carries fresh filesystems whose identities
are recorded as the receipt. Nothing of the previous tenant's
filesystem survives into the replacement.

Credentials do not travel with a node. The per-cluster
authentication material, Kubernetes state, and tenant volume data on
the node are destroyed when it leaves, and a node holds a tenant's
credentials only while it serves that tenant.

Node-local disks are **not encrypted at rest**: the boundary between
one tenant's bytes and the next is the sanitization above, not a
key. Treat node-local data as ephemeral and keep durable state off
the node ([FAQ](https://docs.nationalcompute.com/faq.md)).

## Audit records

Control-plane actions are audited — accepted and refused alike.
Audit records land in append-only database tables (a trigger refuses
updates and deletes) and are exported hourly to write-once object
storage under a locked retention policy, currently 400 days: for
that window nothing in the pipeline — or outside it — can overwrite
or delete an exported record. The sanitization tools emit their own
receipts: the wipe's zero-read verification, the recycle's fresh
filesystem identities.

## API credentials

Org API tokens are stored as SHA-256 hashes and compared in constant
time; the plaintext secret is shown once, at mint, and never stored.
Expiry is mandatory — 365 days at most — and revocation on the
console is immediate. A token is one grant for the whole
organization, and destructive actions (deleting a preserved volume,
changing billing settings) refuse every token, requiring a console
session. [Authentication](https://docs.nationalcompute.com/authentication.md) has the full model.

## Marshall data sharing

Data sharing from Marshall sessions is an organization-level setting in the
console (Settings → Data sharing); since 1 October 2026 the baseline for an
organization that has never set a tier is tier 1, redacted conversation
transcripts, and tier 0 is one click away;
[the tiers are documented here](https://docs.nationalcompute.com/marshall-data-sharing.md). At tier 0
the platform collects usage analytics and the feedback notes members
write to us, never message content. Higher tiers add redacted conversation
transcripts and, at the highest, training artifacts matching published
patterns. Collection stops as soon as an organization lowers its tier;
data collected earlier is kept under the tier in force when it was
collected. Collected data lives in dedicated object storage under the platform's
own account, encrypted at rest by the storage service, with access
limited to platform staff by access policy and every read and write of
it logged; every record carries the consent it was captured
under. Redaction of credentials and
secrets before storage is automated and best effort.

## Web analytics

The console, including its sign-in pages and public pages, and this
documentation site use Google Analytics to count page views and
in-page navigation. Each view sends
the page address, a browser cookie that identifies the browser, and
standard request metadata to Google. It never sends message content,
credentials, or account details. The organization name in a console
address reaches Google as part of that address. Google Signals and
advertising personalization are off. Administration pages carry no tag.

---

# Marshall data sharing

Source: https://docs.nationalcompute.com/marshall-data-sharing/ (this markdown: https://docs.nationalcompute.com/marshall-data-sharing.md)

Marshall data sharing is an organization-level setting with three tiers.
Since 1 October 2026 tier 1 is the baseline for organizations that have never set a tier, and tier 0 is one click away. The tier decides what National Compute collects from your
organization's signed-in Marshall sessions to improve Marshall and to train the model
that powers it.

[Public Marshall](https://docs.nationalcompute.com/public-marshall.md#data-retained) chat without sign-in
does not enter this program and does not store conversation text.

## Tiers

Since 1 October 2026 the baseline for an organization that has never set a
tier is **tier 1**: redacted conversation transcripts are collected. An
organization that set tier 0 itself stays at tier 0, and any member can
lower the tier at any time from Settings → Data sharing.

| Tier | Collects | Never collects | Default |
|---|---|---|---|
| 0 — Usage analytics only | Counts, tool names, timings and outcomes of Marshall turns; error classes; model usage and cost; feedback notes members write to us | Message content; tool arguments or output; files in your storage | No |
| 1 — Analytics and conversation transcripts | Everything in tier 0; conversation transcripts as the model saw them, redacted for credentials before storage | Files in your storage | Yes, for organizations that have never set a tier |
| 2 — Analytics, transcripts and training artifacts | Everything in tier 1; training artifacts your runs write, matching the patterns below, copied from the organization share, your workspaces and the cluster's shared volume | Files whose name or content looks like a credential; files still being written | |

Feedback notes — a thumbs-down note, a `/feedback` message, a task label —
are messages to National Compute and are collected at every tier.

## Training artifacts

At tier 2 the platform copies files that match these patterns. Files are
copied, never moved or modified. A file whose name or content looks like a
credential is skipped. A file written in the last fifteen minutes waits for
the next pass.

| Kind | Patterns | Minimum size |
|---|---|---|
| Checkpoints | `*.pt`, `*.pth`, `*.safetensors`, `*.ckpt`, `*.gguf`, `*.msgpack`, `*.pkl`, `*optim*.pt`; `*.bin` and `*.index.json` beside a checkpoint | 1 MiB |
| Logs | `*.log`, `events.out.tfevents.*`, `trainer_state.json`, `*.out`, `*.err` | 1 KiB |
| Configs | `*.yaml`, `*.yml`, `*.toml`, `*.cfg`, `*.ini`, `*.args`; `*.json` beside a checkpoint | 64 bytes |
| Datasets | `*.jsonl`, `*.parquet`, `*.arrow`, `*.npy`, `*.npz`, `*.tsv`, `*.csv` | 1 MiB |

Skipped by name: keys, certificates, `.env` files, kubeconfigs, SSH keys,
and anything named like a token, secret, credential or password. Skipped
by directory: `.git`, `node_modules`, `__pycache__`, virtual environments,
caches and the tool configuration directories under a home. Model files
downloaded from a public model hub are not collected: the hub's cache
directories are skipped wherever they sit, and a file whose download
record sits beside it is skipped as well. A checkpoint your run saves
into the same directory still counts.

Locations at tier 2: the organization share, each member's workspace, and
the cluster's shared volume (the volume every worker mounts). Files that
exceed 50 GiB are not copied.

## Changing the tier

Console → Settings → Data sharing. Any member of the organization can
change it; the page shows who set the current tier and when. A change takes
effect for every capture after it. An organization may also be placed at a
tier under a written agreement with National Compute, such as a research
grant; the page says so, and changes to that tier go through National
Compute under the agreement.

## Revocation and retention

Lowering the tier stops future collection. Data collected earlier is kept
under the tier in force when it was collected. Deletion requests are handled
case by case where law requires.

## Incentive

Organizations at tier 1 or 2 may receive credits on terms National Compute
publishes in the console. The amount and form may change. Abuse — padding
sessions, synthetic activity, uploading data to earn credits — makes an
organization ineligible, the determination entirely at National Compute's
discretion.

## Redaction

Transcripts and feedback notes pass through automated, best-effort
redaction before storage. It targets private keys, bearer tokens, JWTs,
credentials in URLs, secret-named fields in JSON and YAML, and Kubernetes
Secret objects. It is best effort: do not paste credentials into Marshall.

The program is governed by the
[Terms of Service](https://nationalcompute.com/documents/TOS), the
[Privacy Notice](https://nationalcompute.com/documents/PrivacyNotice) and the
[Data Processing Addendum](https://nationalcompute.com/documents/DPA).

---

# FAQ

Source: https://docs.nationalcompute.com/faq/ (this markdown: https://docs.nationalcompute.com/faq.md)

## Allocation and base load

### How does the allocation model work — is it first-come, first-served, or guaranteed reservations?

Neither: it's **price-ordered**. Every auction tick (roughly 10
seconds) re-runs the market over all demand, and capacity goes to the
highest-value bids. Arrival order matters only as a tie-break — at
equal bids, the older demand wins — and there is no queue to hold a
place in. What plays the role of a guarantee is the
[minimum-duration protection window](https://docs.nationalcompute.com/market.md#minimum-duration-protection):
a freshly granted node cannot be taken by any competing bid while
its window (currently one hour on VM clusters, two hours on Kubernetes
clusters) is open. Beyond that window, you keep capacity by keeping
your ceiling at or above the going rate. The one guaranteed product is a
[base load block](https://docs.nationalcompute.com/market.md#base-load-capacity): a fixed block of nodes for a fixed
term at a fixed rate, outside the auction.

### Can a reservation be secured within ~36 hours of a request?

For burst capacity, faster than that: declared capacity is granted at
the **next auction tick** in which supply exists and your ceiling
clears — seconds to minutes when the market has room. For a committed
[base load block](https://docs.nationalcompute.com/market.md#base-load-capacity), any member of your
organization buys it on the Burst capacity page where buying is open. Elsewhere your
account team arranges it. A block starts as soon as it is sold and its
nodes are delivered ahead of every bid on the island.

### Is this on-demand — do we specify a time length?

For burst capacity, no fixed term in either direction. You hold
capacity as long as you want it and release it whenever you want
(scale-down is immediate; a Kubernetes node still bills its
[one-hour minimum](https://docs.nationalcompute.com/market.md#minimum-duration-protection)). The only
time-shaped elements are the protection window and the one-hour minimum
hold on fresh grants. If you want a committed term, a
[base load block](https://docs.nationalcompute.com/market.md#base-load-capacity) is sold for the posted terms at a
fixed rate.

### How do we do capacity planning under this system?

Three instruments: the [market feed](https://docs.nationalcompute.com/api/market-feed.md) gives you the
going rate per tick and up to a year of history, so you can see what a
given ceiling would have held historically; the capacity read echoes
your own position (declared, granted, pending, and whether a standing
bid can still clear); and your worst-case spend is arithmetic — held
GPUs × your ceiling. The
practical loop is to pick the ceiling that clears at the rate you can
see, then watch `pending` — persistent pending capacity means your
ceiling is under the market or supply is short.

### What are the SLAs?

Base load capacity carries the published
[Base Load Capacity SLA](https://docs.nationalcompute.com/sla.md). Preemptible burst capacity has no
formal SLA; what the mechanism itself guarantees there: the protection
window on fresh grants, billing at the clearing rate and never above your
declared ceiling, and
[unrecorded gaps are never billed](https://docs.nationalcompute.com/billing.md#what-is-never-billed).
For other contractual availability or support commitments, contact us.

## Pay for what you use

### Are we only charged for compute that's running jobs?

You're charged for capacity you **hold**, metered at market-record
boundaries — see [Billing](https://docs.nationalcompute.com/billing.md). For Kubernetes clusters that
converges on "charged when running work, plus a five-minute tail"
automatically: nodes join only when a queued job pulls them and, once
the job ends, shed after a five-minute idle grace billed at that job's
rate unless another job takes the node — so the meter tracks the queue.
For VM clusters you pay for nodes while you hold them, working or idle
— holding is your choice, and lowering `max_gpus` stops the meter.

### If unused reserved capacity is pushed back into the pool, do we get paid even if nobody bids on it?

For burst capacity the situation can't arise: you never own capacity
you have to resell. Released capacity (or a Kubernetes node shed for
idleness) simply returns to the platform pool and **your billing stops
at that moment** — whether anyone else bids on it afterward is the
platform's problem, not yours. A [base load block](https://docs.nationalcompute.com/market.md#base-load-capacity)
is different by design: you pay the fixed rate for the block whether the
nodes are busy or idle, and there is no rebate for idle nodes in the
block — the fixed rate and the no-preemption guarantee are the trade.

## Pricing index

### How are prices set — is there an index?

Pricing here is not computed from an external index at all. Prices are
**set by the auction**: what you pay is determined by the demand
competing with yours, never above your own ceiling. The platform's
price index is its own
[market feed](https://docs.nationalcompute.com/api/market-feed.md) — the actual volume-weighted clearing
rates, published per tick and as a long-run series. Other providers'
list prices are not an input to the price.

### Does any external price feed the index?

No — see above. No external price feeds into what you pay.

### Is the strike price the minimum across available providers?

There is no strike price. Each tick's clearing outcome is set by
competing demand, capped by your own declared ceiling. The
[market feed](https://docs.nationalcompute.com/api/market-feed.md) publishes what actually clears.

## Pricing and auction mechanics

### What is the exact auction structure — generalized second price, VCG, something else?

A repeated sealed-bid **combinatorial auction**, roughly every 10
seconds. Winner determination maximizes total value across all demand
in whole-node units (multi-node groups are atomic — granted whole or
not at all). Payments are second-price in nature: a winner pays for the
demand it displaces — a VCG-style payment with a core adjustment — and
your own cluster's losing demand never sets your price. A bid too low
to ever clear the market is refused on write (`bid-too-low`); the
[price to win ladder](https://docs.nationalcompute.com/api/k8s-market.md) says what clears. Practical
consequences: bidding your true
ceiling is the safe strategy, you never pay above it, and prices are
per cluster rather than uniform — which is why the
[market feed](https://docs.nationalcompute.com/api/market-feed.md) publishes a volume-weighted mean with
a min/max spread rather than a single quote.

### My limit price seems reasonable — why is nothing clearing?

On Kubernetes, read the request's reason before guessing —
`kubectl describe job`, or the console's Workloads page
([queue events](https://docs.nationalcompute.com/api/k8s-bid.md#queue-events)): `Outbid` names the limit
price that wins right now, `BidTooLow` means the request's limit
price — the job's label if set, else the cluster's limit price — can
never clear the market as it stands (the market moved, or the label
undercuts the cluster's price), `BalanceTooLow`
means the org's balance cannot fund two hours of the request,
`PendingSupply` means
protected nodes hold the supply your gang needs. The
other causes are the mundane ones. When the site is genuinely out of
capacity, no bid clears it: the console's Workloads page shows
**capacity unavailable**, raising the limit price does not start the job
sooner, and the queue clears on its own when capacity returns
([when the site has no capacity](https://docs.nationalcompute.com/kubernetes.md#when-the-site-has-no-capacity)).
Market-wide, the [feed](https://docs.nationalcompute.com/api/market-feed.md)'s ticks show `null` when
nothing clears. The last mundane cause is no limit price set at all: no
bid means $0, which never clears.

### We already hold capacity — how do we flex up on short notice?

Raise `max_gpus` (VM) or submit more GPU jobs (Kubernetes). The new
demand competes at the next tick and grants land **in your existing
cluster**, next to what you already hold. Short notice is the norm
here: there's no procurement step, just the market clearing.

### How do we get money back for burst compute sitting idle over a weekend?

You don't get money back — you **stop spending**. Scale to zero (or
just down) on Friday and billing stops with the release; re-declare on
Monday and you re-enter the market at Monday's price. Two caveats to
weigh: released nodes are destroyed (anything on local disk is gone),
and re-entry is at market conditions, not a held price. Kubernetes
clusters do the weekend version automatically — an empty queue sheds
nodes after the idle grace. A [base load block](https://docs.nationalcompute.com/market.md#base-load-capacity)
bills through the weekend: its fixed rate covers the block for the whole
term, used or not.

### Is burst capacity won at auction guaranteed to land on the same cluster as our existing reservation?

Yes. Capacity is granted to the cluster that declared the demand — a
grant is never "somewhere else." A cluster trades in a single market
(its site), so growth lands beside what you hold.

### Isn't preemption disruptive? We can't instantly drain a node.

The mechanism gives you five layers before disruption, and visibility
when it comes:

1. **The protection window** — a fresh grant can't be preempted for
   its first hour (VM node) or first two hours (Kubernetes node), no
   matter what the market does.
2. **Your ceiling is the lever** — you are only reclaimed when the
   market clears *above your declared ceiling*. Work that must not be
   interrupted gets a ceiling above the going rate, which you can watch
   on the [market feed](https://docs.nationalcompute.com/api/market-feed.md).
3. **You pick the victims** — on scale-down, `release` nominations
   choose which nodes go first, and `planned_release` on every read
   shows exactly what a shrink would destroy before you commit it.
4. **Kubernetes requeues itself** — a reclaimed job is suspended with
   its whole gang and requeued as a new request at your current limit
   price; it resumes when granted again, and with the
   [retry-budget spec](https://docs.nationalcompute.com/kubernetes.md#when-the-market-reclaims-a-node)
   in place a reclaim never fails the Job.
5. **A faulty node is replaced** — if our health checks take one of
   your nodes out of service, billing for it stops, the market grants a
   replacement that inherits the remaining protection window, and the
   pods on the faulty node are evicted after a short grace so they
   reschedule. The node's tile reads **Node Fix In Progress** and the
   request shows **Replacement Pending** until the new node joins
   ([details](https://docs.nationalcompute.com/kubernetes.md#when-we-take-a-node-out-of-service)).

Beyond that, this is genuinely preemptible capacity and the honest
advice applies: checkpoint long-running work and keep durable state off
node-local disk.

## Access model

### How do I get an account?

Sign in at [nationalcompute.com](https://nationalcompute.com) with Google
or with an email address that receives a sign-in link. A `.edu`, `.mil` or
`.gov` address gets its own public research organization on the spot, with
$100 of credits to get started, which requests whole GPU nodes from Marshall
through [Public Research](https://docs.nationalcompute.com/public-research.md); an address an existing
organization's rules admit joins that organization;
anything else is placed on the waitlist and emailed when a spot opens —
early access for public users is available in waves
([Sign in and create an account](https://docs.nationalcompute.com/sign-in.md#the-waitlist)).

### Do we submit containers or jobs that you run, or do we get machines we operate ourselves?

You operate it yourself, in both models. A **Kubernetes tenancy** is a
dedicated cluster — you hold the kubeconfig and deploy your own
workloads; the platform only moves nodes in and out as the market
clears ([Kubernetes clusters](https://docs.nationalcompute.com/kubernetes.md)). A **VM tenancy** gives
you nodes over ssh with your registered keys installed
([VM capacity API](https://docs.nationalcompute.com/api/vm-capacity.md)). There is no "submit us a job
and we run it" surface.

### Are VM access and Kubernetes priced differently?

No — same market, same unit. VM and Kubernetes demand clear in the
same per-site auction, both priced in USD per GPU-hour against your
declared ceiling. What differs is the interface: VM capacity is
declared explicitly, Kubernetes demand derives from your jobs.

### How does the bare-metal offering compare to Kubernetes and VM access?

The Kubernetes offering **is** the bare-metal offering: cluster nodes
run the Kubernetes agent directly on the metal, with no hypervisor
layer, dedicated to your cluster while you hold them. The VM offering
runs virtual machines on the same class of GPU hosts and trades that
layer for ssh-level, operate-it-yourself access. Pick by how you want
to drive the hardware, not by price — both clear in the same market.

## Ancillary costs and storage

### Are there fees for logging, monitoring, or ingress/egress on Kubernetes jobs?

No. Billing today is GPU capacity only — the clearing rate for nodes
you hold ([Billing](https://docs.nationalcompute.com/billing.md)). Monitoring dashboards, metrics, and
logs come with the tenancy at no separate charge, and network traffic
is not currently metered or billed.

### How much storage does a Marshall workspace come with?

Two pieces, both included with the tenancy. Each member's workspace has
its own 50 GiB disk, mounted at `/data`: home directory, Marshall's notes,
uploads, anything you want to keep between restarts. An organization with
a workspace hub of its own also has a shared folder at `/org`, visible to
every member, capped at 200 GB. Over the cap, the folder stays readable
and members can delete to free space, but new writes fail until the
organization is back under it; writes return on their own a few minutes
later. Nothing is deleted by the platform. Public research organizations
are single-member and have no shared folder.

### Can you support multi-petabyte storage, and is storage priced off the same index?

Storage is per-site: where a shared filesystem is provisioned, your
capacity read's `environment` field carries the paths and onboarding
facts, and it is not priced off the market — GPU capacity is the only
market-priced resource today. For multi-petabyte requirements, talk to
us about the specific site — large storage is a provisioning
conversation, not a self-serve knob.

---

# API conventions

Source: https://docs.nationalcompute.com/api/ (this markdown: https://docs.nationalcompute.com/api/index.md)

Base URL: `https://nationalcompute.com`. The API lives on the apex host,
not on this docs site.

The machine-readable contracts are the authority — these pages summarize
them:

- [`GET /api/vm/openapi.json`](https://nationalcompute.com/api/vm/openapi.json)
  — VM capacity (OpenAPI 3.1, no auth needed)
- [`GET /api/k8s/openapi.json`](https://nationalcompute.com/api/k8s/openapi.json)
  — the Kubernetes limit price (no auth needed)
- [`GET /api/market/openapi.json`](https://nationalcompute.com/api/market/openapi.json)
  — the price feed, shared by both kinds (no auth needed)
- [`GET /api/billing/openapi.json`](https://nationalcompute.com/api/billing/openapi.json)
  — [billing records](https://docs.nationalcompute.com/api/billing.md): the ledger, daily totals, balance and
  burn rate, spend per workload, unit prices (no auth needed)
- [`GET /api/org/openapi.json`](https://nationalcompute.com/api/org/openapi.json):
  [organization](https://docs.nationalcompute.com/api/organization.md), the roster, join rules, invitations,
  and API token metadata (no auth needed)

## Errors

Every error is plain JSON with a machine code, plus a human `detail`
where it helps:

```json
{"error": "version-mismatch", "detail": "..."}
```

Codes you'll meet across the API: `unauthorized`, `not-found`,
`version-mismatch`, `ambiguous-cluster`, `price-precision`,
`bid-too-low`, `bad-request`, `no-ssh-keys`, `rate-limited`,
`unknown-node`, `static-posture`. Each endpoint page lists the ones it
can return, and the [error reference](https://docs.nationalcompute.com/api/errors.md) tours the catalog; the
OpenAPI contracts are the authority.

## Versions and compare-and-set

Every capacity read carries a `version` — monotonic per cluster, bumped
on every accepted write. Echo it back as `expected_version` on writes:
the write applies only if the version still matches, otherwise you get a
`409 version-mismatch`. This is how you avoid clobbering a concurrent
writer (including your own automation).

## Rate limits

Writes are rate limited **per cluster**: one write (declare, swap, or
limit price) per short cadence, and the OpenAPI description states the exact
figure. Every token also carries a budget of 300 requests per minute
across all routes. The call past either limit returns `429` with
`retry_after_s` and a `Retry-After` header. An edge limit per client
address applies on top. Space your polls: the market feed and the limit
price read change once per tick, so one read every 10 seconds is plenty.

## Prices

Prices are USD, whole cents — at most two decimals. `12.34` is valid,
`12.345` is a `422 price-precision`.

## Nodes and GPUs

- **VM capacity** nodes are identified by **public IP**; node names
  never appear on that surface. The **Kubernetes** surface is the
  exception: the [cluster page reads](https://docs.nationalcompute.com/api/k8s-cluster-pages.md) and
  [`metrics_read`](https://docs.nationalcompute.com/api/metrics.md) address a node by the name the console
  shows.
- The node ↔ GPU conversion is the `gpus_per_node` field on every
  capacity read. Read it from the payload; never hardcode a conversion.
- Every capacity read, the limit price read, and the price feed name the GPU
  class they price as `gpu_model` next to `gpus_per_node`. `gpu_vendor`
  names its maker. `gpu_model` is `null` while the platform has not set
  one for the cluster's island.

## Agent orientation

[`/llms.txt`](https://nationalcompute.com/llms.txt) and
[`/AGENTS.md`](https://nationalcompute.com/AGENTS.md) at the apex root
orient an AI agent driving this API. Both are unauthenticated, like the
OpenAPI contracts. Once a token is in hand, `GET /api/whoami` returns
the org, scopes, and every addressable cluster
(`{name, kind, gpu_vendor, gpu_model}`) — see
[authentication](https://docs.nationalcompute.com/authentication.md#discovering-what-a-token-can-address).

An agent that speaks MCP (Model Context Protocol) reaches the same
surface as tools rather than routes. The hosted
[MCP server](https://docs.nationalcompute.com/api/mcp.md) at `https://nationalcompute.com/mcp` signs its
caller in with OAuth per member and needs no API token. It carries
server-side Kubernetes access, the
[`metrics_read`](https://docs.nationalcompute.com/api/metrics.md) hardware readings,
[`metrics_query`](https://docs.nationalcompute.com/api/metrics.md#free-form-queries-metrics_query) free-form
monitoring queries, [`dashboard_write`](https://docs.nationalcompute.com/api/dashboards.md) Grafana dashboard
documents you own, the `skill_read` and `skill_write` skill collection,
and the [`roadmap_read`](https://docs.nationalcompute.com/api/roadmap.md) hardware roadmap on top of what
these pages document.

---

# MCP server

Source: https://docs.nationalcompute.com/api/mcp/ (this markdown: https://docs.nationalcompute.com/api/mcp.md)

Point your own agent at National Compute. The platform hosts an
[MCP](https://modelcontextprotocol.io) (Model Context Protocol) server
that bundles the whole tenant surface — market data, capacity and limit
prices, billing, hardware metrics, and **server-side Kubernetes
access** — behind one OAuth sign-in. No API token to mint, no kubeconfig
to install, no kubectl on the agent's machine.

```
https://nationalcompute.com/mcp
```

Any MCP capable agent connects: marshall, Claude Code, Codex, Cursor,
claude.ai, or your own. The orientation the server hands every client on
connect states the framing in one line. You are talking to a market
rather than a reservation system, so read `market_read` or
`price_estimate` before pricing anything.

## Connecting

The endpoint is Streamable HTTP with standard MCP OAuth: add it to your
client and complete the browser sign-in when prompted. You sign in as
yourself — a member of your organization — and every action the agent
takes is attributed to you.

**Claude Code**

```sh
claude mcp add --transport http national-compute https://nationalcompute.com/mcp
```

**claude.ai** — Settings → Connectors → *Add custom connector* with the
URL above.

**Cursor** — add to `mcp.json`:

```json
{"mcpServers": {"national-compute": {"url": "https://nationalcompute.com/mcp"}}}
```

If you belong to several organizations, configure the endpoint as
`https://nationalcompute.com/mcp/o/<org>` (your org's name or id); the
server tells you so on the first tool call.

**Headless machines** (a VM with no browser): the OAuth callback lands
on `localhost`, so forward it over your SSH session while you sign in
from your laptop's browser:

```sh
ssh -L <port>:localhost:<port> you@your-vm
```

where `<port>` is the callback port your MCP client prints. Sign-ins are
long-lived — an idle agent re-authenticates after ~90 days, not daily.

## The wire

An unauthenticated `POST` answers `401` with the RFC 9728 challenge that
points a client at sign-in:

```
WWW-Authenticate: Bearer resource_metadata="https://nationalcompute.com/.well-known/oauth-protected-resource/mcp"
```

That metadata document names the authorization server,
`https://auth.nationalcompute.com/application/o/nc-mcp/`. It accepts the
authorization code flow with PKCE, which is what MCP clients use. It
also accepts the device code flow, for a terminal client that cannot
open a browser.

Protocol version `2025-06-18`.

| Fact | Value |
|---|---|
| Method | `POST`; `GET` and `DELETE` answer `405` |
| Response | one JSON-RPC response per POST, `application/json` |
| Sessions | none: no session id, no server-sent events, no server push |
| Batches | refused |
| Methods | `initialize`, `ping`, `tools/list`, `tools/call`; `notifications/*` answer `202` |
| Result size | each tool result is clipped at 40,000 characters, with a suffix naming the limit |

`$NC_MCP_TOKEN` below is the OAuth access token your client obtained,
never an org API token. Start with `initialize`:

```sh
curl -sS https://nationalcompute.com/mcp \
  -H "Authorization: Bearer $NC_MCP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{
       "protocolVersion":"2025-06-18","capabilities":{},
       "clientInfo":{"name":"my-agent","version":"1.0"}}}'
```

```json
{"jsonrpc": "2.0", "id": 1, "result": {
  "protocolVersion": "2025-06-18",
  "capabilities": {"tools": {}},
  "serverInfo": {"name": "national-compute", "version": "1.0.0"},
  "instructions": "National Compute MCP server. Call grid_whoami first: …"
}}
```

`tools/list` answers the whole catalog in one response:

```json
{"jsonrpc": "2.0", "id": 2, "method": "tools/list"}
```

A `tools/call` carries the tool name and its arguments. The result is
one text block holding the tool's JSON:

```json
{"jsonrpc": "2.0", "id": 3, "method": "tools/call", "params": {
  "name": "metrics_read",
  "arguments": {"action": "flags", "cluster": "aurora-prod",
                "kind": "Job", "name": "train-llm", "range": "6h"}}}
```

```json
{"jsonrpc": "2.0", "id": 3, "result": {
  "content": [{"type": "text", "text": "{\n  \"cluster\": \"aurora-prod\", …"}],
  "isError": false
}}
```

A refused tool answers `isError: true` with the refusal JSON in that
same text block. The JSON-RPC error envelope is reserved for protocol
faults.

## The tools

Call `grid_whoami` first: it names your organization and every cluster
you can address. Each cluster entry carries the facts the console gates
its pages on: `market` (`true` = the cluster trades on the market and a
limit price or capacity declaration applies; `false` = a fixed cluster
the platform sizes; `null` = not known yet), `gpus_per_node`,
`shared_volume` (`null` when the storage record could not be read) and
`cpu_worker`. `orgs` lists the organizations you belong to as
`{id, display}`, and `org_bound` with `orgs_note` say whether you can
switch: an OAuth session with several organizations pins one with the
`/mcp/o/<org>` endpoint; a member or workspace token is bound to its
organization and cannot switch. Kubernetes tools take an explicit
`cluster` argument and run **server-side** under an agent service
account minted for you on demand — the credential never reaches the
client.

Read tools:

| Tool | What it does |
|---|---|
| `grid_whoami` | your org (`org_id`, `org_display`, `org_kind`), your identity, your own orgs, every addressable cluster with its `kind`, GPU class, `market` posture, `gpus_per_node`, `shared_volume` and `cpu_worker`; a Slurm entry adds its [login endpoint](https://docs.nationalcompute.com/api/slurm.md#connection-facts) |
| `market_read` | clearing-price ticks (20 rows by default, `limit` up to 500; tail with `after_id`, page back with `before_id`, as on the [price feed](https://docs.nationalcompute.com/api/market-feed.md#tailing-the-live-feed)), bucketed history, or the [price-to-win ladder](https://docs.nationalcompute.com/api/k8s-market.md#the-price-to-win-ladder) |
| `capacity_read` | the standing limit price (k8s) or capacity declaration (VM) |
| [`roadmap_read`](https://docs.nationalcompute.com/api/roadmap.md) | expected landing quarter for a GPU model, from planning; an expectation and never a commitment; no quantities, dates or sources |
| `price_estimate` | a cost band for a job shape from recent clearings — an index, not a quote |
| `billing_read` | transactions, daily rollups, summary (the balance card, the [hold state](https://docs.nationalcompute.com/api/billing.md#hold-state), the [wire transfer rail](https://docs.nationalcompute.com/api/billing.md#wire-transfer)), balance + burn rate, per-workload usage, rates |
| `cluster_read` | live Kubernetes state: pods and events as one summary row per object (`raw: true` for the full objects), logs, any resource, raw GET paths |
| `job_watch` | a bounded poll over a job's pods: phases, restarts, the market's own events with their [verdicts](https://docs.nationalcompute.com/api/k8s-cluster-pages.md#verdicts-in-the-cluster), preemption deadlines, Kueue Workload requeue state, the standing limit price |
| `service_call` | one HTTP request to a cluster-internal Service (the port-forward stand-in) |
| [`metrics_read`](https://docs.nationalcompute.com/api/metrics.md) | what the platform's monitoring saw on a node, or on every node a workload touched, plus flags with the evidence behind them; a node may sit in a Kubernetes or a Slurm cluster |
| [`slurm_read`](https://docs.nationalcompute.com/api/slurm.md) | a Slurm cluster as the console shows it: node states and GPU allocation, the live queue, GPU hours, shared `/home`, one job's record and queue position, its log tail or a grep over it |
| [`metrics_query`](https://docs.nationalcompute.com/api/metrics.md#free-form-queries-metrics_query) | one PromQL expression of your own over your organization's monitoring, bounded and read only |
| [`dashboard_read`](https://docs.nationalcompute.com/api/dashboards.md) | your own Grafana dashboard documents and the platform's tenant dashboards, with panels and variables |
| `docs_read` | this documentation: table of contents, search, pages |
| `recipe_read` | the [starter recipes](https://nationalcompute.com/FIRST-JOB.md) catalog and files |
| `skill_read` | the skill collection: instruction sets Marshall follows for one task, the global collection plus your organization's own skills; `index` lists them with label, source, install count and proposal status, `body` returns one skill's full instructions by id |
| `escalation_read` | your organization's own escalation records with their status (`requested` is the only status; there is no acknowledged state) |
| `context_read` | the guidance the console agent loads at session start: the escalation rules and the notes the platform team wrote for your organization |
| `feedback_read` | your organization's own feedback submissions and whether your next substantive submission earns the [feedback reward](#feedback-rewards) |
| [`checkpoint_read`](https://docs.nationalcompute.com/api/checkpoints.md) | your organization's published [checkpoints](https://docs.nationalcompute.com/api/checkpoints.md) in every state; `list`, `search` (`q` over title, description and file name) and `get` by id. The public feed itself is the [anonymous route](https://docs.nationalcompute.com/api/checkpoints.md#the-public-feed-no-auth), never a member tool's list |
| [`grid_api`](#grid_api-paths) | the live OpenAPI contracts, plus a fixed set of GET paths from the REST API: the console's Workloads, Run History, job, reservation, kubeconfig, storage volume, token and activity reads beside the market, billing and organization feeds |

`grid_api` `action=get` serves the Storage page (`/api/k8s/storage`,
with its [fullness bands](https://docs.nationalcompute.com/api/k8s-cluster-pages.md#fullness-bands)) and the
Cluster Overview charts (`/api/k8s/capacity/history`, with the
[seven health states](https://docs.nationalcompute.com/api/k8s-cluster-pages.md#capacity-history)) among its
paths; an unlisted path refuses and names the set.

Single-phase writes, each on records of your own:

| Tool | What it does |
|---|---|
| [`dashboard_write`](https://docs.nationalcompute.com/api/dashboards.md) | create, update, keep or delete a Grafana dashboard document of your own from a small panel spec; scratch documents expire after 24 hours unless kept |
| `feedback_write` | record product feedback for the National Compute team in one call; nobody replies, and substantive feedback earns credits under the [feedback rewards](#feedback-rewards) terms |
| `skill_write` | `create`, `update` and `delete` one of your organization's own skills (update and delete by the author only; a delete takes the skill's installs with it); `propose` is two-phase and asks the National Compute team to add the skill to the global collection, after a check against the public page policy |

Two-phase tools, the seven that spend money or mutate state:

| Tool | What it does |
|---|---|
| `bid_write` | k8s: set the limit price (`release` is a VM argument: an empty or null `release` on a k8s call is ignored, a non-empty one is refused). VM: declare capacity plus an optional `release` list naming the nodes to give back first ([victim nomination](https://docs.nationalcompute.com/api/vm-capacity.md#declaring-capacity)) |
| `vm_swap` | destroy-and-replace one VM node |
| `ssh_keys` | list org SSH keys (free); add and revoke are two-phase |
| `cluster_apply` | server-side apply of a YAML manifest, with server dry-run previews |
| `cluster_delete` | delete one named object |
| `billing_write` | the [billing actions](https://docs.nationalcompute.com/api/billing.md#writes): `checkout`, `setup` and `portal` answer a hosted page URL in one call for a person to complete; `limit` (tighten only) and `reload` (auto reload, both directions here because a person approves the preview) run in two phases under the human confirmation class |
| `escalate` | send a request the platform cannot self serve to the National Compute team (a Base Load reservation ask, a capacity watch, an unsupported ask, feedback); the preview is the message as the team receives it |
| [`checkpoint_write`](https://docs.nationalcompute.com/api/checkpoints.md) | delete one of your own published checkpoints (`action: delete`); the preview names the item, the confirm removes it for the organization and from the public feed. Publishing carries bytes the wire cannot: that is the [REST family](https://docs.nationalcompute.com/api/checkpoints.md#the-authenticated-family) or Marshall's `checkpoint_publish` inside a workspace |

Node lifecycle is never available: no tool can cordon, drain, taint or
delete a node, on any path, with any confirmation. `cluster_apply` and
`cluster_delete` refuse Node kinds before they reach the cluster.

`metrics_read` has its own page. [Telemetry and flags](https://docs.nationalcompute.com/api/metrics.md)
covers its three actions, the answer shapes, the flag rules and the
refusals, and `metrics_query`, the free-form read next to it.
[Dashboards](https://docs.nationalcompute.com/api/dashboards.md) covers `dashboard_write` and
`dashboard_read`: the panel spec, ownership, scratch expiry and the
caps. `roadmap_read` has its own page as well.
[Checkpoints](https://docs.nationalcompute.com/api/checkpoints.md) covers `checkpoint_read` and `checkpoint_write`, the public feed anyone can read and the REST family that publishes.
[Hardware roadmap](https://docs.nationalcompute.com/api/roadmap.md) states what the roadmap publishes and
what it never publishes. `grid_api get /api/k8s/nodes` serves the
[Cluster Overview board](https://docs.nationalcompute.com/api/k8s-cluster-pages.md#node-verdicts), every
node with the platform's `verdict` word; `grid_api get
/api/k8s/nodes/<node>` answers one machine.

## grid_api paths

`grid_api {action: get, path, params}` serves a fixed set of the
[REST API](https://docs.nationalcompute.com/api/index.md) reads in process, org keyed like every tool. Any
other path reads `404 not-found` with the whole set in `detail`.
Kubernetes paths take `params.cluster` the way the REST routes take
`?cluster=`, and refuse the way those routes refuse: `409
cluster-not-ready`, `404 not-found` for a cluster outside your
organization or a VM cluster, `422 bad-request` for a malformed
parameter, `503 station-unavailable` when the island cannot answer.

| Path | Page |
|---|---|
| `/api/whoami` | [Index](https://docs.nationalcompute.com/api/index.md) |
| `/api/market/ticks`, `/api/market/history` | [Market feed](https://docs.nationalcompute.com/api/market-feed.md) |
| `/api/k8s/bid`, `/api/k8s/storage/volumes` | [Limit price](https://docs.nationalcompute.com/api/k8s-bid.md) |
| `/api/k8s/cluster`, `/api/k8s/cluster/kubeconfig`, `/api/k8s/reservations` | [Kubernetes cluster](https://docs.nationalcompute.com/api/k8s-cluster.md) |
| `/api/k8s/market/ladder`, `/api/k8s/market/demand`, `/api/k8s/market/spend`, `/api/k8s/market/paid`, `/api/k8s/market/bidhistory` | [Kubernetes market](https://docs.nationalcompute.com/api/k8s-market.md) |
| `/api/k8s/workloads`, `/api/k8s/workloads/runs`, `/api/k8s/workloads/history`, `/api/k8s/workloads/cost`, `/api/k8s/workloads/metrics` | [Cluster pages](https://docs.nationalcompute.com/api/k8s-cluster-pages.md) |
| `/api/k8s/jobs/{jobid}`, `/api/k8s/jobs/{jobid}/cost`, `/api/k8s/jobs/{jobid}/network` | [Cluster pages](https://docs.nationalcompute.com/api/k8s-cluster-pages.md) |
| `/api/vm/capacity`, `/api/vm/activity`, `/api/vm/history` | [VM capacity](https://docs.nationalcompute.com/api/vm-capacity.md) |
| `/api/billing/summary`, `/api/billing/balance`, `/api/billing/balance/history`, `/api/billing/transactions`, `/api/billing/daily`, `/api/billing/rates` | [Billing](https://docs.nationalcompute.com/api/billing.md) |
| `/api/org/members`, `/api/org/tokens`, `/api/org/tokens/{token_id}/activity`, `/api/org/activity` | [Organization](https://docs.nationalcompute.com/api/organization.md) |

A path with `{jobid}` or `{token_id}` takes the REST spelling
(`/api/k8s/jobs/<namespace>~<pod>`,
`/api/org/tokens/<token_id>/activity`) or the templated key itself with
the value in `params`. Both spellings are URL decoded the same way, so
`ml%7Etrain-0` and `ml~train-0` name one pod. A value given both ways
must agree; a disagreement reads `422 bad-request` before anything is
read. A `null` in `params` is an omitted value. A pasted REST URL keeps
its query string: `?cluster=<name>&r=1h` fills `params`, and a key set
in `params` wins over the same key in the query string.

Two paths answer differently from their REST routes on this leg:

- `/api/k8s/cluster/kubeconfig` answers JSON, `{cluster, filename,
  media_type, kubeconfig}`: `kubeconfig` holds the same YAML the REST
  route streams, `filename` the name the route's download carries. When
  the platform's own record served the file the route's headers ride
  as fields: `freshness`, `observed_at`, `ingested_at`, and
  `cache_control` (`no-store`).
- `/api/org/tokens/self/activity` reads `422 bad-request`: the MCP sign
  in carries no org API token, so `self` names nothing. Pass a
  `token_id` from `/api/org/tokens`.

The nodes, machine, storage and capacity history reads
(`/api/k8s/nodes`, `/api/k8s/nodes/{node}`, `/api/k8s/storage`,
`/api/k8s/capacity/history`) are served on this leg as well.

[Slurm reads](https://docs.nationalcompute.com/api/slurm.md) covers `slurm_read`:
its four actions, the answer shapes, the node state vocabulary and the
connection facts `grid_whoami` carries for a Slurm cluster.

## Two-phase confirmation

Every tool that spends money or mutates state runs in two phases. The
first call returns a **preview** — the current state, the proposed
change, its projected cost — and a single-use `confirm_token` valid for
**300 seconds**. The same call repeated with identical arguments plus
the token executes. Changed arguments, a reused token, or an expired one
refuse with a typed error; nothing executes without a fresh preview.

Your agent should show you the preview before confirming — but the
pause is enforced server-side either way.

### Human confirmation

A preview that spends money or destroys data carries one more key
beside `confirm_required`:

```json
"human_confirmation": {"required": true,
                       "reason": "destroys the node and every file on its disks"}
```

`reason` is one sentence stating what happens to your organization. A
preview without the key is the ordinary gate. The server decides per
call, so a mixed tool marks only the calls that qualify:

| Call | Marked |
|---|---|
| `vm_swap` | yes |
| `bid_write` with `kind: vm` and a `max_gpus` below the GPUs your granted nodes hold (0 included), or with a non empty `release` list | yes; the reason names how many nodes the declaration releases; `release: []` (clear the nominations) alone is not marked |
| `bid_write` with `kind: vm` that grows or keeps the declaration, or with `kind: k8s` | no |
| `cluster_delete` of a `PersistentVolumeClaim`, `Namespace`, `VolumeSnapshot` or `VolumeSnapshotContent` (any case, singular or plural, `pvc`, `ns`, group qualified spellings) | yes |
| `cluster_delete` of a `StatefulSet` whose `spec.persistentVolumeClaimRetentionPolicy.whenDeleted` is `Delete` | yes |
| `cluster_delete` of a `PersistentVolume` (`pv`) | refused today as an unknown kind (`422 bad-request`); marked if the kind is ever added |
| `cluster_delete` of any other kind (Job, Pod, Deployment, Service, a StatefulSet that keeps its claims, ...) | no |
| `cluster_apply` of a `StatefulSet` document that sets `persistentVolumeClaimRetentionPolicy.whenDeleted` or `whenScaled` to `Delete`, or that lowers `replicas` on a live set whose `whenScaled` is `Delete` | yes |
| `billing_write` with `action: limit` | yes; the reason states that the call lowers what the organization may spend this month |
| `billing_write` with `action: reload` | yes; with `enabled: true` the reason states that the call arms automatic card charges that spend the organization's money without further approval, with `enabled: false` that it changes the automatic top up setting |
| `billing_write` with `action: checkout`, `setup` or `portal` | no; one call answering a hosted page URL, which a person completes in a browser |
| every other `cluster_apply`, `ssh_keys` | no |

The marker, the instruction and the `confirm_token` precede `preview` in
the envelope, so the 40,000 character clip on a large preview never
removes them. The preview of a marked call also carries a `warning`
built from the same reason; a `cluster_delete` preview names the
canonical `resource` plural beside the kind you spelled. A cluster
scoped object (a Namespace, a VolumeSnapshotContent) has no `namespace`
in its preview. A decision read from the live cluster (a StatefulSet's
retention policy, the number of granted nodes) is read again at
execute; when the answer changed inside the window the call refuses
`409 object-changed` and needs a fresh preview.

The server marks the class and verifies nothing more: on the wire a
human's answer and an agent's are the same `confirm_token`. The
platform's own clients (marshall and the workspace agent) ask you in
the terminal for every marked preview. Full Access does not answer it.
A remembered answer does not answer it. A third party client is told
the same in the server's `instructions` and should hold the call for
your explicit answer. The execute leg is unchanged: the same token, the
same 300 second expiry, the same typed refusals.

The token a workspace agent holds cannot run a marked call at all. The
server refuses `403 workspace-human-class-refused` before a preview
exists. Nothing in the workspace can prove a human answered. Use the
console or an MCP client signed in as yourself for those calls; every
other two phase call stays open to the workspace agent.

### Tool annotations

Every `tools/list` entry carries the standard MCP `annotations` object,
so a client can gate by hint before it reads a description:

| Hint | Meaning here |
|---|---|
| `readOnlyHint` | true on every read tool; the tool changes nothing |
| `destructiveHint` | true on every write tool, since none is purely additive: `vm_swap`, `cluster_delete`, `ssh_keys` (revoke), `dashboard_write` (delete), `service_call` (a POST reaches your own Service), `bid_write` (a VM withdraw releases nodes and their disks), `cluster_apply` (an apply can replace a template or shrink replicas), `billing_write` (`limit` and `reload` change what the organization spends; the hosted pages mint a fresh Stripe session) |
| `idempotentHint` | true when repeating the same arguments changes nothing more: every read, `bid_write`, `cluster_apply` |
| `openWorldHint` | false on every tool; they talk to the platform only |

The hint and the class are different facts: a k8s `bid_write`, a
growing VM declaration and an ordinary `cluster_apply` are destructive
by hint and stay the ordinary gate.

Never available through any tool, with any confirmation: minting an org
API token or a member token (your human does it on the console's API
Keys page); inviting or removing members, join rules and domain rules;
buying Base Load; destroying the shared NFS volume (the Storage page's
volume destroy). A Kubernetes `PersistentVolumeClaim` delete is a
different object: it goes through `cluster_delete` under the human
confirmation class. The hosted billing pages (checkout, card setup,
payment portal) are URLs you complete in a browser.

## Feedback rewards

`feedback_write` records how the platform worked for you — the cluster,
scheduling, pricing, the docs, the agent tools, billing — for the
National Compute team to read. Nobody replies to feedback; a question
that needs a human answer is `escalate`. It takes `body` (1 to 8000
characters) and optionally `category` (`cluster`, `scheduling`,
`pricing`, `docs`, `agent_tools`, `billing`, `other`), `rating` (1 to
5), `cluster` and `contact_ok`.

Public-research organizations cannot earn feedback rewards, even after
buying credits. Their feedback is still recorded and read; the reward
is skipped with reason `public_research`.

For eligible standard organizations, substantive feedback (**500 or more
characters** after whitespace is collapsed) deposits **25 credits** to
the organization's balance when submitted. Two limits apply: one rewarded
submission per member per rolling seven days, and at most one reward per
**$1,000 of credits** your organization has loaded or received, over its
lifetime, with the
rewards themselves not counted. Shorter feedback is recorded and read,
and earns nothing.

The response says whether the reward was granted and, if not, why:
`short`, `member_weekly`, `org_cap`, `ineligible`, `disabled` or
`public_research`, with `next_eligible_at` and `org_rewards_remaining`.
`feedback_read` shows
the same eligibility before you write, beside your organization's
earlier submissions. A rewarded submission appears on the
[billing ledger](https://docs.nationalcompute.com/api/billing.md) as a platform grant with the note
"Feedback reward".

Rewards exist to hear real experience. Padded, duplicated or generated
submissions make an organization ineligible for rewards, and that
determination is entirely at National Compute's discretion. An
ineligible organization can still submit feedback; it earns nothing.

## Limits and semantics

- **Rate limit**: 300 requests per minute per member. Every `429`
  carries a `Retry-After` header and `retry_after_s` in the body —
  whether from your budget (`rate-limited`) or from a busy replica
  (`server-busy`, and per-tool `watch-busy` / `service-call-busy` on
  the long-running tools). Back off and retry.
- **Attribution**: every action is yours — the console's Activity page
  shows tool calls "via MCP", and Kubernetes audit logs name your agent
  service account.
- **Access follows membership**: removing a member from the org cuts
  their MCP access within seconds, sign-in or not.
- Errors use the [API error catalog](https://docs.nationalcompute.com/api/errors.md); optimistic concurrency
  rides the previews. Capacity writes capture a version at preview time
  and check it at execute — concurrent changes surface as
  `409 version-mismatch`. `cluster_delete` captures the object's uid the
  same way: if a same-name object replaced the previewed one, execute
  refuses with `409 object-changed` — re-preview and look again.
- `cluster_apply` manifests: at most **1.5 MB** of YAML and 20 objects
  per call.
- The protocol is stateless: no sessions to manage, and previews /
  confirmations work across reconnects.

Prefer raw REST? Everything here wraps the same
[documented APIs](https://docs.nationalcompute.com/api/index.md) — org API tokens keep working unchanged.

---

# Telemetry and flags

Source: https://docs.nationalcompute.com/api/metrics/ (this markdown: https://docs.nationalcompute.com/api/metrics.md)

`metrics_read` answers what the platform's own monitoring saw on the
hardware your job ran on: per GPU temperature, clocks, activity, power,
memory, throttle time and error counters, plus the host's CPU, pressure
and network. It also answers the platform's judgement on those readings
as **flags**, each one carrying the evidence behind it.

The tool lives on the [MCP server](https://docs.nationalcompute.com/api/mcp.md). It is read only and single
phase, so no `confirm_token` is involved. The
[cluster page reads](https://docs.nationalcompute.com/api/k8s-cluster-pages.md) serve the console's own
utilization and machine panels over REST; this tool is the agent-facing
read of the same monitoring, with the flags layer on top. For anything
these three actions do not answer, [`metrics_query`](#free-form-queries-metrics_query)
below runs one PromQL expression of your own over the same monitoring.

| `action` | Target | Answers |
|---|---|---|
| `node` | one node | every reading the platform holds for that node's GPUs and host |
| `workload` | one workload | where it ran over the window, plus each node's readings |
| `flags` | a node or a workload | the platform's judgement: which conditions fired, with evidence |

## Arguments

| Argument | Values | Applies to |
|---|---|---|
| `action` | `node`, `workload`, `flags` | required on every call |
| `cluster` | the cluster name (`grid_whoami` lists them) | required on every call |
| `node` | the node's name | `action=node`; `action=flags` on a node |
| `kind` | `Job`, `JobSet`, `MPIJob`, `Deployment`, `StatefulSet`, `Pod` | `action=workload`; `action=flags` on a workload |
| `name` | the workload's name | the same two |
| `namespace` | defaults to `default` | workload targets |
| `range` | `1h`, `6h`, `24h`, `7d`; defaults to `6h` | optional |
| `start`, `end` | ISO 8601 UTC; `end` defaults to now | optional |
| `detail` | `summary` or `series` | optional |
| `points` | series length after downsampling; defaults to 48, maximum 96 | optional |

`action=flags` takes either `node` or the pair (`kind`, `name`).
Anything else is a `422 bad-request`.

## The window

`range` names a window ending now. `start` and `end` set an explicit
window instead, and the call then ignores `range`. A span longer than
14 days is refused. A `start` at or after `end` is refused.

The bucket width follows from the span and the `points` you asked for:
the span divided by `points`, rounded up to a multiple of 30 seconds,
never below 30 seconds. A 6 hour window at the default 48 points gives
a `step_s` of 450. The answer's `window` echoes the resolved `start`,
`end`, `step_s` and the resulting `points`.

`detail` defaults to `series` for `action=node`. It defaults to
`summary` for `action=workload` and `action=flags`. Under `summary`
each series collapses to its `{mean, min, max}`, except `temp_c`,
`tensor_active_pct` and `power_w`, which keep their values.

## The envelope

Every action answers the same outer object:

| Field | What it carries |
|---|---|
| `cluster`, `gpu_vendor`, `gpu_model` | the cluster and the GPU class its nodes carry |
| `window` | `start`, `end`, `step_s` and `points`, as resolved |
| `reference` | the [comparison number](#the-reference) for the GPU model, or `null` |
| `coverage` | `present` and `missing`: which metric families answered |
| `note` | the units and how to read this answer, in one short paragraph |

A series is
`{"t0": <epoch seconds>, "step_s": n, "values": [...], "mean": x, "min": x, "max": x}`.
A gap inside `values` is `null`. A series the platform holds no reading
for is `null` whole. A `null` is never a zero. `coverage.missing` names
every family that did not answer.

The family names are a fixed vocabulary: `gpu_temp`, `memory_temp`,
`sm_clock`, `mem_clock`, `gpu_util`, `tensor_active`, `sm_active`,
`dram_active`, `power`, `hbm`, `throttle_violations`, `clock_events`,
`link_errors`, `ecc`, `xid`, `host_cpu`, `host_pressure`, `host_nic`,
`north_south`, `health_episodes`.

## action=node

```json
{
  "cluster": "aurora-prod", "gpu_vendor": "amd", "gpu_model": "MI355X",
  "window": {"start": "2026-09-21T12:00:00Z", "end": "2026-09-21T18:00:00Z",
             "step_s": 450, "points": 48},
  "reference": {"gpu_model": "MI355X", "temp_p95_c": 74.5,
                "note": "p95 of GPU temperature for this model over the window, across your nodes and the island's shared pool"},
  "coverage": {"present": ["gpu_temp", "memory_temp", "tensor_active",
                           "power", "hbm", "ecc", "link_errors",
                           "host_pressure", "…"],
               "missing": ["north_south"]},
  "node": {"name": "aurora-prod-n2", "gpus": 8, "state": "ready"},
  "gpus": {
    "3": {
      "temp_c": {"t0": 1758456000, "step_s": 450,
                 "values": [71.0, 83.5, 91.0, 88.2, null],
                 "mean": 83.4, "min": 68.0, "max": 91.0},
      "peak_temp_c": 91.0, "memory_temp_c": {"…": "…"},
      "tensor_active_pct": {"…": "…"}, "power_w": {"…": "…"},
      "hbm_used_gib": {"mean": 241.6, "min": 238.0, "max": 244.9, "…": "…"},
      "hbm_total_gib": 268.2,
      "throttle": {"thermal_s": null, "power_s": null, "clock_events": null,
                   "…": null},
      "errors": {"link_crc": null, "link_replay": null, "link_recovery": null,
                 "pcie_replay": 0, "pcie_recovery": 0, "ecc_sbe": 0,
                 "ecc_dbe": 0, "xid": null}
    }
  },
  "host": {"cpu_busy_pct": {"…": "…"}, "nic": {"errors": 0, "drops": 0, "…": "…"},
           "pressure": {"cpu": {"…": "…"}, "memory": null, "io": {"…": "…"}}},
  "health_episodes": [{"kind": "thermal", "started": "2026-09-21T14:05:00Z",
                       "ended": "2026-09-21T14:46:00Z", "detail": "…"}]
}
```

`gpus` is keyed by the GPU's index on the node:

| Field | Reading |
|---|---|
| `temp_c`, `memory_temp_c` | GPU temperature and memory temperature, °C |
| `sm_clock_mhz`, `mem_clock_mhz` | core clock and memory clock, MHz |
| `util_pct`, `sm_active_pct`, `dram_active_pct` | how busy the GPU, its cores and its memory were, percent |
| `tensor_active_pct` | tensor core activity, percent: the reading that tracks training work |
| `power_w` | power draw, watts |
| `peak_temp_c` | the highest temperature the GPU reached in the window, °C |
| `hbm_used_gib`, `hbm_total_gib` | GPU memory in use, and the GPU's total, GiB. The total is the denominator `memory_pressure` measures against |
| `throttle` | seconds of throttle by cause over the window: `thermal_s`, `power_s`, `board_limit_s`, `sync_boost_s`, `low_util_s`, `reliability_s`; `clock_events` counts each clock-event kind the GPU reports, for example `HW_SLOWDOWN` |
| `errors` | counter increases over the window: `link_crc`, `link_replay`, `link_recovery`, `pcie_replay`, `pcie_recovery`, `ecc_sbe`, `ecc_dbe`, `xid` |

`link_*` counts the GPU-to-GPU fabric links. Those are the NVLink
counters on NVIDIA. The AMD equivalents (XGMI) are not read today, so
`link_*` is `null` on an AMD cluster.

`pcie_recovery` counts PCIe link recoveries over the window. It is
`null` where a maker has no such counter, which is NVIDIA today.

Throttle seconds and error counts are the **increase over the window**,
never a lifetime total. A counter the platform does not have for a GPU
maker reads `null`, field by field. The cluster above is an AMD
cluster, which is why every throttle timer, every `link_*` counter and
`xid` read `null` on it. Its PCIe and ECC counters answer.

`host` carries the node itself:

| Field | Reading |
|---|---|
| `cpu_busy_pct` | host CPU busy, percent |
| `load_per_core` | run queue length per core |
| `pressure.cpu`, `pressure.memory`, `pressure.io` | share of time the host waited on CPU, on memory, or on disk, percent |
| `nic.rx_mbps`, `nic.tx_mbps`, `nic.errors`, `nic.drops` | node network throughput, and error and drop counter increases over the window |
| `north_south.ingress_mbps`, `north_south.egress_mbps` | traffic to and from the internet |

`health_episodes` lists the platform's own recorded episodes for the
node inside the window, as `{kind, started, ended, detail}`. An open
episode reads `"ended": null`.

## action=workload

```json
{
  "workload": {"kind": "Job", "namespace": "default", "name": "train-llm"},
  "placement": [
    {"pod": "train-llm-0", "node": "aurora-prod-n1",
     "from": "2026-09-21T12:04:00Z", "to": null, "phase": "Running"},
    {"pod": "train-llm-1", "node": "aurora-prod-n2",
     "from": "2026-09-21T12:04:00Z", "to": null, "phase": "Running"}],
  "placement_source": "history",
  "nodes": {
    "aurora-prod-n1": {"gpus": {"0": {"temp_c": {"mean": 68.1, "min": 61.0,
                                      "max": 72.4}, "…": "…"}}},
    "aurora-prod-n2": {"…": "…"}}
}
```

`placement` comes from the workload's own scheduling history,
intersected with the window. `placement_source` then reads `history`.
Where the platform holds no history for that window, placement comes
from the pods running now, and `placement_source` reads `current-pods`
instead. Read `placement_source` before you trust a placement.

`nodes` holds one node block per node the workload touched, at
`summary` detail unless you ask for `series`. Each node block carries
its own `health_episodes`, the same shape as on `action=node`.

## action=flags

A one hour window over the same node, opened after the episode above had
already started:

```json
{
  "window": {"start": "2026-09-21T14:20:00Z", "end": "2026-09-21T15:20:00Z",
             "step_s": 90, "points": 40},
  "target": {"workload": {"kind": "Job", "namespace": "default",
                          "name": "train-llm"}, "placement": ["…"]},
  "checked": ["thermal", "utilization", "comm", "host", "memory"],
  "unavailable": ["throttle_violations", "clock_events", "xid"],
  "clean": false,
  "flags": [
    {"kind": "thermal_throttle", "severity": "high",
     "node": "aurora-prod-n2", "gpu": "3", "minutes": 26,
     "from": "2026-09-21T14:20:00Z", "to": "2026-09-21T14:46:00Z",
     "clipped": {"start": true, "end": false},
     "episode": {"from": "2026-09-21T14:05:00Z",
                 "to": "2026-09-21T14:46:00Z"},
     "evidence": {"peak_temp_c": 91.0, "reference_p95_c": 74.5},
     "summary": "GPU 3 on aurora-prod-n2 held at or above 83 °C for 26 min inside this window (peak 91 °C, reference p95 74.5 °C); the episode began 14:05 UTC, before the window"}],
  "note": "this GPU maker reports no throttle timers, clock events or XID events, and no fabric link counters within link_errors; one flag is clipped at the window start"
}
```

Each flag object:

| Field | What it carries |
|---|---|
| `kind`, `severity` | the condition, and how serious this instance of it is |
| `node`, `gpu` | the node, and the index of the GPU the evidence came from |
| `from`, `to`, `minutes` | the sub-window the evidence covers, and its length |
| `clipped` | `{start, end}`: whether the evidence runs past that edge of the window |
| `episode` | the platform's own health episode bounds for this node and kind, or `null` |
| `evidence` | the numbers behind the flag, whichever of them the GPU maker reports: peaks, counter increases, clock events, the reference p95 |
| `summary` | one sentence an agent can quote |

`clipped.start` is `true` when the evidence was already abnormal in the
window's first bucket. `clipped.end` is `true` when it was still
abnormal in the last. Either way `from`, `to` and `minutes` cover only
the part inside the window, so the length is a floor. The answer's
`note` names every clipped flag.

`episode` carries the bounds of the platform's own
[health episode](#actionnode) for that node and kind, where one overlaps
the flag. `to` is `null` while the episode is still open. The field is
`null` where no episode overlaps. Quote the episode bounds when a narrow
window cuts a flag short.

The six kinds, and the rule that fires each one:

| `kind` | Fires when | `severity` |
|---|---|---|
| `thermal_throttle` | thermal throttle time increased, a thermal clock event landed, or the GPU held at or above 83 °C for at least 5 minutes | `high` at 10 minutes or longer, otherwise `warn` |
| `power_cap` | power throttle time increased, or a power clock event landed | `high` at 10 minutes or longer, otherwise `warn` |
| `utilization_collapse` | tensor core activity stayed below half the window's median for at least 10 minutes while the window's pods were Running | `warn` |
| `comm_errors` | a fabric link or PCIe error counter increased, an uncorrectable memory error landed, or a GPU fault event landed | `high` for uncorrectable memory errors and GPU fault events, otherwise `warn` |
| `host_pressure` | the host waited on CPU, memory or disk more than 20% of the time for at least 5 minutes | `warn` |
| `memory_pressure` | GPU memory in use held at or above 95% of the GPU's total for at least 5 minutes | `warn` |

`utilization_collapse` falls back to plain GPU utilization where the
platform has no tensor core reading for that GPU maker.

What a GPU maker cannot report changes what can fire. On AMD today the
platform reads no throttle timers, no clock events and no XID events.
Inside `link_errors` it reads the PCIe counters; the fabric link (XGMI)
counters are absent, so `link_*` is `null` and `link_errors` answers
only in part.

`thermal_throttle` on AMD fires from sustained temperature alone, at or
above 83 °C for five minutes, together with the platform's own health
episodes. `power_cap` cannot be detected there at all, so `power` is
absent from `checked`. `comm_errors` is detectable: it fires from the
ECC counters and the PCIe counters, so `comm` stays in `checked`. GPU
memory temperature answers too. `ecc_sbe` there carries the correctable
and deferred totals together, while `ecc_dbe` carries the uncorrectable
total. The `unavailable` list on such a cluster reads
`throttle_violations`, `clock_events` and `xid`.

## Coverage and clean

A metric family is in one of three states. The difference decides what
an honest answer says:

| State | Meaning |
|---|---|
| `coverage.present` | queried, and the platform has the readings |
| `coverage.missing` | queried, and nothing came back |
| `unavailable` | never queried: this GPU maker structurally has no such reading |

`coverage` rides every action. `unavailable` rides `action=flags`,
beside `checked`. A family the maker structurally lacks appears in
neither `coverage` list, because the platform never asks for it. A
family whose inputs answered only in part is `coverage.present`.

`checked` names only what the cluster's GPU maker can answer, so an
`unavailable` family never holds back a clean result. `clean` is `true`
when no flag fired and every family in `checked` answered. A checked
family that did not answer keeps `clean: false`. `note` then names it.
A healthy cluster whose maker lacks several families reads
`clean: true`, with those families named in `note`. An agent reporting
a clean result should say which families it checked and which the
hardware cannot report at all.

## The reference

`reference.temp_p95_c` is the 95th percentile of GPU temperature over
the same window for the cluster's GPU model, taken across your
organization's own nodes and the island's shared pool, aggregated to
one number. It answers "is this GPU hot for its model" without a second
call. It is not a reading across every tenant on the island. The
aggregate carries no names, no counts and no locations. It is `null`
where the platform could not compute it. A flag then reports its own
temperature with no comparison.

## Refusals

| Code | Status | Meaning |
|---|---|---|
| `bad-request` | 422 | a missing or malformed argument: no `cluster`, an action without its target, a node or workload name outside `[A-Za-z0-9][A-Za-z0-9._-]{0,120}`, a window longer than 14 days or ending at or before its start, or `points` above 96 |
| `not-found` | 404 | a cluster, node or workload outside your organization, or one that does not exist |
| `metrics-unavailable` | 503 | the platform's monitoring did not answer for this cluster; retry |

`cluster` is required, so a call without one is a `422`, never a
`409`. A `409 ambiguous-cluster` survives in one case: your
organization holds two clusters of the same name on two islands. The
refusal lists the candidates.

## What the platform does not see

`metrics_read` reads the hardware and the host. It does not see your
training loop: step time, samples or tokens per second, loss,
dataloader waits, or anything else inside your process. A clean `flags`
answer means the platform's own signals are clean. It means nothing
more. An explanation of a slowdown pairs these signals with the job's
own step timings.

## Evidence for a slow run

1. **Where it ran.** `metrics_read` with `action=workload`,
   `cluster=aurora-prod`, `kind=Job`, `name=train-llm`, `range=6h`
   names every node the job touched and when. Check
   `placement_source`.
2. **What fired.** The same target with `action=flags` returns the
   conditions, each with its node, GPU, sub-window and numbers. A flag
   whose `clipped` reads `true` at either end ran past the window, so
   widen the window or quote its `episode` bounds.
3. **The shape over time.** `action=node`, `node=aurora-prod-n2`,
   `detail=series` shows the flagged node's own curves: temperature
   climbing while clock and tensor core activity fall with it.
4. **The sentence the evidence supports.** "GPU 3 on aurora-prod-n2
   held at or above 83 °C for 41 minutes from 14:05 UTC, peaking at
   91 °C against a reference p95 of 74.5 °C for this GPU model. The
   job's other node stayed clean over the same window."

That evidence supports one claim: this GPU was throttled. It does not
by itself explain the run's wall clock. Read the job's own step timings
over the same window before attributing the slowdown to the hardware or
to the code.

## Free-form queries: `metrics_query`

`metrics_query` runs one PromQL expression of your own over the same
monitoring, scoped to your organization: every cluster you own, nothing
else. It is the open end next to the fixed reads above. Use it for fleet
views, custom aggregations, or a metric `metrics_read` does not summarize.
Like `metrics_read` it is read only and single phase.

| Argument | Values | Meaning |
|---|---|---|
| `query` | PromQL, at most 4000 characters | required |
| `instant` | `true` or `false` (default) | `true` returns one value per series at the window's end; `false` returns a series over the window |
| `range`, `start`, `end`, `points` | as in [The window](#the-window) | the window; `points` sets the step |

The answer carries `org`, `query`, `result_type`, `window`, `series`,
`series_total` and `truncated`. A series over the window is
`{labels, values}` on the inclusive `[start, end]` grid, one slot per
step, `null` where nothing answered; an instant answer is
`{labels, value}`; a scalar answer carries `value` alone. At most 100
series come back. When some were dropped, or when nothing matched, a
`note` says so: an empty result is a red flag to check the metric name
and matchers, not an all-clear.

Labels worth knowing: `cluster` is the cluster name the console shows,
`instance` is the node as the cluster names it, and the device is `gpu`
on NVIDIA nodes and `gpu_id` on AMD nodes. GPU series are `DCGM_FI_DEV_*`
on NVIDIA nodes and `amd_gpu_*` on AMD nodes. To discover names, ask for
`count by (__name__)({__name__=~"amd_gpu_.*"})`.

Refusals: a PromQL error, or a window or `points` outside the catalog,
is a 422 carrying the store's own words; a throttled organization is a
429 with `Retry-After`; an unavailable store is a 503.

---

# Slurm reads

Source: https://docs.nationalcompute.com/api/slurm/ (this markdown: https://docs.nationalcompute.com/api/slurm.md)

`slurm_read` answers what the console shows for a Slurm cluster: the
status page (node states, GPU allocation, the live queue, GPU hours,
shared `/home` usage), one job's page, that job's place in the queue,
and its log. The tool lives on the [MCP server](https://docs.nationalcompute.com/api/mcp.md). It is read only
and single phase. No `confirm_token` is involved. Nothing on this
surface submits, cancels or changes a job: `sbatch`, `scancel` and every
other scheduler command stay on the login node, where your agent runs
them over ssh with the connection facts `grid_whoami` carries (see
[Connection facts](#connection-facts)).

```text
grid_whoami ──▶ the cluster's name, kind slurm, ssh facts
     │
     ▼
slurm_read status ──▶ nodes, queue, GPU hours, /home
slurm_read job    ──▶ one job's record, queue context, GPU metrics
slurm_read queue  ──▶ position and the jobs ahead
slurm_read log    ──▶ stdout tail, or a grep over the whole file
     │
     ▼
metrics_read node ──▶ a Slurm node's GPU and host telemetry
```

Every read is scoped to your organization. A cluster name that is not
one of your organization's Slurm clusters answers `404 not-found`,
whether it belongs to someone else, is a Kubernetes or VM cluster, or
never existed.

## Arguments

| Argument | Values | Applies to |
|---|---|---|
| `cluster` | the cluster name (`grid_whoami` lists them) | required on every call |
| `action` | `status`, `job`, `queue`, `log` | required on every call |
| `jobid` | a Slurm job id: digits, optionally `_<task>` | required for `job`, `queue`, `log` |
| `source` | `live` (default) or `mirror` | `status`; `mirror` answers from the platform's last snapshot without dialing the cluster's island |
| `tail_lines` | 1 to 1000; defaults to 200 | `log`; the lines kept from the end |
| `query` | a fixed string, matched case insensitively | `log`; greps the whole file instead of tailing |
| `file` | an index into the answer's `files` | `log`; a per task log next to the batch stdout; the default is the batch stdout |
| `incarnation` | the `coverage.job.incarnation` of a `job` answer | `log` on a [centrally read island](#centrally-read-islands) |

Each argument belongs to the actions its row names. An argument present
on another action refuses `422 bad-request`; nothing is silently
ignored. `file` and `tail_lines` must be whole numbers. A blank `query`
tails.

## action=status

```json
{
  "age": 3,
  "error": null,
  "nodes": {
    "node-1": {"state": "mixed", "flags": [], "gpus_alloc": 4, "gpus_total": 8,
               "jobs": [{"id": "4242", "user": "ada", "gpus": 4}]},
    "node-2": {"state": "drained", "flags": ["drain"], "gpus_alloc": 0,
               "gpus_total": 8, "jobs": []}
  },
  "queue": [{"id": "4243", "name": "train", "user": "ada", "state": "pending",
             "gpus": 8, "reason": "Resources", "time": "0:00"}],
  "usage": {"month": "2026-09", "users": {"ada": 12.5}, "error": null},
  "storage": {"fs": {"used": 10995116277760, "total": 109951162777600,
                     "free": 98956046500000},
              "dirs": {"ada": 7516192768}, "error": null}
}
```

| Field | What it carries |
|---|---|
| `age` | the age of the scheduler snapshot, seconds |
| `error` | the scheduler feed's error, or `null` |
| `nodes` | one entry per compute node, keyed by the name `sinfo` shows: `state`, `flags`, `gpus_alloc`, `gpus_total`, and `jobs` (the jobs holding GPUs on it) |
| `queue` | `squeue`, live: `id`, `name`, `user`, `state`, `gpus`, and `reason` while pending or `time` while running |
| `usage` | GPU hours this month per user from the scheduler's accounting, with `month`; `error` when accounting did not answer |
| `storage` | the shared `/home` export: `fs` (`used`, `total`, `free`, bytes), `dirs` (bytes per member of this cluster), `error` when the scan did not run |
| `mirror` | `true` when `source=mirror` answered |

`usage` is the scheduler's accounting. The
[billing records](https://docs.nationalcompute.com/api/billing.md) are the authority on what a job cost.
`storage.dirs` lists your organization's members only.

### Node states

A node takes no new jobs when its `state` is one of `down`, `drained`,
`draining`, `error`, `fail`, `failing`, `unknown`, `invalid`, `inval`,
or when `flags` carries `drain`, `not_responding`, `invalid_reg` or
`fail`. The console paints those red. `idle` means free, `mixed` and
`allocated` mean busy, `completing` means a job is finishing on it.
Every other word is Slurm's own state, verbatim.

## action=job

```json
{
  "job": {
    "id": "4242", "name": "train", "user": "ada", "state": "pending",
    "reason": "Resources", "nodes": 2, "nodelist": "", "gpus": 16, "cpus": 448,
    "mem": "1500G", "time_limit": "1-00:00:00", "elapsed": "00:00:00",
    "submitted": "2026-09-29T08:10:00", "started": "", "est_start": "2026-09-29T09:00:00",
    "ended": "", "workdir": "/home/ada/train", "stdout": "/home/ada/train/slurm-4242.out",
    "exit_code": "0:0", "partition": "main", "account": "ada", "qos": "normal",
    "gpu_alloc": {}, "source": "scontrol"
  },
  "queue": {"position": 2, "total": 5, "ahead": [{"id": "4240", "reason": "Priority"}],
            "ahead_total": 1},
  "metrics": {"gpus": [], "series": null, "error": null}
}
```

`job` is the scheduler's record, `scontrol` while the job is live and
`sacct` once accounting holds it (`source` says which). `started` is
empty until the job runs; `est_start` carries the scheduler's estimate
while it waits; `ended` is empty until it ends. `queue` is present only
while `state` is `pending` and is `null` otherwise. `metrics` is the
node scope GPU telemetry over the job's window; its `error` reads
`unavailable` when the metrics store did not answer, never a reason
with internals. A job accounting no longer knows answers
`404 not-found`.

## action=queue

```json
{"jobid": "4242", "state": "pending",
 "queue": {"position": 2, "total": 5, "ahead": [{"id": "4240", "reason": "Priority"}],
           "ahead_total": 1}}
```

`position` counts from 1 among pending jobs in scheduler order; `total`
is the pending count; `ahead` lists the nearest jobs ahead (at most 5)
with their wait reasons; `ahead_total` is the real count ahead. `queue`
is `null` when the job is not pending. This action reads no metrics.

## action=log

```json
{
  "path": "/home/ada/train/slurm-4242.out",
  "lines": ["step 1200 loss 2.31", "step 1210 loss 2.29"],
  "truncated": true,
  "clipped": true,
  "files": ["log-train_4242_0.out", "log-train_4242_1.out"],
  "selected": null,
  "note": null
}
```

The tail reads the batch stdout off the shared `/home` export. `lines`
carries the last `tail_lines` lines; `truncated` is `true` when the
file held more than the platform's own tail window (1000 lines or 512
KiB); `clipped` is `true` when `tail_lines` dropped lines from that
window. `files` lists the per task logs written next to the stdout by
wrapper stacks (`srun` redirections named `*_<jobid>_<rank>.out`), and
`file` selects one by index; `selected` echoes it. `note` explains an
empty or missing log ("log file is empty so far", a log on node local
storage).

With `query` the answer is a grep over the whole file: `lines` are the
matching lines (the last 500 matches at most), `lnos` their line
numbers, `matches` the count of lines returned, `filtered` reads
`true`. When `tail_lines` dropped matches, `clipped` reads `true` and
`total_matches` keeps the file wide count. A `query` longer than 200
characters is cut to 200.

## Centrally read islands

On an island whose reads the platform serves from its own record instead
of dialing the island, every answer carries `coverage`: per source,
`availability` (`complete`, `partial`, `unavailable`), `freshness`,
`observed_at`, and a `reason` when something is missing. The status
answer is then the platform's inventory (`inventory.nodes` with `state`,
`gpus_total`, `gpus_alloc`, `cpus_alloc`, plus `running_count` and
`queue_count`); `jobs`, `queue`, `usage` and `storage` read `null` with
their reasons. The job answer carries `coverage.job.incarnation`, which
`log` needs as `incarnation`. A log answer there carries `first_record`
and `last_record`, the record numbers the returned lines span. A record
that is not complete refuses with `503 station-unavailable` carrying
`code` (the read model's refusal, `central_read_unavailable`), `reason`
(the coverage's own reason, for example `job_incarnation_required` or
`central_store_disabled`) and `coverage`. The message says retry only
when the reason is one a later call can clear.

## Refusals

| Code | Status | Meaning |
|---|---|---|
| `bad-request` | 422 | an unknown `action` or `source`, a missing `jobid`, a malformed job id, an argument outside its action, a fractional `file` or `tail_lines`, `file` outside 0 to 127, `incarnation` on an island that is not centrally read |
| `not-found` | 404 | no Slurm cluster of that name in your organization; a job accounting does not know; per job monitoring not enabled for the cluster (`job`, `queue`, `log`) |
| `cluster-not-ready` | 409 | the cluster is still being built or is not ready; retry |
| `station-unavailable` | 503 | the cluster's island did not answer, or a centrally kept record is not complete (`code`, `reason` and `coverage` say why) |

## Connection facts

`grid_whoami` lists every cluster your organization holds. A `slurm`
entry adds the login endpoint the console's Connect step shows:

| Field | What it carries |
|---|---|
| `ssh_host` | the login address |
| `ssh_port` | the ssh port |
| `ssh_user` | the shared login account, or `null` when the login is your own OIDC identity |
| `ssh_auth` | `key` (keys the platform registered for your organization) or `opk` (a browser sign in through the setup script) |

All four read `null` while the endpoint is unresolved. Kubernetes and VM
entries carry none of these keys. `GET /api/whoami` by
[org API token](https://docs.nationalcompute.com/authentication.md#discovering-what-a-token-can-address)
answers the same body.

## Telemetry

[`metrics_read`](https://docs.nationalcompute.com/api/metrics.md) accepts a Slurm cluster as `cluster` for
`action=node` and `action=flags` with `node`; the node name is the key
`status.nodes` uses. Workload targets (`kind`, `name`) are Kubernetes
only and refuse `422 bad-request` on a Slurm cluster. When your
organization holds a Kubernetes cluster and a Slurm cluster of the same
name, `metrics_read` reads the Kubernetes one.

---

# Dashboards

Source: https://docs.nationalcompute.com/api/dashboards/ (this markdown: https://docs.nationalcompute.com/api/dashboards.md)

`dashboard_write` turns PromQL into a Grafana dashboard document in your
organization's Grafana, owned by the member who asked for it;
`dashboard_read` lists the documents you can show. Both are tools on the
[MCP server](https://docs.nationalcompute.com/api/mcp.md), single phase: no preview, no `confirm_token`. They
exist so an agent shows monitoring in Grafana's own UI, live, instead of
a hand-drawn page. The numbers behind a chart come from
[`metrics_query`](https://docs.nationalcompute.com/api/metrics.md#free-form-queries-metrics_query), which
runs the same expressions.

## Documents

One document per chart. Every `create` writes a new dashboard with its
own uid (`marshall-` followed by 12 hex characters) into a `Marshall`
folder of your organization's Grafana org, tagged `marshall`,
`member:<your username>` and `scratch` or `kept`. Several charts at once
are several panels in one document or several documents. The document
is not editable in Grafana: every change goes through `dashboard_write`.

Ownership is by member. `update`, `keep` and `delete` act only on
documents you created; another member's uid is refused as `404
not-found`, the same answer a missing uid gets. Every document is
visible to the whole organization in Grafana, like every dashboard
there; the tool enumerates only your own.

Every panel reads your organization's own monitoring and nothing else:
the datasource is pinned to your organization's tenant, and PromQL has no
syntax for another one.

## Scratch and kept

A new document is **scratch**: it expires 24 hours after its last use
and is then deleted from Grafana. An `update`, a `dashboard_read get`,
or a console pane showing it counts as use. `keep` names the document,
moves it to the `Saved` folder under `Marshall` and removes the expiry;
the uid does not change, so links stay valid. `delete` removes a
document at once, scratch or kept; there is no undo, but nothing beyond
the document is lost.

| Limit | Value |
|---|---|
| Scratch expiry | 24 hours after last use (currently) |
| Scratch documents per member | 20 |
| Scratch documents per organization | 200 |
| Panels per document | 12 |
| Queries per panel | 8 |
| PromQL per query | 4000 characters |
| Title | 120 characters |

Kept documents do not count against the scratch caps.

## `dashboard_write`

| Argument | Values | Meaning |
|---|---|---|
| `action` | `create`, `update`, `keep`, `delete` | required |
| `dashboard` | a uid of your own | required for `update`, `keep`, `delete` |
| `title` | at most 120 characters | required on `create`; renames on `update` and `keep` |
| `panels` | 1 to 12 panel specs, below | required on `create`; replaces the panels on `update` |
| `range` | `1h`, `6h`, `24h`, `7d` (default `6h`) | the window the dashboard opens on |

`update` takes any of `title`, `panels`, `range` and carries the rest
over from the document; at least one is required. `keep` on a kept
document renames it.

A panel spec is `{title, kind, queries, unit?, thresholds?}`:

| Field | Values | Meaning |
|---|---|---|
| `title` | at most 120 characters | the panel title |
| `kind` | `timeseries`, `stat`, `gauge`, `bargauge`, `table` | lines over the window; one number; a gauge; one bar per series; an instant read as rows |
| `queries` | 1 to 8 of `{expr, legend?}` | PromQL, at most 4000 characters each; `legend` is a Grafana legend template such as `{{instance}}` |
| `unit` | a Grafana unit id (`percent`, `celsius`, `bytes`, `watt`, `short`, …) | optional |
| `thresholds` | up to 4 ascending numbers | optional; colors the panel green below the first, then yellow, orange, red |

The answer carries `dashboard` (the uid), `title`, `folder` (`scratch`
or `saved`), `panels` (`[{id, title, kind}]`, ids `1..n`), `range`,
`version`, `created_at`, `last_used_at`, `expires_at` (`null` once
kept), `url` (the document in your organization's Grafana) and
`member_signed_in`. `member_signed_in` is `false` when you have never
opened Grafana: the first visit to the URL signs you in once, and
charts render after that; `null` means the server could not tell.
`delete` answers `{dashboard, title, deleted: true}`.

## `dashboard_read`

| Argument | Values | Meaning |
|---|---|---|
| `action` | `list`, `get` | required |
| `dashboard` | a uid | required for `get` |

`list` answers `dashboards` (your own documents, the shape above),
`platform` (the platform's tenant dashboards in your organization's
Grafana, each with `uid`, `title`, `tags`, `panels` `[{id, title,
kind}]`, `variables` `[{name, label, hidden, current}]` and `url`),
`caps` (`scratch_used`, `scratch_cap`, `org_scratch_used`,
`org_scratch_cap`), `scratch_ttl_s` and `member_signed_in`. Other
members' documents are not listed.

`get` on your own uid answers the document with its `spec` (the panels
as given) and counts as use; `get` on a platform dashboard's uid answers
its panels and variables.

## Refusals

| Code | When | What to do |
|---|---|---|
| `bad-request` (422) | a missing `title` or `panels`, an unknown `kind` or `action`, an expression over 4000 characters, more than 12 panels or 8 queries; Grafana refused the composed document (its message is in `detail`) | fix the spec |
| `not-found` (404) | a uid that does not exist or is another member's | `dashboard_read list` names yours |
| `dashboard-cap` (409) | the per-member or per-organization scratch cap; `detail` names the least recently used scratch document and `oldest` carries its uid | `keep` or `delete` one, then retry |
| `grafana-unavailable` (503) | Grafana did not answer, or your organization has no Grafana org yet | retry shortly; the org appears with its first monitored cluster |
| `dashboards-unavailable` (503) | the environment has no dashboard store | none; the surface is off there |

The member rate limit and every other rule in [MCP server](https://docs.nationalcompute.com/api/mcp.md#limits-and-semantics) apply.

---

# Checkpoints

Source: https://docs.nationalcompute.com/api/checkpoints/ (this markdown: https://docs.nationalcompute.com/api/checkpoints.md)

A **checkpoint** is a static file a member of an organization published:
an html report, a markdown write-up, an image, a dataset sample, model
weights, any file. It is kept by National Compute and served back on
demand. Every checkpoint belongs to one of two places:

- **the organization** — every member can find, open and download it
  from the console's Checkpoint pane or through the tools below;
- **the public feed** — a global list anyone can read, signed in or not.
  Publishing there is National Compute's own for now: an organization's
  public publish is refused (`403 forbidden`, `reason: public-restricted`)
  and the console offers no Public choice.

The console's Marshall workspace publishes from a document pane (the
Publish button) and browses the organization's checkpoints in a checkpoint
pane; the console's Checkpoints page lists them and the public feed, each
row marked Organization or Public, with a viewer and a link per item;
Marshall itself publishes with `checkpoint_publish` and opens
items with `checkpoint_open` inside a workspace. Two interfaces reach
beyond the console: the anonymous public feed and the authenticated REST
family behind the [MCP server](https://docs.nationalcompute.com/api/mcp.md) tools `checkpoint_read` and
`checkpoint_write`.

## The public feed (no auth)

```
GET https://nationalcompute.com/api/checkpoints/public?q=<text>&limit=<n>&before=<iso>
GET https://nationalcompute.com/api/checkpoints/public/<id>
GET https://nationalcompute.com/api/checkpoints/public/<id>/content[?download=1]
```

The list answers the live public items newest first, plus the platform's
own published documents, as `{checkpoints: [...], more, cursor}`; `q`
searches titles, descriptions and file names; `before` takes the previous
answer's `cursor`. Each item is:

```json
{"id": "…", "title": "…", "description": "…", "filename": "report.html",
 "content_type": "text/html; charset=utf-8", "bytes": 48213, "sha256": "…",
 "visibility": "public", "status": "ready", "status_detail": "Public",
 "publisher": {"org_display": "…", "username": "…"}, "revision": 2,
 "published_at": "…", "created_at": "…",
 "urls": {"content": "/api/checkpoints/public/<id>/content",
          "download": "/api/checkpoints/public/<id>/content?download=1"}}
```

A platform document's `publisher` is `{"name": "National Compute"}`. The
feed shows the publishing organization's display name and the member's
username: both agreed to that when they published publicly.

`/content` streams the stored bytes with the stored content type,
`X-Content-Type-Options: nosniff`, `Accept-Ranges: bytes` (a `Range`
request answers `206`), `Cache-Control: no-store` and, for html, SVG and
XML, a `Content-Security-Policy: sandbox …` header: a document opened
straight from the apex runs in an opaque origin. `?download=1` turns the
disposition into an attachment. Scripts are always attachments. Every
`GET` here carries `Access-Control-Allow-Origin: *`, so a page on any
origin may read the feed; `OPTIONS` answers `204`.

An unknown id is `404 not-found`. The feed lists an item only after review;
an item under review, rejected, or published to its organization alone is
not here.

## The authenticated family

```
GET    /api/checkpoints?scope=org&q&limit&before
POST   /api/checkpoints                      {title, description?, filename, content_type, bytes,
                                              visibility: org|public, upload?: direct|resumable,
                                              sha256?, replace?: true|false|<id>}
PUT    /api/checkpoints/<id>/content         the raw bytes (direct uploads)
POST   /api/checkpoints/<id>/commit          {bytes, sha256?}  (resumable uploads)
GET    /api/checkpoints/<id>
GET    /api/checkpoints/<id>/content[?download=1]
DELETE /api/checkpoints/<id>
```

Bearer: a member token, an OAuth session of the MCP server, or an
organization API token (reads only; a token cannot publish). The same
family answers the console's Marshall workspace on its own session.

**Publishing** is two calls. `POST` records the facts and answers
`{id, revision, status: "pending", visibility, replaced, review: none|required,
upload}`. With `upload: direct` (files up to 32 MiB) the answer carries
`content_url`: `PUT` the bytes there with the declared `Content-Type`. With
`upload: resumable` (any size up to 50 GiB) it carries a one-time storage
`session_url` and `expires_at`: upload the bytes to that session with
resumable `PUT`s (no credential of yours travels with them), then `POST
…/commit` with the final size. Either way the answer to the last call is
the item. Publishing the same file name again to the same place replaces
the item in place — the id stays, `revision` advances; `replace: false`
asks for a new item instead and `409 conflict` with `reason: exists`
says one already lives.

**Visibility.** `org` items are live the moment the bytes land. `public`
is accepted from National Compute's own organization only (`403 forbidden`,
`reason: public-restricted` otherwise; the list's `public_publish` says in
advance). Accepted `public` items enter **review**: the item is visible to the organization with
`status: "review"` and `status_detail: "Under review"`, National Compute
reviews it (an automated check assists; a person decides), and approval
moves it to the feed. A rejection leaves the item with the organization
and tells the publisher why (`rejection_reason`, visible to the creator
alone). A public replacement keeps the previous revision live until the
new one is approved.

**Reading.** The list is your organization's items in every state
(`scope=org`, the only scope; `public` or `all` answer `422 bad-request`).
The feed is read through the anonymous route alone; a public item still
reads by id. Items you may act on carry `can_delete: true`.

**Deleting** is the publisher's alone: `DELETE` by the creator removes the
item for the organization and from the feed; copies already downloaded
are not recalled. Another member's id, an unknown id and a deleted id
answer **`410 gone`** on this family (never `404`, which here means the
feature is off).

### Refusals

| Status | `error` | When |
|---|---|---|
| 404 | `not-found` | checkpoints are not enabled; or the item is unknown on the public feed |
| 410 | `gone` | not your item, deleted, or unknown (bearer family) |
| 413 | `too-large` | over the direct cap (`limit_bytes`) or the per-checkpoint cap |
| 422 | `secret-detected` | the content looks like it carries a credential or token; nothing is stored |
| 422 | `bad-request` | a field is missing or malformed (`detail`) |
| 409 | `conflict` | `reason`: `exists`, `not-pending`, `size-mismatch`, `sha-mismatch`, `object-missing`, `resumable-upload`, `wrong-status` |
| 429 | `org-cap` | the organization's checkpoint storage is full; delete something first |
| 403 | `forbidden` | `reason: public-restricted` — a public publish from an organization the feed is not open to; or an organization API token tried to publish |
| 408 | `upload-timeout` | a resumable session expired before its commit |

| Limit | Value |
|---|---|
| One direct upload (through the API) | 32 MiB |
| One checkpoint (resumable upload) | 50 GiB |
| Checkpoint storage per organization | 200 GiB |
| Title / description / file name | 200 / 4000 / 255 characters |
| List page | 50 items by default, 100 at most |

## The MCP tools

`checkpoint_read` (`action: list | search | get`, `q`, `id`, `limit`,
`before`) answers your organization's items, the owner's view, and reads
any item by id; it never lists the public feed. `checkpoint_write`
(`action: delete`, `id`) deletes one of your own items in two phases: the
preview names the item, the confirm removes it. Publishing carries bytes
the MCP wire cannot, so it stays with the REST family and, inside a
workspace, with Marshall's `checkpoint_publish`. Both tools are absent
while checkpoints are not enabled for the platform.

## What is never published

The content check refuses files that look like credentials or tokens
(`.env`, private keys, kubeconfigs, API keys in text) at every door: the
console, Marshall's tool and the API. Public items pass a content review
before they appear; items that fail stay with their organization.

---

# Hardware roadmap

Source: https://docs.nationalcompute.com/api/roadmap/ (this markdown: https://docs.nationalcompute.com/api/roadmap.md)

`roadmap_read` answers the calendar quarter in which a GPU model is
expected to land on the platform. The quarter is an expectation from
National Compute's capacity planning. It is never a commitment. The
answer carries no quantity, no date inside the quarter and no source.

```text
capacity planning ──▶ platform roadmap ──▶ roadmap_read ──▶ your agent
    (internal)          GPU model + expected quarter, nothing else
```

The tool lives on the [MCP server](https://docs.nationalcompute.com/api/mcp.md). It is read only and single
phase. No `confirm_token` is involved. Every signed in member reads the
same list. The answer is not scoped to an organization.

## Arguments

| Argument | Values | Applies to |
|---|---|---|
| `gpu_model` | a GPU model name as the [capacity reads](https://docs.nationalcompute.com/api/index.md#nodes-and-gpus) spell `gpu_model`; matched case insensitively | optional; omit it for the whole list |

## The answer

A request for one model carries `{"gpu_model": "<model>"}`. Its answer:

```json
{
  "as_of": "2026-09-24T03:50:00Z",
  "age_s": 412,
  "entries": [{"gpu_model": "<model>", "expected_quarter": "2028-Q3"}],
  "note": "Expected landing quarters from National Compute's planning. An expectation, not a commitment: quantities, dates, sources and sites are not published. Ask marshall to record your interest and the team follows up."
}
```

| Field | What it carries |
|---|---|
| `as_of` | when the platform last refreshed the roadmap, ISO 8601 UTC |
| `age_s` | the age of that refresh, seconds |
| `entries` | one entry per GPU model, each `{gpu_model, expected_quarter}` |
| `expected_quarter` | a calendar quarter in UTC, as `YYYY-Qn` |
| `note` | how to read the answer, in one short paragraph |

Read the example as: planning expects that model to land in the third
quarter of 2028, between July and September. The values are
illustrative; the live answer names real models. Nothing in the answer
narrows a quarter further. The answer does not say how many GPUs, on which
day, or from where.

Each GPU model appears once. The quarter shown is the earliest one
planning expects for that model. Capacity that has already landed is
never a roadmap entry. The [capacity reads](https://docs.nationalcompute.com/api/index.md#nodes-and-gpus)
name the GPU class each cluster prices today as `gpu_model`. A model
with no published landing answers `entries: []`. Its `note` says that
no landing is published for that model. That is a valid answer:
nothing firm is planned for it.

## What the roadmap publishes

| Published | Never published |
|---|---|
| the GPU model | quantities |
| the calendar quarter | exact dates |
| | providers or suppliers |
| | sites or regions |
| | deal status |
| | customers |
| | prices |

The right column is absent from the wire rather than redacted from it.
The tool's output is a fixed mapping of the two published fields. No
other value can appear on any question. An agent pressed for a count, a
date or a supplier has nothing to read.

Ask marshall to record your interest in a GPU model. The team follows
up from there. That request works for a model with no entry as well.

## Freshness

The platform refreshes the roadmap from its planning data at regular
intervals. It keeps the last good copy. When a refresh fails, the last
good copy stays in place. `age_s` then keeps growing. Read `age_s`
before you quote an old answer.

## Refusals

| Code | Status | Meaning |
|---|---|---|
| `roadmap-unavailable` | 503 | the roadmap feed has not loaded; retry later |

A roadmap that has never loaded is always a refusal. The tool never
presents it as an empty list. An empty `entries` on a successful answer
means nothing firm is planned. With `gpu_model` given, the empty list
speaks for the named model. Without it, the empty list speaks for every
model.

---

# VM capacity API

Source: https://docs.nationalcompute.com/api/vm-capacity/ (this markdown: https://docs.nationalcompute.com/api/vm-capacity.md)

Declare how many GPUs your organization wants and the most you'll pay
per GPU-hour; the market grants and reclaims nodes to match. Granted
nodes are yours over ssh until the market reclaims them or you scale
down.

Contract: [`GET /api/vm/openapi.json`](https://nationalcompute.com/api/vm/openapi.json).

| Route | Auth | What it does |
|---|---|---|
| `GET /api/vm/capacity` | `capacity:read` | current declaration, allocations, nominations |
| `PUT /api/vm/capacity` | `capacity:write` | declare or update capacity; also scales down |
| `POST /api/vm/capacity/swap` | `capacity:write` | release ONE node and let the market refill its slot |
| `GET /api/vm/activity` | token | what you declared and what the platform did about it, newest first |
| `GET /api/vm/history` | token | declared versus granted, and your ceiling versus the clearing rate, over time |
| `GET /api/vm/orgs/{org}/ssh-keys` | token | list the org's ssh public keys |
| `POST /api/vm/orgs/{org}/ssh-keys` | token | add an ssh public key |
| `DELETE /api/vm/orgs/{org}/ssh-keys/{key_id}` | token | revoke a key |

`{org}` is your organization id — returned as `org_id` by
[`/api/whoami`](https://docs.nationalcompute.com/authentication.md#discovering-what-a-token-can-address),
and shown on the console's **API → Docs** page.

## Before you declare: register an ssh key

A grant installs your org's registered ssh keys on the node — with none
registered the node would be unreachable, so a capacity raise without a
key is refused (`422 no-ssh-keys`):

```sh
curl -X POST -H "Authorization: Bearer $NC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "ci", "public_key": "ssh-ed25519 AAAA... ci@example"}' \
  https://nationalcompute.com/api/vm/orgs/$ORG_ID/ssh-keys
```

Keys install on **future** grants only — adding or revoking a key never
touches nodes you already hold. A key can be scoped to one VM cluster
with the optional `cluster` field; omitted means org-wide. The console's
**SSH Keys** page manages the same list.

## Declaring capacity

```sh
curl -X PUT -H "Authorization: Bearer $NC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "max_gpus": 16,
        "max_price_per_gpu_hour": <price>,
        "expected_version": 7
      }' \
  https://nationalcompute.com/api/vm/capacity
```

| Field | Meaning |
|---|---|
| `max_gpus` | how many GPUs to hold — a multiple of `gpus_per_node` (whole nodes; a non-multiple is a `422`, never rounded). Lowering it is how you scale down; `0` releases everything. |
| `max_price_per_gpu_hour` | your ceiling per GPU-hour, USD, whole cents. A whole node clears at `gpus_per_node` × this. The billed rate is the clearing rate — often lower, never higher. |
| `release` | optional: node public IPs to release first on scale-down. **Replaces** the standing nomination set; omitted = keep current; `[]` = clear all (a nomination tied to an in-flight swap persists until the swap lands); an IP you don't hold is a `409` and nothing is written. |
| `expected_version` | compare-and-set guard — see [conventions](https://docs.nationalcompute.com/api/index.md). |
| `cluster` | only needed with an org-wide token when the org holds several VM clusters. |

Raising `max_gpus` requires the market-managed posture (`409
static-posture` otherwise — on an operator-managed cluster nothing fills
new capacity, so a raise would pend forever). Lowering and `release`
always work.

The [MCP server](https://docs.nationalcompute.com/api/mcp.md)'s `bid_write kind=vm` is the same write in two
phases: `max_gpus`, `max_price_per_gpu_hour` and the optional `release`
list. The preview echoes the nominated nodes, the standing nominations
they replace, and flags any entry that matches no node you hold.

## Reading state

`GET /api/vm/capacity` returns the full picture:

| Field | Meaning |
|---|---|
| `version` | echo back as `expected_version` on writes |
| `writes_armed` | whether the capacity PUT is accepted for this cluster's island at all. `false` means every PUT answers `503 capacity-writes-disabled` while reads keep serving |
| `gpus_per_node` | the node ↔ GPU conversion — read it, never hardcode it |
| `gpu_vendor`, `gpu_model` | the maker and the GPU class this declaration buys. `gpu_model` is `null` while the platform has not set one |
| `declared` | your standing declaration (ceiling, `max_gpus`, `bid_too_low`, `capacity_unavailable`, who updated it and when); `null` until the first PUT |
| `pending` | declared capacity (in whole nodes) not yet covered by an open allocation |
| `allocations[]` | the nodes you hold: `public_ip` (what you ssh to), `state` (`provisioning` / `active` / `reclaiming`), `slot`, `swap` |
| `planned_release[]` | what a shrink will destroy, as of this read |
| `nominations[]` | your standing release nominations, echoed as public IPs |
| `swaps[]` | public IPs with a swap in flight |
| `billing_hold` | your org's billing hold state, or `null`. When set: `state` (`hold` or `grace`), `reason`, and for `hold` a `stage` — `notice` (new capacity paused) or `enforce` (capacity being reclaimed) — plus `enforce_after_ms`. A hold never alters `declared`: the response always echoes exactly what you asked for. |
| `spend_limit_enforced`, `billing_hold_enforced` | whether your organization's monthly spend limit and a balance at or below zero hold its capacity at all. `false` means the platform records the figure and does not act on it for your organization's clusters. The [billing summary](https://docs.nationalcompute.com/api/billing.md#balance-and-totals) carries the same pair |
| `environment` | site-specific onboarding facts (ssh user, environment variables, storage paths) |

A representative response:

```json
{
  "cluster": "acme-research",
  "version": 8,
  "writes_armed": true,
  "gpus_per_node": 8,
  "gpu_vendor": "…",
  "gpu_model": "…",
  "declared": {
    "max_price_per_gpu_hour": …,
    "max_gpus": 24,
    "bid_too_low": false,
    "capacity_unavailable": null,
    "updated_at": "2026-08-26T14:02:11Z",
    "updated_by": "api-token:ci"
  },
  "pending": 1,
  "allocations": [
    {"public_ip": "203.0.113.7",  "granted_at": "2026-08-25T09:14:02Z",
     "state": "active",       "slot": 0, "swap": false},
    {"public_ip": "203.0.113.21", "granted_at": "2026-08-26T13:58:40Z",
     "state": "provisioning", "slot": 1, "swap": false}
  ],
  "planned_release": [],
  "nominations": [],
  "swaps": [],
  "billing_hold": null,
  "environment": {"ssh_user": "tenant"}
}
```

Here the org wants 24 GPUs (3 nodes): one node is active, one is still
provisioning, and one slot is `pending` — declared but not yet granted
by the market.

## Allocation lifecycle

Each held node moves through three states:

1. **`provisioning`** — the market granted the slot and the node is
   being prepared; `public_ip` may still be `null`. Your registered ssh
   keys are installed during this phase.
2. **`active`** — the node is yours over ssh at its `public_ip`. A
   fresh grant carries the [minimum-duration protection
   window](https://docs.nationalcompute.com/market.md#minimum-duration-protection); after it lapses
   the node competes in the market normally.
3. **`reclaiming`** — the node is being taken back: the market cleared
   above your ceiling, you scaled down, or a swap is landing. The node
   will be **destroyed** — anything on local disk is lost, so treat
   node state as ephemeral and keep durable data elsewhere.

You can see a reclaim coming from three signals on the capacity read:
`planned_release` (what a shrink will destroy as of this read),
`nominations` (your own standing choices for who goes first), and
`swaps` (destroys in flight). [Billing](https://docs.nationalcompute.com/billing.md) runs from grant
completion to the start of reclaim.

## Swapping a node

Releases one node — **the node is destroyed and all data on it is
lost** — while its capacity slot stays declared, so the market grants a
replacement once the release lands:

```sh
curl -X POST -H "Authorization: Bearer $NC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"node": "203.0.113.7"}' \
  https://nationalcompute.com/api/vm/capacity/swap
```

The release happens when the swap actuates; a swap that can't actuate
within its freshness window (currently an hour) lapses and the node
simply stays yours. The replacement depends on supply and your limit price.
Retrying the same node while it swaps is an idempotent `200`; swaps on
different nodes stack, spaced by the shared write cadence. To shrink
instead of swap, lower `max_gpus` on the capacity PUT (with `release`
to pick the victim).

## History and activity

`GET /api/vm/activity` is the Instances page's History: applied
declarations and allocation events, newest first, 30 per page. Each
source is read 400 rows deep before the merge, so very old history ages
out. Pages carry 30 entries
(`?page=`, from 0; `more` says whether a next page exists). An entry of
`kind: request` is a declaration the platform applied, with its
`actor`, `max_gpus`, and `max_price_per_gpu_hour`; `who` is `user` for
your organization's own declares and `system` for a platform override.
An entry of `kind: action` is an allocation event: a node granted or
released, named by its public IP in `note`, or a grant attempt unwound.
`?who=user` or `?who=system` filters. Capacity reads never appear.

`GET /api/vm/history?hours=24` is the Instances page's charts as
bucketed series over a window of 0.5 to 8760 hours: `t0` (epoch seconds
of the first bucket), `step` (bucket width in seconds, `max(60, span /
400)`), `max_nodes` (the declared node count in force: `max_gpus`
divided by `gpus_per_node`; `null` before the first declaration),
`granted` (nodes held; `0` when none, never `null`), `max_price` (your
declared ceiling) and `price` (the market's clearing rate), both in USD
per node-hour. Divide by `gpus_per_node` to read them per GPU-hour. A
`null` price is a bucket where nothing was declared or nothing cleared.

## Errors

| Status | When |
|---|---|
| `401` / `403` / `404` | see [authentication](https://docs.nationalcompute.com/authentication.md) |
| `409` | `version-mismatch`, `unknown-node` (a `release`/swap entry that matches no held node), `ambiguous-cluster`, `static-posture`; swaps can also refuse `swap-disabled`, `cluster-gated`, or `node-pinned` |
| `422` | validation — `price-precision`, `bid-too-low` (any `max_price_per_gpu_hour` set too low to ever clear the market, raising or lowering `max_gpus`; a standing price left unchanged can still lower `max_gpus` or release, so a market that moved past your ceiling never traps held capacity), `price-above-maximum` (a `max_price_per_gpu_hour` over the cap the GET echoes as `max_price_cap_per_gpu_hour`; the response carries `maximum`), `no-ssh-keys`, a non-multiple `max_gpus`; the `detail` says exactly what |
| `429` | faster than the per cluster write cadence, or past the token's request budget for the minute; wait `retry_after_s` (the `Retry-After` header carries the same figure) |
| `503` | temporary platform condition — reads keep serving, retry writes later |

The [error reference](https://docs.nationalcompute.com/api/errors.md) covers every code with handling
advice.

The too-low check runs at write time only. When market conditions
move past a standing declaration, the ask keeps losing and its
unfilled capacity never arrives. The GET reports this as
`declared.bid_too_low: true`; check the
[market feed](https://docs.nationalcompute.com/api/market-feed.md) and raise `max_price_per_gpu_hour` to
fill again.

`declared.capacity_unavailable` is the other reason a standing ask can
sit unfilled, and it is not about price. `true` means the site has no
capacity of the requested class right now, so raising the price will
not fill pending slots sooner. The
system retries automatically and slots fill when capacity returns.
`false` means capacity is flowing normally. `null` means no verdict
(the detector is not armed at this site). The same verdict reaches
Kubernetes clusters on the console's Workloads page — see
[when the site has no capacity](https://docs.nationalcompute.com/kubernetes.md#when-the-site-has-no-capacity).

---

# Kubernetes limit price API

Source: https://docs.nationalcompute.com/api/k8s-bid/ (this markdown: https://docs.nationalcompute.com/api/k8s-bid.md)

For a Kubernetes cluster, demand is derived from the cluster's own jobs:
a GPU job is one request priced in whole nodes on the market, capacity
follows the request, and idle nodes return to the pool
([Kubernetes clusters](https://docs.nationalcompute.com/kubernetes.md) covers the full behavior).
There is nothing to declare except the
price — one number per cluster, the **most you'll pay per GPU-hour** —
or per job, through the `nationalcompute.com/limit-price` label.

Contract: [`GET /api/k8s/openapi.json`](https://nationalcompute.com/api/k8s/openapi.json).

| Route | Auth | What it does |
|---|---|---|
| `GET /api/k8s/bid` | `capacity:read` | current limit price, its standing verdict, version |
| `PUT /api/k8s/bid` | `capacity:write` | set (or withdraw) the limit price; `expected_version` makes it compare and set |
| `GET /api/k8s/storage/volumes` | `capacity:read` | your shared storage volumes ([below](#shared-storage-volumes)) |
| `DELETE /api/orgs/{org_id}/k8s/storage/volumes/{volume_id}` | console session only | destroy a preserved volume ([below](#shared-storage-volumes)) |

The same contract carries the five [market analytics](https://docs.nationalcompute.com/api/k8s-market.md)
routes the console's Burst Capacity page reads, and the
[cluster facts and Base Load book](https://docs.nationalcompute.com/api/k8s-cluster.md).

There are no slots, no node counts, no node IPs, and no release verbs on
this surface — submit jobs and the market does the rest.

## Setting the limit price { #setting-the-bid }

```sh
curl -X PUT -H "Authorization: Bearer $NC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"max_price_per_gpu_hour": <price>}' \
  https://nationalcompute.com/api/k8s/bid
```

- The value is USD per GPU-hour, whole cents. The effective billed rate
  is often lower — this is a ceiling, not a price.
- `0` withdraws the limit price.
- A declare too low to ever clear the market is refused —
  `bid-too-low`. Check the [price-to-win ladder](https://docs.nationalcompute.com/api/k8s-cluster-pages.md)
  and the [market feed](https://docs.nationalcompute.com/api/market-feed.md) for what wins now.
- Add `"cluster": "<name>"` (PUT body) or `?cluster=<name>` (GET) only
  with an org-wide token when your org holds several Kubernetes
  clusters.
- Add `"expected_version": <n>` with the `version` the GET returned.
  The write applies only while the version still matches. Otherwise
  the PUT reads `409 version-mismatch` with the live `version` in the
  body and nothing changes. Without the field the write applies
  unconditionally. Your human on the console and your agent share one
  limit price, so send it. The version counts every accepted write on the
  cluster, a withdrawal included, and never restarts: a `0` that
  withdraws the limit price bumps it, and a version read before the withdrawal
  is stale for good.

The PUT returns the applied state — the same body the GET serves:

```json
{
  "cluster": "acme-inference",
  "max_price_per_gpu_hour": …,
  "bid_too_low": false,
  "version": 4,
  "updated_at": "2026-08-26T15:20:07Z",
  "updated_by": "token:8dcbc6",
  "billing_hold": null,
  "spend_limit_enforced": true,
  "billing_hold_enforced": true
}
```

`updated_by` names who wrote the standing limit price: `token:<token_id>`
for an agent, a member's console identity for a human, `null` without a
limit price. Read it before changing a ceiling. A value that is not a token is
a human's decision. The organization's whole trail, human and token
writes side by side, is [`GET /api/org/activity`](https://docs.nationalcompute.com/api/organization.md#the-organizations-trail).

`billing_hold` is your org's billing hold state, or `null`. When set:
`state` (`hold` or `grace`), `reason`, and for `hold` a `stage` —
`notice` (new capacity paused) or `enforce` (capacity being reclaimed) —
plus `enforce_after_ms`. A hold never alters the declared limit price itself.
`spend_limit_enforced` and `billing_hold_enforced` say whether your
organization's spend limit and a balance at or below zero hold its
capacity at all. `false` means the platform records the figure and does
not act on it for your organization's clusters, so the cap is the
instruction a person gave the agent and nothing else stops it.

The body also carries `gpu_model` (the GPU class this limit price buys; `null`
while the platform has not set one), `gpu_vendor` (its maker),
`gpus_per_node` (the auction's supply unit) and `bid_too_low` (`true`
when the standing limit price is too low to ever clear the market,
`null` while nothing is declared);
[how the bid prices the market](#how-the-bid-prices-the-market)
explains the supply unit.

A declare too low to ever clear is refused with `bid-too-low` — the
[error reference](https://docs.nationalcompute.com/api/errors.md) covers the full catalog.

## How the bid prices the market

Each request bids its gang's GPU total × your limit price — the
cluster's declared price, or the job's own
`nationalcompute.com/limit-price` label (USD per GPU-hour) when set;
`"0"` is a real zero limit price, never a fallback to the cluster's price — in
whole nodes, so a request's exposure is its node count ×
`gpus_per_node` × your ceiling. The limit price is read live every auction tick
for the request's whole life: you may change a job's limit price at any
time; an increase is always fine, and a decrease below the price the
job held when its gang started (`Provisioned=True`) voids its
[protection window](https://docs.nationalcompute.com/market.md#minimum-duration-protection).

Capacity sells **in whole nodes**.
[Launch admission](https://docs.nationalcompute.com/kubernetes.md#launch-admission) enforces the
shape that clears. Every pod template's GPU request must be a multiple
of `gpus_per_node`. A smaller request is refused with `JobSizeTooSmall`.
So a request prices its nodes whole. On a cluster with packing onto
the [base load block](https://docs.nationalcompute.com/market.md#base-load-capacity) enabled a
smaller pod is admitted. It runs on the block and bids nothing.

A ceiling can be [too low to ever clear](https://docs.nationalcompute.com/market.md#a-bid-that-can-never-clear):
the write refuses it outright (`422 bid-too-low`), because pending
demand costs nothing but a bid the market can never meet means nothing
will ever arrive. What to bid is read off the market's own numbers —
the [price-to-win ladder](https://docs.nationalcompute.com/api/k8s-cluster-pages.md) and the
[market feed](https://docs.nationalcompute.com/api/market-feed.md) — never off a posted price.

Market conditions can also move past a **standing** bid (the
write-time check never re-runs). You do not have to detect this
yourself:

- The limit price read echoes `bid_too_low`: `true` means the standing
  limit price can never clear at any packing — check the market's
  recent clearing prices and raise it. `null` means there is no
  standing limit price, or no verdict to give.
- The market also says so inside your cluster: the request's reason
  reads `BidTooLow` and its message, posted on the Job as an Event,
  names your own bid's numbers ([queue events](#queue-events)). The
  console's Workloads page shows the verdict as the reason **bid too
  low to ever clear**. The signals clear the moment the bid becomes
  viable.

**No limit price means $0.** Your jobs still create demand, but nothing
is granted until you set a real ceiling.

**CPU-only work is outside the market.** A job requesting no GPUs runs
on the cluster's CPU worker and neither bids nor holds a GPU node:
CPU-only queued work doesn't pull capacity, and a GPU node running only
CPU pods reads as idle and returns to the pool after the five-minute
grace, billed at the departed job's rate (once its one-hour
[minimum hold](https://docs.nationalcompute.com/market.md#minimum-duration-protection) has run).

## Queue events

The market posts its verdicts inside your cluster on the job's
ProvisioningRequest — as the reason of its `Provisioned=False`
condition, with the sentence as the message — and as a Normal Event on
the owning `Job`, `JobSet` or `MPIJob`, one per change of reason
(`kubectl describe job`). Kueue copies the same message into the
Workload's admission check. A new request carries no reason until the
market's first read, one tick after it appears. Reasons, highest
precedence first:

| Reason | Meaning | Clears |
|---|---|---|
| `BalanceTooLow` | the org's balance cannot fund two hours of the request at its limit price; the message names the balance available and what the request costs | after a top-up |
| `BidTooLow` | the limit price is too low to ever clear the market; the message names your own bid's numbers | when the bid becomes viable |
| `NoSupply` | the site holds fewer nodes than the gang needs | when the site grows, or with a smaller gang |
| `PendingSupply` | protected nodes hold the supply the gang needs | on its own, as windows lapse |
| `Outbid` | lost on price; the message names the limit price that wins right now (`outbid · needs over $X per GPU-hour`) | raise the limit price, or wait |
| `ReservedBusy` | the request is smaller than one node and waits for GPUs of your base load block to free; it never bids; the message names the request's GPU count and why it waits | when block GPUs free |
| `ReservedTooSmall` | the request's pods are each smaller than one node and together exceed one node; a request with pods that small runs on one reserved node only and never bids | with whole node pods or with fewer pods that fit one node |
| `Pending` | no verdict this tick (`waiting for the market`): the request was seated this very tick, or the tick left it unfilled | next tick |

The two reserved reasons appear only on clusters with reservation
packing enabled. On such a cluster a request smaller than one node
takes no other market reason. A higher limit price never clears it.

A grant reads `NodeGranted` on the Job once per node; the request's
condition reads `Provisioning` (`k of N granted nodes ready`) until it
turns `Provisioned=True` and the gang starts. On a cluster with packing
enabled a request smaller than one node reads `waiting for N GPUs to
free on reserved node <node>` while it waits for GPUs. Once seated it
reads `reserved node <node> is ready with N GPUs for this job`. A
reclaim rides the request's `PreemptionNotice` condition and
`NodePreempting` events on the node and its pods; at the deadline the
whole job is suspended and requeued as a new request
([when the market reclaims a node](https://docs.nationalcompute.com/kubernetes.md#when-the-market-reclaims-a-node)).
A granted node that leaves the cluster after `Provisioned=True` fails the
request the same way (`Failed=True`, reason `MarketRevoked`) and the job
requeues as a new request.
A site out of capacity shows on the console's Workloads page as
**capacity unavailable**; the request's own reason carries no shortage
token and reads `PendingSupply`, `Outbid` or `Pending` as usual
([when the site has no capacity](https://docs.nationalcompute.com/kubernetes.md#when-the-site-has-no-capacity)).

## Shared storage volumes

Clusters provisioned with shared storage carry a shared volume
([Kubernetes clusters](https://docs.nationalcompute.com/kubernetes.md#the-shared-volume)). The same
contract lists your organization's volumes and deletes a preserved one.
Both routes are scoped to your organization: a volume outside your org
is a `404`, the same as one that does not exist.

| Route | Auth | What it does |
|---|---|---|
| `GET /api/k8s/storage/volumes` | `capacity:read` | your volumes: attached and preserved |
| `DELETE /api/orgs/{org_id}/k8s/storage/volumes/{volume_id}` | console session only | destroy one preserved volume (irreversible) |

The list takes an API token. The organization is the token's own and
nothing identifies it on the wire; the list is organization wide
whatever the token's binding, so scripts can inventory your volumes.

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/k8s/storage/volumes
```

Each volume row carries `volume_id`, `name`, `label`, `cluster` (`null`
for a preserved volume), `status` (`attached` or `preserved`),
`provisioned` and `used` bytes (`null` until the platform's storage scan
lands, never `0`), `mounted` (how many nodes currently mount it, `null`
when unknown), `billing` (the current charge where storage billing is
enabled for your site, else `null`) and `preserved`
(`{former_cluster, deleted_at, used_at_delete}` for a preserved volume,
else `null`).

`label` is the name the console shows for the volume: `<cluster>-shared`,
with the former cluster for a preserved volume. `name` is the storage
name. The two can differ; when they do, the console shows the storage
name under the label. `name` is the value the delete's `confirm` takes.
`label` is a display name and is not unique: two volumes can share it
after a cluster is deleted and created again under the same name.
Scripts key on `volume_id` or `name`, never on `label`.

### Deleting a volume is a console action

The DELETE takes **no API token**. Sign in to the console and use
**Delete volume** on the Storage page
([seeing and deleting your volume](https://docs.nationalcompute.com/kubernetes.md#seeing-and-deleting-your-volume)).
A request carrying a `Bearer` token is refused with
`403 session-required`, whatever the token's scope.

A limit price is reversible. Data destruction is not. An irreversible delete
keeps a human in the loop: a member of your organization, signed in,
typing the volume's storage name back.

The console's delete rides the same route, with the same body
(`{"confirm": "<volume name>"}`) and the same refusals:

- `confirm` must be the volume's `name`, the storage name shown under
  its label. The display label is refused. Anything else is
  `confirmation-mismatch` and nothing is destroyed.
- Only a **preserved** volume can be deleted here. An attached one is
  `volume-attached`: delete the cluster first (the volume detaches with
  it and is preserved by default), or opt to delete the storage with
  the cluster.
- A preserved volume a node still holds is `volume-held`; retry in a
  few minutes.
- The delete is accepted asynchronously: the response is
  `{destroyed, operation}` and the volume leaves the list once it is
  gone. There is no undo.

## Tokens

Token minting is the normal org token flow ([authentication](https://docs.nationalcompute.com/authentication.md));
a token can be bound to a Kubernetes cluster. A k8s-bound token is
refused on the VM surfaces and a VM-bound one is refused here.

---

# Market analytics

Source: https://docs.nationalcompute.com/api/k8s-market/ (this markdown: https://docs.nationalcompute.com/api/k8s-market.md)

Five read only routes publish what the console's Burst Capacity page
shows for a Kubernetes cluster: the price it takes to win nodes, what
the nodes you hold are assessed, and where your demand sits.

| Route | What it serves |
|---|---|
| `GET /api/k8s/market/ladder` | the price to win ladder: per job size in nodes, the minimum bid that won that many nodes at each clearing, and the newest tick's quote |
| `GET /api/k8s/market/spend` | the rate assessed on each node you hold this hour, and the sum |
| `GET /api/k8s/market/paid` | the mean rate your nodes were assessed per bucket, and how many nodes |
| `GET /api/k8s/market/bidhistory` | the limit price in force over time |
| `GET /api/k8s/market/demand` | your cluster's GPUs by band: `bidding`, `provisioning`, `scheduled`, `reserved` |

Contract: [`GET /api/k8s/openapi.json`](https://nationalcompute.com/api/k8s/openapi.json).

All five take an org API token and resolve the cluster the way the
[limit price read](https://docs.nationalcompute.com/api/k8s-bid.md) does: a token bound to a Kubernetes cluster reads its
own; an org wide token resolves when the organization holds one
Kubernetes cluster and otherwise passes `?cluster=<name>` or gets a
`409 ambiguous-cluster`. A token bound to a VM cluster is refused with
`422 bad-request`. A cluster outside your organization is a `404`, the
same as one that does not exist.

Money is **USD per node-hour** on every route, like the
[price feed](https://docs.nationalcompute.com/api/market-feed.md). Divide by the response's own
`gpus_per_node` to compare with a per GPU-hour limit price.

The windowed routes (`ladder`, `paid`, `bidhistory`, `demand`) take
`?hours=` (default 24, clamped to 1 to 8760 hours) and return `t0`
(epoch seconds of the first bucket) and `step` (bucket width in
seconds, growing with the window: `max(300, span / 400)`). The demand
window caps at 720 hours, the 30 days its store keeps. Within one
window the series share the grid: a paid bucket and a demand bucket at
the same offset from `t0` describe the same interval, and demand
carries one extra bucket at the window's end.

!!! note "Feeds arm per market"
    `enabled: false` means the feed is not armed for your cluster's
    market and the data comes empty, not an error. The limit price
    history has no market gate: the declaration is your organization's
    own record.

## The price to win ladder

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  "https://nationalcompute.com/api/k8s/market/ladder?hours=24"
```

The ladder is the outsider's quote: for each job size `k` in whole
nodes, the minimum bid a fresh cluster with no standing position needed
to win `k` nodes at that clearing. It is the honest entry cost for a job
of `k` nodes, and it is quoted per node-hour as the whole job's total
divided by `k`.

- `sizes` lists the sizes that quoted at least once in the window;
  `price` holds one series per size, keyed by the size as a string.
- `rungs` lists every size the ladder quotes, whatever the window.
- `now` is the newest tick's quote per rung and `now_ts` that tick.
  `now[k]` is `null` where no bid at any price wins that size right
  now. Two causes: every node the job would need sits inside a
  [protection window](https://docs.nationalcompute.com/market.md#minimum-duration-protection), or the
  island has fewer nodes than the size asks for. The console shows that
  rung as **Preemption Protected**. `now` is `null` as a whole when no
  tick landed in the window.
- A `null` in a `price` series means no tick quoted that size in that
  bucket. A quote is never held forward into a later bucket.

The [MCP server](https://docs.nationalcompute.com/api/mcp.md)'s `market_read source=ladder` returns this
body unchanged.

## What your nodes are assessed

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/k8s/market/spend
```

`nodes` maps each node you hold to the rate the newest auction assessed
it, `total_usd_hr` is their sum, and `site_rates` is the anonymous range
assessed across the whole market that tick as `{lo, hi}` (`null` when
nothing cleared). These are the rates the meter bills for the current
hour ([billing](https://docs.nationalcompute.com/billing.md)).

`GET /api/k8s/market/paid` is the same figure over time: `price[]` is
the mean rate across your nodes per bucket and `nodes[]` the mean number
of nodes billed, both `null` where you held nothing.

## Your limit price over time { #your-bid-over-time }

`GET /api/k8s/market/bidhistory` returns `price[]`, the limit price in force
at each bucket instant, per node-hour (the limit price PUT's per GPU-hour
figure times `gpus_per_node`). It is `null` before the first declaration
and while the limit price is withdrawn.

## Your demand by band

`GET /api/k8s/market/demand` returns `bands`, four series of GPUs per
bucket: `bidding` (requests waiting on the market), `provisioning`
(granted, nodes joining), `scheduled` (running), and `reserved` (the
cluster's live [Base Load](https://docs.nationalcompute.com/market.md#base-load-capacity) block,
reported beside the three bands and never subtracted from them).

---

# Cluster facts and the Base Load book

Source: https://docs.nationalcompute.com/api/k8s-cluster/ (this markdown: https://docs.nationalcompute.com/api/k8s-cluster.md)

Three read only routes answer what an agent asks right after
[`/api/whoami`](https://docs.nationalcompute.com/authentication.md#discovering-what-a-token-can-address):
what this Kubernetes cluster is, how to connect to it, and what Base
Load it can hold.

| Route | What it serves |
|---|---|
| `GET /api/k8s/cluster` | the cluster's facts: GPU shape, market posture, storage, endpoints |
| `GET /api/k8s/cluster/kubeconfig` | the cluster's kubeconfig, by token ([below](#the-kubeconfig)) |
| `GET /api/k8s/reservations` | the Base Load book: rate sheet, availability by size, your blocks |

Contract: [`GET /api/k8s/openapi.json`](https://nationalcompute.com/api/k8s/openapi.json).

Every route takes an org API token and resolves the cluster the way the
[limit price read](https://docs.nationalcompute.com/api/k8s-bid.md) does: a token bound to a Kubernetes cluster reads its
own; an org wide token resolves when the organization holds one
Kubernetes cluster and otherwise passes `?cluster=<name>` or gets a
`409 ambiguous-cluster`. A token bound to a VM cluster is refused with
`422 bad-request`. A cluster outside your organization is a `404`, the
same as one that does not exist.

## The cluster's facts

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/k8s/cluster
```

| Field | Meaning |
|---|---|
| `cluster`, `kind`, `status` | the name, `k8s`, and the platform's status word (`ready` for a usable cluster) |
| `market` | `true` when the cluster trades on the market; `false` for a fixed cluster the platform sizes |
| `nodes`, `gpus` | the nodes and GPUs the cluster holds right now |
| `gpus_per_node` | the node to GPU conversion every bid uses |
| `cpu_worker` | `true` when the cluster carries a [CPU worker](https://docs.nationalcompute.com/kubernetes.md#the-cluster-cpu-worker) |
| `shared_volume` | `true` when a [shared volume](https://docs.nationalcompute.com/kubernetes.md#the-shared-volume) is bound to the cluster |
| `nccl_env` | the environment the platform injects for high performance networking on this cluster, as `{VAR: value}`; `null` when there is none |
| `gpu_vendor` | which vendor's runtime the nodes carry, so you pick the matching manifests |
| `gpu_model` | the GPU class the cluster's island sells, what a limit price buys. `null` while the platform has not set one |
| `base_load_offered` | `true` when [Base Load](https://docs.nationalcompute.com/market.md#base-load-capacity) is sold for clusters where this one runs |
| `kubeconfig_url` | the kubeconfig download on this API, `GET /api/k8s/cluster/kubeconfig?cluster=<name>`, with your token ([below](#the-kubeconfig)). Signing in is the device code flow the kubeconfig triggers |
| `api_url`, `ingress_url`, `ingress_port` | the API server and the workload ingress endpoint, or `null` where the platform does not derive them for this cluster. The kubeconfig carries the API server regardless |
| `island_stale` | `null` while the platform hears the island's heartbeat; `{since}` once it has lost the island, and every node read then shows the last snapshot ([island staleness](https://docs.nationalcompute.com/api/k8s-cluster-pages.md#island-staleness)) |
| `feed_error`, `feed_age_s` | the last error of the platform's pull of the node health feed (`null` when the last pull landed), and the data age of what that feed serves in seconds: the time since the newest landed pull plus the platform's cache age at that pull (`null` when no pull has ever landed) |

A `404` right after a cluster is created means its record has not
landed yet. Retry.

## The kubeconfig

```sh
curl -fsSL -H "Authorization: Bearer $NC_TOKEN" \
  "https://nationalcompute.com/api/k8s/cluster/kubeconfig?cluster=<name>" \
  -o ~/.kube/<name>.yaml
```

`GET /api/k8s/cluster/kubeconfig` answers the cluster's kubeconfig as
`application/yaml`: a public client and the cluster's certificate, no
secret inside. The first `kubectl` call prints the device code sign in
link a person approves. The cluster resolves like every read here, from
the token's binding or `?cluster=`. Another organization's cluster can
never answer, whatever its name. When your own organization carries one
name on two clusters, the read answers `409 ambiguous-cluster` and the
platform team resolves it.

| Status | Meaning |
|---|---|
| `409 cluster-not-ready` | the cluster is still building |
| `503 station-unavailable` | the island cannot publish the file right now. Retry |

On the MCP server the same read is a [`grid_api` path](https://docs.nationalcompute.com/api/mcp.md#grid_api-paths)
answering JSON with the YAML inside.

The public onboarding path at `access.nationalcompute.com` resolves a
bare name without a token. When more than one cluster carries the name
it answers `409 ambiguous-cluster` and points here.

## The Base Load book

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/k8s/reservations
```

The response is what the console's Burst Capacity page shows under Base
Load, read only:

- `offered` is `false` when no rates are posted where the cluster runs.
  `checkout_open` is `false` while self serve buying is closed there.
- `rates` is the posted sheet, USD per GPU-hour keyed by size in GPUs
  and then by term in days. `days` lists the posted terms.
- `sizes[]` is the size matrix: each entry's `chips`, `nodes`, whether
  it is `available` now, else `sold_out_until` as an epoch second
  (`null` when no live block's end would free it).
- `reservations[]` is the cluster's own book: blocks that are live, and
  blocks that ended or were cancelled within 30 days, each with `id`
  (the block's handle), `nodes`, `chips`, `rate_usd_per_gpu_hr`,
  `starts_at` and `ends_at` (epoch seconds), and `status` (`scheduled`,
  `active`, `ended`, `cancelled`).
- `deposit_cap_ucredits` is the deposit rule's cap in microcredits
  (1,000,000 = $1): a purchase needs the smaller of the block's price
  and this figure on the balance.

Buying is a console action ([Base Load](https://docs.nationalcompute.com/market.md#base-load-capacity)).
This API has no purchase route, and neither does the MCP server, where
the book is a [`grid_api` path](https://docs.nationalcompute.com/api/mcp.md#grid_api-paths).

---

# Cluster pages by token

Source: https://docs.nationalcompute.com/api/k8s-cluster-pages/ (this markdown: https://docs.nationalcompute.com/api/k8s-cluster-pages.md)

Thirteen read only routes serve what the console's Workloads, job,
Cluster Overview, machine, and Storage pages show for a Kubernetes
cluster, so an agent reads the platform's own view instead of rebuilding
it from kubectl.

| Route | What it serves |
|---|---|
| `GET /api/k8s/workloads` | the Workloads page: running, pending and recent jobs with their state and waiting verdict, services with their load balancer endpoints and verdict flags, GPU totals, the market block with the price to win |
| `GET /api/k8s/workloads/runs?before=` | Run History, newest first |
| `GET /api/k8s/workloads/history?kind=&ns=&wname=` | one workload's scheduling history |
| `GET /api/k8s/workloads/cost?kind=&ns=&wname=` | one workload's market cost |
| `GET /api/k8s/workloads/metrics?kind=&ns=&wname=&r=` | one workload's utilization and traffic |
| `GET /api/k8s/jobs/{ns}~{pod}` | one pod's page: detail, GPU metrics, history, load test results |
| `GET /api/k8s/jobs/{ns}~{pod}/cost` | one pod's cost panel |
| `GET /api/k8s/jobs/{ns}~{pod}/network?r=` | one pod's internet traffic |
| `GET /api/k8s/nodes` | the Cluster Overview board: every node, its GPUs, health, pods, and the platform's `verdict` word per node ([below](#node-verdicts)) |
| `GET /api/k8s/nodes/{node}` | one node's machine page, with its `verdict` |
| `GET /api/k8s/nodes/{node}/network?r=` | one node's network volume |
| `GET /api/k8s/capacity/history?hours=` | the Cluster Overview charts: GPUs by health state over time, demand against capacity |
| `GET /api/k8s/storage` | the Storage page: NVMe and HBM per node, shared home and shared volume fullness, for every cluster in your organization |

Contract: [`GET /api/k8s/openapi.json`](https://nationalcompute.com/api/k8s/openapi.json).
The workload and job routes are also [`grid_api` paths](https://docs.nationalcompute.com/api/mcp.md#grid_api-paths)
on the MCP server, same bodies and refusals; the node and storage
routes are not.

All thirteen take an org API token and resolve the cluster the way the
[limit price read](https://docs.nationalcompute.com/api/k8s-bid.md) does: a token bound to a Kubernetes cluster reads its
own; an org wide token resolves when the organization holds one
Kubernetes cluster and otherwise passes `?cluster=<name>` or gets a
`409 ambiguous-cluster`. A token bound to a VM cluster is refused with
`422 bad-request`. A cluster outside your organization is a `404`, the
same as one that does not exist.

## What the platform knows that kubectl does not

These reads carry the platform's own judgement about your cluster:
the market verdict on each waiting request, the price and protection
state of each running gang, node health as the platform's checkers
see it, cost attributed to a workload or a pod, and history that
outlives the cluster's own events. The raw objects stay in kubectl.

## Refusals

| Code | Status | Meaning |
|---|---|---|
| `cluster-not-ready` | 409 | the cluster is still being built or is not ready; retry |
| `not-found` | 404 | no such cluster, node, or pod in your organization |
| `bad-request` | 422 | a malformed range (`r` is `1h`, `6h`, `24h`, or `7d`), workload identity, pod id, `before` cursor, or a non numeric `hours` |
| `station-unavailable` | 503 | the cluster's island did not answer, or its reads are served in a way this token cannot reach yet |

## Reading the Workloads page

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/k8s/workloads
```

`running`, `pending` and `recent` list the jobs with their GPUs, nodes,
state, reason and events; `services` carries each Service with its
load balancer endpoint; `market` is the page's market block (the limit price,
each node's price, protection windows, the price to win, the reserved
block); `mirror` gives the age of each data kind, so a dark island's
last snapshot ages honestly instead of reading fresh.

The waiting verdicts an agent branches on ride this read. When the
market has a verdict for a waiting pod, `pending[].reason` is one of
these eight strings:

| `reason` | Meaning | What to do |
|---|---|---|
| `outbid` | lost on price; `win_price_per_gpu_hour` carries the price that wins when the market quoted one | raise the limit price, or wait |
| `waiting for available supply` | protected nodes hold the supply the gang needs | wait; windows lapse on their own |
| `cluster too small for this request` | the site holds fewer nodes than the gang needs | a smaller gang |
| `waiting for capacity; none available at this site right now` | the site is out of stock for the class; a higher limit price cannot help | wait; the platform retries |
| `waiting for reserved GPUs` | the job is smaller than one node and waits for GPUs of your base load block to free; it never bids on the market; a higher limit price cannot help | wait |
| `reserved GPUs too few for this request` | the job's pods are each smaller than one node and together ask for more GPUs than one node holds; a job with pods that small runs on one reserved node only; it never bids | whole node pods to run across nodes; fewer pods to fit one node |
| `bid too low to ever clear` | no bid this low can ever win a node; `bid_too_low` reads `true` on the row and its Service | check the price to win and raise the limit price |
| `node provisioning` | a grant is inbound | nothing |

The two reserved verdicts appear only on clusters with reservation
packing enabled. There a job smaller than one node runs on your
[base load block](https://docs.nationalcompute.com/market.md#base-load-capacity). It never bids.

Any other value is scheduler state, never a market verdict: the
scheduler's own reason for a pod the market is not judging
(`Unschedulable`, `ContainerCreating`, `ImagePullBackOff`, a static
cluster's words), the request's condition message verbatim, or the
empty string before the first verdict. Branch on the two Service
booleans and the eight verdicts. Treat everything else as scheduler
state and watch the pod with kubectl.

`services[].capacity_unavailable` and `services[].bid_too_low` carry
the two verdicts as booleans per Service. `services[].requests[].reason`
is the market's full sentence for each request, naming your own bid's
numbers. Each request entry, under a job row or a Service, carries
`gpus` (the request's GPU total) and `sub_node`. `sub_node` reads
`true` when each pod of the request asks for fewer GPUs than one node.
On a cluster with reservation packing enabled such a request runs only
on your reservation.
`market.price_to_win` maps a job size in nodes to the price per GPU
hour that wins it right now, and `market.protection` lists the open
protection windows per node. `market.reserved` is your base load
block with its nodes and the seat each request holds on them. Its
`free_gpus` sums the GPUs still open for sub node work across the
block's nodes this tick. Nodes a whole node request holds are
excluded. `free_gpus` reads `null` until the platform reports it. A
request smaller than one node starts when one reserved node has its
`gpus` free and no whole node job is ahead of it.

## Verdicts in the cluster

The market posts the same verdicts inside the cluster, as Events on the
Job (one per change of reason) and as the reason of the request's
`Provisioned` condition. The MCP `job_watch` tool reads them and maps
each to the sentence above. One table covers every spelling:

| `pending[].reason` on the wire | Event reason on the Job | `job_watch` field |
|---|---|---|
| `outbid` | `Outbid` | `market_events[].verdict` reads `outbid` |
| `waiting for available supply` | `PendingSupply` | `market_events[].verdict` reads `waiting for available supply` |
| `cluster too small for this request` | `NoSupply` | `market_events[].verdict` reads `cluster too small for this request` |
| `waiting for capacity; none available at this site right now` | `CapacityUnavailable`, a Warning Event on the pod; the request itself reads `PendingSupply`, `Outbid` or `Pending` | `market_events[].verdict` reads `waiting for capacity; none available at this site right now` |
| `bid too low to ever clear` | `BidTooLow` | `market_events[].verdict` reads `bid too low to ever clear`; `standing_bid.bid_too_low` |
| `node provisioning` | `NodeGranted`, one per granted node | `node_granted_events`; `market_events[].verdict` reads `node provisioning` |
| `waiting for reserved GPUs` | `ReservedBusy` | `market_events[].verdict` reads `waiting for reserved GPUs` |
| `reserved GPUs too few for this request` | `ReservedTooSmall` | `market_events[].verdict` reads `reserved GPUs too few for this request` |
| the empty string before the first verdict | `Pending` | `market_events[].verdict` reads `null` |
| `billing_hold` (an object; no reason sentence) | `BalanceTooLow` | `market_events[].verdict` reads `null`; `capacity_read` carries `billing_hold` |
| `preempt_until` on the running row | `NodePreempting` on the node and its pods; `PreemptionRescinded` on the node when the notice is withdrawn | `preempting.pods[].preempt_at`, `preempting.earliest` |
| the `requeued` state and `requeued_at` | the Kueue Workload's `Evicted` and `Requeued` conditions and `status.requeueState.count` | `workloads[].requeue_count`, `workloads[].evicted`, `workloads[].requeued` |
| the `requeued` state; `requests[].phase` reads `revoked` and its message names the departed nodes | `MarketRevoked`, a Warning Event on the Job when every granted node left the cluster | `market_events[].verdict` reads `requeued` |

`PreemptionNotice=True` (reason `MarketPreemption`) is a condition on
the ProvisioningRequest. It is not an Event, so `job_watch` does not
read it; `cluster_read` does. The two reserved verdicts appear only on
clusters with reservation packing enabled.

`job_watch` answers `market_events` from one page of the namespace's
Events, scoped to the watched pods and their owning jobs (name the job
in the selector while it has no pods), at most 30, newest last;
`node_granted_events` comes off the same page. `market_events_truncated`
reads `true` when the page overflowed and precise per reason reads
filled it; a refused Events read answers `null` with
`market_events_note`. `preempting` lists every watched pod carrying the
`marketplace.nationalcompute.com/preempt-at` annotation with the
earliest deadline. `workloads` lists the Kueue Workloads behind the
watched pods, narrowed by the pods' Job uid or queue label before the
fetch, at most 20; when none names the watched job as its owner every
fetched Workload answers with `workloads_note`; it reads `null` with
`workloads_note` when the cluster cannot answer.

`billing_hold` is your organization's billing hold state, or `null`,
the same object the [limit price read](https://docs.nationalcompute.com/api/k8s-bid.md#setting-the-bid)
carries. While `state` reads `hold` the market places nothing new for
the cluster. At `stage` `notice` new placements stop. At `enforce`
capacity is reclaimed. Pending rows then carry no market verdict until
the hold lifts. Read the hold before you treat an empty `reason` as
scheduler state.

Run History pages by `?before=<the previous page's last ended stamp>`.
A workload's history, cost and metrics take its identity as `kind`,
`ns` and `wname`, the values the Workloads page shows.

## Reading a pod, a node, the storage

A pod's page takes `{namespace}~{pod}` as its id. A deleted pod still
answers from the platform's record; a pod unknown to both the cluster
and the record is a `404`.

The board and the machine page name nodes the way the console does.
`?source=mirror` on either answers from the platform's mirror without
dialing the island, the console's first paint; `island_stale` marks a
dark island ([below](#island-staleness)).

The Storage page is organization wide: one entry per cluster with the
NVMe and HBM readings the console plots (`nvme` and `hbm`, each with
its rows and averages), `pending` where the island's storage telemetry
has not landed, `scanned` (whether a reading exists) with `age` (its
age in seconds), the cluster's `quota`, `used`, `members`, `node_count`,
whether the cluster trades on the market, and `shared`: the shared
volumes bound to the cluster with `provisioned`, `used`, `mountpoint`
and each node's `mounted` state (`true`, `false`, or `null` when the
node has not been swept). Sizes are bytes from the platform's scan.
`null` means not scanned. It is never 0.

### Fullness bands

The Storage page paints every fullness reading in one of three bands.
The body carries the same verdict, so an agent reacts to the band the
console shows instead of picking its own thresholds.

| Field | Reading | Where |
|---|---|---|
| `thresholds` | `{warn_pct: 90, crit_pct: 99}`, the constants | top level, once |
| `nvme.rows[].level` | node scratch: `used` of `total` | each NVMe row |
| `hbm.rows[].level` | GPU memory: `used` of `total` | each HBM row |
| `shared[].level` | the shared volume: `used` of `provisioned` | each bound volume |
| `quota_level` | the shared home: `used` of `quota` | each cluster |

`level` is `crit` at or above `crit_pct`, `warn` above `warn_pct`, `ok`
below, and `null` when the reading is unknown (an unreachable node, a
volume not yet scanned, a cluster with no quota set). A `null` band is
never `ok`. The bands are instantaneous readings of the latest scan.
[`metrics_read`](https://docs.nationalcompute.com/api/metrics.md) `action=flags` judges GPU memory over a
window with its own rule (`memory_pressure`). The two can disagree by
design.

## Capacity history

`GET /api/k8s/capacity/history?hours=24` serves the two charts on the
Cluster Overview: GPUs by health state over time, and demand against
capacity. `hours` defaults to 24 and clamps to 0.5 .. 8760 (currently
one year).

```json
{"t0": 1759100000, "step": 216, "source": "pg",
 "states": [{"label": "healthy", "data": [16, 16, 8]},
            {"label": "unschedulable", "data": [0, 0, 8]}],
 "total": [16, 16, 16],
 "scheduled": [8, 8, 8], "demand": [0, 8, 8], "available": [8, 8, 0]}
```

`t0` is the first bucket in epoch seconds and `step` the bucket width
in seconds; every array holds one value per bucket. `states` is one
series per health state the cluster's nodes were in during the window,
GPU weighted (a node counts its GPUs). A state absent from the window
is absent from the list. There is no zero series for it. `total`
stacks every state per bucket.

`states[].label` is the platform's health vocabulary, seven words:

| `label` | Meaning | Schedulable |
|---|---|---|
| `healthy` | nothing known bad | yes |
| `degraded` | hardware checks failing while the scheduler still places work | yes |
| `unschedulable` | the scheduler places nothing: NotReady, a platform cordon, a node in lifecycle error | no |
| `cordoned` | parked by your own cluster admin; Ready and fault free; still billed | no |
| `moving` | an operation owns the node; recorded without a cluster, so it rarely appears on a cluster's own series | no |
| `unreachable` | the control plane lost the node; outranks `unschedulable` | no |
| `unknown` | the platform cannot say: the cluster snapshot was missing or the node was absent from it | no |

Schedulable capacity is `healthy` plus `degraded`. The console's GPU
Health chart merges those two into its healthy band and draws
`unschedulable` as unhealthy; `cordoned`, `unreachable`, `unknown` and
`moving` get their own bands only when they occurred in the window.

`scheduled` (GPUs held by running work), `demand` (GPUs asked for by
waiting work) and `available` (schedulable capacity minus `scheduled`,
floored at 0) are the demand overlay. A `null` bucket inside them means
no sampler beat was in reach: unknown, never 0. The three keys are
absent when the demand record is unreadable; the health series still
answers. `source` is `pg` for the record store and `live` for a single
current point when the store is off; `error` then names the
degradation.

A cluster whose reads the platform serves centrally answers `503
station-unavailable` here, the same as the board and the machine page.

## Node verdicts

Every row of the board and the `node` object of the machine page carry
`verdict`, the platform's own health word for the node. It is the word
the console tile shows, derived on the server from the same inputs the
payload carries (`k8snodes`, the lifecycle `status`, the `unreachable`
mark). The object is
`{sev, word, detail, healthy, schedulable, lost, observed}`. `sev` is
`ok`, `warn` or `err`. `detail` is the hover text, `""` when there is
none. `healthy` and `schedulable` are the board's section split:
healthy nodes land in Healthy, unhealthy nodes that still schedule in
"Unhealthy, schedulable", the rest in Unschedulable, and `lost` nodes
in Unreachable. `schedulable` is Kubernetes truth (Ready and not
cordoned) whatever the word says. `observed` is `false` when the
scheduler feed behind the verdict did not answer on this read
([below](#island-staleness)). `verdict` is `null` only when the
platform could not derive one; the board still answers.

| `word` | `sev` | Meaning | What to do |
|---|---|---|---|
| `""` | `ok` | Ready, schedulable, no platform marker | nothing |
| `unobserved` | `warn` | the scheduler feed did not answer on this read (the island is stale, the feed is absent or erroring, or it carries no rows for the cluster); nothing judged the node, and `detail` names the cause | read `island_stale` and `feed_error`; retry |
| `unreachable` | `err` | the control plane lost the node: the kubelet has been silent for more than 300 seconds, `lost` is `true`, and `detail` carries the last known state | wait; the platform already sees it |
| `NotReady` | `err` | the node is not Ready, within the 300 second threshold | wait; a kubelet restart clears it |
| `Node Fix In Progress` | `warn` | our health checks took the node out of service: cordoned with a platform marker; billing for it stopped; a replacement is being provisioned ([mechanism](https://docs.nationalcompute.com/kubernetes.md#when-we-take-a-node-out-of-service)) | nothing; the request row reads Replacement Pending until the replacement joins |
| `unhealthy` | `err` | the health check daemon reports failing checks, named in `detail`; the node still schedules unless it is also cordoned | move work off the node, or cordon it |
| `tainted` | `warn` | schedulable, with a taint the platform's health checker placed | as `unhealthy` |
| `cordoned` | `warn` | cordoned without a platform marker: your own cordon; `detail` lists the taints | uncordon when you are done |
| `missing` | `err` | the cluster answered and this node is absent from its node list | check `kubectl get nodes` |
| `creating`, `joining`, `error` | `warn`; `err` for `error` | the platform's lifecycle status while the node is not ready. Lifecycle is never health | wait |

A node mid move leaves the list: the board drops it and its machine
read answers `404`, so no `moving` word rides the wire. A node with a
scheduler row is judged from that row. A node with no row on a board
that lists other nodes is `missing`. A node with no row on a feed that
did not answer is `unobserved`, unless the platform's own record marks
it `unreachable` or its lifecycle status is not `ready`; those words
stand, with `observed: false`.

## Island staleness

Two fields tell a dark island from a broken node. `island_stale` on
the board, the machine page and [`GET /api/k8s/cluster`](https://docs.nationalcompute.com/api/k8s-cluster.md#the-clusters-facts)
is `null` while the platform hears the island's heartbeat and `{since}`
once it has lost the island. Every verdict then reads
`observed: false`: a node the dark feed would have read as healthy
reads `unobserved`, a node with a word of its own keeps it. `feed_error`
on `GET /api/k8s/cluster` is the last error of the platform's pull of
the node health feed, `null` when the last pull landed. `feed_age_s` is
the data age of what that feed serves, in seconds: the time since the
newest landed pull plus the platform's cache age at that pull, `null`
when no pull has ever landed. A failing pull leaves the age growing;
it never resets. `k8snodes.error` on the board names a node feed the
island could not answer on this read, and its rows then read
`observed: false`. Read these before acting on a red verdict.

On the MCP server the same two bodies are `grid_api get /api/k8s/nodes`
(the board) and `grid_api get /api/k8s/nodes/<node>` (one machine, the
REST spelling; the literal `/api/k8s/nodes/{node}` with `node` in
`params` is the same call), with `params` `cluster` and `source`
(`mirror` for the platform's mirror, no island dial). A malformed node
name is `422 bad-request`, an unknown one `404 not-found`.

---

# Price feed

Source: https://docs.nationalcompute.com/api/market-feed/ (this markdown: https://docs.nationalcompute.com/api/market-feed.md)

Two read-only endpoints publish what capacity clears at in your market,
so you can tune your ceiling against the going rate instead of blind
increments.

| Route | What it serves |
|---|---|
| `GET /api/market/ticks` | one row per auction tick (~30s), newest first, 7-day window — the live feed |
| `GET /api/market/history` | bucketed series, up to 365 days — the trend feed |

The machine-readable contract is
[`GET /api/market/openapi.json`](https://nationalcompute.com/api/market/openapi.json)
— the feed's own document, since every cluster kind reads the same feed.

Both take an org API token. "Your market" is the site your cluster trades
in — the feed is **uniform per market**: every caller in it gets the
same payload, and nothing caller-specific rides it (your own position
lives on the [capacity read](https://docs.nationalcompute.com/api/vm-capacity.md) or the
[k8s limit price](https://docs.nationalcompute.com/api/k8s-bid.md)).

**Picking the market.** A token bound to a cluster reads that cluster's
market with no extra input. An org-wide token must pass
`?cluster=<one of your clusters>` (either kind) on **every** call — a
missing selector is a `422`. This is always explicit because your
clusters can sit on different sites, and each site is its own market.

!!! note "Feeds arm per market"
    Both responses carry an `enabled` flag. `false` means the feed is
    not yet armed for your cluster's market and the data comes empty —
    not an error.

## The price is an index, not a quote

Published prices are what winners were **assessed**, never bids.
Pricing is discriminatory per cluster, so there is no single per-tick
clearing price to publish — the feed publishes statistics:

- `price` — the volume-weighted mean clearing rate, **USD per
  node-hour**. Divide by the response's own `gpus_per_node` to compare
  against your per-GPU-hour ceiling.
- `price_min` / `price_max` (ticks only) — the spread of rates assessed
  that tick.
- `price` is `null` when nothing cleared; `0.0` is a real rate (an
  uncontended market can clear at zero).
- `gpu_model` and `gpu_vendor`: the GPU class these prices are for and
  its maker. `gpu_model` is `null` while the platform has not set one.

An agent bidding exactly the published mean has no clearing guarantee.

## Tailing the live feed

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  "https://nationalcompute.com/api/market/ticks?limit=50"
```

Rows carry `id` (the keyset cursor) and `at`. Two cursors, mutually
exclusive:

- `after_id=<newest id you hold>` — tail live: you usually get 0–1 rows.
- `before_id=<oldest id you hold>` — page backward; `next_before_id` in
  the response is the next page's cursor (`null` = exhausted).

Passing both is a `422`. The [MCP server](https://docs.nationalcompute.com/api/mcp.md)'s `market_read
source=ticks` takes the same two cursors and `limit`, and its `grid_api
get /api/market/ticks` passes them through unchanged.

## The trend feed

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  "https://nationalcompute.com/api/market/history?hours=168"
```

Returns `t0` (epoch seconds of the first bucket), `step` (bucket width
in seconds — it grows with the window), and `price[]` (volume-weighted
mean per bucket, `null` where nothing cleared).

## The tuning loop

1. Read the going rate here.
2. Read your position on `GET /api/vm/capacity` (declared ceiling,
   pending, allocations).
3. Adjust the ceiling on the capacity PUT (or the
   [k8s limit price](https://docs.nationalcompute.com/api/k8s-bid.md)). A ceiling under the going rate leaves
   capacity pending and can shed nodes you hold.
4. Check what you actually pay on
   [`GET /api/billing/balance`](https://docs.nationalcompute.com/api/billing.md#balance-and-burn-rate): the
   hourly rate the meter last assessed, by kind, cluster and unit price.

---

# Billing records

Source: https://docs.nationalcompute.com/api/billing/ (this markdown: https://docs.nationalcompute.com/api/billing.md)

Your organization's ledger by org API token: every charge down to the
node and the price window, daily totals in your calendar, the balance
now with the hourly burn rate and the balance over time, the balance
card with the card on file and the auto reload setting, spend
attributed per workload, and the unit prices in force. Five writes
start the console's billing actions for a person: three hand back a
hosted page URL, one tightens the spend limit, one configures auto
reload.

Contract: [`GET /api/billing/openapi.json`](https://nationalcompute.com/api/billing/openapi.json).

| Route | What it serves |
|---|---|
| `GET /api/billing/transactions` | the itemized ledger: every charge and deposit, newest first, one entry per resource per price window |
| `GET /api/billing/daily` | one row per calendar day: charges, deposits, the resources charged and their unit-hours |
| `GET /api/billing/summary` | balance, lifetime totals, this month's charges, spend limit, hold state, the card on file, the auto reload setting |
| `GET /api/billing/balance` | the balance now, the hourly burn rate the meter last assessed, and what it is made of by kind, cluster and unit price |
| `GET /api/billing/balance/history` | the balance as a series over a window: `start`, `step_s`, one value per bucket |
| `GET /api/billing/usage/workloads` | spend attributed per workload over a preset range |
| `GET /api/billing/rates` | the unit prices in force for your organization |
| `POST /api/billing/checkout` | `{usd}`: a hosted payment page URL for a person to complete; the deposit lands when they pay |
| `POST /api/billing/setup` | a hosted page URL that saves a card and charges nothing |
| `POST /api/billing/portal` | the payment portal URL: receipts and card changes |
| `PUT /api/billing/limit` | `{usd}`: set the monthly spend limit where none exists, or lower it |
| `PUT /api/billing/reload` | `{enabled, threshold_usd, amount_usd}`: the auto reload setting, a standing charge on the saved card whenever the balance runs low; a token may only disable it, lower the amount or raise the threshold |

Any of your organization's tokens reads these routes, cluster-bound or
org-wide. The organization is the token's own and nothing identifies it
on the wire. There is no cluster selector: the ledger is organization
wide, and each charge entry names its cluster. No call on this wire
moves money by itself: the three hosted page routes return a URL that
a person opens in a browser, and no card detail ever rides the token.
Auto reload is the one standing authorization. Once enabled it charges
the saved card whenever the balance falls under the threshold, so a
token may only tighten it: enabling it stays a console action.

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  "https://nationalcompute.com/api/billing/daily?limit=31"
```

## Units

Amounts are integers in **microcredits**: fields ending in `_ucredits`,
where 1,000,000 microcredits = 1 credit = $1. Stamps are **RFC 3339
UTC** to the second (`2026-09-10T14:00:00Z`). Where a route aligns to
your calendar it takes `tz_offset_min`, the JavaScript
`getTimezoneOffset()` value (minutes UTC is *ahead* of local; `0` is UTC
days; a value outside ±900 is refused with `422 bad-request`).

## The ledger

`GET /api/billing/transactions` returns one row per ledger transaction,
newest first:

```json
{
  "transactions": [
    {
      "type": "charge",
      "start": "2026-09-10T14:00:00Z",
      "end": "2026-09-10T14:15:00Z",
      "amount_ucredits": …,
      "source": "metered",
      "note": null,
      "balance_after_ucredits": 3998500000,
      "charges": [
        {
          "kind": "burst",
          "cluster": "acme-train",
          "resource_id": "…",
          "resource_type": "GPU_…",
          "units": 8,
          "unit_hourly_price_ucredits": …,
          "start": "2026-09-10T14:00:00Z",
          "end": "2026-09-10T14:15:00Z",
          "amount_ucredits": …
        }
      ]
    }
  ],
  "next": "1757513700000.2"
}
```

| Field | Meaning |
|---|---|
| `type` | `charge` or `deposit` |
| `start`, `end` | the window the row accounts for — metering windows are about 15 minutes |
| `amount_ucredits` | always positive; `type` gives the sign |
| `source` | `metered` (the platform's compute meter), `inference` ([Marshall model usage](https://docs.nationalcompute.com/billing.md#marshall-model-usage) charged to credits), `public_research` (a meter row made only of Public Research pool lease entries), `card` (a card payment), `airdrop` (the credits a [public research organization](https://docs.nationalcompute.com/sign-in.md#public-research-organizations) receives at sign-up), `manual` (platform-authored: a wire deposit, a grant, an adjustment) |
| `note` | a platform-authored note on a manual row, else `null` |
| `charges[]` | the per-resource entries a metered row is made of — empty on deposits and manual rows; they sum to `amount_ucredits` |
| `balance_after_ucredits` | the balance after this row in ledger order: the newest row's is the balance now, and each older row's is the next newer one plus a charge or minus a deposit. Rows a filter hides still move it |
| `next` | the page cursor; pass it back as `before` for the next page. A full page always carries one (keyset paging cannot know whether more exists); a short or empty page ends the walk, `next` then `null` |

### Charge entries

Each entry is one resource over one price window, so a metered row lists
every node you held in that window, once per rate change:

| Field | Meaning |
|---|---|
| `kind` | `burst`: market capacity at the clearing rate. `base_load`: a [base load block](https://docs.nationalcompute.com/market.md#base-load-capacity) at its fixed rate. `fixed`: an attached node at the price posted for your organization. `inference`: one member's [Marshall model usage](https://docs.nationalcompute.com/billing.md#marshall-model-usage) over one metering pass. `public_research`: one whole node held under [Public Research](https://docs.nationalcompute.com/public-research.md#billing), at the flat rate in force when the lease was granted |
| `cluster` | the cluster the resource served; `null` on an `inference` or `public_research` entry |
| `resource_id` | the node's identity, `reservation:<id>` for a base load block, `public_research:<request id>` for a Public Research lease (the node rides `node`), or the shared volume's id |
| `resource_type` | the priced type — a GPU type, a CPU type, or `STORAGE_SHARED_GIB`. GPU and CPU type names carry the hardware model; read them from the payload |
| `units` | GPUs for a GPU type, nodes for a CPU type, GiB of used bytes for storage |
| `unit_hourly_price_ucredits` | the rate per unit-hour assessed for this window |
| `start`, `end` | the entry's own window inside the row's |
| `principal`, `model`, `prompt_tokens`, `completion_tokens` | `inference` entries only: the member (username), the public model id the calls named, and the token counts in the pass. `resource_type` and `resource_id` are `null` there; `units` and `unit_hourly_price_ucredits` are `0` |
| `principal_kind` | `inference` entries only: `member` when a person's credential made the calls (`principal` is then the username). `token` when a credential other than a person's made them (`principal` is then that credential's id) |
| `node`, `site`, `gpus`, `rate_ucr_per_gpu_hour`, `start_ms`, `end_ms` | `public_research` entries only: the node the lease held, the site id it ran on, the node's GPUs (the billed `units`), the flat rate per GPU-hour fixed when the lease was granted in microcredits (the same figure as `unit_hourly_price_ucredits`), and the entry's window as epoch milliseconds beside `start` and `end`. `resource_type` is the node's GPU type |
| `amount_ucredits` | `(end − start) / 1 h × units × unit_hourly_price_ucredits`, rounded to the microcredit |

### Query parameters

| Parameter | Meaning |
|---|---|
| `limit` | rows per page, 1–100; default 25 |
| `before` | the previous page's `next` (opaque) |
| `type` | `charge` or `deposit` |
| `source` | `metered`, `inference`, `public_research`, `card`, `airdrop`, or `manual` |
| `since`, `until` | a half-open window on the row's `end`: RFC 3339 (`2026-09-01T00:00:00Z`; a bare date is midnight UTC) or epoch milliseconds |

A bad value is `422 bad-request`; `detail` names the parameter.

### What the ledger read hides

- **Metered rows totalling $0** never appear — a metering window that
  cleared at $0 leaves an auditable zero row in the books, but the read
  returns neither it nor a day made only of such rows. Manual $0 rows
  stay (an adjustment can carry a `note` worth reading).
- **Who acted** never leaves: a manual row names no platform operator.
  `source` is the class, nothing finer.

## Daily totals

`GET /api/billing/daily` folds the ledger into one row per calendar day
in your zone, newest first:

```json
{
  "days": [
    {
      "date": "2026-09-10",
      "start": "2026-09-10T00:00:00Z",
      "end": "2026-09-11T00:00:00Z",
      "today": true,
      "charges_ucredits": …,
      "deposits_ucredits": 0,
      "transactions": 96,
      "resources": [
        {"kind": "burst", "resource_type": "GPU_…",
         "unit_hourly_price_ucredits": …,
         "units": 16, "unit_hours": 192.0}
      ],
      "clusters": ["acme-train"],
      "balance_end_ucredits": 3904000000
    }
  ],
  "next": "20706"
}
```

| Field | Meaning |
|---|---|
| `date` | `YYYY-MM-DD` in your zone (`tz_offset_min`) |
| `start`, `end` | the day's bounds as UTC stamps |
| `today` | the current local day — its totals are still running |
| `charges_ucredits`, `deposits_ucredits` | the day's totals |
| `transactions` | ledger rows folded into the day |
| `resources[]` | the day's charges per (`kind`, resource type, unit price), most unit-hours first: `kind` is `burst` / `base_load` / `fixed` / `public_research` as on a charge entry (`inference` charges fold into no resource line), `units` the distinct units charged that day, `unit_hours` their metered time |
| `clusters` | cluster names touched that day |
| `balance_end_ucredits` | the balance at the day's end; for today, the balance now |

`limit` counts days (1–100, default 31); `before` takes the previous
page's `next`, with the same full-page rule as the ledger. For the rows
behind a day, read the
[ledger](#the-ledger) with `since` and `until` set to its bounds.

## Balance and burn rate

`GET /api/billing/balance` is the one call a budget script polls: the
balance now, the hourly rate the meter last assessed your organization,
and what that rate is made of.

```json
{
  "balance_ucredits": 4213500000,
  "burn_ucredits_hr": …,
  "as_of": "2026-09-11T17:05:00Z",
  "billing_hold": null,
  "resources": [
    {"kind": "burst", "cluster": "acme-train", "resource_type": "GPU_…",
     "units": 16, "unit_hourly_price_ucredits": …,
     "rate_ucredits_hr": …},
    {"kind": "fixed", "cluster": "acme-train",
     "resource_type": "STORAGE_SHARED_GIB", "units": 800,
     "unit_hourly_price_ucredits": …, "rate_ucredits_hr": …}
  ],
  "clusters": [{"cluster": "acme-train", "rate_ucredits_hr": …}]
}
```

| Field | Meaning |
|---|---|
| `balance_ucredits` | the balance now; negative when charges outran deposits |
| `burn_ucredits_hr` | microcredits per hour: the sum of units × unit price over every resource still held when the newest metering window closed. A node released or a rate replaced mid-window is not part of it; the entry that replaced it is |
| `as_of` | the close of the metering window the rate was read from; `null` before the first charge |
| `billing_hold` | the hold state the [summary](#balance-and-totals) and the capacity reads carry, or `null` |
| `resources[]` | the rate by (`kind`, `cluster`, `resource_type`, `unit_hourly_price_ucredits`), largest first, with the `units` held at that price; `kind` is `burst` / `base_load` / `fixed` / `public_research` (a live [Public Research](https://docs.nationalcompute.com/public-research.md#billing) lease, `cluster` null). Jobs holding nodes at different clearing rates are separate lines, so the fold says what each limit price is costing |
| `clusters[]` | the rate per cluster, largest first |

The rate is read from the meter, not the market, so it reconciles with
the [ledger](#the-ledger) to the microcredit and covers every kind the
meter bills: burst, base load, attached nodes and storage, on VM and
Kubernetes clusters alike. The meter closes one window every 5 minutes,
about 5 minutes after the fact, so the figure is 5 to 10 minutes behind
real time and changes at most every 5 minutes; polling faster buys
nothing. The meter writes no window in which nothing was billed, so a
newest window older than 20 minutes reads as a rate of 0, with `as_of`
still naming it. For one Kubernetes cluster's live market rate, ahead
of the meter, read
[`GET /api/k8s/market/spend`](https://docs.nationalcompute.com/api/k8s-market.md#what-your-nodes-are-assessed).

### Balance over time

`GET /api/billing/balance/history` returns the balance as a series over
the last `hours` (1 to 8760, default 24; a value out of range is
`422 bad-request`). The path is the operation; `hours` rides the query
string.

```json
{"start": "2026-09-10T17:10:00Z", "step_s": 300,
 "balance_ucredits": [4300000000, 4300000000, 4289333333]}
```

`step_s` is the bucket width, `max(300, span / 400)` seconds, so a
window is at most 400 points; bucket `i` covers
`(start + i × step_s, start + (i + 1) × step_s]` and the last bucket
closes now. `balance_ucredits[i]` is the balance at the close of bucket
`i`: a balance is a step function, so buckets with no ledger event carry
the last value forward, seeded from the last event before the window,
and `0` before the first ever event. For the exact stamp of a deposit or
charge read the [ledger](#the-ledger), whose rows carry
`balance_after_ucredits`; for a day's closing balance read the
[daily totals](#daily-totals).

## Balance and totals

`GET /api/billing/summary`:

| Field | Meaning |
|---|---|
| `balance_ucredits` | the current balance; negative when charges outran deposits |
| `deposited_ucredits`, `charged_ucredits` | lifetime totals |
| `month` | `start`, `end`, and `charged_ucredits` for the current calendar month in your zone |
| `spend_limit_ucredits` | the monthly spend limit, or `null`; set on the console, or [tightened by token](#writes) |
| `spend_limit_used_pct` | `month.charged_ucredits` as a percentage of the limit, one decimal; `null` without a limit. The console's limit bar warns at 75 and turns critical at 99. The value can exceed 100 |
| `billing_hold` | the [hold state](#hold-state), or `null` — the same object the [capacity read](https://docs.nationalcompute.com/api/vm-capacity.md#reading-state) carries |
| `spend_limit_enforced` | `true` when reaching the spend limit holds your organization's capacity. `false` when the platform records the limit and does not act on it for your organization's clusters; the cap is then the instruction a person gave the agent and nothing else stops it |
| `billing_hold_enforced` | `true` when a balance at or below zero holds your organization's capacity. `false` when the platform records the balance and does not hold on it for your organization's clusters. `spend_limit_enforced` requires it |
| `base_load` | `true` when your organization is on a [base load](https://docs.nationalcompute.com/market.md#base-load-capacity) arrangement |
| `purchases` | `true` when self serve credit purchases are open to your organization on the console's [Billing page](https://docs.nationalcompute.com/billing.md#buying-credits) |
| `payment_method` | the card on file as `{brand, last4}`, or `null` when none is saved |
| `card_max_usd`, `card_window_days` | the card cap: your organization's card payments may total `card_max_usd` USD inside any rolling `card_window_days` days (currently $50,000 per 7 days). Every card payment counts, checkout and auto reload alike |
| `card_window_remaining_usd` | whole dollars a card payment can still carry right now: the cap minus the card payments inside the window, never negative. It rises as payments age out of the window |
| `card_window_blocked` | `true` while `card_window_remaining_usd` is below the smallest purchase: no card payment goes through right now |
| `min_usd`, `max_usd` | the smallest and the largest single checkout, whole dollars (platform constants; the card cap still applies) |
| `auto_reload` | the auto reload setting: `enabled`, `threshold_ucredits`, `amount_ucredits`, `last` (`status`, `reason`, `at`) for the newest attempt whether or not auto reload is enabled now (`null` before the first), and `cooldown_until`: the stamp until which a failed attempt pauses new ones, or `null` |
| `wire` | the [wire transfer rail](#wire-transfer), or `null` while `purchases` is `false` |
| `recent_deposits[]` | the last 20 deposits: `at`, `amount_ucredits`, `source` (`card`, `airdrop` or `manual`) |
| `inference` | [Marshall model usage](https://docs.nationalcompute.com/billing.md#marshall-model-usage) charged to credits, dollars with cents: `billed` (`true` when model usage is charged to the organization's credits), `free_tier_usd` (the lifetime free allowance per member; `0` when model usage is billed from the first call), `free_tier_remaining_usd` (what the caller has left of it; `null` for an org token, which has no person behind it, or while the allowance is `0`), `charged_month_to_date_usd` and `charged_lifetime_usd` (the organization's charged model usage) |
| `public_research` | [Public Research](https://docs.nationalcompute.com/public-research.md#billing) leases, as `{gpu_hours_month_to_date, charged_month_to_date_usd}`: GPU-hours this calendar month from the lease records (an open lease counts up to now; a free rate still counts) and the charged amount from the ledger's `public_research` entries, dollars with cents. Both `null` while the ledger cannot be read. The MCP `billing_read` summary view carries the same object |

These read what the console's balance card shows. The [writes](#writes)
start three of the console's actions for a person (buying credits,
saving a card, opening the payment portal), tighten the spend limit and
configure auto reload.

### Hold state

`billing_hold` is one object on the summary, the [balance read](#balance-and-burn-rate)
and the capacity reads, or `null` when nothing holds or counts down.

| Field | Meaning |
|---|---|
| `state` | `hold`: an active hold. `grace`: a countdown to a hold |
| `reason` | `balance` (the balance at or below zero), `spend_limit` (the month's charges at `spend_limit_ucredits`), or `balance+spend_limit` (both at once) |
| `stage` | `hold` only. `notice`: growth is blocked and nothing is reclaimed yet. `enforce`: capacity is being reclaimed |
| `enforce_after_ms` | `hold` only: epoch milliseconds when `stage` turns `enforce` |
| `remaining_min` | `grace` only: minutes until the hold opens. The grace is 0 by default, so this state is rare |
| `base_load_min` | present only while your organization holds a live [base load](https://docs.nationalcompute.com/market.md#base-load-capacity) block during a `balance` condition: minutes left before the block is cancelled |

A deposit cures a `balance` condition. Only a raised or cleared limit,
or the month rolling over, cures a `spend_limit` one. `billing_hold_enforced`
and `spend_limit_enforced` say whether either can open for your
organization at all.

### Wire transfer

`wire` carries what the console's Billing page shows under Wire
transfer, for amounts the card cap refuses. It is `null` while
`purchases` is `false`: an organization on a reserved arrangement pays
by invoice.

| Field | Meaning |
|---|---|
| `bank_name`, `aba_routing`, `bank_address` | the receiving bank; `aba_routing` serves domestic wires and ACH |
| `aba_routing_alt` | the routing number to use when the sending bank does not recognize `aba_routing` |
| `swift_bic`, `intermediary_swift_bic` | international wires: the receiving bank's SWIFT code and the intermediary bank the wire must name |
| `beneficiary_name`, `account_number`, `account_kind`, `beneficiary_address` | the beneficiary |
| `reference` | your organization's memo for the current UTC day. It rolls at midnight UTC: read it on the day of the wire. It must ride the wire's reference or memo field, or the deposit is not matched to your organization promptly |
| `notify_email` | the address to email when the wire is sent |

The bank and beneficiary values are platform constants and appear on
this wire and the console only.

## Spend per workload

`GET /api/billing/usage/workloads?range=7d` re attributes charge
entries to the jobs and pods that occupied the charged nodes, from the
platform's occupancy samples. The figures are approximate by design:
idle capacity and spend that cannot be attributed ride as their own
labeled series, never smeared across jobs.

| Parameter | Meaning |
|---|---|
| `range` | one of `1d`, `7d`, `30d`, `90d`, `mtd`, `ytd`, `lastmonth` (default `7d`); anything else is `422 bad-request` |
| `tz_offset_min` | the calendar the buckets use, as on the other reads |

The response carries `starts[]` (one stamp per bucket) and `end`,
`labels[]` (`HH:00` for `1d`, `MM-DD` otherwise), `today_index` (the
bucket holding today, `null` for `lastmonth`), and `groups` with two
groupings, `workload` and `user`. Each grouping has `order[]`, `series`
(per name: `spend[]` in microcredits and `unit_ms[]`, milliseconds of
unit time, per bucket) and `meta` (per name: `label`, `who`, `cluster`,
`unit`, `handover_ucredits`; the user grouping adds `workloads`, how
many workloads the user ran). `who` names the person who started the
workload: the Slurm username, or the creator of the Kubernetes object
as the cluster's audit log recorded it (the console's "Started by").
A workload with no recorded creator carries its Helm release or app
group instead. A workload with neither carries an empty `who`, and the
user grouping lists it under "No user recorded". Series names are
workload ids plus three sentinels: `IDLE` (held capacity no job used),
`UNATTRIBUTED` (spend no sample explains) and `OTHER` (the fold of the
smaller workloads, the way the console's chart folds them). The payload
is served through a cache: `cache_age` is its age in seconds, and
`stale: true` means a refresh is in flight, so read again in a moment
for fresh figures.

A range nobody has read yet is computed on first request. When the
computation takes more than a few seconds, the call answers
`202 Accepted` with `{"pending": true, "retry_after_s": 3}` and a
`Retry-After` header instead of holding the connection. Poll the same
URL after that many seconds. The `200` lands once the figures are
ready, and later reads of the same range come from the cache.

## Prices in force

`GET /api/billing/rates` lists every resource type priced specifically
for your organization right now:

```json
{"rates": [
  {"resource_type": "GPU_…", "label": "…", "unit": "GPU",
   "unit_hourly_price_ucredits": …, "since": "2026-09-01T00:00:00Z"}
]}
```

`unit` is `GPU`, `CPU`, or `node`. Burst capacity does not appear here —
it bills at the [clearing rate](https://docs.nationalcompute.com/billing.md#what-the-meter-measures),
which the [price feed](https://docs.nationalcompute.com/api/market-feed.md) publishes.

A public research organization also sees its
[Public Research](https://docs.nationalcompute.com/public-research.md#billing) rate as one line per
site that offers the program: `resource_type` the node's GPU type,
`label` `Public Research · <GPU model> GPU`, `unit` `GPU`,
`unit_hourly_price_ucredits` the flat rate a lease granted now is
fixed at, and `since` the stamp the rate was last set. A free rate is
listed at `0`; a standard organization never sees the line. The MCP
`billing_read` rates view carries the same rows.

## Writes

The five writes are the console's billing actions an agent starts for
the person it works for. Bodies are JSON; every amount is a whole
number of dollars.

```sh
curl -X POST -H "Authorization: Bearer $NC_TOKEN" \
  -H "Content-Type: application/json" -d '{"usd": 250}' \
  https://nationalcompute.com/api/billing/checkout
```

```json
{"url": "https://checkout.stripe.com/c/pay/…"}
```

| Route | Body | What comes back |
|---|---|---|
| `POST /api/billing/checkout` | `{"usd": 250}` | a hosted payment page URL ($1 = 1 credit). Hand it to a person; the page never offers the organization's saved card, the person enters a card, so the URL alone cannot pay. The deposit lands on the ledger when they pay, so read the [summary](#balance-and-totals) again afterwards. Card payments are capped at `card_max_usd` total per rolling `card_window_days` days per organization. `card_window_remaining_usd` on the summary is what a card can still carry right now. Larger amounts go by wire transfer |
| `POST /api/billing/setup` | none | a hosted page URL that saves a card and charges nothing. The on ramp for auto reload |
| `POST /api/billing/portal` | none | the payment portal URL: receipts and card changes. Before the first purchase there is no customer record to manage |
| `PUT /api/billing/limit` | `{"usd": 500}` | `{"ok": true, "spend_limit_ucredits": 500000000}`. Sets the monthly spend limit where none exists, or lowers the one in force |
| `PUT /api/billing/reload` | `{"enabled": true, "threshold_usd": 50, "amount_usd": 150}` | `{"ok": true, "enabled": true, "threshold_ucredits": 50000000, "amount_ucredits": 150000000}`: the setting as stored. Tightens the auto reload setting: lowers `amount_usd`, raises `threshold_usd`, or disarms it with `{"enabled": false}` (nothing else needed; the stored amounts stay for the next enable). A write that changes nothing is a no op |

The spend limit is the cap a person sets on the month's charges. Read
`spend_limit_enforced` on the summary before relying on it. When it is
`true`, reaching the limit holds capacity until the limit is raised or
the month rolls over. When it is `false`, the platform records the
limit and does not act on it for your organization's clusters. The
flag differs between organizations: read it, never assume it. A token
may only tighten the limit:
raising the limit, or clearing it with `null`, is refused
`403 session-required` and stays a console action. The hosted page URLs
are short lived; request a fresh one if it expired before the person
opened it.

Auto reload is a standing spending authorization: every charge it fires
runs off session with no further approval and counts against the card
cap. A token may only tighten it, the spend limit's posture: disable it,
lower `amount_usd`, or raise `threshold_usd`. Enabling it, raising the
amount, or lowering the threshold is refused `403 session-required` and
stays a console action for a person. An update sends `enabled: true`
with both amounts (`threshold_usd` 1 to 1000000, `amount_usd` within
`min_usd` and `max_usd`; the bounds apply to every write, a disable
included). A failed charge pauses attempts for 24 hours, or until a
change is saved on the console, whichever comes first; the summary's
`auto_reload.cooldown_until` names the stamp, and a no op by token never
moves it. On the [MCP server](https://docs.nationalcompute.com/api/mcp.md#two-phase-confirmation) the
same write runs in both directions, because a person approves the
preview there: `billing_write` carries `human_confirmation` on it.

## Errors

| Status | Code | When |
|---|---|---|
| `401` | `unauthorized` | missing, malformed, expired, or revoked token |
| `403` | `reserved-arrangement` | credits are managed by the platform team for your organization; purchases are not self serve |
| `403` | `session-required` | raising or clearing the spend limit by token, or enabling auto reload, raising its amount or lowering its threshold by token: console actions |
| `409` | `card-window` | the card cap for the rolling window is spent (`card_window_remaining_usd` is `0`); wait for payments to age out of the window, or pay by wire transfer |
| `409` | `no-purchases` | the payment portal before the first purchase: nothing to manage yet |
| `409` | `no-card` | enabling auto reload with no card on file; `POST /api/billing/setup` hands back the page that saves one |
| `422` | `bad-request` | a query value or body field out of range or malformed; `detail` names it |
| `422` | `card-cap` | `usd` above what a card can still carry this window (`card_window_remaining_usd` on the summary); lower the amount, or pay larger amounts by wire transfer |
| `429` | `rate-limited` | past the token's request budget for the minute (300 by default; the vm contract states the figure); wait `retry_after_s` |
| `503` | `purchases-unavailable` | self serve purchases are off |
| `503` | `station-unavailable` | the ledger or the payment provider is unreachable; retry |

Every error is the plain `{"error": code}` body of the
[API conventions](https://docs.nationalcompute.com/api/index.md).

---

# Organization

Source: https://docs.nationalcompute.com/api/organization/ (this markdown: https://docs.nationalcompute.com/api/organization.md)

These routes publish what the console's Organization and API Keys
pages show: who is in your organization, who is invited, which API
tokens exist, and what each token has done.

| Route | What it serves |
|---|---|
| `GET /api/org/members` | the roster, the auto join rules, the outstanding invitations |
| `GET /api/org/tokens` | every API token the organization holds, as metadata |
| `GET /api/org/tokens/{token_id}/activity` | one token's activity trail; `self` is the calling token |
| `DELETE /api/org/tokens/self` | the calling token revokes itself |
| `GET /api/org/activity` | the organization's trail: every member's and every token's recorded write, newest first |

Contract: [`GET /api/org/openapi.json`](https://nationalcompute.com/api/org/openapi.json).

Every route takes an org API token. The organization is the token's
own and nothing identifies it on the wire. Any of your organization's
tokens reads these routes, cluster bound or org wide. The one write is
a token revoking itself. Adding or removing a member, adding a rule,
minting a token, and revoking another token are console actions.
Removing a member revokes every token that member minted, and retiring
the organization revokes every token it holds; those revokes appear in
the [activity trail](#activity) under the actor `system:offboard`.

## Members and invitations

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/org/members
```

`members[]` is the roster: `username`, `email`, `name`, `is_active`
(`false` when the identity provider has suspended the account; `true`
means active, or unknown while the identity provider read is degraded),
`last_login` (RFC 3339, or `null`), and `role`. Every member holds the
one organization role, `admin`. When the identity provider read is
degraded, `email` and `name` read empty and `last_login` reads `null`
rather than stale.

`rules[]` are the auto join rules, each `{id, kind, value}`. A `domain`
rule admits every sign in whose email carries that domain. An `email`
rule invites one address. `invited[]` lists the email rules no member
has consumed yet, the rows the console shows as Invited. An invitation
counts as consumed once a member carries the email, or its username
form when the identity provider read is degraded. An invitation made
from the console also emails that address a one time sign in link. The
link is currently valid for 3 days. A member who signs in with a Google
account under that address can ignore the link.

## Tokens

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/org/tokens
```

`tokens[]` lists every token, live and inactive, newest first:
`token_id` (the middle segment of `national_compute_<id>_<secret>`),
`name`, `scopes` as recorded at mint, `cluster` (the bound cluster, or
`null` for an org wide token), `expires_at`, `created_by`,
`created_at`, `last_used_at`, and `revoked_at` (`null` while live). A
secret is shown once, at mint, on the console. It never appears here,
and neither does its hash. The organization's kind, `standard` or
`public_research`, is on
[`GET /api/whoami`](https://docs.nationalcompute.com/authentication.md#discovering-what-a-token-can-address)
as `org_kind`.

## Activity

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/org/tokens/self/activity
```

The trail of one token, newest first. `self` is the calling token; a
`token_id` from the token list reads any token of your organization.
An id outside your organization reads `404 not-found`, the same as an
unknown id.

`entries[]` are the console audit rows the token wrote as actor. Writes
are always recorded: capacity declares and swaps, limit prices, ssh key adds
and revokes. The token's own mint and revoke rows are included, with
the console user who performed them as `actor`. Two reads are recorded
on every call as well, the VM capacity read and the ssh key list. Those
rows stay out of the trail unless you pass `reads=1`. Other reads are
not recorded. `last_used_at` on the token list says whether a token is
in use.

Each entry carries `id`, `ts` (RFC 3339 UTC), `actor` (`token:<id>`
for the token's own calls), `action` (for example `vm_capacity_put`,
`k8s_bid_put`, `vm_key_add`, `vm_token_mint`, `vm_token_revoke`; with
`reads=1` also `vm_capacity_get` and `vm_keys_get`), `subject` (a
cluster name, or the organization for key and token rows), and
`detail` (the recorded arguments, free text).

Paging is by entry id. `limit` is 100 by default and clamps to 500.
When `more` is `true`, pass the last entry's `id` as `before` for the
next page. A `limit` or `before` that is not an integer, a `before`
that is not positive, or a `reads` other than `1` or `0` reads
`422 bad-request`.

The console shows the same trail: the API Keys page has an Activity
toggle on every token row, with a show reads switch in the panel. On
the MCP server the token list and both trails are
[`grid_api` paths](https://docs.nationalcompute.com/api/mcp.md#grid_api-paths); `self` names no token there.

## The organization's trail

```sh
curl -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/org/activity
```

Every recorded write in your organization, newest first, human and
token side by side. Three kinds of rows arrive: every write any of
your tokens made, every row about the organization itself (token mint
and revoke, the spend limit, the billing pages, member and rule
changes), and your members' console writes on your clusters, tied to
each cluster's own island. A limit price or a capacity declaration a member set
on the console reads here with the member as `actor`, so an agent
learns whose hand moved a ceiling before it acts on the change.
Platform admin actions on your clusters are not included. Platform
system rows about the organization are. A console row whose actor ties
to no member, or whose cluster is not one of yours on its island, reads
absent. Rows of another organization never appear, even from a cluster
elsewhere that carries your organization's name or from a person who
belongs to both organizations.

Entries carry the same fields as the token trail. Read rows hide
unless `reads=1`. Paging is by entry id: `limit` is 100 by default and
clamps to 500, and when `more` is `true`, pass the last entry's `id` as
`before`. The response is `{org_id, entries, more}`.

## Rate limit

Every token carries a budget of requests per minute across all routes
(300 by default; the vm contract states the figure in force). The call
past it reads `429 rate-limited` with `retry_after_s`, and the
`Retry-After` header carries the same figure.

## Revoking your own token

```sh
curl -X DELETE -H "Authorization: Bearer $NC_TOKEN" \
  https://nationalcompute.com/api/org/tokens/self
```

The calling token stops working at once. The next call with it reads
`401 unauthorized`. The response is `{ok, token_id, revoked_at}`. The
audit row carries `actor` `token:<id>`, so the trail shows the token
closing itself. No token can revoke another token. That stays a console
action on the API Keys page.

---

# Error reference

Source: https://docs.nationalcompute.com/api/errors/ (this markdown: https://docs.nationalcompute.com/api/errors.md)

Every error is plain JSON with a machine code, plus a human `detail`
where it helps:

```json
{"error": "version-mismatch", "detail": "..."}
```

Some errors carry extra keys — `unknown-node` echoes the unresolved
entry under `name`, `ambiguous-cluster` lists the candidate `clusters`,
`rate-limited` carries `retry_after_s`. The OpenAPI contracts are the
authority; this page is the tour.

## Auth and addressing

| Code | Status | Meaning | Fix |
|---|---|---|---|
| `unauthorized` | 401 | missing, malformed, expired, or revoked token | mint or re-issue on the console's API Keys page |
| `session-required` | 403 | an API token on a console-only action: the [volume delete](https://docs.nationalcompute.com/api/k8s-bid.md#deleting-a-volume-is-a-console-action), [raising or clearing the spend limit](https://docs.nationalcompute.com/api/billing.md#writes), or [loosening auto reload](https://docs.nationalcompute.com/api/billing.md#writes) (enabling it, raising its amount, lowering its threshold); every token is refused | sign in to the console and use the Storage or Billing page |
| `reserved-arrangement` | 403 | a [billing write](https://docs.nationalcompute.com/api/billing.md#writes) for an organization whose credits the platform team manages | contact the platform team |
| `not-found` | 404 | no cluster of the right kind matches the token, the `{org}` in the path is not the token's organization, or a node, pod or volume named on a cluster page read is not in your organization | check the token's binding; a VM-bound token is refused on the k8s surfaces and vice versa |
| `ambiguous-cluster` | 409 | org-wide token, several clusters of that kind; or the public onboarding path (`access.nationalcompute.com/api/onboard/<name>`) when more than one cluster carries the name | pass `cluster` (query on GETs, body on writes); on the onboarding path fetch the kubeconfig by token, [`GET /api/k8s/cluster/kubeconfig`](https://docs.nationalcompute.com/api/k8s-cluster.md#the-kubeconfig) |

## Validation (422)

| Code | Meaning |
|---|---|
| `price-precision` | prices are USD, whole cents — at most 2 decimals |
| `bid-too-low` | a declare too low to ever clear the market that would *acquire* capacity; check the [market feed](https://docs.nationalcompute.com/api/market-feed.md) for what wins and raise the price. Scale-downs are never price-gated (a k8s limit price of `0` always withdraws) |
| `no-ssh-keys` | a VM capacity raise with no registered ssh key — the node would be unreachable ([register one first](https://docs.nationalcompute.com/api/vm-capacity.md#before-you-declare-register-an-ssh-key)) |
| `bad-request` | malformed body; the `detail` says exactly what — including a non-multiple `max_gpus` (whole nodes only, never rounded) and the retired per-node field spellings (the detail gives the conversion) |
| `card-cap` | a [checkout](https://docs.nationalcompute.com/api/billing.md#writes) above what a card can still carry this window; the summary's `card_window_remaining_usd` says how much that is, larger amounts go by wire transfer |
| `confirmation-mismatch` | a volume delete whose `confirm` is not the volume's storage name (its `name` field, shown under the label; the label is refused); nothing is destroyed ([shared storage volumes](https://docs.nationalcompute.com/api/k8s-bid.md#shared-storage-volumes)) |

## Concurrency and state (409)

| Code | Meaning |
|---|---|
| `version-mismatch` | your `expected_version` no longer matches — re-read, reconcile, retry ([conventions](https://docs.nationalcompute.com/api/index.md#versions-and-compare-and-set)) |
| `unknown-node` | a `release` or swap entry resolves to no node you hold; the entry is echoed under `name`, and **nothing is written** |
| `static-posture` | a `max_gpus` raise on a cluster not under market management — nothing would fill the new capacity; lowering and `release` still work |
| `swap-disabled`, `cluster-gated`, `node-pinned` | a swap the platform can't take right now — the surface isn't enabled for your site, the cluster is gated, or the node is pinned; the swap is refused whole |
| `volume-attached` | a delete of a shared volume that is attached to a live cluster; delete the cluster first (the volume detaches with it and is preserved by default) or opt to delete the storage with the cluster |
| `volume-held` | a delete of a preserved volume a node still holds; retry in a few minutes, contact support if it persists |
| `cluster-not-ready` | a [cluster page read](https://docs.nationalcompute.com/api/k8s-cluster-pages.md) or the [kubeconfig download](https://docs.nationalcompute.com/api/k8s-cluster.md#the-kubeconfig) on a cluster that is still being built or is not ready; retry |
| `card-window` | a [checkout](https://docs.nationalcompute.com/api/billing.md#writes) while the rolling card cap is spent (`card_window_remaining_usd` is `0`); wait for payments to age out of the window, or pay by wire transfer |
| `no-purchases` | the [payment portal](https://docs.nationalcompute.com/api/billing.md#writes) before the first purchase; nothing to manage yet |
| `no-card` | enabling [auto reload](https://docs.nationalcompute.com/api/billing.md#writes) with no card on file; the setup page saves one |

## Pacing and platform

| Code | Status | Meaning |
|---|---|---|
| `rate-limited` | 429 | faster than the per cluster write cadence, or more than 300 requests in one minute with this token; wait `retry_after_s` |
| `purchases-unavailable` | 503 | self serve purchases are off, so a [billing write](https://docs.nationalcompute.com/api/billing.md#writes) that needs them is refused; load credits through the console's Billing page or contact the platform team |
| `station-unavailable`, — | 503 | temporary platform condition — reads keep serving, retry writes later |

## Admission denials (Kubernetes)

These denials arrive from your cluster's API server at `kubectl apply`
time, not from this HTTP API — see
[launch admission](https://docs.nationalcompute.com/kubernetes.md#launch-admission) and
[the max cluster size](https://docs.nationalcompute.com/kubernetes.md#the-max-cluster-size):

| Denial | Meaning |
|---|---|
| `JobSizeTooSmall` | a pod template's GPU request is not a multiple of `gpus_per_node` on a cluster without packing onto the base load block; GPU workloads on the market occupy full nodes |
| `JobSizeNotNodeAligned` | on a cluster with packing onto the base load block enabled a pod template requests more GPUs than one node and not a whole number of nodes; pods smaller than one node and whole node multiples are admitted there |
| `BalanceTooLow` | the org balance does not cover two hours of the job at the current limit price; the message names the required and current balance |
| `StationOwned` | an edit of a Kueue object the platform owns — the ClusterQueue `market`, its flavors, admission check, request config and priority classes — these are read-only to cluster users; the max cluster size is set by the platform |

`BalanceTooLow` is also a queue reason: the market re-checks the same
two hours at every tick while the request waits, and a request the
balance can no longer fund stays pending under that reason — never
refused — until a top-up ([queue events](https://docs.nationalcompute.com/api/k8s-bid.md#queue-events)).

## Handling advice

- Treat the machine code as the contract; the `detail` text can change.
- On `409 version-mismatch`, always re-read before retrying — the state
  that moved under you may change what you want to write.
- On `429`, respect `retry_after_s` rather than tight-looping; writes
  are paced per cluster, so a second client on the same cluster shares
  your budget.