Skip to content

Object storage recipes

The starter recipes keep their durable state in your organization's object storage bucket when the cluster holds the Secret nc-storage. They keep it on the shared volume when one is mounted. The pod decides at start: NC_BUCKET in its environment means the bucket steps run; a network filesystem at /mnt/shared means the volume steps run. Both can be true. A cluster with neither runs the same files and keeps nothing once the market takes the node back. The files are served at access.nationalcompute.com/first-job/, which also lists them as a catalog. FIRST-JOB.md walks an agent through each one.

How a recipe finds the bucket

Every recipe reads the Secret through envFrom with optional: true:

envFrom:
  - secretRef:
      name: nc-storage
      optional: true

On a cluster without the Secret the pod starts with no bucket variables and the bucket steps are skipped. On a cluster with it the five variables land in the environment and the AWS CLI reads them as they are (using the bucket from a pod). The recipes also set AWS_EC2_METADATA_DISABLED=true, so a client with no credential fails at once instead of probing for one. The Secret exists in the default namespace only. Every recipe runs there.

Logs in the bucket

The market takes a GPU node back minutes after its job finishes. The job's kubectl logs output goes with the node. The training recipes (nanogpt-job.yaml, nanogpt-job-amd.yaml, nanogpt-multinode.yaml, nanogpt-multinode-amd.yaml) tee their output into a file on an emptyDir. When NC_BUCKET is set they upload that file to the bucket:

  • every 60 seconds, from a loop that runs beside the training process;
  • once more when the process exits, whatever the exit status, from an EXIT trap. A stop or a market reclaim can end the pod before the trap runs. The bucket then holds the log as of the last upload, at most 60 seconds old.

The object is logs/<job>/<pod>.log: logs/nanogpt/<pod>.log for the one node run, one object per pod under logs/nanogpt-mn/ for the two node run. Each pod writes its own key because a key accepts one write per second. Each upload replaces the whole object: the endpoint has no append. There is no live tail. A reader sees the log as of the last upload. The lag is the upload interval, 60 seconds. Where the volume is mounted the same tee still writes /mnt/shared/logs/; both copies exist then.

The pattern, for a job of your own:

envFrom:
  - secretRef: { name: nc-storage, optional: true }
env:
  - name: POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - { name: AWS_EC2_METADATA_DISABLED, value: "true" }
  - { name: AWS_REQUEST_CHECKSUM_CALCULATION, value: when_required }
  - { name: AWS_RESPONSE_CHECKSUM_VALIDATION, value: when_required }
command: ["bash", "-c"]
args:
  - |
    set -o pipefail                          # the Job fails when train.py fails
    log=/logs/train.log                      # an emptyDir mounted at /logs
    if [ -n "${NC_BUCKET:-}" ]; then
      key="s3://$NC_BUCKET/logs/my-job/$POD_NAME.log"
      ( while sleep 60; do aws s3 cp "$log" "$key" >/dev/null 2>&1 || true; done ) &
      UP=$!
      trap 'kill "$UP" 2>/dev/null; aws s3 cp "$log" "$key" >/dev/null 2>&1 || true' EXIT
    fi
    python train.py 2>&1 | tee -a "$log"

The upload needs an S3 client in the pod. The recipes install one at start. When the install fails the run goes on and the log stays out of the bucket. A failed upload never fails the run: the loop swallows the error and tries again a minute later.

Reading a log

bucket-logs-reader.yaml is a Job pinned to the cluster CPU worker, which the market never reclaims. It runs the AWS CLI image with envFrom the Secret, lists logs/<JOB>/ and prints one object to standard output: the newest under the prefix, or the object KEY names. Its env block sets JOB to nanogpt and KEY to empty. backoffLimit: 0 keeps a failure visible. ttlSecondsAfterFinished: 120 removes the Job two minutes after it finishes, so applying it again prints a fresh copy.

kubectl --context <cluster> apply -f https://access.nationalcompute.com/first-job/bucket-logs-reader.yaml
kubectl --context <cluster> logs -f job/bucket-logs-reader

To read another job's log, set JOB to its name (nanogpt-mn for the two node run). To read one particular object, set KEY to its whole key. Download the file, set the variable in its env block and apply your copy.

Any machine that holds the pair reads the same object with no cluster at all. The volume never leaves the cluster's nodes. The bucket does:

aws s3 ls "s3://$NC_BUCKET/logs/nanogpt/"
aws s3 cp "s3://$NC_BUCKET/logs/nanogpt/<pod>.log" -

Stream to standard output with -. Never copy to /dev/null: the CLI stages a temporary file beside the destination and fails there.

Weights through the bucket

A serve recipe loads its checkpoint from the node's NVMe. Getting the checkpoint there is the slow part of a first start. The bucket holds the durable copy, which outlives every node and every cluster.

Staging a checkpoint

kimi-k3-weights-to-bucket.yaml and inkling-weights-to-bucket.yaml are Jobs that download a checkpoint from the public model hub that publishes it and upload it, many files at a time, into s3://$NC_BUCKET/models/<name>/, in the layout the serve recipe keeps on node NVMe. They run on the CPU worker, so the stage starts while the GPU node is still arriving. Where the catalog lists a vendor variant, apply the variant for the cluster's gpu_vendor: the Inkling checkpoint differs by GPU vendor.

kubectl --context <cluster> apply -f https://access.nationalcompute.com/first-job/kimi-k3-weights-to-bucket.yaml
kubectl --context <cluster> logs -f job/<name>    # kubectl apply prints the Job's name

Two small objects mark the stage. models/<name>/.staging is a heartbeat the Job refreshes while it runs. models/<name>/.complete is written when every file is up. A Job that finds every shard and the config already in the bucket writes .complete and exits at once. There is no free space check: the bucket has no size. The staged bytes count against the soft cap: the Kimi K3 checkpoint (about 1.5 TB) fits under the default 2 TiB; a second checkpoint of that size crosses it, which is a warning today. Do not rotate the credential while a staging Job runs: the pod holds the pair it loaded at start, which stops working ten minutes after a rotate. Apply the Job again afterwards.

Serving from node NVMe

The serve recipes (kimi-k3-serve-amd.yaml, inkling-serve-amd.yaml, inkling-serve-nvidia.yaml) put the weights on the node's NVMe in an init container before the engine starts, in this order:

  1. Already on this node's NVMe: nothing to copy.
  2. .complete in the staged copy on the shared volume, where one is mounted: a parallel copy from the volume. The volume comes first where both the volume and the bucket hold a complete copy.
  3. .complete under models/<name>/ in the bucket: a parallel pull from the bucket onto the NVMe. A pull that stops is tried again up to five times and resumes at the file level. A file already on the NVMe at the bucket's size is not pulled again.
  4. Otherwise the model hub, onto the NVMe.

While a staging Job's heartbeat is fresh (under ten minutes old) the init container waits for it. Nothing is downloaded twice. The engine always loads from the NVMe. Its compile caches live on the shared volume where one is mounted and on the node's NVMe otherwise. Deleting the serve recipe leaves the staged copy in the bucket for the next start. The serve steps themselves (apply, watch, the first request) are in FIRST-JOB.md and on the console's Getting Started page.

The Getting Started page

The console's Getting Started page shows the bucket variant of each Serve step on a cluster without a shared volume and the volume variant where one is mounted. Where both exist the Serve steps take the volume variant and name the bucket path. The logs section shows the volume reader where the volume is mounted and the bucket reader where the organization holds a bucket. Both readers show where both stores exist. On a cluster with neither, the Stage step and the logs section are not shown. The Serve path then downloads the checkpoint from the model hub onto the node.

When a shared volume is the better store

Throughput is not the reason. With many streams in flight the bucket moves data as fast as the shared volume in our measurements: a GPU node wrote 64 GiB to the bucket in under a minute and read it back faster, and a 527 GiB checkpoint staged from the model hub into the bucket in about fourteen minutes from the CPU worker. One stream is far slower, which is why the recipes use many. A shared volume is the better store when a workload needs a file system: a live tail of an open file, appends, random writes, many small files, or a path that several nodes read and write at once. Support adds one on request. The recipes then take the volume path unchanged. The bucket keeps the logs, the results and the copy that outlives the cluster.