# Object storage recipes The starter recipes keep their durable state in your organization's [object storage bucket](https://docs.nationalcompute.com/object-storage.md) when the cluster holds the Secret `nc-storage`. They keep it on the [shared volume](https://docs.nationalcompute.com/kubernetes.md#the-shared-volume) when one is mounted. The pod decides at start: `NC_BUCKET` in its environment means the bucket steps run; a network filesystem at `/mnt/shared` means the volume steps run. Both can be true. A cluster with neither runs the same files and keeps nothing once the market takes the node back. The files are served at [`access.nationalcompute.com/first-job/`](https://access.nationalcompute.com/first-job/), which also lists them as a catalog. [FIRST-JOB.md](https://nationalcompute.com/FIRST-JOB.md) walks an agent through each one. ## How a recipe finds the bucket Every recipe reads the Secret through `envFrom` with `optional: true`: ```yaml envFrom: - secretRef: name: nc-storage optional: true ``` On a cluster without the Secret the pod starts with no bucket variables and the bucket steps are skipped. On a cluster with it the five variables land in the environment and the AWS CLI reads them as they are ([using the bucket from a pod](https://docs.nationalcompute.com/object-storage.md#using-the-bucket-from-a-pod)). The recipes also set `AWS_EC2_METADATA_DISABLED=true`, so a client with no credential fails at once instead of probing for one. The Secret exists in the `default` namespace only. Every recipe runs there. ## Logs in the bucket The market takes a GPU node back minutes after its job finishes. The job's `kubectl logs` output goes with the node. The training recipes (`nanogpt-job.yaml`, `nanogpt-job-amd.yaml`, `nanogpt-multinode.yaml`, `nanogpt-multinode-amd.yaml`) `tee` their output into a file on an `emptyDir`. When `NC_BUCKET` is set they upload that file to the bucket: - every 60 seconds, from a loop that runs beside the training process; - once more when the process exits, whatever the exit status, from an `EXIT` trap. A stop or a market reclaim can end the pod before the trap runs. The bucket then holds the log as of the last upload, at most 60 seconds old. The object is `logs//.log`: `logs/nanogpt/.log` for the one node run, one object per pod under `logs/nanogpt-mn/` for the two node run. Each pod writes its own key because a key accepts [one write per second](https://docs.nationalcompute.com/object-storage.md#using-the-bucket-from-a-pod). Each upload replaces the whole object: the endpoint has no append. There is no live tail. A reader sees the log as of the last upload. The lag is the upload interval, 60 seconds. Where the volume is mounted the same `tee` still writes `/mnt/shared/logs/`; both copies exist then. The pattern, for a job of your own: ```yaml envFrom: - secretRef: { name: nc-storage, optional: true } env: - name: POD_NAME valueFrom: { fieldRef: { fieldPath: metadata.name } } - { name: AWS_EC2_METADATA_DISABLED, value: "true" } - { name: AWS_REQUEST_CHECKSUM_CALCULATION, value: when_required } - { name: AWS_RESPONSE_CHECKSUM_VALIDATION, value: when_required } command: ["bash", "-c"] args: - | set -o pipefail # the Job fails when train.py fails log=/logs/train.log # an emptyDir mounted at /logs if [ -n "${NC_BUCKET:-}" ]; then key="s3://$NC_BUCKET/logs/my-job/$POD_NAME.log" ( while sleep 60; do aws s3 cp "$log" "$key" >/dev/null 2>&1 || true; done ) & UP=$! trap 'kill "$UP" 2>/dev/null; aws s3 cp "$log" "$key" >/dev/null 2>&1 || true' EXIT fi python train.py 2>&1 | tee -a "$log" ``` The upload needs an S3 client in the pod. The recipes install one at start. When the install fails the run goes on and the log stays out of the bucket. A failed upload never fails the run: the loop swallows the error and tries again a minute later. ### Reading a log `bucket-logs-reader.yaml` is a Job pinned to the [cluster CPU worker](https://docs.nationalcompute.com/kubernetes.md#the-cluster-cpu-worker), which the market never reclaims. It runs the AWS CLI image with `envFrom` the Secret, lists `logs//` and prints one object to standard output: the newest under the prefix, or the object `KEY` names. Its `env` block sets `JOB` to `nanogpt` and `KEY` to empty. `backoffLimit: 0` keeps a failure visible. `ttlSecondsAfterFinished: 120` removes the Job two minutes after it finishes, so applying it again prints a fresh copy. ```sh kubectl --context apply -f https://access.nationalcompute.com/first-job/bucket-logs-reader.yaml kubectl --context logs -f job/bucket-logs-reader ``` To read another job's log, set `JOB` to its name (`nanogpt-mn` for the two node run). To read one particular object, set `KEY` to its whole key. Download the file, set the variable in its `env` block and apply your copy. Any machine that holds the pair reads the same object with no cluster at all. The volume never leaves the cluster's nodes. The bucket does: ```sh aws s3 ls "s3://$NC_BUCKET/logs/nanogpt/" aws s3 cp "s3://$NC_BUCKET/logs/nanogpt/.log" - ``` Stream to standard output with `-`. Never copy to `/dev/null`: the CLI stages a temporary file beside the destination and fails there. ## Weights through the bucket A serve recipe loads its checkpoint from the node's NVMe. Getting the checkpoint there is the slow part of a first start. The bucket holds the durable copy, which outlives every node and every cluster. ### Staging a checkpoint `kimi-k3-weights-to-bucket.yaml` and `inkling-weights-to-bucket.yaml` are Jobs that download a checkpoint from the public model hub that publishes it and upload it, many files at a time, into `s3://$NC_BUCKET/models//`, in the layout the serve recipe keeps on node NVMe. They run on the CPU worker, so the stage starts while the GPU node is still arriving. Where the catalog lists a vendor variant, apply the variant for the cluster's `gpu_vendor`: the Inkling checkpoint differs by GPU vendor. ```sh kubectl --context apply -f https://access.nationalcompute.com/first-job/kimi-k3-weights-to-bucket.yaml kubectl --context logs -f job/ # kubectl apply prints the Job's name ``` Two small objects mark the stage. `models//.staging` is a heartbeat the Job refreshes while it runs. `models//.complete` is written when every file is up. A Job that finds every shard and the config already in the bucket writes `.complete` and exits at once. There is no free space check: the bucket has no size. The staged bytes count against the [soft cap](https://docs.nationalcompute.com/object-storage.md#the-soft-cap): the Kimi K3 checkpoint (about 1.5 TB) fits under the default 2 TiB; a second checkpoint of that size crosses it, which is a warning today. Do not rotate the credential while a staging Job runs: the pod holds the pair it loaded at start, which stops working ten minutes after a [rotate](https://docs.nationalcompute.com/object-storage.md#rotation). Apply the Job again afterwards. ### Serving from node NVMe The serve recipes (`kimi-k3-serve-amd.yaml`, `inkling-serve-amd.yaml`, `inkling-serve-nvidia.yaml`) put the weights on the node's NVMe in an init container before the engine starts, in this order: 1. Already on this node's NVMe: nothing to copy. 2. `.complete` in the staged copy on the shared volume, where one is mounted: a parallel copy from the volume. The volume comes first where both the volume and the bucket hold a complete copy. 3. `.complete` under `models//` in the bucket: a parallel pull from the bucket onto the NVMe. A pull that stops is tried again up to five times and resumes at the file level. A file already on the NVMe at the bucket's size is not pulled again. 4. Otherwise the model hub, onto the NVMe. While a staging Job's heartbeat is fresh (under ten minutes old) the init container waits for it. Nothing is downloaded twice. The engine always loads from the NVMe. Its compile caches live on the shared volume where one is mounted and on the node's NVMe otherwise. Deleting the serve recipe leaves the staged copy in the bucket for the next start. The serve steps themselves (apply, watch, the first request) are in [FIRST-JOB.md](https://nationalcompute.com/FIRST-JOB.md) and on the console's Getting Started page. ## The Getting Started page The console's Getting Started page shows the bucket variant of each Serve step on a cluster without a shared volume and the volume variant where one is mounted. Where both exist the Serve steps take the volume variant and name the bucket path. The logs section shows the volume reader where the volume is mounted and the bucket reader where the organization holds a bucket. Both readers show where both stores exist. On a cluster with neither, the Stage step and the logs section are not shown. The Serve path then downloads the checkpoint from the model hub onto the node. ## When a shared volume is the better store Throughput is not the reason. With many streams in flight the bucket moves data as fast as the shared volume in our measurements: a GPU node wrote 64 GiB to the bucket in under a minute and read it back faster, and a 527 GiB checkpoint staged from the model hub into the bucket in about fourteen minutes from the CPU worker. One stream is far slower, which is why the recipes use many. A shared volume is the better store when a workload needs a file system: a live tail of an open file, appends, random writes, many small files, or a path that several nodes read and write at once. Support adds one on request. The recipes then take the volume path unchanged. The bucket keeps the logs, the results and the copy that outlives the cluster.