# Object storage Every organization with a Kubernetes cluster holds one bucket. Public Research organizations have none. The bucket is the durable store for datasets, checkpoints and outputs. It outlives every cluster. It is reachable from every site and from outside the platform through an S3 compatible endpoint. One access key pair opens it. The platform writes that pair into every Kubernetes cluster of the organization as the Secret `nc-storage` in the `default` namespace. The console's **Storage** page shows the same pair on request. ## The bucket The platform creates the bucket at the organization's first Kubernetes cluster. The name is assigned by the platform (`nc-` and 16 hex characters). The Storage page shows the bucket under **Object storage** on every cluster of the organization, with the endpoint, the bucket's name and the access key ID. An organization whose clusters predate object storage and whose Storage page shows no **Object storage** block asks support. The bucket belongs to the organization. Nothing about it is per cluster: every Kubernetes cluster of the organization, on every site, reaches the same bucket with the same pair. The bucket survives cluster deletion. Deleting the organization's last cluster does not delete it. Slurm clusters and VM clusters receive no Secret ([below](#what-the-bucket-is-not)). ## The credential One access key pair per organization: an access key ID and a secret access key. Every member of the organization shares it. There are no per member credentials, no per cluster credentials and no prefix scoping. The pair grants read and write on the whole bucket. ### The Secret `nc-storage` The platform writes the pair into every Kubernetes cluster of the organization as the Secret `nc-storage` in the `default` namespace, within minutes of a cluster's creation and again at every rotate. The Secret carries five keys, named so that the common S3 clients read them with no mapping: | Key | Value | |---|---| | `AWS_ACCESS_KEY_ID` | the access key ID | | `AWS_SECRET_ACCESS_KEY` | the secret access key | | `AWS_ENDPOINT_URL` | the S3 compatible endpoint | | `AWS_DEFAULT_REGION` | `auto` | | `NC_BUCKET` | the bucket's name | The Secret is platform managed (label `app.kubernetes.io/managed-by: nationalcompute-station`). An edit to it is overwritten at the next push. Deleting it revokes nothing. The platform writes it again at the next rotate. The Secret exists in the `default` namespace only. A pod in another namespace cannot reference it. Copy it into that namespace yourself. A copy does not follow a rotate. The Storage page shows the Secret's state on the picked cluster: **present** once the write landed, **pending** with a short reason while it has not, and **Kubernetes clusters only** on a Slurm or VM cluster. The platform retries a pending write after five minutes at first and then at growing intervals up to six hours. A Secret that stays pending is a support matter. ### Show and Rotate on the Storage page **Show credentials** displays the pair. Each display is recorded in your organization's activity trail with the member who asked. **Rotate credentials** replaces it ([below](#rotation)). Both are console actions. No API token and no agent tool can show or rotate the pair. ## Using the bucket from a pod `envFrom` the Secret. The five variables land in the container's environment. boto3 and the AWS CLI read them as they are: no endpoint flag, no credentials file, no region to choose. ```yaml apiVersion: batch/v1 kind: Job metadata: name: bucket-smoke spec: backoffLimit: 0 ttlSecondsAfterFinished: 120 template: spec: restartPolicy: Never containers: - name: aws image: public.ecr.aws/aws-cli/aws-cli:2.38.5 envFrom: - secretRef: name: nc-storage env: # the endpoint does not implement the default checksum mode # of current AWS CLI and boto3 releases - { name: AWS_REQUEST_CHECKSUM_CALCULATION, value: when_required } - { name: AWS_RESPONSE_CHECKSUM_VALIDATION, value: when_required } command: ["sh", "-c"] args: - | echo hello > /tmp/hello aws s3 cp /tmp/hello "s3://$NC_BUCKET/smoke/hello" aws s3 ls "s3://$NC_BUCKET/smoke/" ``` The Job requests no GPUs, so it runs at once on the [cluster CPU worker](https://docs.nationalcompute.com/kubernetes.md#the-cluster-cpu-worker) and never touches the market. A GPU job uses the same `envFrom` block. The same round trip in Python: ```python import os import boto3 s3 = boto3.client("s3") # endpoint, region and keys come from the environment bucket = os.environ["NC_BUCKET"] s3.put_object(Bucket=bucket, Key="smoke/hello", Body=b"hello\n") print(s3.get_object(Bucket=bucket, Key="smoke/hello")["Body"].read()) ``` The two checksum variables in the Job apply to boto3 as well. An older client release that does not read `AWS_ENDPOINT_URL` takes the endpoint explicitly: `--endpoint-url "$AWS_ENDPOINT_URL"` on the AWS CLI, `endpoint_url=os.environ["AWS_ENDPOINT_URL"]` on a boto3 client. Multipart uploads work as on any S3 endpoint. An object is currently at most 5 TiB. A key accepts one write per second. A second write to the same key within that second fails. Give concurrent writers their own keys. The pair works from anywhere on the internet with any S3 client: a laptop, a VM node, a Slurm login node, a Marshall workspace. Set the five variables from the Storage page's **Show credentials** and the same commands run. The starter recipes use the same `envFrom` block, with `optional: true` so the files run on a cluster without the Secret: [object storage recipes](https://docs.nationalcompute.com/object-storage-recipes.md). ## Rotation **Rotate credentials** on the Storage page replaces the whole pair. Both halves change: the access key ID and the secret access key. The platform writes the new pair into every Kubernetes cluster of the organization and shows it once. - The old pair keeps working for ten minutes after the rotate, then stops. Every copy you saved stops working with it at that point. - The Secret `nc-storage` is updated in every Kubernetes cluster of the organization within seconds. A cluster the write cannot reach reads **pending** on the Storage page and receives the pair when the platform's retry lands. - A pod that loaded the Secret through `envFrom` keeps the old pair until it restarts. Kubernetes injects the environment at container start and never updates it. Restart the consuming pods within the ten minutes after a rotate. A pod that starts in the seconds between the rotate and the update on its cluster loads the old pair too. A volume mount of the Secret refreshes on its own within about a minute. Rotate when a member leaves the organization and after any suspected exposure of the pair. Member removal revokes that member's tokens and sign in. It does not change the bucket pair. A departed member may hold a copy. The activity trail records who pressed **Show** on the console. It is not a record of who holds the pair: any member, workload or agent account in a cluster reads the Secret without a record. ## The soft cap Each bucket carries a soft cap, currently 2 TiB. Usage is sampled hourly. The Storage page shows the bytes and objects held against the cap. Over the cap the page shows a warning and asks you to contact support, who raise the cap on request. Nothing is refused at the cap today: writes continue while the warning shows. The bucket is not billed today. ## Reading the bucket by API `GET /api/k8s/storage/bucket` with an org API token answers the bucket's facts, its usage against the cap and the Secret's state on each Kubernetes cluster of the organization. It never carries the secret access key ([limit price API](https://docs.nationalcompute.com/api/k8s-bid.md#the-object-storage-bucket)). On the MCP server the same body is `grid_api get /api/k8s/storage/bucket`. Show and Rotate have no route and no tool. ## What the bucket is not - **No public access.** There is no public URL, no anonymous read and no per object sharing link. Whoever holds the pair reads and writes. Nobody else does. - **One pair.** The pair opens the whole bucket for the whole organization. Scoping by member, by cluster or by prefix does not exist. - **No versioning and no lifecycle rules.** The platform keeps no object versions, sets no expiry rules and takes no backups of the bucket. A deleted object is gone. - **Kubernetes clusters only.** The Secret lands in Kubernetes clusters. Slurm clusters and VM clusters receive nothing. The pair itself works from any machine. - **No tenant delete.** No console action and no route deletes the bucket or revokes its pair. Ask support. A bucket is destroyed only when it is empty. - **No append and no tail.** A write replaces the whole object. A log kept in the bucket reads as of its last upload ([logs in the bucket](https://docs.nationalcompute.com/object-storage-recipes.md#logs-in-the-bucket)). - **No POSIX mount.** For a filesystem shared by every node of a cluster, ask support for a [shared volume](https://docs.nationalcompute.com/kubernetes.md#the-shared-volume).