> For the complete documentation index, see [llms.txt](https://docs.zerve.ai/self-managed/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zerve.ai/self-managed/configuration/builds.md). # Builds The `build` block of the [Execution Environment Manifest](/self-managed/configuration/execution-environment-manifest.md) controls how Zerve builds the container images behind Environments. ```yaml build: builder: { ... } # required — where and how images are built defaultImageRegistry: { ... } # required — where built images are pushed extraImageRegistries: [] # optional — additional registries to authenticate against imageConfigs: { ... } # required — base images and package sources per language ``` Snippets below start at the manifest root, so in your values file they go under `executor.execEnvManifest`. Fields marked `(ref)` accept a [reference value](/self-managed/configuration/execution-environment-manifest.md#reference-values) instead of a literal. ## Choosing a builder Available builders are: * BuildKit * BuildKit on Armada (for clusters fronted by an Armada scheduler) Both run BuildKit as a pod in the cluster; they differ only in how that pod is submitted. BuildKit caches layers between builds, so a rebuild that only adds a package doesn't start from scratch. Builds run in a separate namespace from the Zerve application. The chart provisions that namespace and grants Zerve the required permissions in it, so the `namespace` set below must match `executor.runtimeNamespace.name` in your values file (`zerve-runtime` by default). {% tabs %} {% tab title="BuildKit on Kubernetes" %} ```yaml build: builder: type: k8s_buildkit buildkitImage: nexus.example.com/docker/moby/buildkit:v0.30.0-rootless # optional fluentbitImage: nexus.example.com/ghcr.io/fluent/fluent-bit:5.0 # optional contextDownloaderImage: nexus.example.com/docker/library/alpine:3.24.1 # optional extraArgs: [] # optional — extra BuildKit flags k8s: cluster: endpoint: https://kubernetes.default.svc # (ref) required token: # (ref) required ref: file path: /var/run/secrets/kubernetes.io/serviceaccount/token caPath: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt # optional verify: true # optional — set false to skip API server cert checks namespace: zerve-runtime # required — namespace where build pods run serviceAccountName: default # optional, default: "default" nodeSelector: # optional — constrain builds to specific nodes workload-type: builder imagePullSecrets: # optional — for pulling the builder images themselves - name: regcred memory: # optional — resource requests/limits for build pods, request: "8Gi" # defaults shown limit: "8Gi" cpu: request: "2" ephemeralStorage: request: "10G" limit: "20G" # BuildKit's layer cache lives here ``` `contextDownloaderImage` is the init container that fetches the build context. It defaults to `alpine`. A substitute image must provide a `wget` with TLS support (`busybox` does not). {% endtab %} {% tab title="BuildKit on Armada" %} For clusters where workloads are submitted through [Armada](https://armadaproject.io/) instead of directly to the Kubernetes API. Armada accepts only a PodSpec, with two consequences: image pull secrets must already exist in the target namespace and are referenced by name, and taints require explicit `tolerations`. ```yaml build: builder: type: armada_buildkit buildkitImage: nexus.example.com/docker/moby/buildkit:v0.30.0-rootless # optional fluentbitImage: nexus.example.com/ghcr.io/fluent/fluent-bit:5.0 # optional contextDownloaderImage: nexus.example.com/docker/library/alpine:3.24.1 # optional extraArgs: [] # optional armada: server: &armadaServer endpoint: { ref: env, name: ARMADA_ENDPOINT } # (ref) required — host:port, no scheme insecure: false # optional — plain gRPC channel, dev/trusted networks only caPath: /etc/ssl/certs/internal-ca.crt # optional — custom CA; system trust store if unset binocularsEndpoint: { ref: env, name: ARMADA_BINOCULARS_ENDPOINT } # (ref) optional auth: # optional, defaults to type: none type: oidc tokenUrl: { ref: env, name: ARMADA_TOKEN_URL } clientId: { ref: env, name: ARMADA_CLIENT_ID } clientSecret: { ref: env, name: ARMADA_CLIENT_SECRET } scopes: [armada] # audience: { ref: env, name: ARMADA_AUDIENCE } # some providers (e.g. Auth0) need this queue: zerve # required — Armada queue to submit to namespace: zerve-runtime # required — namespace the pods land in priority: 1.0 # optional, default: 1.0 serviceAccountName: default # optional, default: "default" imagePullSecrets: # optional — must already exist in `namespace` - name: regcred tolerations: # optional - key: dedicated operator: Equal # Equal (default, compares `value`) | Exists value: armada-execution # effect: NoSchedule # omit to tolerate the taint regardless of effect # tolerationSeconds: 300 nodeSelector: { } # optional memory: { request: "4Gi", limit: "4Gi" } # optional, defaults shown cpu: { request: "2", limit: "2" } ephemeralStorage: { request: "10G", limit: "10G" } ``` **`armada.server.auth`** accepts four types: | Type | Fields | Notes | | -------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | | `none` | — | Default. The only type compatible with `insecure: true`. | | `oidc` | `tokenUrl`, `clientId`, `clientSecret`, `scopes`, `audience` | OAuth2 client credentials; tokens are issued and refreshed automatically. Recommended for production. | | `basic` | `username`, `password` | | | `bearer` | `token` | Pre-issued static token, for testing or out-of-band injection. | Credentials are sent as gRPC call credentials, which require a secure channel, so any type other than `none` is rejected in combination with `insecure: true`. `binocularsEndpoint` enables live log streaming from Armada's Binoculars service. If unset, logs are shipped to object storage by the Fluent Bit sidecar and only become available after the job finishes. The `server` block is also required for [Armada code execution](/self-managed/configuration/code-execution.md). Define it once as a YAML anchor (`&armadaServer` above) and reference it from the `execution` block. {% endtab %} {% endtabs %} {% hint style="warning" %} Every image Zerve pulls has a default, and every one of those defaults points at a public registry — Docker Hub for BuildKit, Fluent Bit, the context downloader and both language base images. That's there so a test install runs without any registry configuration; it is not a production posture. Builds will reach out to the public internet, they'll break when a tag moves or a rate limit is hit, and nothing about the image is under your control. For anything beyond a trial, mirror all five into your own registry and set them explicitly: | Field | Default | Where it's set | | ------------------------ | -------------------------------- | --------------------------------------------------------- | | `buildkitImage` | `moby/buildkit:v0.30.0-rootless` | `build.builder` | | `fluentbitImage` | `fluent/fluent-bit:5.0` | `build.builder`, and `execution.k8s` / `execution.armada` | | `contextDownloaderImage` | `alpine` | `build.builder` | | `baseImage` (Python) | `python:3.11-slim-bookworm` | `build.imageConfigs.python` | | `baseImage` (R) | `posit/r-base:4.5-bookworm` | `build.imageConfigs.r` | An air-gapped install has no choice: none of the defaults are reachable, so all five must be overridden before a build will succeed. {% endhint %} `extraArgs` passes additional flags to `buildctl`, for options not covered by the manifest. It cannot be used to allow a registry served over plain HTTP; use `insecure` on the registry instead (below). ## Pod security Zerve sets the build container's `securityContext` itself, so there's nothing to configure — but the cluster's admission policy has to allow it. Both builders run the same rootless BuildKit image and ask for the same thing: | Setting | Value | | -------------------------- | ------------------------------------- | | `runAsUser` / `runAsGroup` | 1000, with `runAsNonRoot: true` | | `allowPrivilegeEscalation` | `true` | | Capabilities | drop `ALL`, add `SETUID` and `SETGID` | | Seccomp profile | `Unconfined` | The pod also runs with `automountServiceAccountToken: false`, because the rootless image has a read-only root filesystem and the projected token mount fails against it. The counter-intuitive part is that the *rootless* builder asks for more privilege rather than less. BuildKit runs its daemon inside a user namespace, and RootlessKit sets that up by executing the setuid `newuidmap` / `newgidmap` helpers. The kernel won't honour those when `allowPrivilegeEscalation` is false, and it refuses to exec a binary whose file capabilities exceed the bounding set — hence `SETUID` and `SETGID`. `CLONE_NEWUSER` is then blocked by most default seccomp profiles, hence `Unconfined`. The daemon itself runs as uid 1000 and needs nothing beyond that. {% hint style="warning" %} This does not satisfy the `restricted` Pod Security Standard, which requires `allowPrivilegeEscalation: false` and a `RuntimeDefault` seccomp profile. The build namespace needs `baseline`, or an exception from whatever enforces policy on your cluster — a Kyverno or OPA rule scoped to that namespace. Without one the build pod will be rejected before it starts, and builds have to run outside the cluster instead. {% endhint %} ## Where built images are pushed `defaultImageRegistry` is the registry that built images are pushed to and pulled from. ```yaml build: defaultImageRegistry: url: nexus.example.com # required — registry URL, no trailing slash insecure: false # optional, default: false — see below caPath: /etc/ssl/certs/internal-ca.crt # optional — CA verifying the registry's certificate auth: # optional — omit for an anonymous registry type: basic username: { ref: env, name: NEXUS_USERNAME } password: { ref: env, name: NEXUS_PASSWORD } ``` ### Registries with a private CA If the registry presents a certificate from a private or internal CA, point `caPath` at your CA bundle instead of reaching for `insecure` — verification stays on. The path is read by the executor service, so mount the bundle into its pods the same way as in the [Vault TLS example](/self-managed/configuration/vault.md#tls), with `extraVolumes` on the `api` and `worker` workloads. Zerve ships the certificate into the build pod itself, so nothing has to exist in the build namespace beforehand. `caPath` is per registry and covers the base image pull, the push and the layer cache. Like `insecure`, it applies to the build only — see the warning below. ### Registries without trusted TLS `insecure` marks a registry served over plain HTTP, or over TLS with a certificate the builder does not trust. The typical use case is an in-cluster registry deployed with a test install. ```yaml build: defaultImageRegistry: url: registry.zerve-system.svc.cluster.local:5000 insecure: true ``` The flag is per registry, so a build that pushes to an insecure registry can still pull its base image from a secured one. Registries are matched on the exact `url` string, as credentials are, so `nexus.example.com` and `nexus.example.com:443` count as two different registries. {% hint style="danger" %} `insecure` applies to the **build** only. Pulling the finished image is done by the container runtime — the kubelet, for Kubernetes execution — which needs its own configuration to accept an untrusted registry. This setting does not enable run pods to pull from an HTTP registry. {% endhint %} ### Additional registries `extraImageRegistries` lists additional registries Zerve authenticates against when pulling base images. Same structure as `defaultImageRegistry`, as a list. ```yaml build: extraImageRegistries: - url: https://index.docker.io/v1/ auth: type: basic username: { ref: env, name: DOCKERHUB_USER } password: { ref: env, name: DOCKERHUB_PASSWORD } - url: registry.zerve-system.svc.cluster.local:5000 insecure: true ``` ## Language runtimes `imageConfigs` defines the base image and package sources for each language runtime. `python` is required; `r` is required only if `r` is listed under [`execution.languages`](/self-managed/configuration/code-execution.md). {% tabs %} {% tab title="Python" %} ```yaml build: imageConfigs: python: imageRepo: zerve-docker/python-executor # required — repository for built images, # relative to defaultImageRegistry # optional — defaults to python:3.11-slim-bookworm on Docker Hub; mirror it baseImage: nexus.example.com/docker/library/python:3.11-slim-bookworm uvVersion: "0.10.12" # optional — uv release installed into the image from defaultIndex defaultIndex: # required — Python package index for uv and user dependencies name: zerve-python url: https://nexus.example.com/repository/python-group/simple auth: # optional type: basic username: { ref: env, name: NEXUS_USERNAME } password: { ref: env, name: NEXUS_PASSWORD } aptSources: # optional — replaces the base image's apt sources - types: [deb] uris: ['https://nexus.example.com/repository/debian'] suites: [bookworm, bookworm-updates] components: [main] signedBy: /usr/share/keyrings/debian-archive-keyring.gpg aptAuth: # optional — credentials for private apt repos - machine: nexus.example.com login: { ref: env, name: NEXUS_USERNAME } password: { ref: env, name: NEXUS_PASSWORD } requiredAptPackages: # optional — system packages baked into every image - build-essential - libpq-dev extraCaCertPaths: # optional — CAs to trust during the build; see below - /etc/ssl/certs/internal-ca.crt ``` A Debian (bookworm) base image is recommended: `aptSources` and `aptAuth` use deb822 and netrc format respectively, and the runner's build dependencies use Debian package names. {% hint style="info" %} `uvVersion` pins the uv release used to resolve and install Python packages. uv is installed from `defaultIndex`, so the index must serve a `uv` wheel at that version. (Older sample manifests set `uvImage` instead; that key is no longer used and is ignored if present.) {% endhint %} {% endtab %} {% tab title="R" %} ```yaml build: imageConfigs: r: imageRepo: zerve-docker/r-executor # required # optional — defaults to posit/r-base:4.5-bookworm on Docker Hub; mirror it baseImage: nexus.example.com/docker/posit/r-base:4.5-bookworm defaultIndex: # required — R package repository name: r-group url: https://nexus.example.com/repository/r-group auth: type: basic username: { ref: env, name: NEXUS_USERNAME } password: { ref: env, name: NEXUS_PASSWORD } aptSources: [ ] # optional — same shape as Python aptAuth: [ ] # optional requiredAptPackages: [] # optional extraCaCertPaths: [ ] # optional — same shape as Python ``` R packages are installed with `pak`. Use a repository that serves Linux binaries for your base image's distribution — a Posit Package Manager `__linux__/bookworm` URL, or a Nexus R group proxying one — as a source-only CRAN mirror makes builds considerably slower. {% endtab %} {% endtabs %} ### Package indexes with a private CA `caPath` on a registry covers what the builder pulls and pushes. It does not cover what happens *inside* the build: `apt-get`, `pip`, `uv`, `pak` and `git` make their own TLS connections to your apt mirror and package indexes. If those are served with a private CA, list the bundles in `extraCaCertPaths`, per language. These are paths on the executor service, mounted the same way as `caPath`. Zerve copies them into the build context and the build steps that need them are pointed at the result, combined with the base image's own trust store so public hosts keep working. The distribution trust store in the image is not modified and nothing is registered as image-wide trust, so user code running in the finished Environment does not pick these CAs up. Rotating a CA invalidates the layer cache from that point, so the next build reinstalls packages. ### Runner package Every Environment image includes a Zerve runner package, which executes user code inside the image. Releases ship it inside the `executor` images under `/app/includes`, and that's what `runnerPackage` defaults to — `/app/includes/zerve.whl` for Python, `/app/includes/Rzerve.tar.gz` for R. Leave the field out and every Environment gets the runner matching your release. Setting it explicitly is for pinning a build other than the bundled one, usually at Zerve's request while chasing down a problem. The alternatives to `file` all fetch over the network, so they're of limited use to a self-managed install: | Source | Fields | Notes | | ------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `file` | `path` (ref) | A `.whl` or `.tar.gz` on the container filesystem — what the default uses. Versioned by the hash of its contents, so replacing the file triggers a rebuild. | | `index` | `name`, `version` | Installs from `defaultIndex`, if you've mirrored the package into your own index. | | `url` | `name`, `url` (ref) | A direct download URL. | | `github_repo` | `repo`, `ref` | A `Zerve-AI` repository at a commit or tag; needs outbound access to GitHub. | However it resolves, the runner version is an input to the preseeded-environment sync check below: a new version queues a rebuild of every organisation-wide Environment. ### Authenticating to the package index `defaultIndex.auth` accepts `basic` (username/password) value only. ## Pre-seeded Environments `prebuiltImages` is the set of ready-made Environments Zerve provides to the whole organisation. ```yaml prebuiltImages: python: images: # required — preset names to make available - Minimal - "ML Starter" default: Minimal # required — must be one of `images` r: images: - Minimal default: Minimal ``` `default` is the Environment to be used by default in all Notebooks, per language. It must appear in that language's `images` list, otherwise the manifest is rejected. The names are not free-form: each one refers to a package set from Zerve's built-in preset list. Unrecognised names are skipped with a warning during the sync described below, and the Environment is never created. Supported image names are: **Python** * `Default` * `Minimal` * `ML Starter` * `Deep Learning (CPU)` * `Deep Learning (CUDA 12.6)` * `Natural Language Processing` * `Computer Vision` * `Geospatial` * `Spark Connect` * `FastAPI` * `Streamlit` * `Gradio` * `Dash` * `Flask` **R** * `Default` * `Minimal` * `ML Starter` * `Natural Language Processing` * `Computer Vision` * `Geospatial` All of these images ship a sensible default environment for a specific work domain, but the choices are opinionated and library versions might not match your expectations. You will be able to create Environments based on your own requirements in the product directly. ### Building the pre-seeded Environments Each listed preset is built on install and upgrade by a post-install job, which reconciles the list against the existing Environments. ```yaml executor: workloads: syncDefaultImages: enabled: true ``` Enable it once the rest of the manifest is in place. It is the first component to exercise the manifest end to end, so a misconfigured registry, builder or package index surfaces as a failed install hook rather than as a broken Environment later. {% hint style="info" %} Every listed preset must be built before it can be used, and the larger presets install a substantial number of packages, which the install hook waits for. For a first install — particularly an air-gapped one, where the package indexes are not yet verified — start with `Minimal` and add presets once builds succeed. {% endhint %} `prebuiltImages` is optional. It defaults to Python / `Minimal` as the only Environment. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.zerve.ai/self-managed/configuration/builds.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.