Published on

Shipping Software Updates Into an Air-Gapped Environment

Authors

A critical CVE drops on a Friday. In your SaaS, this is a non-event. You cut a patch, your pipeline builds it, canary takes it, and by Monday every tenant is running the fix. Nobody outside the on-call channel even knows it happened.

Now picture the same CVE, but the affected system runs inside a customer's air-gapped network. No inbound access. No outbound access. Your deploy pipeline ends at their firewall, and on the other side of that firewall is a box you have never seen, three internal networks deep, that has not talked to the public internet since the day it was installed. The fix is done in twenty minutes. Getting it onto that box takes three weeks and a change-advisory board.

This is the part of enterprise deployment that nobody writes about, because it is unglamorous and specific and there is no vendor slide for it. But if you sell into defense, healthcare, finance, or critical infrastructure, this is the difference between a product that survives an audit and one that quietly rots because it can never be patched. This post is about how you actually move a trustworthy update across an air gap, and how you keep a system patchable for the years it will live out there.

The mental shift: delivery is a supply chain that ends at a wall

The first thing to internalize is that everything your SaaS pipeline does for free, you now do by hand, on the far side of a boundary you cannot cross.

In a connected deployment, the artifact and its trust travel together over the network at the moment of deploy. The node pulls the image from your registry, the registry proves who you are over TLS, the package manager checks a signature against a keyserver it can reach. Every link in that chain assumes connectivity.

Air-gapped delivery breaks every one of those assumptions at once. So you stop thinking of "deploy" as a network event and start thinking of it as producing a physical artifact that carries its own proof. Everything the far side needs to trust and run the update has to be inside the bundle, because the far side cannot ask anyone anything. The bundle is the unit of delivery, the unit of trust, and the unit of audit. Get that object right and the rest is logistics.

There are four hard problems, in order: build a complete bundle, make it self-verifying, get it across the gap, and apply it without breaking the system you cannot see. Then one problem that outlives all of them: keep it patchable when the customer is running a version from eighteen months ago.

Build a complete, deterministic bundle

"Complete" is doing a lot of work in that sentence. A connected install pulls dozens of things implicitly: base images, sidecars, the ingress controller, a database operator, language runtime packages, the vulnerability database your scanner uses. Every single implicit pull is a landmine in an air-gapped environment. The install will get 90 percent of the way through and then block forever on one docker pull of some transitive dependency nobody remembered.

So the bundle has to contain the full closure of everything the system needs to run, resolved ahead of time. In a Kubernetes-shaped product that means:

  • Every container image, including the ones you did not write (ingress, cert-manager, the CSI driver, your database)
  • The Helm charts or manifests, with image references rewritten to the customer's internal registry
  • An SBOM for every image
  • A signature and provenance attestation for the whole thing
  • The installer and its preflight checks
  • A manifest that lists exactly what is inside, with a digest for each item

The way to collect images reliably is to resolve everything by digest, never by tag. Tags are mutable and they drift. A :latest or even :1.24.3 that resolved to one image when you built the bundle can resolve to a different one later, and in air-gapped delivery "later" might be a year. Pin to sha256 digests everywhere and the bundle becomes reproducible.

A simple collection step, using oras to pull everything into a local OCI layout, looks like this:

# images.txt is a digest-pinned list, e.g.
#   registry.example.com/app@sha256:9f2a...
#   registry.example.com/ingress@sha256:4c81...
mkdir -p bundle/oci
while read -r image; do
  oras copy --to-oci-layout "$image" "bundle/oci:$(basename "${image%@*}")"
done < images.txt

# Charts, installer, SBOMs, and the manifest go alongside
cp -r charts installer sboms bundle/
sha256sum bundle/oci/* bundle/charts/* bundle/installer/* > bundle/MANIFEST.sha256
tar -C bundle -czf release-2.7.4.tar.gz .

The MANIFEST.sha256 is the spine of the whole thing. It is how the far side confirms, byte for byte, that what arrived is what you shipped, before it trusts a single file.

Make the bundle self-verifying

A checksum manifest proves the bundle is intact. It does not prove the bundle is yours. Anyone can produce a tarball with a valid internal manifest. In a regulated environment the customer's security team will, correctly, refuse to install anything they cannot cryptographically tie back to you. And they have to do that verification with no ability to call your key server, your registry, or any external trust root.

The answer is to sign the bundle with a key whose public half the customer already holds, established once at onboarding, and to ship the signature and provenance inside the bundle. cosign handles this well, and its keyful mode works entirely offline, which is exactly what you want here. Avoid the keyless flow that depends on a public transparency log and an external OIDC provider, because that flow assumes connectivity the far side does not have.

Sign on your side:

# One-time: the customer receives cosign.pub through an out-of-band channel
# during onboarding and pins it in their verification policy.
cosign sign-blob --key cosign.key release-2.7.4.tar.gz > release-2.7.4.tar.gz.sig

# Ship the SBOM and provenance as attestations too, so the far side can audit
# what is inside without unpacking and trusting it first.
cosign attest-blob --key cosign.key \
  --predicate sbom.spdx.json --type spdxjson \
  release-2.7.4.tar.gz > release-2.7.4.tar.gz.att

Verify on the far side, air-gapped, using only the pinned public key:

cosign verify-blob --key cosign.pub \
  --signature release-2.7.4.tar.gz.sig release-2.7.4.tar.gz \
  || { echo "SIGNATURE INVALID, refusing to proceed"; exit 1; }

# Only after the signature passes do we trust the manifest and check integrity
tar -xzf release-2.7.4.tar.gz -C staged/
( cd staged && sha256sum -c MANIFEST.sha256 ) \
  || { echo "MANIFEST MISMATCH, refusing to load"; exit 1; }

Two properties matter here. The verification fails closed: an invalid signature or a mismatched manifest stops the install, it does not warn and continue. And it uses nothing but the bundle and a public key the customer already trusts. No network, no external log, no OCSP call. That is the whole game in an air gap: trust that travels inside the artifact.

Get it across the gap

Now the physical part. There are three common transfer mechanisms, in increasing order of paranoia.

A DMZ transfer host or an internal artifact mirror that both sides can reach at different times is the easiest. You push the signed bundle to a staging area on the boundary, and an operator inside pulls it across. There is still an air gap in the sense that no session ever spans both sides, but the bundle moves over a wire.

A data diode is a one-way physical link: photons go in one direction and there is literally no return path, enforced in hardware. This is common in defense and critical infrastructure. The consequence for you is that there is no acknowledgment, no retry, no TCP. You are shipping a blob over a one-way pipe, so the transfer protocol has to tolerate loss by design: forward error correction, redundant sends, and a manifest so the receiver can detect a truncated transfer and reject it.

Sneakernet is a removable drive, physically carried, scanned, and mounted on the inside. Slower and more human, but for the highest-assurance environments it is still the norm.

You do not get to choose which one the customer uses, so the bundle has to be transport-agnostic. That is another argument for the self-verifying design: whether the bundle arrives over a diode, a DMZ hop, or a USB drive that sat in a security desk for a day, the receiver runs the exact same verification, and either the signature is valid or it is not. The transport is someone else's problem. The trust is yours, and it rides inside the file.

One practical note: size discipline matters more than you expect. A bundle that is 40 GB because you shipped every layer of every image uncompressed will make a diode transfer take hours and a sneakernet process take a day. Deduplicate image layers, compress, and split large bundles into verifiable parts so a failed transfer does not restart from zero.

Load it without any external pulls

The bundle is inside. Now it has to run, and this is where the second wave of implicit-connectivity assumptions bites. Kubernetes will happily try to pull an image from docker.io the moment a pod schedules, and in an air-gapped cluster that pull hangs until it times out and the pod crash-loops. Every image reference in every manifest has to point at the customer's internal registry, and nothing may ever reach outward.

So the load step pushes every image from the bundle into the internal registry, and the charts are rendered with image references rewritten to that registry:

# Push every image from the OCI layout into the customer's internal registry
for img in staged/oci/*; do
  name=$(basename "$img")
  oras copy --from-oci-layout "$img" "internal-registry.corp.local/app/$name"
done

# Render charts against the internal registry, never the public one
helm template app staged/charts/app \
  --set global.registry=internal-registry.corp.local/app \
  --set global.imagePullPolicy=IfNotPresent > rendered.yaml

Then you enforce the invariant rather than trusting it. A cluster policy that rejects any image not from the internal registry turns "we should not pull externally" into "we cannot," which is the only version an auditor believes. With a policy engine like Kyverno:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: only-internal-registry
spec:
  validationFailureAction: Enforce
  rules:
    - name: block-external-images
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "images must come from the internal registry"
        pattern:
          spec:
            containers:
              - image: "internal-registry.corp.local/*"

Set imagePullPolicy to IfNotPresent, not Always. An Always policy sends the kubelet back to the registry on every restart, which is wasted work in a static environment and a failure mode if the registry is briefly unavailable. In an air-gapped cluster the image you loaded is the image you run, so pull it once and keep it.

Apply atomically, and make rollback boring

Here is the constraint that shapes everything about the apply step: you are not the one running it. An operator inside the customer's environment runs the upgrade, following your runbook, and if it goes wrong you cannot SSH in and fix it. You might not even hear about the failure for hours. So the upgrade has to be atomic, self-checking, and trivially reversible by someone who did not write the software.

Atomic means the system is either fully on the old version or fully on the new one, never a torn state where half the services upgraded and half did not. Reversible means rollback is a single, obvious command that a stressed operator can run at 2 a.m. without understanding the internals.

The pattern that holds up is a versioned release with a pointer you can flip. Stage the new version completely alongside the old, run health checks against it, and only then move the pointer. If the checks fail, the pointer never moves and the old version is still live.

# Stage the new release next to the current one, don't touch live yet
ln -s /opt/app/releases/2.7.4 /opt/app/releases/staged

# Preflight: schema compatibility, config validity, dependency versions
/opt/app/releases/staged/bin/preflight --current /opt/app/current \
  || { echo "preflight failed, live version untouched"; exit 1; }

# Health-check the staged release in place, then flip the current pointer
# atomically. A symlink swap via rename() is atomic on POSIX filesystems.
ln -sfn /opt/app/releases/2.7.4 /opt/app/current.new
mv -T /opt/app/current.new /opt/app/current
systemctl reload app

# Rollback is the same move in reverse, and it is in the runbook verbatim:
#   ln -sfn /opt/app/releases/2.7.3 /opt/app/current.new
#   mv -T /opt/app/current.new /opt/app/current && systemctl reload app

On Kubernetes the same idea shows up as keeping the previous Helm release and its images resident so helm rollback app works with no network, or as a blue-green swap where you cut traffic back to the old deployment. The mechanism differs; the property is identical. Rollback must not require anything the customer does not already have on the inside.

The failure mode to design against is the irreversible schema migration. If your upgrade runs a migration that drops a column or rewrites data in place, rollback is now impossible even though your binaries swap cleanly, because the old version cannot read the new schema. In an environment where you cannot supervise the upgrade, that is unacceptable. Which leads directly to the problem that outlives every single release.

The version-skew tax nobody budgets for

In your SaaS you run exactly one version in production, the one you shipped this morning. You can make a breaking change on Tuesday and delete the old code path on Wednesday, because there is nothing left running it.

Air-gapped customers destroy that assumption. Upgrades are a manual, scheduled, change-controlled event, and they happen slowly. It is completely normal to have customers running a version you shipped eighteen months and a dozen releases ago. Some of them will skip versions. You are not supporting one version, you are supporting the whole spread at once, and every schema and API change you make has to survive that spread.

This has hard architectural consequences, and they are the real cost of this market.

Migrations must be backward compatible, always. The discipline is the expand-and-contract pattern: never change a column, add a new one, write to both, backfill, and only remove the old one several releases later once you are certain nothing in the supported window still reads it. A migration is safe only if the previous version can still run against the migrated schema. That is what makes rollback possible, and rollback possible is what makes the whole update model safe.

APIs need a real deprecation window measured in releases, not weeks. An internal component talking to another internal component cannot assume they are on the same version during an upgrade, because upgrades are not atomic across every service. Version your internal APIs and keep old handlers alive across the supported window.

Feature flags ship as config the customer controls, not as a service you toggle. You cannot reach in and flip a flag. So a new feature lands dark, gated behind a config value the customer sets when they are ready, and the old path stays intact until the flag has been on across enough of the supported window that you can retire it.

Define the supported window explicitly and enforce it in the upgrade tool. "We support the last N releases" has to be a real number, published, and checked by preflight. If a customer on version 2.1 tries to jump straight to 2.9 and your migrations only chain safely across four releases, the upgrade tool must refuse and route them through the intermediate steps. Silent best-effort across an unbounded gap is how you corrupt someone's data.

None of this is exotic. It is the same backward-compatibility discipline good teams already aspire to. The difference is that in air-gapped delivery it is not aspirational. Skip it once and you ship an update that a customer cannot roll back and cannot recover from, in an environment where you have no hands on the box.

Keeping security current when nothing can phone home

One last nerve, because it is the reason this market exists. The whole point of shipping vulnerability fixes fast is undercut if the customer's scanner cannot tell them what is vulnerable in the first place. Vulnerability scanners depend on a feed that updates several times a day, and in an air gap that feed is as stranded as everything else.

So the vulnerability database becomes another signed artifact you ship on a cadence. A scanner like Trivy can run fully offline against a database you package and transfer the same way you transfer releases:

# On your side, on a schedule: pull the latest DB and bundle it
trivy image --download-db-only --cache-dir ./trivy-db
tar -czf trivy-db-2026-07-15.tar.gz -C trivy-db .
cosign sign-blob --key cosign.key trivy-db-2026-07-15.tar.gz > trivy-db-2026-07-15.tar.gz.sig

# Inside, after verifying the signature: scan with no network at all
trivy image --skip-db-update --offline-scan \
  --cache-dir /opt/trivy-db internal-registry.corp.local/app/app@sha256:9f2a...

The cadence of that database bundle is a promise you make to the customer, and it belongs in the contract. If you ship it monthly, they are a month behind the world on known vulnerabilities, and both of you should be clear-eyed about that. The realistic goal is not zero lag. It is a known, bounded, documented lag with a clean process to close it when something urgent lands.

What this really buys you

Every technique here comes back to one idea. In an air-gapped deployment you cannot reach across the boundary, so you push everything the far side needs across it once, inside a single object that carries its own proof, and you make the far side able to verify, apply, and reverse it without asking anyone anything.

That object, the signed self-verifying bundle, is the actual product surface in this market. Teams that treat it as an afterthought, a tarball someone scripts together the week before a deal closes, lose the deal or, worse, win it and then cannot patch the thing they sold. Teams that treat the bundle with the same rigor as the software inside it get to sell into the environments that pay the most and trust the least.

The CVE will still drop on a Friday. You still cannot reach the box. But if the bundle is right, the customer's operator can verify your fix, stage it, health-check it, and flip a pointer, with a rollback command sitting right there if it goes wrong. Three weeks and a change-advisory board, yes. But a patchable system at the end of it, which in a regulated environment is the only kind worth shipping.