Containers and Reproducible Deployments

Goal

Package a service so its software environment can be recreated, then record the exact image, model, runtime, settings, and host requirements needed to reproduce a deployment.

Start with two machines that both say latest

A teammate says, “The service works on my machine.” You start what appears to be the same deployment on another machine:

service:latest

but the behavior is different. The tag latest was rebuilt between the two runs, so the same human-readable label now points to different image content.

A container helps with the first part of this problem. It packages a filesystem and runtime settings so the same software environment can start on compatible machines. That removes many accidental differences in libraries and files.

But a container is only reproducible when you can identify the exact container you used. An image tag such as latest is a movable label. An image digest, often written with a sha256 value, identifies exact image content.

For example:

human label: service:latest
exact image: sha256:9f...

If an incident happened under the second value, you can point back to that exact image instead of guessing which build latest meant at the time.

Record more than the image

The image is only one part of an AI deployment. Also record the model and revision, quantization settings, serving-runtime version, service version, and important resource settings. The same container image can behave differently if it loads a different model artifact.

image_digest: sha256:9f…
model_revision: model-abc123
runtime_version: 0.9.2
quantization: int8

Pin build inputs when you can. Use dependency lock files, an exact base image, and a controlled set of build files. This reduces accidental changes between builds. Keep secrets out of the image and inject them at deployment time.

Containers do not include the hardware

OCI defines common container rules for files, processes, isolation, and resource limits. You do not need to implement OCI. Remember one idea: what is inside the image and how the host runs it are separate.

Hardware is outside the image too. A GPU container still needs compatible drivers, devices, and host runtime support. "Runs in a container" does not mean "runs on every machine."

Verify artifacts before readiness

Startup checks should verify required artifacts before the service becomes ready. If the model revision is missing, the tokenizer is incompatible, or device memory is too small, readiness should fail before traffic arrives.

Keep deployment settings reviewable. Environment variables, memory limits, ports, model locations, and rollout settings should come from version-controlled or otherwise auditable configuration, not from forgotten shell commands.

To reproduce a deployment, keep four pieces of evidence: the exact image, model/runtime settings, environment needs, and repeatable startup/health checks. Together they help you rebuild an incident or roll back to a known version.

Predict

Two deployments both use the tag service:latest, but the tag was rebuilt between them. Is the tag enough to prove identical image content?

No; an immutable image digest provides stronger evidence of the exact image contentYes; a tag name always refers to one permanent imageYes, if both deployments use the same API port

Run the Docker-environment Lab preflight

This activity is registered for the Docker-oriented production environment used in the second half of Level 14. Start with the deterministic Python preflight so service-logic failures remain distinguishable from container, GPU, cluster, or credential setup problems.

Run:

python3 labs/notebooks/level-14/l14-09-deployment-identity.py

The Lab compares a mutable image tag with immutable deployment identity fields.

1. Run it unchanged. Both the tag comparison and the identity comparison should report same.
2. In candidate, change only "tag": "serve:1.2" to "tag": "serve:latest". Before rerunning, predict which comparison should change while the content identity stays fixed.
3. Rerun and confirm the tag changed but the immutable identity is still the same.
4. Restore the tag. Then, as a separate one-variable experiment, predict what happens if only "image_digest": "sha256:aaa" becomes "sha256:bbb". Make that single edit and rerun.
5. Confirm image_digest appears in changed identity fields. A tag change and an image-content change are not equivalent evidence.

Loading lab…

Quick Check

1. What does an image digest provide?

A replacement for application configurationAn identifier for exact image contentA guarantee that the model quality is acceptable

2. What should remain outside the image?

The application source codeDeployment-injected secretsThe service executable itself

3. Why record model revision separately from image identity?

Because model revision stops mattering after trainingBecause containers cannot read model filesBecause one service image can load different model artifacts

0 of 3 questions answered.

Explain it back

Write a deployment identity record containing image digest, service version, model revision, runtime version, and config hash. Explain what each field lets you reproduce.

Key Takeaways

• Containers package the runtime environment but do not replace version evidence.
• Prefer immutable image identity for reproducibility.
• Record model and serving-runtime versions separately.
• Keep secrets outside image content.
• Readiness should verify model, device, and startup requirements.

Next Lesson

Next, L14.10 — Observability for AI Systems connects API, queue, runtime, and deployment evidence.

References

• Docker, Documentation.
• Open Container Initiative, Runtime Specification.