Compiling a model
Put the compiler's own cache on the volume, so only the first worker pays the compile cost.
Introduction
torch.compile on an image classifier, with the compiler's own cache on the volume, so only the first worker pays the compile cost.
The first thing this example teaches is why it is a container at all: the code path has no compiler.
Building the image
# A -devel image, not a -runtime one. torch.compile hands its kernels to Triton,
# and Triton shells out to a C compiler at runtime, so the toolchain has to be in
# the image. This is the reason this example is a container at all.
FROM pytorch/pytorch:2.14.1-cuda13.0-cudnn9-devel
# --break-system-packages because this image's python is system managed and pip
# refuses otherwise (PEP 668). Inside a container there is no other environment
# to break.
RUN pip install --no-cache-dir --break-system-packages timm==1.0.30 pillow==13.0.0
# Set here rather than in code: torch reads all three as it loads, and the first
# import anywhere in the process has already fixed them.
ENV HF_HOME=/weights/huggingface \
TORCHINDUCTOR_CACHE_DIR=/weights/inductor \
TRITON_CACHE_DIR=/weights/triton \
PORT=8080
# Fail the build, not the first request. There is no GPU here, so this checks the
# imports rather than the model: enough to catch a wheel that does not load.
RUN python -c "import torch, timm; from PIL import Image; print(torch.__version__, timm.__version__)"
WORKDIR /app
COPY server.py /app/
EXPOSE 8080
CMD ["python", "/app/server.py"]The cache paths are set in ENV, not in code, because torch reads all of them as it loads, and the first import anywhere in the process has already fixed them. Setting them after that silently does nothing.
--break-system-packages is needed to pip install at all, because this image's python is system managed and pip refuses without it. A container has one environment, so there is nothing else to install into.
The last RUN imports torch, timm and pillow at build time. There is no GPU during a build, so it checks the imports and not the model. That catches a wheel that does not load. Without it, a broken wheel shows up only after a deploy and a request.
Declaring the contract
configVersion: 2
port: 8080
probes:
readiness: /ready
liveness: /health
timeouts:
requestSeconds: 120
startupSeconds: 2400
endpoints:
- path: classify
default: true
input:
type: object
additionalProperties: false
required: [image]
properties:
image:
type: string
description: The image to classify, base64 encoded, with no data URI prefix.
top:
type: integer
minimum: 1
maximum: 10
description: How many labels to return.
output:
type: object
properties:
labels:
type: array
description: The most likely labels, best first.
items:
type: object
properties:
label:
type: string
index:
type: integer
probability:
type: number
compiled:
type: boolean
description: Whether the graph this request ran was a compiled one.
seconds:
type: numberstartupSeconds asks for 2400, the maximum, because a first compile is longer than the 300 second default and the platform adds 30 minutes on top of that. A worker that has not answered by then is restarted, so a long compile looks like a crash loop.
The toolchain the compile needs
torch.compile passes its kernels to Triton, and Triton shells out to a C compiler at runtime. A code worker has none.
Measured from inside one with shutil.which:
| Tool | Found |
cc, gcc, g++, clang | none |
as, ld | none |
make, ninja | none |
nvcc, ptxas | none |
python3-config | /usr/local/bin/python3-config |
A code app that compiles anyway crash-loops: the container exits with code 3 on every attempt, and the task stays pending with no error while apps errors stays empty. The reason appears only in apps logs, inside a Triton traceback that ends Failed to find C compiler.
So the image is a -devel one, which ships the toolchain. A dependency you cannot install any other way is the clearest reason to choose the container path. A server you already wrote is a weaker reason.
Where the compile runs
"""The server the platform talks to, and the readiness gate that hides the compile.
Runware hands this server the request's ``payload`` member alone, not the
envelope around it, so a local curl and a real invocation send the same body.
"""
import base64
import io
import json
import os
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import torch
from PIL import Image
import timm
PORT = int(os.environ.get("PORT", "8080"))
MODEL = "resnet50.a1_in1k"
SIZE = 224
DEFAULT_TOP = 5
state = {"ready": False, "compiled": False, "error": None}
def warm():
"""Load, compile and exercise the model. Runs in a thread so the server can
answer its readiness probe with a 503 while this is still going."""
try:
model = timm.create_model(MODEL, pretrained=True).eval().cuda()
config = timm.data.resolve_data_config({}, model=model)
transform = timm.data.create_transform(**config)
try:
labels = timm.data.ImageNetInfo()
except Exception:
labels = None
# No cudagraphs. Capturing a graph records the addresses its tensors live
# at, and a worker that is checkpointed and restored does not get the same
# addresses back, so a captured graph replays against memory that moved.
compiled = torch.compile(model, mode="max-autotune-no-cudagraphs", dynamic=False)
# This pass is the compile. It writes the cache the volume keeps, and it is
# why readiness takes minutes the first time and seconds afterwards.
with torch.inference_mode():
compiled(torch.zeros(1, 3, SIZE, SIZE, device="cuda"))
state.update(model=compiled, transform=transform, labels=labels, compiled=True, ready=True)
except Exception as problem: # noqa: BLE001
state["error"] = f"{type(problem).__name__}: {problem}"
def classify(raw, top):
image = Image.open(io.BytesIO(base64.b64decode(raw, validate=True))).convert("RGB")
batch = state["transform"](image).unsqueeze(0).cuda()
started = time.time()
with torch.inference_mode():
probabilities = state["model"](batch).softmax(dim=-1)[0]
seconds = time.time() - started
scores, indices = probabilities.topk(top)
labels = state["labels"]
return {
"labels": [
{
"label": labels.index_to_description(int(i)) if labels else str(int(i)),
"index": int(i),
"probability": round(float(s), 5),
}
for s, i in zip(scores.tolist(), indices.tolist())
],
"compiled": state["compiled"],
"seconds": round(seconds, 4),
}
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def _send(self, status, body):
raw = json.dumps(body).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
def do_GET(self):
if self.path == "/health":
return self._send(200, {"status": "ok"})
if self.path == "/ready":
if state["error"]:
return self._send(500, {"status": "failed", "error": state["error"]})
if not state["ready"]:
# The whole cold start mechanism: no traffic until the compile is
# done, so no caller ever waits on it inside a request.
return self._send(503, {"status": "compiling"})
return self._send(200, {"status": "ok"})
return self._send(404, {"error": "not found"})
def do_POST(self):
if self.path.rstrip("/") not in ("/classify", ""):
return self._send(404, {"error": f"no endpoint at {self.path}"})
if not state["ready"]:
return self._send(503, {"error": "still compiling"})
try:
length = int(self.headers.get("Content-Length", "0"))
payload = json.loads(self.rfile.read(length) or b"{}")
except (ValueError, json.JSONDecodeError) as problem:
return self._send(400, {"error": f"bad body: {problem}"})
top = payload.get("top")
try:
return self._send(200, classify(payload["image"], int(top) if top else DEFAULT_TOP))
except KeyError:
return self._send(400, {"error": "image is required"})
except Exception as problem: # noqa: BLE001
return self._send(500, {"error": str(problem)})
def log_message(self, fmt, *args):
print(f"{self.address_string()} {fmt % args}", flush=True)
threading.Thread(target=warm, daemon=True).start()
ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever()The server starts, returns 503 on /ready, and compiles in a background thread. The platform sends no traffic until that turns 200, so no caller ever waits on a compile inside a request.
A server that reports ready before its weights are loaded receives a request it cannot serve.
mode="max-autotune-no-cudagraphs" is deliberate. Capturing a CUDA graph records the addresses of its tensors, and a worker that is checkpointed and restored does not get the same addresses back, so a captured graph replays against memory that moved.
Deploying the app
runware serverless deploy --from https://runware.ai/docs/serverless/examples/compiled-model.ziprunware serverless deploy --container . \
--id compiled-model \
--gpu-type rtx-pro-6000 \
--volume /weights--container points at the directory, not at a file, and that directory has the Dockerfile and the container.yaml beside whatever the Dockerfile copies.
Caches on a volume
A volume stores anything your app generates once and should not generate again, not only weights. The volumes page lists a compiled kernel cache as one of them.
The weights here are 100MB and the compile is the expensive part. Loading a model is the inverse, and both examples exist for that reason.
Calling the app
import { createClient } from '@runware/sdk'
const client = await createClient({ apiKey: process.env.RUNWARE_API_KEY })
const task = await client.invoke({
appId: 'compiled-model',
endpointPath: 'classify',
payload: {
image: '<base64 JPEG or PNG>',
top: 5
}
}, {
deliveryMethod: 'sync'
})import asyncio
import os
from runware import InvokeOptions, Runware
async def main():
client = Runware(api_key=os.environ["RUNWARE_API_KEY"])
try:
task = await client.invoke(
"compiled-model",
"classify",
{
"image": "<base64 JPEG or PNG>",
"top": 5
},
options=InvokeOptions(delivery_method="sync"),
)
finally:
await client.close()
asyncio.run(main())curl https://api.serverless.runware.ai/v1/apps/compiled-model/invoke-sync/classify \
-H "Authorization: Bearer $RUNWARE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"taskId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
"payload": {
"image": "<base64 JPEG or PNG>",
"top": 5
}
}'cat <<'EOF' | runware serverless apps invoke compiled-model classify --sync -f -
{
"image": "<base64 JPEG or PNG>",
"top": 5
}
EOF{
"taskId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
"payload": {
"image": "<base64 JPEG or PNG>",
"top": 5
}
}A container app receives the payload your server expects, with no parameter name wrapping it, because there is no handler signature to bind to.
The labels come back at output.labels, best first. A container app has no predictions wrapper: output is your server's response body as it was. A code app puts its handler result at output.predictions[0].
seconds is the forward pass alone, so you can see the compile's effect with no queue time in the number.
What the run cost
| Call | Time |
| Build | 10m41s |
| First call, empty volume: weights plus the compile | 14m24s |
| A restart, compile cache already on the volume | 19s |
| A warm worker | 1.1s |
| The forward pass itself, warm | 3.2ms |
The first compile runs 45 times as long as a restart that reads the cache. Every worker after the first one is cheap for that reason.
The first forward pass after a restart took 2.59 seconds against 3.2ms warm, so a worker that has read the cache still needs one pass to reach that speed.
max-autotune makes the first start this long. mode="default" compiles in a fraction of the time and keeps most of the benefit. Use it when you do not want to pay a 14 minute cold start even once.