---
title: Resuming a long job | Runware Docs
url: https://runware.ai/docs/serverless/examples/resumable-job
description: Checkpoint to storage every worker can reach, so the next call picks the job up where the last one left it.
relatedDocuments:
  - https://runware.ai/docs/serverless/volumes
  - https://runware.ai/docs/serverless/secrets
  - https://runware.ai/docs/serverless/compute-and-scaling
---
## Introduction

Work that takes longer than you want to hold a connection open, driven as **many short calls that resume**. Each call runs a slice, writes a checkpoint, and tells you whether to call again.

The training loop here is a small stand-in. What the example is about is the checkpoint and where it goes.

## The app

**model.py**:

```python
"""Resuming a long job: run a slice, checkpoint, answer, resume.

The training here is a stand-in, deliberately small, because the subject is the
checkpoint and where it goes. Swap the three lines in ``_step`` for real work and
nothing else about the shape changes.
"""

from __future__ import annotations

import io
import os

from pydantic import BaseModel, Field

from runware_serverless import serve

STEPS_PER_CALL = 200
"""Small enough that every call returns progress you can watch, and that a worker
lost between two calls costs one slice rather than the whole job.

A code app does not apply a request timeout, so this is not a limit being worked
around. It is a choice about what a failure costs."""

BUCKET = os.environ.get("S3_BUCKET", "")
ENDPOINT = os.environ.get("S3_ENDPOINT", "")
REGION = os.environ.get("S3_REGION", "")
"""Any S3-compatible store. All three come from secrets, which the platform puts
in the worker's environment, so nothing about the store is in this file.

Leave ``S3_ENDPOINT`` and ``S3_REGION`` unset for AWS and boto3 resolves both the
usual way. Other stores need the endpoint, and some, Cloudflare R2 among them,
want the region set to ``auto``."""

class Request(BaseModel):
    job: str = Field(
        description="Names the checkpoint. The same name resumes, a new one starts over.",
        min_length=1,
        max_length=64,
        pattern=r"^[A-Za-z0-9._-]+$",
    )
    total_steps: int = Field(default=1000, ge=1, le=1_000_000)

class Progress(BaseModel):
    job: str
    steps_done: int
    total_steps: int
    done: bool = Field(description="False means call again with the same job name.")
    loss: float
    resumed_from: int = Field(description="The step the checkpoint this call started from held.")

@serve
class ResumableJob:
    requirements = ["torch==2.14.1", "boto3==1.43.123"]

    def load(self) -> None:
        if hasattr(self, "store"):
            return

        import boto3
        import torch

        if not BUCKET:
            raise RuntimeError(
                "S3_BUCKET is unset. Attach the secrets before invoking: "
                "runware serverless secrets attach resumable-job --secret s3"
            )

        self.torch = torch
        self.store = boto3.client(
            "s3",
            endpoint_url=ENDPOINT or None,
            region_name=REGION or None,
            aws_access_key_id=os.environ["S3_ACCESS_KEY_ID"],
            aws_secret_access_key=os.environ["S3_SECRET_ACCESS_KEY"],
        )

    def predict(self, request: Request) -> Progress:
        state = self._restore(request.job)
        started_at = state["steps_done"]

        model, optimizer = self._build(state)
        loss = 0.0

        remaining = min(STEPS_PER_CALL, request.total_steps - started_at)
        for _ in range(max(remaining, 0)):
            loss = self._step(model, optimizer)

        steps_done = started_at + max(remaining, 0)
        self._save(request.job, model, optimizer, steps_done)

        return Progress(
            job=request.job,
            steps_done=steps_done,
            total_steps=request.total_steps,
            done=steps_done >= request.total_steps,
            loss=round(loss, 6),
            resumed_from=started_at,
        )

    def _key(self, job: str) -> str:
        return f"checkpoints/{job}.pt"

    def _restore(self, job: str) -> dict[str, object]:
        try:
            raw = self.store.get_object(Bucket=BUCKET, Key=self._key(job))["Body"].read()
        except self.store.exceptions.NoSuchKey:
            return {"steps_done": 0}

        # weights_only is the default from torch 2.6 and this is our own file, but
        # a checkpoint read back from object storage is still input, so say so.
        return self.torch.load(io.BytesIO(raw), weights_only=True)

    def _build(self, state: dict[str, object]) -> tuple[object, object]:
        torch = self.torch

        torch.manual_seed(0)
        model = torch.nn.Sequential(
            torch.nn.Linear(64, 256), torch.nn.GELU(), torch.nn.Linear(256, 1)
        ).to("cuda")
        optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)

        if "model" in state:
            model.load_state_dict(state["model"])
            optimizer.load_state_dict(state["optimizer"])

        return model, optimizer

    def _step(self, model: object, optimizer: object) -> float:
        torch = self.torch

        inputs = torch.randn(256, 64, device="cuda")
        targets = inputs.sum(dim=1, keepdim=True)

        loss = torch.nn.functional.mse_loss(model(inputs), targets)
        optimizer.zero_grad(set_to_none=True)
        loss.backward()
        optimizer.step()

        return float(loss.detach())

    def _save(self, job: str, model: object, optimizer: object, steps_done: int) -> None:
        buffer = io.BytesIO()
        self.torch.save(
            {
                "model": model.state_dict(),
                "optimizer": optimizer.state_dict(),
                "steps_done": steps_done,
            },
            buffer,
        )
        self.store.put_object(
            Bucket=BUCKET, Key=self._key(job), Body=buffer.getvalue()
        )
```

## Deploying the app

One secret holds one value, and `--value-file -` reads it from stdin so it never reaches the process list.

```bash
printf '%s' "$S3_BUCKET"            | runware serverless secrets set S3_BUCKET --value-file -
printf '%s' "$S3_ACCESS_KEY_ID"     | runware serverless secrets set S3_ACCESS_KEY_ID --value-file -
printf '%s' "$S3_SECRET_ACCESS_KEY" | runware serverless secrets set S3_SECRET_ACCESS_KEY --value-file -
```

Those three exist before the deploy does, because the app attaches them as it is created.

resumable-jobrtx-pro-6000[source](https://github.com/Runware/runware-serverless-examples/tree/main/resumable-job)

[One-click deploy](https://runware.ai/serverless/apps?template=resumable-job)

**From a URL**:

```bash
runware serverless deploy --from https://runware.ai/docs/serverless/examples/resumable-job.zip
```

**From a clone**:

```bash
runware serverless deploy model.py \
  --id resumable-job \
  --gpu-type rtx-pro-6000 \
  --secret S3_BUCKET \
  --secret S3_ACCESS_KEY_ID \
  --secret S3_SECRET_ACCESS_KEY
```

The secret names are the environment variable names the code reads, so nothing needs mapping. Attaching one under a different name is `--secret NAME=ENV_VAR`.

**`S3_ENDPOINT` and `S3_REGION` are the optional two.** Leave both unset and boto3 resolves them the way it does for AWS. Any other S3-compatible store needs the endpoint, and some want the region as well.

<table><tbody><tr><td>Store</td><td><code>S3_ENDPOINT</code></td><td><code>S3_REGION</code></td></tr><tr><td>AWS S3</td><td>unset</td><td>unset</td></tr><tr><td>Cloudflare R2</td><td><code>https://&lt;account&gt;.r2.cloudflarestorage.com</code></td><td><code>auto</code></td></tr><tr><td>MinIO and most others</td><td>the server's URL</td><td>unset, or whatever it expects</td></tr></tbody></table>

Neither is a credential, so they go in as plain variables rather than secrets:

```bash
runware serverless deploy model.py --id resumable-job --gpu-type rtx-pro-6000 \
  --secret S3_BUCKET --secret S3_ACCESS_KEY_ID --secret S3_SECRET_ACCESS_KEY \
  --env-var S3_ENDPOINT=https://<account>.r2.cloudflarestorage.com \
  --env-var S3_REGION=auto
```

## What ends a long run

`requestTimeoutSecs` applies to **container apps only**. A code app stores the field and does not apply it, so a long handler is not interrupted at some published ceiling.

What ends a long run is everything else. A worker is checkpointed and restored. `load` is retried and your class re-instantiated when the GPU is short at startup. An idle worker is scaled away.

**A long job on one worker loses all of its work if that worker goes away.**

## Where the checkpoint goes

A [volume](https://runware.ai/docs/serverless/volumes) is **node-local**: the worker that serves your next call may be on a different node and find an empty directory.

A volume makes a *restart* cheap, it does not make a *resume* possible.

> [!WARNING]
> A volume is the right place for weights, a compile cache or an index you rebuild on start, because losing one costs time and nothing else. It is the wrong place for the only copy of your progress.

So the checkpoint goes to object storage, which every worker can reach regardless of where it landed.

## Resuming by job name

The same name resumes the same checkpoint, a new name starts fresh.

There is no job registry to keep in step with the storage. `resumed_from` in each answer is the step the checkpoint it started from held, so a call that resumed correctly says so.

A job name becomes part of an object key, so `pattern` on the field keeps a slash or a `..` out of that key.

> [!WARNING]
> The check still costs a worker. Measured: a job named `../etc/passwd` is refused by this app after a worker has woken, not by the platform before it. The platform enforces the types it published and `additionalProperties`. The value constraints, `pattern`, `minimum`, `maximum` and a list's length, **all reach the worker**.

## Calling the app

**TypeScript**:

```typescript
import { createClient } from '@runware/sdk'

const client = await createClient({ apiKey: process.env.RUNWARE_API_KEY })

const task = await client.invoke({
  appId: 'resumable-job',
  endpointPath: 'predict',
  payload: {
    request: {
      job: 'nightly',
      total_steps: 1000
    }
  }
}, {
  deliveryMethod: 'sync'
})
```

**Python**:

```python
import asyncio
import os

from runware import InvokeOptions, Runware

async def main():
    client = Runware(api_key=os.environ["RUNWARE_API_KEY"])
    try:
        task = await client.invoke(
            "resumable-job",
            "predict",
            {
                "request": {
                    "job": "nightly",
                    "total_steps": 1000
                }
            },
            options=InvokeOptions(delivery_method="sync"),
        )
    finally:
        await client.close()

asyncio.run(main())
```

**cURL**:

```bash
curl https://api.serverless.runware.ai/v1/apps/resumable-job/invoke-sync/predict \
  -H "Authorization: Bearer $RUNWARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "taskId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "payload": {
      "request": {
        "job": "nightly",
        "total_steps": 1000
      }
    }
  }'
```

**CLI**:

```bash
cat <<'EOF' | runware serverless apps invoke resumable-job predict --sync -f -
{
  "request": {
    "job": "nightly",
    "total_steps": 1000
  }
}
EOF
```

**JSON**:

```json
{
  "taskId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "payload": {
    "request": {
      "job": "nightly",
      "total_steps": 1000
    }
  }
}
```

Then call it again with the same `job` until `done` is true. Size the slice by what a failure should cost: a bigger slice loses more work on each failure, a smaller one pays the fixed per-call overhead more often.

## What the run cost

Five calls to one job of 1000 steps, on an RTX PRO 6000:

<table><tbody><tr><td>Call</td><td><code>resumed_from</code></td><td><code>steps_done</code></td><td><code>loss</code></td><td>Time</td></tr><tr><td>1, cold worker</td><td>0</td><td>200</td><td>0.0444</td><td><strong>106s</strong></td></tr><tr><td>2</td><td>200</td><td>400</td><td>0.0331</td><td>1.8s</td></tr><tr><td>3</td><td>400</td><td>600</td><td>0.0251</td><td>1.1s</td></tr><tr><td>4</td><td>600</td><td>800</td><td>0.0189</td><td>1.3s</td></tr><tr><td>5</td><td>800</td><td>1000, <code>done</code></td><td>0.0142</td><td>1.5s</td></tr></tbody></table>

**The loss column is what proves the resume worked.** A step counter only shows that a number was stored. A loss that keeps falling across five separate calls shows that the **optimizer state** survived the round trip, which is easy to get wrong and silent when you do.

A sixth call under a new job name came back at `resumed_from: 0` with a loss of 0.0444, exactly the first call's, which is what a fresh start should look like.