Compute and scaling

Choose a GPU, set the bounds the platform scales between, and change any of it on a running app without a rebuild.

Introduction

An app's configuration says what hardware its workers run on and how many of them there may be. The platform stays inside those bounds and decides the rest.

{
  "gpuType": "l40s",
  "maxWorkers": 4,
  "idleTtlSecs": 300,
  "scalingDelaySecs": 30,
  "minWorkers": 0
}

Four of these are required: gpuType, maxWorkers, idleTtlSecs and scalingDelaySecs. The rest have defaults.

gpuType is a lowercase catalog code such as l40s or h100. It must match an id from GET /v1/gpu-types that currently has capacity, so the list you get back is what you can actually deploy on rather than everything that exists.

Every app must pin a GPU type. Omitting it fails with a 422 before a build starts, because an unpinned app would fall through to whatever default the deployer renders.

fallbackGpuType names a substitute GPU type for the app. Placement uses it when the pool behind gpuType has no recorded inventory, or when no complete block is free there. It stays unused when placement is merely contending for a busy pool, when a scan reaches its limit, or when your credit does not cover gpuType.

A worker placed on the substitute reports it as its own gpuType, and one app's workers can sit on both types at once. The substitute bills at its own catalog rate, including when that rate is higher than the type you preferred, and a commitment bought for gpuType does not cover it.

The value has to differ from gpuType and resolve to capacity in the same cluster, and it has to grant the gpusPerWorker you asked for. Send an empty string to clear it.

More than one GPU per worker

A worker holds its GPUs as one group the cluster grants indivisibly, so gpusPerWorker is a group size: 1, 2, 4 or 8.

The size also has to be one the cluster behind your gpuType grants. A size it does not grant answers 422 naming /configuration/gpusPerWorker, because the pod would have nowhere to run.

Not every GPU type can hold a group at all. GET /v1/gpu-types reports multiGpuSupport on each entry, and a type that reports false holds exactly one GPU per worker.

Raising it above 1 needs a redeploy, not only a new number: an image built for a single rank holds every GPU it was granted and uses one.

GPU is the only compute type. computeType defaults to it and naming cpu is a 422.

Bounds, not instructions

The platform starts and stops workers for you. You set the edges.

Field	What it does	Default
`minWorkers`	The floor. `0` allows the app to release every worker when nothing is arriving	`0`
`maxWorkers`	The ceiling, and your spend limit. Required	none
`idleTtlSecs`	How long an idle worker survives before it is reclaimed. Required	none
`scalingDelaySecs`	The damping window before a scaling decision acts. Required	none
`minAvailableWorkers`	Idle workers held above current demand, so a burst lands on workers that are already up. Must be lower than `maxWorkers`	none
`availableWorkersPct`	The same buffer as a percentage of current demand, rounded up. When both are set, the larger applies	none
`startupTimeoutSecs`	How long readiness has before the pod is failed, from 1 to 2400	platform
`requestTimeoutSecs`	How long the platform waits on one forwarded request, from 1 to 900. Container apps only	platform

requestTimeoutSecs bounds the worker, not your HTTP wait. invoke-sync still answers 504 at the platform deadline so you can go on polling the task. A code app stores and returns the field but does not apply it, because it keeps the platform read ceiling.

On a container app a later deploy overwrites both from container.yaml, out of its timeouts.requestSeconds and timeouts.startupSeconds. Setting them through the API and then deploying that document puts the document's values back.

The buffer only holds while work is queued. An idle app still scales down to minWorkers, so the floor is what holds capacity through a quiet stretch. While demand holds steady, the platform skips a buffer below about a tenth of current demand.

Your credit balance can lower the ceiling. When your organization's credit cannot back every worker up to maxWorkers at deploy time, the platform applies a lower ceiling and reports it on the app as effectiveMaxWorkers. The app scales only up to it. maxWorkers keeps the value you set, and a top-up raises a running app's ceiling back without a redeploy.

The trade you are actually making

Scaling to zero is the reason this product is cheap, and the reason a cold request is slow. A request arriving at an app with no workers waits for one to start and for your model to load.

The levers are few and they pull against each other.

minWorkers above zero keeps capacity ready and removes the cold start for the traffic that fits inside it. You pay for that GPU whether or not anything arrives.

A buffer keeps spare workers ahead of a burst while traffic is flowing, and releases them when the queue empties.

A longer idleTtlSecs keeps a worker alive through gaps in traffic, so the second request after a quiet minute is fast. You pay for the idle time you did not use.

A longer scalingDelaySecs stops a brief spike from starting workers you will not need. It also delays the response to a real one.

Caching weights on a volume attacks the same problem from the other side: a worker that already has the weights on disk starts faster than one that has to fetch them, and that is where most of a cold start goes.

Changing it on a running app

Configuration lives on the app, separate from your source, and can be changed while the app is serving.

runware serverless apps scale <appId> --max-workers 16

Every field except computeType can be patched this way. A field you omit is left alone, so a patch touches only what you name.

Clearing a value by sending null is not supported. Omit the field instead, which leaves it unchanged. The one exception is fallbackGpuType, which an empty string clears.

Two consequences worth knowing. Changing gpuType affects only workers created after the change, so the ones already running finish on the hardware they started on. And a patch is rejected with a 422 if the new type has no capacity on offer, which is the same check that applies at create.

On a running app, a change that raises minWorkers or moves to another gpuType is checked against your credit balance and returns 402 when the balance cannot back it. A change that lowers capacity is never refused that way, though the rollout it starts reads the balance again like any other.

What it costs you in versions

A configuration change produces a new version but not a new build. The image is carried forward untouched, so tuning your scaling is fast and cheap, and the version history still records exactly what changed.

That is also why your scaling survives a redeploy of your code: the values live on the app.