Rate Limits

Introduction

Runware runs on shared infrastructure with finite processing capacity. Queue-based systems balance load across users while prioritizing active workloads, and the REST API adds a per-organization rate limit that sits far above typical traffic.

Under load, latency rises first. Once a model's queue is over capacity, new requests for that model are rejected right away with a retry hint.

Current approach:

• High rate limit - A per-organization limit on the REST API sits far above typical traffic.
• Shared queues - Requests are processed through model-specific queues with finite capacity.
• Fair allocation - Processing capacity is distributed fairly across active users.
• Graceful degradation - Under load, latency increases first. Once a queue is over capacity, new requests are rejected.

What to expect

Under normal conditions, requests are processed quickly. When platform capacity is reached, here is what happens:

1. Queueing - Requests enter a queue and wait for available processing capacity.

2. Increased latency - Generation time increases as queue depth grows. This is normal and expected.

3. Overload rejection - When a model's queue is over capacity, new requests for that model are rejected immediately with a retry hint. See Overload rejections.

4. Timeout potential - Extremely long queues may result in timeout errors after waiting too long.

5. Transient failures - During peak demand, some requests may fail with retryable errors.

{
"errors": [
  {
    "code": "serviceOverloaded",
    "message": "The service is temporarily unavailable due to high demand. Please retry your request in a few moments.",
    "taskType": "imageInference",
    "retryAfter": 7,
    "taskUUID": "f4cdf306-e3dc-4a58-ad64-0ad049518e2d"
  }
]
}

Third-party provider limits

We integrate with external AI providers like OpenAI, Black Forest Labs, ByteDance, and others. Their rate limits and capacity constraints affect our service, even when our infrastructure has available capacity.

What this means:

• Provider throttling appears as our errors - You'll see 429, 503, or increased latency when upstream providers hit their limits.
• We can't control their capacity - Some providers enforce strict concurrency or rate limits that cascade to our service.
• Model-specific constraints - Check individual model documentation for provider-specific limitations.

This is an inherent part of working with third-party AI services, and why a retry with backoff belongs in your client.

Best practices

async function sendWithRetry(tasks: object[], maxRetries = 5) {
  for (let attempt = 0; ; attempt++) {
    const response = await fetch('https://api.runware.ai/v1', {
      method: 'POST',
      headers: {
        'Authorization': 'Bearer ' + RUNWARE_API_KEY,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify(tasks),
    })

    const overCapacity = response.status === 429 || response.status === 503
    if (!overCapacity || attempt === maxRetries) {
      return response.json()
    }

    const hint = Number(response.headers.get('Retry-After'))
    const seconds = hint > 0 ? hint : 2 ** attempt
    await new Promise(resolve => setTimeout(resolve, (seconds + Math.random()) * 1000))
  }
}

import random
import time
import httpx

def send_with_retry(tasks, max_retries=5):
    for attempt in range(max_retries + 1):
        response = httpx.post(
            'https://api.runware.ai/v1',
            headers={
                'Authorization': f'Bearer {RUNWARE_API_KEY}',
                'Content-Type': 'application/json',
            },
            json=tasks,
            timeout=None,
        )

        over_capacity = response.status_code in (429, 503)
        if not over_capacity or attempt == max_retries:
            return response.json()

        hint = response.headers.get('Retry-After', '')
        seconds = int(hint) if hint.isdigit() else 2 ** attempt
        time.sleep(seconds + random.random())

Model-specific capacity

Some models have limited infrastructure due to hardware requirements, demand patterns, or provider constraints:

High-demand models - Popular models may experience longer queues during peak hours.

Resource-intensive models - Video generation and large models require significant GPU resources, limiting concurrent capacity. Check individual model documentation for specific constraints.

Third-party provider models - Models from external providers are subject to their capacity constraints and rate limits.

What's coming

We're actively developing improvements to provide more predictable performance:

Tiered concurrency limits - Predictable limits based on usage tier, with clear thresholds and automatic scaling.

Capacity reservations - Guaranteed throughput for enterprise users with dedicated infrastructure.

Real-time capacity metrics - API endpoints showing current queue depth and estimated processing times.

Serverless scaling - Dynamic model loading for improved availability and reduced cold starts.

These improvements will provide enterprise-grade reliability while maintaining platform accessibility for all users.

Summary

Current state:

• Shared infrastructure with finite capacity, and a high per-organization rate limit on the REST API
• Queueing under load, then rejection with a retry hint once a queue is over capacity
• Third-party provider limits affect service availability

Your responsibilities:

• Retry capacity errors, waiting for Retry-After when the response carries it
• Manage concurrency (2-4 concurrent requests recommended)
• Monitor latency and adjust request patterns

Need guarantees?

• Contact sales for dedicated capacity, SLAs, and custom infrastructure

Follow these and your integration absorbs a capacity constraint instead of failing on it.