Introduction to Video Inference

Introduction

Video inference enables video generation and transformation. Generate videos from text descriptions, transform existing videos, or use images to guide video generation, all powered by our Sonic Inference Engine®.

For the core API parameters common to all Runware tasks, please refer to our Getting Started Introduction. That guide covers the fundamental concepts that apply across all Runware APIs.

This page covers the common parameters and concepts shared across all video inference operations. Understanding these fundamentals will help you work effectively with our video generation APIs.

[
  {
    // Core parameters
    "taskType": "videoInference",
    "taskUUID": "24cd5dff-cb81-4db5-8506-b72a9425f9d1",
    "includeCost": true,

    // Video-related parameters
    "outputType": "URL",
    "outputFormat": "MP4",
    "outputQuality": 95,
    "numberResults": 2,
    "checkNSFW": false,

    // Task parameters
    "model": "klingai:5@3",
    "positivePrompt": "A cat playing with a ball of yarn, high quality, detailed",
    "duration": 10,
    "width": 1920,
    "height": 1080
  }
]

Key considerations for video generation

Working with video introduces several important concepts that affect how you structure requests and handle responses.

  // Automatic distribution
  "frameImages": [{ "inputImage": "aac49721-1964-481a-ae78-8a4e29b91402" }],

  // Explicit frame number
  "frameImages": [{ "inputImage": "aac49721-1964-481a-ae78-8a4e29b91402", "frame": 0 }],

  // Named frame position
  "frameImages": [{ "inputImage": "aac49721-1964-481a-ae78-8a4e29b91402": "frame": "first" }]

Common video inference parameters

While each video operation has its specific parameters, several parameters are common across all video inference tasks.

outputType

"URL" Default: URL

Specifies the output type in which the video is returned. Currently, only URL delivery is supported for video outputs.

• URL: The video is returned as a URL string using the videoURL parameter in the response object.

outputFormat

"MP4" | "WEBM" Default: MP4

Specifies the format of the output video. Supported formats are: MP4 and WEBM.

• MP4: MPEG-4 video format, widely compatible and recommended for most use cases.
• WEBM: WebM video format, optimized for web delivery and smaller file sizes.

outputQuality

integer Min: 20 Max: 99 Default: 95

Sets the compression quality of the output video. Higher values preserve more quality but increase file size, lower values reduce file size but decrease quality.

uploadEndpoint

string

This parameter allows you to specify a URL to which the generated video will be uploaded as binary video data using the HTTP PUT method. For example, an S3 bucket URL can be used as the upload endpoint.

When the video is ready, it will be uploaded to the specified URL.

numberResults

integer Min: 1 Max: 4 Default: 1

Specifies how many videos to generate for the given parameters. Each video will have the same parameters but different seeds, resulting in variations of the same concept.

If seed is set, it will be incremented by 1 (+1) for each video generated.

Here's a simple text-to-video request illustrating the common parameters:

[
  {
    "taskType": "videoInference",
    "taskUUID": "24cd5dff-cb81-4db5-8506-b72a9425f9d1",
    "model": "klingai:5@3",
    "positivePrompt": "A peaceful sunset over a calm lake with gentle waves",
    "duration": 10,
    "width": 1920,
    "height": 1090,
    "outputFormat": "MP4",
    "outputQuality": 95,
    "numberResults": 1,
    "includeCost": true
  }
]

Common response fields

All video inference operations return a consistent set of fields:

videoUUID

string UUID v4

A unique identifier for the generated video. This UUID can be used to reference the video in subsequent operations or for tracking purposes.

The videoUUID is different from the taskUUID. While taskUUID identifies the generation request, videoUUID identifies the specific video output.

videoURL

string

If outputType is set to URL, this parameter contains the URL of the video to be downloaded.

seed

integer

The seed value that was used to generate this video. This value can be used to reproduce the same video when using identical parameters in another request.

Video inference operations use asynchronous processin, so responses come in two stages. The initial response confirms task acceptance, while final results are retrieved through polling.

// Success acknowledgment
{
  "data": [
    "taskType": "videoInference",
    "taskUUID": "24cd5dff-cb81-4db5-8506-b72a9425f9d1"
  ]
}

// Or error response
{
  "errors": [
    {
      "code": "unsupportedDuration",
      "message": "Invalid value for duration parameter. This duration is not supported by the model architecture.",
      "parameter": "duration",
      "type": "float",
      "documentation": "https://runware.ai/docs/en/video-inference/video-reference#request-duration",
      "taskUUID": "24cd5dff-cb81-4db5-8506-b72a9425f9d1",
      "allowedValues": [6]
    }
  ]
}

{
  // Successes
  "data": [
    {
      "taskType": "videoInference",
      "taskUUID": "24cd5dff-cb81-4db5-8506-b72a9425f9d1",
      "status": "success",
      "videoUUID": "b7db282d-2943-4f12-992f-77df3ad3ec71",
      "videoURL": "https://im.runware.ai/video/ws/0.5/vi/b7db282d-2943-4f12-992f-77df3ad3ec71.mp4",
      "cost": 0.18
    }
  ],
  // Errors
  "errors": [
    {
      "code": "timeoutProvider",
      "status": "error",
      "message": "The external provider did not respond within the timeout window. The request was automatically terminated.",
      "documentation": "https://runware.ai/docs/en/video-inference/api-reference",
      "taskUUID": "24cd5dff-cb81-4db5-8506-b72a9425f9d1"
    }
  ]
}