Embeddings

Embed many texts in one request, and batch them inside your handler because the platform will not do it for you.

Introduction

A text embedding model that takes a list of texts and returns one vector per text.

bge-small-en-v1.5 is MIT and ungated, so you deploy it with no token.

The app

model.py
"""Many inputs in one request: a list of texts in, a list of vectors out."""

from __future__ import annotations

from pydantic import BaseModel, ConfigDict, Field

from runware_serverless import serve

MODEL = "BAAI/bge-small-en-v1.5"
"""MIT and ungated, 384 dimensions, small enough that the GPU is never the limit."""

CACHE = "/weights"
"""Mount a volume here. Small as this model is, a checkpoint captures the worker's
filesystem, so weights left outside a volume are copied into every one."""

MAX_TEXTS = 256
"""The largest list this app accepts.

Measured: the platform checks the types it published and rejects a wrong one in
under half a second without waking anything, but it does not check a list's
length, so a request carrying more than this reaches the worker and fails there.
This app enforces the value constraint itself, and a request that breaks one
wakes a worker before it is refused."""

ENCODE_BATCH = 64
"""How many texts cross the GPU at once inside one request. Unrelated to how many
the request carries: the platform gives a worker one task at a time, so batching
is something you do inside your handler or not at all."""


class Request(BaseModel):
    # Without this pydantic ignores a field it does not know, so a caller who
    # misspells one gets a silent success. An unknown field is an error here.
    model_config = ConfigDict(extra="forbid")

    texts: list[str] = Field(
        min_length=1,
        max_length=MAX_TEXTS,
        description="The texts to embed. One request, one pass over the GPU.",
    )
    normalize: bool = Field(
        default=True,
        description="Return unit vectors, so a dot product is a cosine similarity.",
    )


class Embedded(BaseModel):
    vectors: list[list[float]] = Field(description="One vector per input, in order.")
    dimensions: int
    count: int


@serve
class Embeddings:
    requirements = [
        "torch==2.14.1",
        "transformers==5.21.0",
        "sentence-transformers==6.2.0",
    ]

    def load(self) -> None:
        if hasattr(self, "model"):
            return

        # Here and not at the top of the file. The build imports this file with the
        # platform's own dependencies, not yours, so nothing in requirements exists
        # until a worker runs. A `with imports():` block at the top of the file does
        # the same, and is what to use when a name has to be visible outside a method.
        from sentence_transformers import SentenceTransformer

        self.model = SentenceTransformer(MODEL, cache_folder=CACHE, device="cuda")
        self.model.encode(["warmup"])

    def predict(self, request: Request) -> Embedded:
        vectors = self.model.encode(
            request.texts,
            batch_size=ENCODE_BATCH,
            normalize_embeddings=request.normalize,
        )

        return Embedded(
            vectors=vectors.tolist(),
            dimensions=int(vectors.shape[1]),
            count=len(request.texts),
        )

Deploying the app

embeddingsrtx-pro-6000source
From a URL
runware serverless deploy --from https://runware.ai/docs/serverless/examples/embeddings.zip
From a clone
runware serverless deploy model.py \
  --id embeddings \
  --gpu-type rtx-pro-6000 \
  --volume /weights

The model is 130MB, small enough that you might skip the volume. Mount it anyway: a checkpoint captures the worker's filesystem, so an unmounted download is copied into every one.

Batching inside one request

A worker takes one task at a time. There is no setting that lets several requests share a worker, so the only batching available to you is the batching you put inside one request.

That is why the request takes a list rather than a string.

Measured on an RTX PRO 6000, against a worker already serving:

Texts in one requestTime on the workerPer text
10.62s620ms
80.41s52ms
320.54s17ms
1280.80s6.2ms
2560.65s2.5ms

The time barely moves between one text and 256, and eight came back faster than one, because on a warm worker almost all of it is fixed overhead rather than model work.

Sending those 256 texts one per request would pay 256 queue hops and 256 times that fixed cost for the same work.

What the platform checks

Check this before you rely on a schema to keep bad requests away from your worker. Measured against this app:

RequestWhat happened
texts as a string422 in 0.4s, nothing woken
texts holding integers422, with a pointer per bad index
normalize as a string422 in 0.1s
An empty list, against min_lengthreached the worker and failed there
257 texts, against max_lengthreached the worker and failed there

A type is checked before anything wakes. A value constraint is enforced by the app, not the platform, and a request that breaks one wakes a worker before it is refused.

Pydantic ignores an unknown field by default, so a caller who writes normalise instead of normalize gets a silent success and the wrong behavior. model_config = ConfigDict(extra="forbid") turns that into an error, and the platform publishes and enforces it: the same misspelled request went from a completed task on a woken worker to a 422 in half a second with nothing woken.

Calling the app

The class declares no @endpoint, so @serve exposes predict at the default path.

import { createClient } from '@runware/sdk'

const client = await createClient({ apiKey: process.env.RUNWARE_API_KEY })

const task = await client.invoke({
  appId: 'embeddings',
  endpointPath: 'predict',
  payload: {
    request: {
      texts: [
        'a red bicycle',
        'a blue car'
      ]
    }
  }
})
import asyncio
import os

from runware import Runware


async def main():
    client = Runware(api_key=os.environ["RUNWARE_API_KEY"])
    try:
        task = await client.invoke(
            "embeddings",
            "predict",
            {
                "request": {
                    "texts": [
                        "a red bicycle",
                        "a blue car"
                    ]
                }
            },
        )
    finally:
        await client.close()


asyncio.run(main())
curl https://api.serverless.runware.ai/v1/apps/embeddings/invoke-async/predict \
  -H "Authorization: Bearer $RUNWARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "taskId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "payload": {
      "request": {
        "texts": [
          "a red bicycle",
          "a blue car"
        ]
      }
    }
  }'
cat <<'EOF' | runware serverless apps invoke embeddings predict -f -
{
  "request": {
    "texts": [
      "a red bicycle",
      "a blue car"
    ]
  }
}
EOF
{
  "taskId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "payload": {
    "request": {
      "texts": [
        "a red bicycle",
        "a blue car"
      ]
    }
  }
}

The vectors come back at output.predictions[0].vectors, one per input and in the same order. They are unit length unless you ask for normalize: false, so a dot product between two of them is their cosine similarity.

Measured on an RTX PRO 6000: a 2m57s build, 112s for the first call with an empty volume, 14 seconds for a restart that finds the weights already there, and 0.6 seconds on a worker already serving.