SAM 2.1 Hiera Large

Meta's Segment Anything 2.1, run without a prompt so it finds every part of an image on its own. It answers with a label map, one PNG the size of the image where each pixel holds the id of the part it belongs to and the smallest part wins wherever parts overlap, so nothing is hidden by what contains it. Each part also comes back with its own area and bounding box, and with its own mask on request. The point grid and the two quality filters are request parameters, so how finely it divides an image is the caller's decision.

Image generationl40sMetaApache 2.0facebook/sam2.1-hiera-large

What runs

This is the whole app. The entry names the model and what serves it, and the build turns that into these files: there is nothing else to write.

model.py
"""SAM 2.1 Hiera Large, generated from a declarative template."""

from __future__ import annotations

import base64
import io
from typing import Literal

from pydantic import BaseModel, ConfigDict, Field

from runware_serverless import serve

MODEL = "facebook/sam2.1-hiera-large"
REVISION = "665f8e2ad61cf5f53d65644ff27c8ee525124610"
CACHE = "/weights"

# The grid of query points and the two filters applied to what each one produces,
# all three as this checkpoint's own generator sets them. transformers defaults
# the predicted-IoU floor to SAM 1's 0.88, so this file states 0.8 instead of
# inheriting it: these weights are SAM 2.1 and 0.8 is the value they were
# released with.
POINTS_PER_SIDE = 32
PRED_IOU_THRESH = 0.8
STABILITY_SCORE_THRESH = 0.95

# A ceiling on the input rather than on the weights. The checkpoint is under a
# gigabyte and what bounds a request is the mask set: each query point produces
# three masks at the image's own resolution, and every part that survives
# filtering is held at that resolution until the response is built. This is a
# guard above the largest size anyone has asked for, not a measured limit, and
# it is here because the alternative is the container's own kill, which reports
# as a platform fault rather than as an image too large.
MAX_PIXELS = 12_000_000

# What a 16-bit label map can number. A grid of 32 points produces at most 3072
# masks before deduplication, so this is out of reach: the check says so rather
# than expecting to fire.
MAX_PARTS = 65_535

# What the warm-up segments. Shapes rather than a blank canvas, because a blank
# one is filtered down to nothing and the paths that build a response would then
# go unexercised until somebody's first call.
WARMUP_SIZE = (768, 768)
WARMUP_SHAPES = ((64, 64, 360, 300), (420, 120, 700, 420), (150, 420, 600, 700))


class Rle(BaseModel):
    """One part as an uncompressed run length encoding."""

    size: list[int] = Field(description="Height then width, in pixels.")
    counts: list[int] = Field(
        description=(
            "Alternating run lengths beginning with a run of zeros, read down each column"
            " rather than along each row. That is the layout pycocotools reads."
        ),
    )


class Part(BaseModel):
    id: int = Field(description="This part's value in the label map. Parts are numbered from one.")
    score: float = Field(description="What the model predicts its own mask quality to be.")
    area: int = Field(
        description=(
            "Pixels in this part's own mask, which is not what it covers in the label map:"
            " a part overlapped by a smaller one keeps its own area here and loses those"
            " pixels there."
        ),
    )
    box: list[int] = Field(description="Left, top, right, bottom, in pixels.")
    rle: Rle | None = Field(default=None, description="Set when masks is rle.")
    mask: str | None = Field(default=None, description="A one bit PNG, set when masks is png.")


class Request(BaseModel):
    # An input this model does not read is a line the caller believed was doing
    # something, so it is refused rather than dropped. Without this, pydantic
    # ignores it: a prompt sent to a model that takes none, or a misspelled
    # threshold, both come back as a result that silently answered a different
    # request.
    model_config = ConfigDict(extra="forbid")

    image: str = Field(description="The image to segment, base64 encoded.")
    points_per_side: int = Field(
        default=POINTS_PER_SIDE,
        ge=1,
        le=64,
        description=(
            "The model is queried at every point of a square grid of this many points a side,"
            " and the parts it finds are whatever those queries return once duplicates are"
            " removed. Raising it finds smaller parts and costs the square of the change:"
            " the whole request scales with the point count."
        ),
    )
    points_per_batch: int = Field(
        default=64,
        ge=1,
        le=1024,
        description=(
            "How many of those points are queried at once. It does not change the result,"
            " only how much of the card the run holds at its peak, since each point in a"
            " batch carries three masks at the full size of the image."
        ),
    )
    pred_iou_thresh: float = Field(
        default=PRED_IOU_THRESH,
        ge=0,
        le=1,
        description=(
            "Drops a mask the model scores below this on its own quality. Lower it to keep"
            " parts the model is unsure of."
        ),
    )
    stability_score_thresh: float = Field(
        default=STABILITY_SCORE_THRESH,
        ge=0,
        le=1,
        description=(
            "Drops a mask whose shape changes when the cutoff that binarizes it is moved."
            " This is what removes the parts with no settled edge."
        ),
    )
    min_area: int = Field(
        default=0,
        ge=0,
        description=(
            "Drops a part whose mask covers fewer pixels than this. An area floor on the"
            " whole mask, so it removes small parts and does not reach inside a large one."
        ),
    )
    max_parts: int = Field(
        default=0,
        ge=0,
        le=MAX_PARTS,
        description=(
            "Keeps at most this many parts, the highest scoring first, or all of them at"
            " zero. The cap is applied before the parts are ordered by area, so what it"
            " drops is what the model was least sure of rather than what is smallest."
        ),
    )
    masks: Literal["none", "rle", "png"] = Field(
        default="none",
        description=(
            "Whether each part also carries its own mask, and in which form. The label map"
            " already describes every part, so this is for a caller that needs the parts"
            " separately and is worth the size it adds."
        ),
    )


class Segmented(BaseModel):
    label_map: str = Field(
        description=(
            "A 16-bit grayscale PNG the size of the image, where each pixel holds the id of"
            " the part it belongs to and zero means no part. Where parts overlap the"
            " smallest one wins, so a part is never hidden by the one containing it."
        ),
    )
    count: int = Field(description="How many parts the label map numbers.")
    width: int
    height: int
    parts: list[Part]


@serve
class GeneratedSegmentation:
    def load(self) -> None:
        if hasattr(self, "generator"):
            return

        # Imported here rather than at the top of the file. The build imports
        # this file with the platform's own dependencies, so nothing above
        # exists until a worker runs.
        import torch
        from transformers import AutoImageProcessor, AutoModelForMaskGeneration, pipeline

        self.torch = torch

        # float32, and not the bfloat16 the reference implementation autocasts
        # to. The mask generator ends in a non-maximum suppression that
        # transformers calls as batched_nms(boxes=mask_boxes.float(),
        # scores=iou_scores), casting the boxes and leaving the scores at
        # whatever the model produced, and torchvision's CPU kernel requires
        # both to carry the same type. So a bfloat16 model raises "dets should
        # have the same type as scores" on the last step of every call,
        # measured here on 2026-11-19, and the card has room for the wider
        # masks.
        model = AutoModelForMaskGeneration.from_pretrained(
            MODEL,
            revision=REVISION,
            cache_dir=CACHE,
            dtype=torch.float32,
            device_map="cuda",
        )

        # The image processor on its own rather than through AutoProcessor. This
        # repository names a video processor as its processor class, and the mask
        # generator reads nothing but the image side.
        processor = AutoImageProcessor.from_pretrained(
            MODEL, revision=REVISION, cache_dir=CACHE
        )

        generator = pipeline("mask-generation", model=model, image_processor=processor)

        # Kept in a local until the warm-up has run, because load is retried and
        # the guard at the top of it reads this attribute. Assigning first would
        # mean a failed warm-up is skipped on the retry, and the worker then
        # reports ready without having completed a single forward pass.
        self._warm(generator)
        self.generator = generator

    def _warm(self, generator) -> None:
        """Segment a synthetic image at the default settings, inside load.

        The first call allocates, picks kernels and walks every branch that
        builds a response. Paying it here puts a failure in the readiness check,
        where it is reported, instead of in somebody's first request.
        """
        from PIL import Image, ImageDraw

        canvas = Image.new("RGB", WARMUP_SIZE, "black")
        drawing = ImageDraw.Draw(canvas)
        for index, shape in enumerate(WARMUP_SHAPES):
            drawing.rectangle(shape, fill=(40 + index * 90, 180 - index * 50, 90 + index * 70))

        self._segment(generator, canvas, Request(image=""))

    def predict(self, request: Request) -> Segmented:
        from PIL import Image, ImageOps

        image = Image.open(io.BytesIO(base64.b64decode(request.image)))

        # Before the convert, because convert drops the EXIF and the orientation
        # with it. A photograph off a phone carries its rotation there rather
        # than in its pixels, so a mask produced without this is correct for an
        # image nobody sees.
        image = ImageOps.exif_transpose(image).convert("RGB")

        if image.width * image.height > MAX_PIXELS:
            raise ValueError(
                f"the image is {image.width}x{image.height}, which is"
                f" {image.width * image.height} pixels against a ceiling of {MAX_PIXELS}."
            )

        parts, label_map = self._segment(self.generator, image, request)

        return Segmented(
            label_map=label_map,
            count=len(parts),
            width=image.width,
            height=image.height,
            parts=parts,
        )

    def _segment(self, generator, image, request: Request):
        """Every part of one image, and the label map numbering them."""
        import numpy as np

        found = generator(
            image,
            points_per_crop=request.points_per_side,
            points_per_batch=request.points_per_batch,
            pred_iou_thresh=request.pred_iou_thresh,
            stability_score_thresh=request.stability_score_thresh,
            output_bboxes_mask=True,
            output_rle_mask=request.masks == "rle",
        )

        masks = found["masks"]
        scores = [float(one) for one in found["scores"]]
        boxes = [[int(side) for side in one] for one in found["bounding_boxes"]]
        rles = found.get("rle_mask") or [None] * len(masks)

        # The area is summed once and carried, because a sum over a mask at ten
        # megapixels is not free and the floor below reads it as well as the
        # ordering further down.
        #
        # One pixel is the floor even when min_area is zero. Deduplication can
        # leave a mask with nothing in it, and an empty part would take an id in
        # the label map that no pixel carries.
        kept = []
        for mask, score, box, rle in zip(masks, scores, boxes, rles):
            area = int(mask.sum())
            if area < max(request.min_area, 1):
                continue
            kept.append({"mask": mask, "score": score, "box": box, "rle": rle, "area": area})

        # Scored first, so a cap keeps what the model was most sure of, then by
        # area so the painting below ends on the smallest part.
        if request.max_parts:
            kept.sort(key=lambda part: part["score"], reverse=True)
            kept = kept[: request.max_parts]
        kept.sort(key=lambda part: part["area"], reverse=True)

        if len(kept) > MAX_PARTS:
            raise ValueError(
                f"{len(kept)} parts is more than a 16-bit label map can number. Set"
                " max_parts, or raise min_area."
            )

        label = np.zeros((image.height, image.width), dtype=np.uint16)
        parts = []
        for number, part in enumerate(kept, start=1):
            # In this order, largest first, so a part contained in another one
            # is painted after it and keeps its own pixels.
            label[part["mask"].numpy()] = number
            parts.append(Part(
                id=number,
                score=part["score"],
                area=part["area"],
                box=part["box"],
                rle=Rle(**part["rle"]) if part["rle"] is not None else None,
                mask=self._mask(part["mask"]) if request.masks == "png" else None,
            ))

        return parts, self._encode(label)

    def _mask(self, mask) -> str:
        """One part on its own, as a one bit PNG."""
        from PIL import Image

        bilevel = Image.fromarray(
            mask.to(self.torch.uint8).mul(255).numpy(), mode="L"
        ).convert("1")

        return self._file(bilevel)

    def _encode(self, label) -> str:
        """The label map as a 16-bit grayscale PNG.

        Sixteen bits rather than eight because a part id is a number and not a
        shade: a grid of 32 points regularly finds more than the 255 an 8-bit
        image can tell apart, and the ones it would lose are the small parts the
        caller asked for by segmenting at all.
        """
        from PIL import Image

        return self._file(Image.fromarray(label))

    @staticmethod
    def _file(image) -> str:
        buffer = io.BytesIO()
        image.save(buffer, format="PNG")

        return base64.b64encode(buffer.getvalue()).decode("ascii")
runware.yaml
configVersion: 1
id: sam-2-1-hiera-large
name: SAM 2.1 Hiera Large
source:
  type: code
  entry: model.py
compute:
  gpuType: l40s
build:
  requirements:
    - torch==2.14.1
    - torchvision==0.29.1
    - transformers==5.21.0
    - huggingface-hub==2.4.0
    - accelerate==1.15.0
    - pillow==13.0.0
volumes:
  - /weights

Deploying

Deploy it as it is, or take the files above and deploy your own copy. Nothing you change reaches back to this page.

sam-2-1-hiera-largel40ssource
From a URL
runware serverless deploy --from https://runware.ai/docs/serverless/examples/sam-2-1-hiera-large.zip
From a clone
runware serverless deploy model.py \
  --id sam-2-1-hiera-large \
  --gpu-type l40s \
  --requirement torch==2.14.1 \
  --requirement torchvision==0.29.1 \
  --requirement transformers==5.21.0 \
  --requirement huggingface-hub==2.4.0 \
  --requirement accelerate==1.15.0 \
  --requirement pillow==13.0.0 \
  --volume /weights

Calling it

One request that works, against the app you just deployed. The fields are the ones the entry declares above, and anything it does not declare is refused rather than ignored.

import { createClient } from '@runware/sdk'

const client = await createClient({ apiKey: process.env.RUNWARE_API_KEY })

const task = await client.invoke({
  appId: 'sam-2-1-hiera-large',
  endpointPath: 'predict',
  payload: {
    request: {
      image: '<base64 of the image to segment>'
    }
  }
})
import asyncio
import os

from runware import Runware


async def main():
    client = Runware(api_key=os.environ["RUNWARE_API_KEY"])
    try:
        task = await client.invoke(
            "sam-2-1-hiera-large",
            "predict",
            {
                "request": {
                    "image": "<base64 of the image to segment>"
                }
            },
        )
    finally:
        await client.close()


asyncio.run(main())
curl https://api.serverless.runware.ai/v1/apps/sam-2-1-hiera-large/invoke-async/predict \
  -H "Authorization: Bearer $RUNWARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "taskId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "payload": {
      "request": {
        "image": "<base64 of the image to segment>"
      }
    }
  }'
cat <<'EOF' | runware serverless apps invoke sam-2-1-hiera-large predict -f -
{
  "request": {
    "image": "<base64 of the image to segment>"
  }
}
EOF
{
  "taskId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "payload": {
    "request": {
      "image": "<base64 of the image to segment>"
    }
  }
}

What it costs

This entry is pinned to l40s, which is the card it was tested on rather than the only one it runs on: --gpu-typedeploys it somewhere else. You pay for that card by the second while a worker holds it, and for nothing else. Pricing covers what meters, and the rates are published per GPU-second.