OIDC connectors

Introduction

An OIDC connector lets your workers reach AWS, Google Cloud or any service that trusts OpenID Connect without a stored key. Runware signs a short-lived token for each worker, your cloud account trusts that token through federation, and your code uses the AWS or Google SDK unchanged.

import boto3

s3 = boto3.client("s3")   # no access key anywhere

Compare that with a secret that holds an access key. The key is valid until someone rotates it, it works from any machine that has a copy, and it is one more thing to keep out of logs. A connector holds no credential at all. The token proves which app in which environment is asking, it expires within the hour, and your cloud decides what that app may do.

How it works

1. You create a connector in your organization. It says where the token goes: an IAM role, a Google workload identity provider, or an audience of your own.
2. You attach it to an app. Its workers start getting a token file and the environment variables the SDKs read.
3. You tell your cloud to trust the Runware issuer, and only for tokens whose subject names your app.
4. When the SDK needs credentials, it sends the token to the cloud's security token service and gets temporary credentials back.

The platform renews the token file while the worker runs, with no restart and nothing for your code to do.

The values your cloud needs

Every trust rule you write is built from three claims of the token.

An aws connector carries runware:<organization uuid> as its audience unless you set your own, and a generic one carries the value you set. A gcp connector always carries the provider's full resource name, the value Google expects.

//iam.googleapis.com/projects/<project number>/locations/global/workloadIdentityPools/<pool id>/providers/<provider id>

The subject names the organization, the environment and the app, in that order.

worker:org:<organization uuid>:env:<environment uuid>:app:<app id>

The order of the sub segments and the : between them are stable, so a pattern such as worker:org:<organization uuid>:env:<environment uuid>:* keeps matching every app of one environment. The worker: prefix means a rule written for workers matches no other kind of Runware token.

The environment segment is the environment's id, not its name, so a staging copy of an app never matches a production trust rule, and a new environment that reuses a deleted one's name gets none of its trust. The app segment is the app id, so an app you delete and create again with the same id in the same environment matches the same rule.

curl "https://api.serverless.runware.ai/v1/apps/my-app/oidc-connectors?environment=production" \
  -H "Authorization: Bearer $RUNWARE_API_KEY"

{
  "data": [
    {
      "name": "s3-reader",
      "kind": "aws",
      "audience": "runware:0190a6a8-0000-7000-8000-00000000000a",
      "issuer": "https://oidc.serverless.runware.ai",
      "aws": { "roleArn": "arn:aws:iam::123456789012:role/runware-s3-reader" },
      "subject": "worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:app:my-app",
      "subjectPattern": "worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:*",
      "id": "...",
      "createdAt": "...",
      "updatedAt": "..."
    }
  ]
}

Connecting to AWS

curl -X POST https://api.serverless.runware.ai/v1/oidc-connectors \
  -H "Authorization: Bearer $RUNWARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "s3-reader",
    "kind": "aws",
    "aws": { "roleArn": "arn:aws:iam::123456789012:role/runware-s3-reader" }
  }'

curl -X POST https://api.serverless.runware.ai/v1/apps/my-app/oidc-connectors \
  -H "Authorization: Bearer $RUNWARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "connectorName": "s3-reader" }'

aws iam create-open-id-connect-provider \
  --url https://oidc.serverless.runware.ai \
  --client-id-list runware:<organization uuid>

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.serverless.runware.ai"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.serverless.runware.ai:aud": "runware:<organization uuid>",
          "oidc.serverless.runware.ai:sub": "worker:org:<organization uuid>:env:<environment uuid>:app:my-app"
        }
      }
    }
  ]
}

aws iam create-role \
  --role-name runware-s3-reader \
  --assume-role-policy-document file://trust-policy.json

aws iam attach-role-policy \
  --role-name runware-s3-reader \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

import boto3

@serve
class Thumbnails:
    def load(self) -> None:
        self.s3 = boto3.client("s3", region_name="eu-west-1")

    @endpoint
    def list_inputs(self, payload: dict) -> dict:
        page = self.s3.list_objects_v2(Bucket="my-inputs", MaxKeys=10)
        return {"keys": [item["Key"] for item in page.get("Contents", [])]}

Connecting to Google Cloud

curl -X POST https://api.serverless.runware.ai/v1/oidc-connectors \
  -H "Authorization: Bearer $RUNWARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "gcs-reader",
    "kind": "gcp",
    "gcp": {
      "workloadIdentityProvider": "projects/123456789012/locations/global/workloadIdentityPools/runware/providers/runware-workers",
      "serviceAccountEmail": "gcs-reader@my-project.iam.gserviceaccount.com"
    }
  }'

curl -X POST https://api.serverless.runware.ai/v1/apps/my-app/oidc-connectors \
  -H "Authorization: Bearer $RUNWARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "connectorName": "gcs-reader" }'

gcloud iam workload-identity-pools create runware \
  --location=global \
  --display-name="Runware Serverless"

gcloud iam workload-identity-pools providers create-oidc runware-workers \
  --location=global \
  --workload-identity-pool=runware \
  --issuer-uri=https://oidc.serverless.runware.ai \
  --attribute-mapping="google.subject=assertion.sub,attribute.org_id=assertion.org_id,attribute.environment_id=assertion.environment_id,attribute.app_id=assertion.app_id" \
  --attribute-condition="assertion.sub.startsWith('worker:org:<organization uuid>:env:<environment uuid>:')"

gcloud iam service-accounts add-iam-policy-binding \
  gcs-reader@my-project.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member="principal://iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/runware/subject/worker:org:<organization uuid>:env:<environment uuid>:app:my-app"

gcloud storage buckets add-iam-policy-binding gs://my-inputs \
  --role=roles/storage.objectViewer \
  --member="principal://iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/runware/subject/worker:org:<organization uuid>:env:<environment uuid>:app:my-app"

from google.cloud import storage

@serve
class Thumbnails:
    def load(self) -> None:
        self.gcs = storage.Client(project="my-project")

    @endpoint
    def list_inputs(self, payload: dict) -> dict:
        blobs = self.gcs.list_blobs("my-inputs", max_results=10)
        return {"names": [blob.name for blob in blobs]}

Connecting to your own service

The generic kind is for anything else that can verify an OpenID Connect token: your own API, a partner's service, or a third-party product with an OIDC trust setting. You choose the audience, and the worker gets the raw token file.

curl -X POST https://api.serverless.runware.ai/v1/oidc-connectors \
  -H "Authorization: Bearer $RUNWARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "billing-api",
    "kind": "generic",
    "audience": "https://billing.example.com"
  }'

An audience that starts with runware: is reserved, except your own organization's runware:<organization uuid>. Attach the connector to the app as above.

import os
import requests

def call_billing(body: dict) -> dict:
    with open(os.environ["RUNWARE_OIDC_TOKEN_FILE"]) as f:
        token = f.read().strip()

    response = requests.post(
        "https://billing.example.com/usage",
        json=body,
        headers={"Authorization": f"Bearer {token}"},
        timeout=10,
    )
    response.raise_for_status()
    return response.json()

import jwt
import requests

ISSUER = "https://oidc.serverless.runware.ai"
AUDIENCE = "https://billing.example.com"
TRUSTED_PREFIX = "worker:org:<organization uuid>:env:<environment uuid>:"

discovery = requests.get(f"{ISSUER}/.well-known/openid-configuration", timeout=5).json()
keys = jwt.PyJWKClient(discovery["jwks_uri"])

def verify(token: str) -> dict:
    signing_key = keys.get_signing_key_from_jwt(token)
    claims = jwt.decode(
        token,
        signing_key.key,
        algorithms=["RS256"],
        audience=AUDIENCE,
        issuer=ISSUER,
        leeway=60,
        options={"require": ["iss", "sub", "aud", "exp", "iat"]},
    )
    if not claims["sub"].startswith(TRUSTED_PREFIX):
        raise PermissionError("token is not from a trusted app")
    return claims

What the worker gets

The platform mounts a read-only directory at /var/run/secrets/runware/oidc and sets the variables for each attached kind. The file variables hold a path inside that directory, at the name below. The first two come with an aws connector, the third with a gcp one and the fourth with a generic one.

AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE and RUNWARE_OIDC_TOKEN_FILE belong to the connector only while one of that kind is attached. Attaching to an app that already sets one of them as an environment variable or a secret returns 422, and so does setting one later. Whatever else sets one of them, such as an older version the app is pinned to, the worker gets the connector's value. An app with no connector of that kind can set them as before.

GOOGLE_APPLICATION_CREDENTIALS and RUNWARE_OIDC_CONFIG_DIGEST are reserved on every app, with or without a connector, and setting either returns 422. The platform sets RUNWARE_OIDC_CONFIG_DIGEST on workers that have a connector, and your code has no use for it. The token directory is also reserved for volumes: a mount path on it, under it or over it, such as /var, returns 422.

The token

A token is a JWT signed with RS256, and its header carries the kid of the signing key.

AWS trust policies can match only aud and sub. A Google attribute condition or your own verifier can match any claim, for example deployment_id to trust one deployment only.

A token lives at most one hour. Do not depend on an exact lifetime, because it may get shorter. The platform replaces the token file at half its lifetime, including on a worker that was resumed after scaling to zero, and the AWS and Google SDKs read the new file when their credentials run out.

Key rotation needs nothing from you. New keys are published at the issuer before they sign anything, and old keys stay until every token they signed has expired. If a signing key is ever compromised, Runware removes it and tells every affected organization when the exposure started and ended, which subjects were used, and what to revoke.

Limits

• One connector of each kind per app. Attaching a second aws connector, or the same one twice, returns 409.
• A connector belongs to the organization, not to an environment. Any app in any environment can attach it, and the environment shows up only in sub.
• Changes reach workers when they roll. Attaching, updating and detaching roll the app's live deployment. An app that is not live, or one with a rollout already in progress, picks the change up on its next rollout.
• Detaching does not revoke access at once. A worker that still runs keeps getting tokens until it rolls, a token stays valid until it expires, and the cloud credentials it was exchanged for stay valid for their own lifetime. The worst case is one hour of token plus the credential lifetime: one hour on Google, and on AWS the role's maximum session duration, one hour by default and up to twelve. To cut access at once, remove the trust or the grant on your cloud's side, and on AWS revoke the role's active sessions.
• A connector cannot be deleted while any app attaches it, a stopped or failed one included. Detach it first, or delete the app.
• The audience is one string, never a list, of at most 256 characters with no whitespace.
• A name is 2 to 63 characters: lowercase letters, digits and hyphens, starting with a letter and ending with a letter or a digit. The name and the kind are fixed once created.
• An update replaces the whole configuration, so send every field you want to keep. An aws update without audience goes back to the default.

Troubleshooting