OIDC connectors
Let your workers call AWS, Google Cloud or your own services with short-lived tokens that Runware signs, instead of cloud keys you store and rotate.
Introduction
An OIDC connector lets your workers reach AWS, Google Cloud or any service that trusts OpenID Connect without a stored key. Runware signs a short-lived token for each worker, your cloud account trusts that token through federation, and your code uses the AWS or Google SDK unchanged.
import boto3
s3 = boto3.client("s3") # no access key anywhereCompare that with a secret that holds an access key. The key is valid until someone rotates it, it works from any machine that has a copy, and it is one more thing to keep out of logs. A connector holds no credential at all. The token proves which app in which environment is asking, it expires within the hour, and your cloud decides what that app may do.
How it works
- You create a connector in your organization. It says where the token goes: an IAM role, a Google workload identity provider, or an audience of your own.
- You attach it to an app. Its workers start getting a token file and the environment variables the SDKs read.
- You tell your cloud to trust the Runware issuer, and only for tokens whose subject names your app.
- When the SDK needs credentials, it sends the token to the cloud's security token service and gets temporary credentials back.
The platform renews the token file in place while the worker runs, so your code only ever reads it.
The values your cloud needs
Every trust rule you write is built from three claims of the token.
| Claim | Value |
iss | https://oidc.serverless.runware.ai, also returned as issuer on every connector |
aud | What your cloud checks the token against, one string that depends on the kind. Returned as audience on the connector |
sub | Which app in which environment is asking, returned as subject on an attachment |
An aws connector carries runware:<organization uuid> as its audience unless you set your own, and a generic one carries the value you set. A gcp connector always carries the provider's full resource name, the value Google expects.
//iam.googleapis.com/projects/<project number>/locations/global/workloadIdentityPools/<pool id>/providers/<provider id>The subject names the organization, the environment and the app, in that order.
worker:org:<organization uuid>:env:<environment uuid>:app:<app id>The order of the sub segments and the : between them are stable, so a pattern such as worker:org:<organization uuid>:env:<environment uuid>:* keeps matching every app of one environment. The worker: prefix means a rule written for workers matches no other kind of Runware token.
The environment segment is the environment's id, not its name, so a staging copy of an app never matches a production trust rule, and a new environment that reuses a deleted one's name gets none of its trust. The app segment is the app id, so an app you delete and create again with the same id in the same environment matches the same rule.
The audience does not keep other Runware customers out. You choose a gcp provider name and a generic audience, and anyone could choose the same one. Every trust rule you write must check sub, or org_id where your verifier reads claims. A rule that checks only aud can accept tokens from apps that are not yours.
Find your values
Every command on this page reads its values from the shell, so set them once and the rest is copy and paste. Start with the two you already have.
export RUNWARE_API_KEY=... # the key you call the Serverless API with
export APP_ID=my-app # the app the connector is attached toThe organization and environment ids come next, and the connector reports them rather than you assembling them. Attach it to an app and list the app's connectors: each one carries the exact subject its tokens use, and a subjectPattern for every app of that environment.
curl "https://api.serverless.runware.ai/v1/apps/$APP_ID/oidc-connectors?environment=production" \
-H "Authorization: Bearer $RUNWARE_API_KEY"{
"data": [
{
"name": "s3-reader",
"kind": "aws",
"audience": "runware:0190a6a8-0000-7000-8000-00000000000a",
"issuer": "https://oidc.serverless.runware.ai",
"aws": { "roleArn": "arn:aws:iam::123456789012:role/runware-s3-reader" },
"subject": "worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:app:my-app",
"subjectPattern": "worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:*",
"id": "...",
"createdAt": "...",
"updatedAt": "..."
}
]
}Read them out of the subject, which is worker:org:<organization uuid>:env:<environment uuid>:app:<app id>.
export ORG_UUID=0190a6a8-0000-7000-8000-00000000000a
export ENV_UUID=0190a6a8-0000-7000-8000-0000000000e1The ids are not shown anywhere else yet, so the app has to exist before you can read its subject. The cloud side can wait: the connector only names the role or provider, and the worker asks for credentials only when your code does.
Last, your cloud account's own identifier, which is the one value Runware does not hold. Set the one your cloud uses.
# on AWS
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
# on Google Cloud
export PROJECT_ID=my-project
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')Google wants the project number, not the project id. It is the all-digit one, and a provider name built with the id is rejected.
Connecting to AWS
Create and attach the connector
Name the IAM role you will create. The role does not have to exist yet, because the connector only records its ARN.
export ROLE_NAME=runware-s3-reader
curl -X POST https://api.serverless.runware.ai/v1/oidc-connectors \
-H "Authorization: Bearer $RUNWARE_API_KEY" \
-H "Content-Type: application/json" \
-d @- <<JSON
{
"name": "s3-reader",
"kind": "aws",
"aws": { "roleArn": "arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME" }
}
JSONThe response carries issuer and audience. Leave audience out, as here, and it is runware:$ORG_UUID. Then attach the connector to the app, and add ?environment=<name> for a copy outside production.
curl -X POST https://api.serverless.runware.ai/v1/apps/$APP_ID/oidc-connectors \
-H "Authorization: Bearer $RUNWARE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "connectorName": "s3-reader" }'Attaching rolls the app's live deployment, so new workers start with the token.
Register the issuer in IAM
Create an IAM OIDC identity provider for the issuer, with the connector's audience as its client ID. One provider serves every app of your organization, because the default audience is the same for all of them.
aws iam create-open-id-connect-provider \
--url https://oidc.serverless.runware.ai \
--client-id-list runware:$ORG_UUIDIf you already have the provider and set a custom audience on a connector, add it with aws iam add-client-id-to-open-id-connect-provider.
Create the role and its trust policy
The trust policy names the provider as a federated principal and pins both claims: StringEquals on aud, and StringEquals or StringLike on sub.
cat > trust-policy.json <<JSON
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::$ACCOUNT_ID:oidc-provider/oidc.serverless.runware.ai"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.serverless.runware.ai:aud": "runware:$ORG_UUID",
"oidc.serverless.runware.ai:sub": "worker:org:$ORG_UUID:env:$ENV_UUID:app:$APP_ID"
}
}
}
]
}
JSONThe exact subject above admits one app in one environment. To admit every app of the environment, move sub into a StringLike block with the subjectPattern value, which ends in :*. Then create the role with the name the connector records, and give it only the permissions the app needs.
aws iam create-role \
--role-name $ROLE_NAME \
--assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy \
--role-name $ROLE_NAME \
--policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccessUse it from your app
The worker has AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE set, which every AWS SDK reads on its own. Create the client once, in load, and keep it: it exchanges the token for credentials and does it again when they expire.
import boto3
@serve
class Thumbnails:
def load(self) -> None:
self.s3 = boto3.client("s3", region_name="eu-west-1")
@endpoint
def list_inputs(self, payload: dict) -> dict:
page = self.s3.list_objects_v2(Bucket="my-inputs", MaxKeys=10)
return {"keys": [item["Key"] for item in page.get("Contents", [])]}Make sure that it works
sts:GetCallerIdentity needs no permissions of its own and is free to call, so it proves the exchange without touching a bucket. Deploy this endpoint and call it once.
import boto3
@serve
class Check:
def load(self) -> None:
self.sts = boto3.client("sts")
@endpoint
def identity(self, payload: dict) -> dict:
return {"arn": self.sts.get_caller_identity()["Arn"]}The ARN it returns is the assumed role, so an answer means the token reached AWS, the trust policy matched and the role was assumed. Anything else is a trust problem, and the AWS entries below name each one.
Connecting to Google Cloud
Create and attach the connector
The connector names the workload identity provider by its full resource name. The name is fixed by the ids you choose, so you can create the connector before the pool exists. Name the pool and the provider, and build the member string every grant below reuses.
export POOL_ID=runware
export PROVIDER_ID=runware-workers
export SERVICE_ACCOUNT=gcs-reader@my-project.iam.gserviceaccount.com
export POOL=projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$POOL_ID
export MEMBER=principal://iam.googleapis.com/$POOL/subject/worker:org:$ORG_UUID:env:$ENV_UUID:app:$APP_IDcurl -X POST https://api.serverless.runware.ai/v1/oidc-connectors \
-H "Authorization: Bearer $RUNWARE_API_KEY" \
-H "Content-Type: application/json" \
-d @- <<JSON
{
"name": "gcs-reader",
"kind": "gcp",
"gcp": {
"workloadIdentityProvider": "$POOL/providers/$PROVIDER_ID",
"serviceAccountEmail": "$SERVICE_ACCOUNT"
}
}
JSONserviceAccountEmail is optional. With it, the worker impersonates that service account after the exchange. Without it, you grant roles to the federated identity directly. The audience of a gcp connector is always the provider's name, the value Google expects, so leave it out.
Attach it to the app and read the subject, exactly as for AWS:
curl -X POST https://api.serverless.runware.ai/v1/apps/$APP_ID/oidc-connectors \
-H "Authorization: Bearer $RUNWARE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "connectorName": "gcs-reader" }'Create the pool and the provider
Point the provider at the issuer URL, map the claims you want to write conditions on, and restrict it with an attribute condition on the subject.
gcloud iam workload-identity-pools create $POOL_ID \
--location=global \
--display-name="Runware Serverless"
gcloud iam workload-identity-pools providers create-oidc $PROVIDER_ID \
--location=global \
--workload-identity-pool=$POOL_ID \
--issuer-uri=https://oidc.serverless.runware.ai \
--attribute-mapping="google.subject=assertion.sub,attribute.org_id=assertion.org_id,attribute.environment_id=assertion.environment_id,attribute.app_id=assertion.app_id" \
--attribute-condition="assertion.sub.startsWith('worker:org:$ORG_UUID:env:$ENV_UUID:')"Leave --allowed-audiences out. Google then accepts the provider's own name, which is what the token carries.
Do not upload a copy of the Runware keys to the provider. Google lets a provider hold an uploaded JWKS instead of reading it from the issuer, but Runware rotates its signing keys, and an uploaded set stops working at the next rotation. Use --issuer-uri as above, so Google fetches the current keys itself.
Grant access
With a service account on the connector, let the app's identity impersonate it:
gcloud iam service-accounts add-iam-policy-binding $SERVICE_ACCOUNT \
--role=roles/iam.workloadIdentityUser \
--member="$MEMBER"Without one, grant the role on the resource to the same member:
gcloud storage buckets add-iam-policy-binding gs://my-inputs \
--role=roles/storage.objectViewer \
--member="$MEMBER"To grant every app of the environment at once, swap the subject for a mapped attribute, and keep the attribute condition that pins your organization.
gcloud storage buckets add-iam-policy-binding gs://my-inputs \
--role=roles/storage.objectViewer \
--member="principalSet://iam.googleapis.com/$POOL/attribute.environment_id/$ENV_UUID"Use it from your app
The worker has GOOGLE_APPLICATION_CREDENTIALS pointing at an external account credential file, which google-auth and every Google client library read as Application Default Credentials. Create the client once and keep it.
from google.cloud import storage
@serve
class Thumbnails:
def load(self) -> None:
self.gcs = storage.Client(project="my-project")
@endpoint
def list_inputs(self, payload: dict) -> dict:
blobs = self.gcs.list_blobs("my-inputs", max_results=10)
return {"names": [blob.name for blob in blobs]}Pass project explicitly: a federated credential does not always carry one. For lower level use, google.auth.default() returns the same credentials.
Make sure that it works
Reading the project's own metadata proves the chain on a project with no billing, so you can confirm the setup before you put a bucket behind it. Grant roles/browser, which reads that and nothing else.
gcloud projects add-iam-policy-binding $PROJECT_ID \
--role=roles/browser \
--member="$MEMBER"Then call Resource Manager from an endpoint, with the credentials the worker already has.
import google.auth
from google.auth.transport.requests import AuthorizedSession
@serve
class Check:
def load(self) -> None:
credentials, _ = google.auth.default(
scopes=["https://www.googleapis.com/auth/cloud-platform"]
)
self.session = AuthorizedSession(credentials)
@endpoint
def project(self, payload: dict) -> dict:
response = self.session.get(
"https://cloudresourcemanager.googleapis.com/v3/projects/my-project"
)
return {"status": response.status_code}A 200 means the token was accepted, exchanged and authorized. A 403 means the exchange worked and the grant did not reach this member, and the exchange itself fails earlier with one of the Google entries below.
Connecting to your own service
The generic kind is for anything else that can verify an OpenID Connect token: your own API, a partner's service, or a third-party product with an OIDC trust setting. You choose the audience, and the worker gets the raw token file.
curl -X POST https://api.serverless.runware.ai/v1/oidc-connectors \
-H "Authorization: Bearer $RUNWARE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "billing-api",
"kind": "generic",
"audience": "https://billing.example.com"
}'An audience that starts with runware: is reserved, except your own organization's runware:<organization uuid>. Attach the connector to the app as above.
Sending the token
RUNWARE_OIDC_TOKEN_FILE names the file. Read it on every request instead of once at startup, because the platform replaces it before it expires.
import os
import requests
def call_billing(body: dict) -> dict:
with open(os.environ["RUNWARE_OIDC_TOKEN_FILE"]) as f:
token = f.read().strip()
response = requests.post(
"https://billing.example.com/usage",
json=body,
headers={"Authorization": f"Bearer {token}"},
timeout=10,
)
response.raise_for_status()
return response.json()Verifying the token
Your service reads the issuer's discovery document at https://oidc.serverless.runware.ai/.well-known/openid-configuration, which names the JWKS URL. Fetch the keys from there and cache them by kid, and fetch again when a token arrives with a kid you have not seen, because a rotation adds new keys. Then check:
- the signature, with
RS256only, issequalshttps://oidc.serverless.runware.ai,audequals the connector's audience,expis in the future andnbfis not, with a small allowance for clock skew,subequals the subject you trust or starts with your environment's prefix, ororg_idequals your organization's id.
import jwt
import requests
ISSUER = "https://oidc.serverless.runware.ai"
AUDIENCE = "https://billing.example.com"
TRUSTED_PREFIX = "worker:org:<organization uuid>:env:<environment uuid>:"
discovery = requests.get(f"{ISSUER}/.well-known/openid-configuration", timeout=5).json()
keys = jwt.PyJWKClient(discovery["jwks_uri"])
def verify(token: str) -> dict:
signing_key = keys.get_signing_key_from_jwt(token)
claims = jwt.decode(
token,
signing_key.key,
algorithms=["RS256"],
audience=AUDIENCE,
issuer=ISSUER,
leeway=60,
options={"require": ["iss", "sub", "aud", "exp", "iat"]},
)
if not claims["sub"].startswith(TRUSTED_PREFIX):
raise PermissionError("token is not from a trusted app")
return claimsThe example uses PyJWT with its crypto extra, pip install "pyjwt[crypto]" requests. Any OIDC library that reads a JWKS URL does the same checks.
The jti claim is unique per token, so a service that must refuse a replay can record the ones it has seen until they expire.
What the worker gets
The platform mounts a read-only directory at /var/run/secrets/runware/oidc and sets the variables for each attached kind. The file variables hold a path inside that directory, at the name below. The first two come with an aws connector, the third with a gcp one and the fourth with a generic one.
| Variable | Value |
AWS_ROLE_ARN | The role on the aws connector |
AWS_WEB_IDENTITY_TOKEN_FILE | aws-token |
GOOGLE_APPLICATION_CREDENTIALS | gcp-credential-configuration.json |
RUNWARE_OIDC_TOKEN_FILE | generic-token |
AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE and RUNWARE_OIDC_TOKEN_FILE belong to the connector only while one of that kind is attached. Attaching to an app that already sets one of them as an environment variable or a secret returns 422, and so does setting one later. Whatever else sets one of them, such as an older version the app is pinned to, the worker gets the connector's value. An app with no connector of that kind can set them as before.
GOOGLE_APPLICATION_CREDENTIALS and RUNWARE_OIDC_CONFIG_DIGEST are reserved on every app, with or without a connector, and setting either returns 422. The platform sets RUNWARE_OIDC_CONFIG_DIGEST on workers that have a connector, and your code has no use for it. The token directory is also reserved for volumes: a mount path on it, under it or over it, such as /var, returns 422.
The token
A token is a JWT signed with RS256, and its header carries the kid of the signing key.
| Claim | Value |
iss, aud, sub | As described above |
org_id | Your organization's id |
environment_id | The id of the environment the app runs in |
app_id | The app id |
deployment_id | The id of the deployment the worker belongs to |
worker | The worker's name |
connector | The connector's name |
jti | Unique per token |
iat, nbf, exp | When it was issued, when it becomes valid, and when it expires |
AWS trust policies can match only aud and sub. A Google attribute condition or your own verifier can match any claim, for example deployment_id to trust one deployment only.
A token lives at most one hour. Do not depend on an exact lifetime, because it may get shorter. The platform replaces the token file at half its lifetime, including on a worker that was resumed after scaling to zero, and the AWS and Google SDKs read the new file when their credentials run out.
Runware rotates the signing keys. New keys are published at the issuer before they sign anything, and old keys stay until every token they signed has expired. If a signing key is ever compromised, Runware removes it and tells every affected organization when the exposure started and ended, which subjects were used, and what to revoke.
Limits
- One connector of each kind per app. Attaching a second
awsconnector, or the same one twice, returns409. - A connector belongs to the organization, not to an environment. Any app in any environment can attach it, and the environment shows up only in
sub. - Changes reach workers when they roll. Attaching, updating and detaching roll the app's live deployment. An app that is not live, or one with a rollout already in progress, picks the change up on its next rollout.
- Detaching does not revoke access at once. A worker that still runs keeps getting tokens until it rolls, a token stays valid until it expires, and the cloud credentials it was exchanged for stay valid for their own lifetime. The worst case is one hour of token plus the credential lifetime: one hour on Google, and on AWS the role's maximum session duration, one hour by default and up to twelve. To cut access at once, remove the trust or the grant on your cloud's side, and on AWS revoke the role's active sessions.
- A connector cannot be deleted while any app attaches it, a stopped or failed one included. Detach it first, or delete the app.
- The audience is one string, never a list, of at most 256 characters with no whitespace.
- A name is 2 to 63 characters: lowercase letters, digits and hyphens, starting with a letter and ending with a letter or a digit. The name and the kind are fixed once created, and
site-storeis reserved for a platform connector, so it is not available for one of yours. - An update replaces the whole configuration, so send every field you want to keep. An
awsupdate withoutaudiencegoes back to the default.
Troubleshooting
On AWS
The IAM OIDC provider is missing, or its URL is not exactly
https://oidc.serverless.runware.ai, with no trailing slash.
The provider's client ID list does not contain the connector's
audience.
The trust policy does not match the token. Compare its
subwith thesubjectthe API returns, check the environment, and check thataudmatches. Also check that the connector'sroleArnnames this role.
AWS could not reach the issuer for a moment.
boto3 raises it when the worker has no connector. Check that the connector is attached in the app's environment, and that the deployment has rolled since.
On Google Cloud
The pool or the provider does not exist, is disabled, or the connector's
workloadIdentityProvidernames another one. Use the project number, not the project id.
When the message names the audience, the provider has
--allowed-audiencesset to something else. Clear it, so Google accepts its own name.When it names the signature or the key on a setup that worked before, the provider holds an uploaded JWKS. Switch it to the issuer URL.
The condition does not match the token. Compare it with the
subjectthe API returns.
The app's identity has no
roles/iam.workloadIdentityUseron the service account, or the member's subject does not match.
The exchange worked and Google refused the call itself. The service account, or the federated identity without one, lacks a role on the resource.
From the Serverless API
| Status | Cause |
409 on create | Your organization already has a connector of that name |
409 on attach | The app already has a connector of that kind, or this one |
409 on delete | An app still attaches the connector |
409 on deploying an older version | That version sets one of the connector's variables. Remove it, or detach the connector first |
422 on attach, or on setting a variable or a secret | A name the connector sets is already set on the app, or you are setting one while the connector is attached. GOOGLE_APPLICATION_CREDENTIALS and RUNWARE_OIDC_CONFIG_DIGEST return it on every app |
422 on create or update | A field breaks its rule, for example a custom audience on a gcp connector, or another organization's runware: audience |
403 | Your organization is not active, so its connectors cannot change |