Skip to main content

Serverless Workers on GCP Cloud Run - Java SDK

View Markdown

On a GCP Cloud Run worker pool, you run a standard long-lived Temporal Worker. Register Workflows and Activities the same way you would with any other Java Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific runtime or handler. The one addition to a standard Worker is Worker Versioning, which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see Deploy a Serverless Worker on GCP Cloud Run.

Create a versioned Worker​

Build the Worker as you would any long-running Java Worker, then set WorkerDeploymentOptions on WorkerOptions to declare the Worker Deployment Version and turn versioning on.

The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace:

package example;

import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowClientOptions;
import io.temporal.common.VersioningBehavior;
import io.temporal.common.WorkerDeploymentVersion;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerDeploymentOptions;
import io.temporal.worker.WorkerFactory;
import io.temporal.worker.WorkerOptions;

String apiKey = System.getenv("TEMPORAL_API_KEY");

WorkflowServiceStubs service =
    WorkflowServiceStubs.newServiceStubs(
        WorkflowServiceStubsOptions.newBuilder()
            .setTarget(System.getenv("TEMPORAL_ADDRESS"))
            .setEnableHttps(true)
            .addApiKey(() -> apiKey)
            .build());

WorkflowClient client =
    WorkflowClient.newInstance(
        service,
        WorkflowClientOptions.newBuilder()
            .setNamespace(System.getenv("TEMPORAL_NAMESPACE"))
            .build());

WorkerFactory factory = WorkerFactory.newInstance(client);

Worker worker =
    factory.newWorker(
        System.getenv("TEMPORAL_TASK_QUEUE"),
        WorkerOptions.newBuilder()
            .setDeploymentOptions(
                WorkerDeploymentOptions.newBuilder()
                    .setUseVersioning(true)
                    .setVersion(new WorkerDeploymentVersion("my-app", "build-1"))
                    .setDefaultVersioningBehavior(VersioningBehavior.PINNED)
                    .build())
            .build());

worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class);
worker.registerActivitiesImplementations(new GreetingActivitiesImpl());

factory.start();

The two arguments to WorkerDeploymentVersion are the deployment name and the build ID, and together they identify the Worker Deployment Version. Both values must match the version you create with temporal worker deployment create-version in the deployment guide, or the Worker polls under a version the WCI does not manage.

Every Workflow needs a versioning behavior, either PINNED or AUTO_UPGRADE. Setting setDefaultVersioningBehavior as shown above covers every Workflow on the Worker. To set the behavior per Workflow instead, annotate the Workflow method with @WorkflowVersioningBehavior:

import io.temporal.common.VersioningBehavior;
import io.temporal.workflow.WorkflowVersioningBehavior;

public class GreetingWorkflowImpl implements GreetingWorkflow {
  @Override
  @WorkflowVersioningBehavior(VersioningBehavior.PINNED)
  public String run(String name) {
    // ...
  }
}

For general Worker setup and options that are not specific to Cloud Run, see Run a Worker.

Configure the Temporal connection​

Read the Namespace, address, and Task Queue from environment variables you set on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager rather than passing it in plaintext. The Worker above reads TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, TEMPORAL_API_KEY, and TEMPORAL_TASK_QUEUE, so the same image can run against any Namespace.

addApiKey takes a supplier, which the SDK calls on each request. Rotate the key by returning a new value from the supplier instead of restarting the Worker.

For TLS client certificates instead of an API key, see Connect to Temporal Cloud.

Package the Worker image​

Cloud Run runs one JVM per instance, so give the JVM a heap sized to the instance rather than to the host. Java reads the container's memory limit and defaults the maximum heap to a quarter of it, which leaves most of a small instance unused. Set -XX:MaxRAMPercentage to raise that share:

CMD ["java", "-XX:MaxRAMPercentage=75", "-jar", "/app/worker.jar"]

A Cloud Run Worker Pool defaults to 512 MiB per instance. Raise --memory when you create the pool if your Worker needs more.

Keep Activities safe across scale-in​

The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing. An instance running a long Activity can be stopped mid-execution.

Use Activity Heartbeats so a retry resumes from the last recorded progress instead of starting over:

public class GreetingActivitiesImpl implements GreetingActivities {
  @Override
  public String process(List<String> items) {
    for (int i = 0; i < items.size(); i++) {
      Activity.getExecutionContext().heartbeat(i);
      // ... process items.get(i)
    }
    return "done";
  }
}

For how scale-in decisions are made, see Serverless Workers on GCP Cloud Run.

Configure OpenTelemetry​

Run an OpenTelemetry Collector as a sidecar in the Worker Pool. The Cloud Run OpenTelemetry plugin exports metrics and traces by OTLP gRPC to the Collector at localhost:4317. It configures the endpoint and service name from Cloud Run defaults.

Create the plugin and register it on WorkflowServiceStubsOptions before you create the Client:

gcp/cloud-run/opentelemetry/src/main/java/io/temporal/samples/gcp/cloudrun/CloudRunWorker.java

ClientConfigProfile profile = ClientConfigProfile.load();
CloudRunOpenTelemetryPlugin telemetryPlugin = CloudRunOpenTelemetryPlugin.newBuilder().build();

WorkflowServiceStubsOptions serviceOptions =
    WorkflowServiceStubsOptions.newBuilder(profile.toWorkflowServiceStubsOptions())
        .setPlugins(telemetryPlugin)
        .build();
WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(serviceOptions);

Configure the Collector sidecar to receive OTLP gRPC on localhost:4317, export traces to Google Cloud, and export metrics to Google Managed Service for Prometheus:

gcp/cloud-run/opentelemetry/collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

processors:
  # Batch traces for throughput. Do not add this processor to the cumulative metrics pipeline:
  # a shutdown flush can otherwise be batched with a recent periodic export of the same series.
  batch/traces:
    send_batch_max_size: 200
    send_batch_size: 200
    timeout: 5s
  memory_limiter:
    # This is the collector's memory polling cadence, not the SDK metric export interval.
    check_interval: 1s
    limit_percentage: 65
    spike_limit_percentage: 20
  resourcedetection:
    detectors: [gcp]
    timeout: 10s
  # Avoid collisions with labels that Google Managed Service for Prometheus adds.
  transform/collision:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["exported_location"], attributes["location"])
          - delete_key(attributes, "location")
          - set(attributes["exported_cluster"], attributes["cluster"])
          - delete_key(attributes, "cluster")
          - set(attributes["exported_namespace"], attributes["namespace"])
          - delete_key(attributes, "namespace")
          - set(attributes["exported_job"], attributes["job"])
          - delete_key(attributes, "job")
          - set(attributes["exported_instance"], attributes["instance"])
          - delete_key(attributes, "instance")
          - set(attributes["exported_project_id"], attributes["project_id"])
          - delete_key(attributes, "project_id")
  # The Telemetry API expects the Google Cloud project in gcp.project_id.
  transform/set_project_id:
    error_mode: ignore
    trace_statements:
      - set(resource.attributes["gcp.project_id"], resource.attributes["gcp.project.id"]) where resource.attributes["gcp.project.id"] != nil
      - set(resource.attributes["gcp.project_id"], resource.attributes["cloud.account.id"]) where resource.attributes["gcp.project_id"] == nil and resource.attributes["cloud.account.id"] != nil

exporters:
  googlemanagedprometheus:
  # Google Cloud's supported OTLP path for traces is the Telemetry API.
  otlp:
    endpoint: telemetry.googleapis.com:443
    compression: none
    balancer_name: pick_first
    auth:
      authenticator: googleclientauth

extensions:
  # Cloud Run container dependencies require a startup probe. This endpoint is also used for the
  # collector liveness probe in worker-pool.yaml.
  health_check:
    endpoint: 0.0.0.0:13133
  googleclientauth:

service:
  extensions:
    - health_check
    - googleclientauth
  pipelines:
    metrics/otlp:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, transform/collision]
      exporters: [googlemanagedprometheus]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, transform/set_project_id, batch/traces]
      exporters: [otlp]
  # Feed collector self-metrics back through the metrics pipeline.
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: grpc
                endpoint: http://localhost:4317
                insecure: true

On shutdown, stop the WorkerFactory, then run telemetryPlugin.newFlushHook() before the Cloud Run termination window ends. For a Collector configuration that exports traces to Google Cloud and metrics to Google Managed Service for Prometheus, see the Java Cloud Run OpenTelemetry sample.

Set a Worker identity​

Use WorkerIdPlugin to identify each Worker instance as <instance-id>@<revision>. Fetch the Cloud Run metadata during process startup, then register the plugin on WorkflowClientOptions. Workers created from that Client inherit the identity.

gcp/cloud-run/workerid/src/main/java/io/temporal/samples/gcp/cloudrun/workerid/CloudRunWorker.java

GoogleCloudRunMetadata metadata = GoogleCloudRunMetadata.fetch();

String address = envOrDefault(ADDRESS_ENV, DEFAULT_ADDRESS);
String namespace = envOrDefault(NAMESPACE_ENV, DEFAULT_NAMESPACE);
String taskQueue = envOrDefault(TASK_QUEUE_ENV, DEFAULT_TASK_QUEUE);

// Plaintext connection to the Temporal Service. Configure TLS or an API key here for a secured
// Service such as Temporal Cloud.
WorkflowServiceStubs service =
    WorkflowServiceStubs.newServiceStubs(
        WorkflowServiceStubsOptions.newBuilder().setTarget(address).build());

// Register WorkerIdPlugin on the client. It sets the derived worker identity
// ({instanceId}@{revision}) on the client, and workers created from the client inherit it.
// Passing the already-fetched metadata avoids a second call to the Cloud Run metadata server.
WorkflowClient client =
    WorkflowClient.newInstance(
        service,
        WorkflowClientOptions.newBuilder()
            .setNamespace(namespace)
            .setPlugins(new WorkerIdPlugin(metadata))
            .build());

Fetching the metadata fails when the Worker runs outside Cloud Run. For a complete example, see the Java Cloud Run Worker Id sample.