Skip to content

GCP Messaging (Pub/Sub, Cloud Tasks, Eventarc, Managed Kafka)

Scope

Pub/Sub (topics, subscriptions, schemas, BigQuery subscriptions), Cloud Tasks (managed task queues), Eventarc (event-driven triggers), Managed Service for Apache Kafka, message patterns and delivery semantics.

Checklist

  • [Critical] Choose messaging service based on pattern: Pub/Sub for asynchronous publish/subscribe with decoupled producers and consumers (at-least-once or exactly-once delivery), Cloud Tasks for managed task queues with rate limiting and scheduled delivery to HTTP targets, Eventarc for event-driven triggers from GCP services and custom sources to Cloud Run, GKE, or Workflows, Managed Service for Apache Kafka for workloads requiring Kafka protocol compatibility or migrating from self-managed Kafka
  • [Recommended] Configure Pub/Sub subscriptions with appropriate acknowledgment deadline (10 seconds to 600 seconds, default 10s); extend deadline using modAckDeadline for long-running processing; messages redelivered after deadline expiry lead to duplicate processing if not handled idempotently
  • [Recommended] Enable Pub/Sub exactly-once delivery on subscriptions that require it: prevents redelivery of acknowledged messages within the deduplication window; requires pull subscriptions (not push); consumers must still be idempotent for edge cases and message replay scenarios
  • [Optional] Configure Pub/Sub ordering keys for FIFO ordering within a key: messages with the same ordering key are delivered in publish order to pull subscribers; ordering keys limit throughput to 1 MB/s per key; unacknowledged messages block subsequent messages with the same ordering key
  • [Recommended] Set up Pub/Sub dead-letter topics: configure maximum delivery attempts (5-100, default no DLQ) and dead-letter topic destination; messages exceeding max attempts are forwarded to the dead-letter topic with delivery attempt metadata; monitor DLQ subscription for investigation and reprocessing
  • [Optional] Design Pub/Sub schemas for message validation: Avro or Protocol Buffer schemas registered in Schema Registry; schemas enforce structure at publish time (messages failing validation are rejected); supports schema revisions with compatibility checking (backward, forward, full)
  • [Recommended] Configure Pub/Sub BigQuery subscriptions for zero-code analytics ingestion: messages written directly to BigQuery tables with automatic schema mapping; supports both write and use topic schema modes; eliminates the need for a Dataflow pipeline for simple ingestion patterns
  • [Recommended] Set up Cloud Tasks queues with rate limiting: max dispatches per second (rate at which tasks are dispatched to the target), max concurrent dispatches (parallel execution limit), and retry configuration (max attempts, min/max backoff, max doublings); rate limiting protects downstream services from overload
  • [Recommended] Design Eventarc triggers for event-driven architectures: system events from 130+ GCP services (Cloud Storage object finalized, BigQuery job completed, Firestore document written) via Cloud Audit Logs or direct events; custom events published to Eventarc channels via Pub/Sub
  • [Optional] Configure Eventarc destinations: Cloud Run services (HTTP push), GKE services (HTTP push via GKE destinations), Workflows (start workflow execution); each trigger creates an underlying Pub/Sub subscription; use event filters to narrow trigger scope (resource name, method name, service name)
  • [Optional] Implement Pub/Sub message retention and replay: configure topic-level message retention (10 minutes to 31 days) for replay capability; seek subscriptions to a timestamp or snapshot to replay historical messages; useful for recovering from consumer bugs or onboarding new consumers
  • [Recommended] Monitor messaging with Cloud Monitoring: Pub/Sub oldest unacked message age (critical alert for consumer lag), subscription/num_undelivered_messages (backlog depth), Cloud Tasks queue depth and dispatch rate; set alerting policies on consumer health metrics
  • [Recommended] Design Pub/Sub for cross-project and cross-region messaging: Pub/Sub topics are global resources (publish from any region, messages routed to the nearest Google datacenter); use IAM to grant cross-project publish/subscribe access; message storage location can be restricted to specific regions using message storage policy
  • [Optional] Evaluate Managed Service for Apache Kafka for Kafka-native workloads: fully managed Apache Kafka clusters with Kafka protocol compatibility; suitable for teams with existing Kafka expertise, Kafka Connect ecosystems, or migrating from self-managed Kafka; supports standard Kafka client libraries and tooling

Why This Matters

GCP's messaging services cover four distinct patterns. Pub/Sub is a general-purpose message bus for decoupling producers from consumers with at-least-once delivery and optional exactly-once semantics. Cloud Tasks is a managed task queue for dispatching work items to HTTP endpoints with rate control -- it is not a general messaging system. Eventarc provides the reactive event routing layer connecting GCP service events to processing targets. Managed Service for Apache Kafka provides Kafka protocol-compatible managed clusters for teams invested in the Kafka ecosystem. Using the wrong service leads to overengineered solutions or missing capabilities.

Pub/Sub's ordering key behavior is a common source of bugs: when ordering is enabled and a message fails acknowledgment, all subsequent messages with the same ordering key are blocked until the failed message is acknowledged or nacked. This provides strict ordering but can cause entire key-scoped message streams to stall. Applications must handle nack properly or messages accumulate indefinitely.

BigQuery subscriptions represent a major simplification for analytics pipelines. Previously, ingesting Pub/Sub messages into BigQuery required a Dataflow streaming pipeline (with its operational overhead and cost). BigQuery subscriptions eliminate this intermediate step for straightforward ingestion, though Dataflow remains necessary for transformation, enrichment, or windowed aggregation.

Common Decisions (ADR Triggers)

  • Pub/Sub vs Cloud Tasks -- Pub/Sub is publish/subscribe: multiple subscribers can independently consume copies of each message, and producers are fully decoupled from consumers. Cloud Tasks is a task queue: each task is delivered to exactly one handler, with explicit rate limiting, scheduling (deliver at a future time), and deduplication. Use Pub/Sub for event fan-out, stream processing, and multi-consumer patterns. Use Cloud Tasks for work dispatch with rate control (API rate limiting, webhook delivery) and delayed execution.
  • Pub/Sub vs Managed Kafka -- Pub/Sub is serverless with automatic scaling, per-message pricing, and no cluster management. Managed Kafka provides Kafka protocol compatibility, partition-based ordering, and Kafka Connect ecosystem but requires cluster capacity planning. Use Pub/Sub for new GCP-native workloads. Use Managed Kafka for Kafka ecosystem migrations, existing Kafka client libraries, or workloads requiring Kafka-specific features (compacted topics, consumer groups with partition assignment).
  • Pub/Sub push vs pull subscriptions -- Push subscriptions deliver messages via HTTP POST to an endpoint (Cloud Run, App Engine, HTTPS webhook); Google manages the polling and delivery. Pull subscriptions require consumers to actively pull messages (streaming pull for low latency or synchronous pull for batch). Push is simpler for serverless targets and does not require consumer infrastructure. Pull provides more control (batching, flow control, exactly-once support) and works behind firewalls. Use push for Cloud Run and serverless. Use pull for GKE, Compute Engine, or when exactly-once delivery is needed.
  • Eventarc vs direct Pub/Sub subscription for GCP events -- Eventarc provides a unified event model with CloudEvents format, filtering by event attributes, and direct integration with Cloud Run and Workflows. Direct Pub/Sub consumption of Cloud Audit Logs requires setting up a log sink to Pub/Sub and parsing raw audit log entries. Eventarc is simpler for reactive processing of GCP events. Direct Pub/Sub provides more flexibility for complex filtering, batch processing, or non-supported event sources.
  • Pub/Sub ordering keys vs application-level ordering -- Pub/Sub ordering keys guarantee delivery order within a key but limit throughput (1 MB/s per key) and block on failed messages. Application-level ordering uses sequence numbers in message attributes with consumer-side reordering buffers. Pub/Sub ordering is simpler but creates head-of-line blocking risk. Application-level ordering is more complex but decouples ordering from delivery. Use Pub/Sub ordering for moderate-throughput, critical-ordering scenarios (financial transactions). Use application-level ordering for high-throughput or when blocking is unacceptable.
  • BigQuery subscription vs Dataflow for analytics ingestion -- BigQuery subscriptions provide zero-code direct ingestion with automatic schema mapping and no infrastructure management. Dataflow provides transformation (filtering, enrichment, format conversion), windowed aggregation, and joining with other data sources. Use BigQuery subscriptions when messages match the target table schema directly. Use Dataflow when transformation or aggregation is needed before loading.
  • Pub/Sub message storage policy -- By default, Pub/Sub stores messages in the nearest Google Cloud region to the publisher. Message storage policies restrict storage to specific regions for data residency compliance (e.g., messages must remain in europe-west1). Restricted policies may increase publish latency if publishers are in different regions. Use unrestricted storage for lowest latency. Use restricted policies for data sovereignty requirements.

Reference Architectures

Event-Driven Microservices with Pub/Sub

Order service publishes OrderCreated events to Pub/Sub topic (Avro schema enforced). Three subscriptions: (1) payment-sub -> Cloud Run payment service (push), (2) inventory-sub -> GKE inventory service (pull with exactly-once), (3) analytics-sub -> BigQuery subscription (direct to analytics.orders table). Dead-letter topics on payment-sub and inventory-sub (max 5 attempts). Ordering keys on order ID for per-order sequencing on payment subscription. Cloud Monitoring alerting on oldest_unacked_message_age > 5 minutes.

Managed Rate-Limited API Integration

Cloud Tasks queue with max dispatches per second = 50 (matching downstream API rate limit), max concurrent dispatches = 10. Cloud Scheduler creates periodic tasks to sync data from external API. Tasks target Cloud Run service that calls external API, processes response, and writes to Cloud SQL. Retry policy: max 10 attempts, 10s min backoff, 300s max backoff. Failed tasks after max retries written to Pub/Sub dead-letter topic for manual review. Cloud Monitoring on queue depth and dispatch success rate.

Reactive Infrastructure Automation with Eventarc

Eventarc triggers: (1) Cloud Storage object finalized in upload bucket -> Cloud Run function processes file (image resize, virus scan, metadata extraction), (2) Cloud SQL instance created (via audit log) -> Workflows execution configures backup policy and monitoring, (3) Secret Manager secret rotation event -> Cloud Run function updates dependent services. Event filters scoped to specific buckets, projects, and resource types. Each trigger creates managed Pub/Sub infrastructure automatically.

High-Throughput Streaming Pipeline

Application publishers -> Pub/Sub topic (message storage policy: us-central1 region). Two consumer paths: (1) Dataflow streaming pipeline for real-time aggregation (windowed counts, anomaly detection) -> BigQuery streaming insert to real-time dashboard table, (2) BigQuery subscription for raw event archival to cold storage table (partitioned by publish time, clustered by device ID). Topic retention = 7 days for replay capability. Subscription message retention = 7 days matching topic. Cloud Monitoring alert on subscription backlog exceeding 100,000 messages.

See Also

  • providers/gcp/serverless.md -- Cloud Run and event-driven architectures
  • providers/gcp/containers.md -- GKE workloads consuming Pub/Sub
  • providers/gcp/observability.md -- monitoring messaging pipelines