A Terraform module for creating a PubSub Subscriber for Google Cloud Pub/Sub.
This module supports Terraform version 1 and is compatible with the Terraform Google Provider version 5.
This module is part of our Infrastructure as Code (IaC) framework that enables our users and customers to easily deploy and manage reusable, secure, and production-grade cloud infrastructure.
- Module Features
- Getting Started
- Module Argument Reference
- Module Outputs
- External Documentation
- Module Versioning
- About Mineiros
- Reporting Issues
- Contributing
- Makefile Targets
- License
This module implements the following Terraform resources
google_pubsub_subscription
and supports additional features of the following modules:
Most common usage of the module:
module "terraform-google-pubsub-subscription" {
source = "[email protected]:mineiros-io/terraform-google-pubsub-subscription.git?ref=v0.1.0"
name = "subscription-name"
topic = "topic-resource"
project = "resource-project"
}
See variables.tf and examples/ for details and use-cases.
name = "name-of-sub" topic = "topic-resource" project = "project-a"
-
name
: (Requiredstring
)Name of the subscription.
-
topic
: (Requiredstring
)A reference to a Topic resource.
-
project
: (Requiredstring
)The project in which the resource belongs. If it is not provided, the provider project is used.
-
labels
: (Optionalmap(string)
)A set of key/value label pairs to assign to this Subscription.
Default is
{}
. -
ack_deadline_seconds
: (Optionalnumber
)This value is the maximum time after a subscriber receives a message before the subscriber should acknowledge the message. After message delivery but before the ack deadline expires and before the message is acknowledged, it is an outstanding message and will not be delivered again during that time (on a best-effort basis). For pull subscriptions, this value is used as the initial value for the ack deadline. To override this value for a given message, call
subscriptions.modifyAckDeadline
with the correspondingackId
if using pull. The minimum custom deadline you can specify is10
seconds. The maximum custom deadline you can specify is600
seconds (10 minutes). If this parameter is0
, a default value of10
seconds is used. For push delivery, this value is also used to set the request timeout for the call to the push endpoint. If the subscriber never acknowledges the message, the Pub/Sub system will eventually redeliver the message. -
message_retention_duration
: (Optionalstring
)How long to retain unacknowledged messages in the subscription's backlog, from the moment a message is published. If
retainAckedMessages
istrue
, then this also configures the retention of acknowledged messages, and thus configures how far back in time asubscriptions.seek
can be done. Defaults to 7 days. Cannot be more than 7 days (604800s
) or less than 10 minutes (600s
). A duration in seconds with up to nine fractional digits, terminated bys
.Default is
"604800s"
.Example:
600.5s
-
retain_acked_messages
: (Optionalbool
)Indicates whether to retain acknowledged messages. If
true
, then messages are not expunged from the subscription's backlog, even if they are acknowledged, until they fall out of themessageRetentionDuration
window. -
filter
: (Optionalstring
)The subscription only delivers the messages that match the filter. Pub/Sub automatically acknowledges the messages that don't match the filter. You can filter messages by their attributes. The maximum length of a filter is 256 bytes. After creating the subscription, you can't modify the filter.
-
enable_message_ordering
: (Optionalbool
)If
true
, messages published with the sameorderingKey
inPubsubMessage
will be delivered to the subscribers in the order in which they are received by the Pub/Sub system. Otherwise, they may be delivered in any order. -
enable_exactly_once_delivery
: (Optionalbool
)If
true
,The message sent to a subscriber is guaranteed not to be resent before the message's acknowledgement deadline expires. An acknowledged message will not be resent to a subscriber. Note that subscribers may still receive multiple copies of a message when enable_exactly_once_delivery is true if the message was published multiple times by a publisher client. These copies are considered distinct by Pub/Sub and have distinct messageId values -
expiration_policy_ttl
: (Optionalstring
)A policy that specifies the conditions for this subscription's expiration. A subscription is considered active as long as any connected subscriber is successfully consuming messages from the subscription or is issuing operations on the subscription. If
expirationPolicy
is not set, a default policy with ttl of 31 days will be used. If it is set but ttl is "", the resource never expires. The minimum allowed value forexpirationPolicy.ttl
is 1 day. A duration in seconds with up to nine fractional digits, terminated by 's'. -
dead_letter_policy
: (Optionalobject(dead_letter_policy)
)A policy that specifies the conditions for dead lettering messages in this subscription. If
dead_letter_policy
is not set, dead lettering is disabled. The Cloud Pub/Sub service account associated with this subscription's parent project (i.e.,service-{project_number}@gcp-sa-pubsub.iam.gserviceaccount.com)
must have permission toAcknowledge()
messages on this subscription.The
dead_letter_policy
object accepts the following attributes:-
dead_letter_topic
: (Optionalstring
)The name of the topic to which dead letter messages should be published. Format is
projects/{project}/topics/{topic}
. The Cloud Pub/Sub service account associated with the enclosing subscription's parent project (i.e.,service-{project_number}@gcp-sa-pubsub.iam.gserviceaccount.com
) must have permission toPublish()
to this topic. The operation will fail if the topic does not exist. Users should ensure that there is a subscription attached to this topic since messages published to a topic with no subscriptions are lost. -
max_delivery_attempts
: (Optionalnumber
)The maximum number of delivery attempts for any message. The value must be between 5 and 100. The number of delivery attempts is defined as 1 + (the sum of number of NACKs and number of times the acknowledgement deadline has been exceeded for the message). A NACK is any call to
ModifyAckDeadline
with a 0 deadline. Note that client libraries may automatically extendack_deadlines
. This field will be honored on a best effort basis. If this parameter is 0, a default value of 5 is used.
-
-
retry_policy
: (Optionalobject(retry_policy)
)A policy that specifies how Pub/Sub retries message delivery for this subscription. If not set, the default retry policy is applied. This generally implies that messages will be retried as soon as possible for healthy subscribers.
RetryPolicy
will be triggered on NACKs or acknowledgement deadline exceeded events for a given message.The
retry_policy
object accepts the following attributes:-
minimum_backoff
: (Optionalstring
)The minimum delay between consecutive deliveries of a given message. Value should be between 0 and 600 seconds. A duration in seconds with up to nine fractional digits, terminated by
s
.Example:
3.5s
-
maximum_backoff
: (Optionalstring
)The maximum delay between consecutive deliveries of a given message. Value should be between 0 and 600 seconds. A duration in seconds with up to nine fractional digits, terminated by
s
.Example:
3.5s
-
-
push_config
: (Optionalobject(push_config)
)If push delivery is used with this subscription, this field is used to configure it. An empty
pushConfig
signifies that the subscriber will pull and ack messages using API methods.The
push_config
object accepts the following attributes:-
oidc_token
: (Optionalobject(oidc_token)
)If specified, Pub/Sub will generate and attach an OIDC JWT token as an Authorization header in the HTTP request for every pushed message.
The
oidc_token
object accepts the following attributes:-
service_account_email
: (Requiredstring
)Service account email to be used for generating the OIDC token. The caller (for
subscriptions.create
,subscriptions.patch
, andsubscriptions.modifyPushConfig
RPCs) must have theiam.serviceAccounts.actAs
permission for the service account. -
audience
: (Optionalstring
)Audience to be used when generating OIDC token. The audience claim identifies the recipients that the JWT is intended for. The audience value is a single case-sensitive string. Having multiple values (array) for the audience field is not supported. More info about the OIDC JWT token audience here: https://tools.ietf.org/html/rfc7519#section-4.1.3 Note: if not specified, the Push endpoint URL will be used.
-
-
no_wrapper
: (Optionalobject(no_wrapper)
)When set, the payload to the push endpoint is not wrapped. Sets the data field as the HTTP body for delivery.
The
no_wrapper
object accepts the following attributes:-
write_metadata
: (Requiredstring
)When true, writes the Pub/Sub message metadata to x-goog-pubsub-: headers of the HTTP request. Writes the Pub/Sub message attributes to : headers of the HTTP request.
-
-
push_endpoint
: (Requiredstring
)A URL locating the endpoint to which messages should be pushed. For example, a Webhook endpoint might use "https://example.com/push".
-
attributes
: (Requiredstring
)Endpoint configuration attributes. Every endpoint has a set of API supported attributes that can be used to control different aspects of the message delivery. The currently supported attribute is
x-goog-version
, which you can use to change the format of the pushed message. This attribute indicates the version of the data expected by the endpoint. This controls the shape of the pushed message (i.e., its fields and metadata). The endpoint version is based on the version of the Pub/Sub API. If not present during thesubscriptions.create
call, it will default to the version of the API used to make such call. If not present during asubscriptions.modifyPushConfig
call, its value will not be changed.subscriptions.get
calls will always return a valid version, even if the subscription was created without this attribute. The possible values for this attribute are:- v1beta1: uses the push format defined in the v1beta1 Pub/Sub API.
- v1 or v1beta2: uses the push format defined in the v1 Pub/Sub API.
-
-
bigquery_config
: (Optionalobject(bigquery_config)
)If delivery to BigQuery is used with this subscription, this field is used to configure it. Either
push_config
orbigquery_config
can be set, but not both. If both are empty, then the subscriber will pull and ack messages using API methods."The
bigquery_config
object accepts the following attributes:-
table
: (Requiredstring
)The name of the table to which to write data, of the form
{projectId}.{datasetId}.{tableId}
-
use_topic_schema
: (Optionalbool
)When
true
, use the topic's schema as the columns to write to in BigQuery, if it exists. Only one of use_topic_schema and use_table_schema can be set. -
use_table_schema
: (Optionalbool
)When true, use the BigQuery table's schema as the columns to write to in BigQuery. Messages must be published in JSON format. Only one of use_topic_schema and use_table_schema can be set.
-
write_metadata
: (Optionalbool
)When
true
, write the subscription name, messageId, publishTime, attributes, and orderingKey to additional columns in the table. The subscription name, messageId, and publishTime fields are put in their own columns while all other message properties (other than data) are written to a JSON object in the attributes column. -
drop_unknown_fields
: (Optionalbool
)When
true
anduse_topic_schema
istrue
, any fields that are a part of the topic schema that are not part of the BigQuery table schema are dropped when writing to BigQuery. Otherwise, the schemas must be kept in sync and any messages with extra fields are not written and remain in the subscription's backlog. -
service_account_email
: (Optionalstring
)The service account to use to write to BigQuery. If not specified, the Pub/Sub service agent, service-{project_number}@gcp-sa-pubsub.iam.gserviceaccount.com, is used.
-
-
cloud_storage_config
: (Optionalobject(cloud_storage_config)
)If delivery to Cloud Storage is used with this subscription, this field is used to configure it. Either pushConfig, bigQueryConfig or cloudStorageConfig can be set, but not combined. If all three are empty, then the subscriber will pull and ack messages using API methods.
The
cloud_storage_config
object accepts the following attributes:-
bucket
: (Requiredstring
)User-provided name for the Cloud Storage bucket. The bucket must be created by the user. The bucket name must be without any prefix like "gs://".
-
filename_prefix
: (Optionalstring
)(Optional) User-provided prefix for Cloud Storage filename.
-
filename_suffix
: (Optionalstring
)(Optional) User-provided suffix for Cloud Storage filename. Must not end in "/".
-
max_duration
: (Optionalstring
)(Optional) The maximum duration that can elapse before a new Cloud Storage file is created. Min 1 minute, max 10 minutes, default 5 minutes. May not exceed the subscription's acknowledgement deadline. A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".
-
max_bytes
: (Optionalnumber
)(Optional) The maximum bytes that can be written to a Cloud Storage file before a new file is created. Min 1 KB, max 10 GiB. The maxBytes limit may be exceeded in cases where messages are larger than the limit.
-
avro_config
: (Optionalobject(avro_config)
)If set, message data will be written to Cloud Storage in Avro format.
The
avro_config
object accepts the following attributes:-
write_metadata
: (Optionalbool
)When true, write the subscription name, messageId, publishTime, attributes, and orderingKey as additional fields in the output.
-
-
-
iam
: (Optionallist(iam)
)A list of IAM access.
Default is
[]
.Example:
iam = [{ role = "roles/viewer" members = ["user:[email protected]"] }]
Each
iam
object in the list accepts the following attributes:-
members
: (Optionalset(string)
)Identities that will be granted the privilege in role. Each entry can have one of the following values:
allUsers
: A special identifier that represents anyone who is on the internet; with or without a Google account.allAuthenticatedUsers
: A special identifier that represents anyone who is authenticated with a Google account or a service account.user:{emailid}
: An email address that represents a specific Google account. For example, [email protected] or [email protected].serviceAccount:{emailid}
: An email address that represents a service account. For example, [email protected].group:{emailid}
: An email address that represents a Google group. For example, [email protected].domain:{domain}
: A G Suite domain (primary, instead of alias) name that represents all the users of that domain. For example, google.com or example.com.computed:{identifier}
: An existing key fromvar.computed_members_map
.
Default is
[]
. -
role
: (Optionalstring
)The role that should be applied. Note that custom roles must be of the format
[projects|organizations]/{parent-name}/roles/{role-name}
. -
authoritative
: (Optionalbool
)Whether to exclusively set (authoritative mode) or add (non-authoritative/additive mode) members to the role.
Default is
true
.
-
-
policy_bindings
: (Optionallist(policy_binding)
)A list of IAM policy bindings.
Example:
policy_bindings = [{ role = "roles/viewer" members = ["user:[email protected]"] }]
Each
policy_binding
object in the list accepts the following attributes:-
role
: (Requiredstring
)The role that should be applied.
-
members
: (Optionalset(string)
)Identities that will be granted the privilege in
role
.Default is
var.members
. -
condition
: (Optionalobject(condition)
)An IAM Condition for a given binding.
Example:
condition = { expression = "request.time < timestamp(\"2022-01-01T00:00:00Z\")" title = "expires_after_2021_12_31" }
The
condition
object accepts the following attributes:-
expression
: (Requiredstring
)Textual representation of an expression in Common Expression Language syntax.
-
title
: (Requiredstring
)A title for the expression, i.e. a short string describing its purpose.
-
description
: (Optionalstring
)An optional description of the expression. This is a longer text which describes the expression, e.g. when hovered over it in a UI.
-
-
-
computed_members_map
: (Optionalmap(string)
)A map of members to replace in
members
of various IAM settings to handle terraform computed values.Default is
{}
.
-
module_enabled
: (Optionalbool
)Specifies whether resources in the module will be created.
Default is
true
. -
module_depends_on
: (Optionallist(dependency)
)A list of dependencies. Any object can be assigned to this list to define a hidden external dependency.
Default is
[]
.Example:
module_depends_on = [ null_resource.name ]
The following attributes are exported in the outputs of the module:
-
iam
: (list(iam)
)The iam resource objects that define access to the GCS bucket.
-
subscription
: (object(subscription)
)All attributes of the created subscription.
-
module_enabled
: (bool
)Whether this module is enabled.
- PubSub: https://cloud.google.com/pubsub
- Subscriber Overview: https://cloud.google.com/pubsub/docs/subscriber
This Module follows the principles of Semantic Versioning (SemVer).
Given a version number MAJOR.MINOR.PATCH
, we increment the:
MAJOR
version when we make incompatible changes,MINOR
version when we add functionality in a backwards compatible manner, andPATCH
version when we make backwards compatible bug fixes.
- Backwards compatibility in versions
0.0.z
is not guaranteed whenz
is increased. (Initial development) - Backwards compatibility in versions
0.y.z
is not guaranteed wheny
is increased. (Pre-release)
Mineiros is a remote-first company headquartered in Berlin, Germany that solves development, automation and security challenges in cloud infrastructure.
Our vision is to massively reduce time and overhead for teams to manage and deploy production-grade and secure cloud infrastructure.
We offer commercial support for all of our modules and encourage you to reach out if you have any questions or need help. Feel free to email us at [email protected] or join our Community Slack channel.
We use GitHub Issues to track community reported issues and missing features.
Contributions are always encouraged and welcome! For the process of accepting changes, we use Pull Requests. If you'd like more information, please see our Contribution Guidelines.
This repository comes with a handy Makefile.
Run make help
to see details on each available target.
This module is licensed under the Apache License Version 2.0, January 2004. Please see LICENSE for full details.
Copyright © 2020-2022 Mineiros GmbH