From 86529b2e45fea8fb849e8ac16ca52d2edcbe6156 Mon Sep 17 00:00:00 2001 From: Jacob Cable Date: Tue, 15 Aug 2023 12:16:17 +0100 Subject: [PATCH 1/5] feat: add KMS key management param --- firestore-bigquery-export/PREINSTALL.md | 18 ++++++++++ firestore-bigquery-export/README.md | 10 ++++-- firestore-bigquery-export/extension.yaml | 8 +++++ .../package-lock.json | 4 +-- .../package.json | 2 +- .../src/bigquery/index.ts | 7 ++++ .../__snapshots__/config.test.ts.snap | 1 + .../functions/__tests__/config.test.ts | 4 ++- .../functions/package-lock.json | 14 ++++---- .../functions/package.json | 2 +- .../functions/src/config.ts | 1 + .../functions/src/index.ts | 1 + .../scripts/gen-schema-view/package-lock.json | 36 +++++++++---------- .../scripts/import/package-lock.json | 12 +++---- 14 files changed, 82 insertions(+), 38 deletions(-) diff --git a/firestore-bigquery-export/PREINSTALL.md b/firestore-bigquery-export/PREINSTALL.md index d541130ce..b1fff3f5e 100644 --- a/firestore-bigquery-export/PREINSTALL.md +++ b/firestore-bigquery-export/PREINSTALL.md @@ -47,6 +47,24 @@ Prior to sending the document change to BigQuery, you have an opportunity to tra The response should be indentical in structure. +#### Using Customer Managed Encryption Keys + +By default, BigQuery encrypts your content stored at rest. BigQuery handles and manages this default encryption for you without any additional actions on your part. + +If you want to control encryption yourself, you can use customer-managed encryption keys (CMEK) for BigQuery. Instead of Google managing the key encryption keys that protect your data, you control and manage key encryption keys in Cloud KMS. + +For more general information on this, see [the docs](https://cloud.google.com/bigquery/docs/customer-managed-encryption). + +To use CMEK and the Key Management Service (KMS) with this extension +1. [Enable the KMS API in your Google Cloud Project](https://console.cloud.google.com/apis/enableflow?apiid=cloudkms.googleapis.com). +2. Create a keyring and keychain in the KMS. Note that the region of the keyring and key *must* match the region of your bigquery dataset +3. Grant the BigQuery service account permission to encrypt and decrypt using that key. The Cloud KMS CryptoKey Encrypter/Decrypter role grants this permission. +4. When installing this extension, enter the resource name of your key. It will look something like the following: +``` +projects//locations//keyRings//cryptoKeys/ +``` +If you follow these steps, your changelog table should be created using your customer-managed encryption. + #### Backfill your BigQuery dataset This extension only sends the content of documents that have been changed -- it does not export your full dataset of existing documents into BigQuery. So, to backfill your BigQuery dataset with all the documents in your collection, you can run the [import script](https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/IMPORT_EXISTING_DOCUMENTS.md) provided by this extension. diff --git a/firestore-bigquery-export/README.md b/firestore-bigquery-export/README.md index 01aaf24e7..dbfb65b55 100644 --- a/firestore-bigquery-export/README.md +++ b/firestore-bigquery-export/README.md @@ -77,8 +77,6 @@ To install an extension, your project must be on the [Blaze (pay as you go) plan **Configuration Parameters:** -* Cloud Functions location: Where do you want to deploy the functions created for this extension? You usually want a location close to your database. For help selecting a location, refer to the [location selection guide](https://firebase.google.com/docs/functions/locations). - * BigQuery Dataset location: Where do you want to deploy the BigQuery dataset created for this extension? For help selecting a location, refer to the [location selection guide](https://cloud.google.com/bigquery/docs/locations). * Project Id: Override the default project bigquery instance. This can allow updates to be directed to a bigquery instance on another project. @@ -102,18 +100,26 @@ To install an extension, your project must be on the [Blaze (pay as you go) plan * BigQuery SQL table clustering: This parameter will allow you to set up Clustering for the BigQuery Table created by the extension. (for example: `data,document_id,timestamp`- no whitespaces). You can select up to 4 comma separated fields. The order of the specified columns determines the sort order of the data. Available schema extensions table fields for clustering: `document_id, timestamp, event_id, operation, data`. +* Maximum number of synced documents per second: This parameter will set the maximum number of syncronised documents per second with BQ. Please note, any other external updates to a Big Query table will be included within this quota. Ensure that you have a set a low enough number to componsate. Defaults to 10. + * Backup Collection Name: This (optional) parameter will allow you to specify a collection for which failed BigQuery updates will be written to. * Transform function URL: Specify a function URL to call that will transform the payload that will be written to BigQuery. See the pre-install documentation for more details. * Use new query syntax for snapshots: If enabled, snapshots will be generated with the new query syntax, which should be more performant, and avoid potential resource limitations. +* Cloud KMS key name: Instead of Google managing the key encryption keys that protect your data, you control and manage key encryption keys in Cloud KMS. If this parameter is set, the extension will specify the KMS key name when creating the BQ table. + **Cloud Functions:** * **fsexportbigquery:** Listens for document changes in your specified Cloud Firestore collection, then exports the changes into BigQuery. +* **syncBigQuery:** A task-triggered function that gets called on BigQuery sync + +* **setupBigQuerySync:** Runs configuration for sycning with BigQuery + **APIs Used**: diff --git a/firestore-bigquery-export/extension.yaml b/firestore-bigquery-export/extension.yaml index eb95afb17..64ad39b09 100644 --- a/firestore-bigquery-export/extension.yaml +++ b/firestore-bigquery-export/extension.yaml @@ -322,6 +322,14 @@ params: default: no required: true + - param: KMS_KEY_NAME + label: Cloud KMS key name + description: >- + Instead of Google managing the key encryption keys that protect your data, you control and manage key encryption keys in Cloud KMS. + If this parameter is set, the extension will specify the KMS key name when creating the BQ table. + type: string + required: false + events: - type: firebase.extensions.big-query-export.v1.sync.start description: Occurs on a firestore document write event. diff --git a/firestore-bigquery-export/firestore-bigquery-change-tracker/package-lock.json b/firestore-bigquery-export/firestore-bigquery-change-tracker/package-lock.json index d925bc222..0a74b746f 100644 --- a/firestore-bigquery-export/firestore-bigquery-change-tracker/package-lock.json +++ b/firestore-bigquery-export/firestore-bigquery-change-tracker/package-lock.json @@ -1,12 +1,12 @@ { "name": "@firebaseextensions/firestore-bigquery-change-tracker", - "version": "1.1.26", + "version": "1.1.27", "lockfileVersion": 2, "requires": true, "packages": { "": { "name": "@firebaseextensions/firestore-bigquery-change-tracker", - "version": "1.1.26", + "version": "1.1.27", "license": "Apache-2.0", "dependencies": { "@google-cloud/bigquery": "^4.7.0", diff --git a/firestore-bigquery-export/firestore-bigquery-change-tracker/package.json b/firestore-bigquery-export/firestore-bigquery-change-tracker/package.json index 9eda9e327..b42b027ea 100644 --- a/firestore-bigquery-export/firestore-bigquery-change-tracker/package.json +++ b/firestore-bigquery-export/firestore-bigquery-change-tracker/package.json @@ -5,7 +5,7 @@ "url": "github.com/firebase/extensions.git", "directory": "firestore-bigquery-export/firestore-bigquery-change-tracker" }, - "version": "1.1.26", + "version": "1.1.27", "description": "Core change-tracker library for Cloud Firestore Collection BigQuery Exports", "main": "./lib/index.js", "scripts": { diff --git a/firestore-bigquery-export/firestore-bigquery-change-tracker/src/bigquery/index.ts b/firestore-bigquery-export/firestore-bigquery-change-tracker/src/bigquery/index.ts index 8ac26f2f9..b71ed08da 100644 --- a/firestore-bigquery-export/firestore-bigquery-change-tracker/src/bigquery/index.ts +++ b/firestore-bigquery-export/firestore-bigquery-change-tracker/src/bigquery/index.ts @@ -60,6 +60,7 @@ export interface FirestoreBigQueryEventHistoryTrackerConfig { backupTableId?: string | undefined; useNewSnapshotQuerySyntax?: boolean; skipInit?: boolean; + kmsKeyName?: string | undefined; } /** @@ -389,6 +390,12 @@ export class FirestoreBigQueryEventHistoryTracker } const options: TableMetadata = { friendlyName: changelogName, schema }; + if (this.config.kmsKeyName) { + options["encryptionConfiguration"] = { + kmsKeyName: this.config.kmsKeyName, + }; + } + //Add partitioning await partitioning.addPartitioningToSchema(schema.fields); diff --git a/firestore-bigquery-export/functions/__tests__/__snapshots__/config.test.ts.snap b/firestore-bigquery-export/functions/__tests__/__snapshots__/config.test.ts.snap index 85af9a448..1845171b0 100644 --- a/firestore-bigquery-export/functions/__tests__/__snapshots__/config.test.ts.snap +++ b/firestore-bigquery-export/functions/__tests__/__snapshots__/config.test.ts.snap @@ -13,6 +13,7 @@ Object { "datasetLocation": undefined, "initialized": false, "instanceId": undefined, + "kmsKeyName": "test", "location": "us-central1", "maxDispatchesPerSecond": 10, "tableId": "my_table", diff --git a/firestore-bigquery-export/functions/__tests__/config.test.ts b/firestore-bigquery-export/functions/__tests__/config.test.ts index aa1634a37..46f8bbffa 100644 --- a/firestore-bigquery-export/functions/__tests__/config.test.ts +++ b/firestore-bigquery-export/functions/__tests__/config.test.ts @@ -17,8 +17,10 @@ const environment = { TABLE_ID: "my_table", TRANSFORM_FUNCTION: "", CLUSTERING: "data,timestamp", + KMS_KEY_NAME: "test", }; +//@ts-ignore const { config } = global; describe("extension config", () => { @@ -46,8 +48,8 @@ describe("extension config", () => { datasetId: environment.DATASET_ID, tableId: environment.TABLE_ID, clustering: clustering(environment.CLUSTERING), + kmsKeyName: environment.KMS_KEY_NAME, }; - expect(config()).toMatchSnapshot(env); }); diff --git a/firestore-bigquery-export/functions/package-lock.json b/firestore-bigquery-export/functions/package-lock.json index 5e5fcdfaf..be13d5e77 100644 --- a/firestore-bigquery-export/functions/package-lock.json +++ b/firestore-bigquery-export/functions/package-lock.json @@ -7,7 +7,7 @@ "name": "firestore-bigquery-export", "license": "Apache-2.0", "dependencies": { - "@firebaseextensions/firestore-bigquery-change-tracker": "^1.1.26", + "@firebaseextensions/firestore-bigquery-change-tracker": "^1.1.27", "@google-cloud/bigquery": "^4.7.0", "@types/chai": "^4.1.6", "@types/express-serve-static-core": "4.17.30", @@ -486,9 +486,9 @@ } }, "node_modules/@firebaseextensions/firestore-bigquery-change-tracker": { - "version": "1.1.26", - "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.26.tgz", - "integrity": "sha512-ZfEyFAuM7PFjfImF9IuWyQCITgUugTypQxNTO1E1X8jYJ9Ker4fn2isIHhnAF2Rf40vYJjp4iB6XVRJCkHI3qw==", + "version": "1.1.27", + "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.27.tgz", + "integrity": "sha512-yyVo6qRvTZ/8pd/+czJgaIXv6bi6tRPscXiZYn2AoIgwwuAvtS6gr9oQ+tJOqhd65idbeBYPvS8kI3CYE22IIQ==", "dependencies": { "@google-cloud/bigquery": "^4.7.0", "@google-cloud/resource-manager": "^3.0.0", @@ -11046,9 +11046,9 @@ } }, "@firebaseextensions/firestore-bigquery-change-tracker": { - "version": "1.1.26", - "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.26.tgz", - "integrity": "sha512-ZfEyFAuM7PFjfImF9IuWyQCITgUugTypQxNTO1E1X8jYJ9Ker4fn2isIHhnAF2Rf40vYJjp4iB6XVRJCkHI3qw==", + "version": "1.1.27", + "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.27.tgz", + "integrity": "sha512-yyVo6qRvTZ/8pd/+czJgaIXv6bi6tRPscXiZYn2AoIgwwuAvtS6gr9oQ+tJOqhd65idbeBYPvS8kI3CYE22IIQ==", "requires": { "@google-cloud/bigquery": "^4.7.0", "@google-cloud/resource-manager": "^3.0.0", diff --git a/firestore-bigquery-export/functions/package.json b/firestore-bigquery-export/functions/package.json index fe4bca774..1f473daf9 100644 --- a/firestore-bigquery-export/functions/package.json +++ b/firestore-bigquery-export/functions/package.json @@ -13,7 +13,7 @@ "author": "Jan Wyszynski ", "license": "Apache-2.0", "dependencies": { - "@firebaseextensions/firestore-bigquery-change-tracker": "^1.1.26", + "@firebaseextensions/firestore-bigquery-change-tracker": "^1.1.27", "@google-cloud/bigquery": "^4.7.0", "@types/chai": "^4.1.6", "@types/express-serve-static-core": "4.17.30", diff --git a/firestore-bigquery-export/functions/src/config.ts b/firestore-bigquery-export/functions/src/config.ts index ff51ad4e9..49154c033 100644 --- a/firestore-bigquery-export/functions/src/config.ts +++ b/firestore-bigquery-export/functions/src/config.ts @@ -56,4 +56,5 @@ export default { maxDispatchesPerSecond: parseInt( process.env.MAX_DISPATCHES_PER_SECOND || "10" ), + kmsKeyName: process.env.KMS_KEY_NAME, }; diff --git a/firestore-bigquery-export/functions/src/index.ts b/firestore-bigquery-export/functions/src/index.ts index 1d9874a41..a6f7e340e 100644 --- a/firestore-bigquery-export/functions/src/index.ts +++ b/firestore-bigquery-export/functions/src/index.ts @@ -45,6 +45,7 @@ const eventTracker: FirestoreEventHistoryTracker = bqProjectId: config.bqProjectId, useNewSnapshotQuerySyntax: config.useNewSnapshotQuerySyntax, skipInit: true, + kmsKeyName: config.kmsKeyName, }); logs.init(); diff --git a/firestore-bigquery-export/scripts/gen-schema-view/package-lock.json b/firestore-bigquery-export/scripts/gen-schema-view/package-lock.json index 1f3a6ca65..060640627 100644 --- a/firestore-bigquery-export/scripts/gen-schema-view/package-lock.json +++ b/firestore-bigquery-export/scripts/gen-schema-view/package-lock.json @@ -546,9 +546,9 @@ "integrity": "sha512-tGyy4dAjRIEwI7BzsB0lynWgOpfqjUdq91XXAlIWD2OwKBH7oCl/GZG/HT4BOHrTlPMOASlMQ7veyTqpmRcrNA==" }, "node_modules/@firebaseextensions/firestore-bigquery-change-tracker": { - "version": "1.1.25", - "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.25.tgz", - "integrity": "sha512-8pY8X0i0sknjIszik3Zh2WJF/3tfGAhgusnxUFaQKsyVaMlfir090oOZswewRDGWCGjBR2rNw79l4xuccaQ9Rw==", + "version": "1.1.27", + "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.27.tgz", + "integrity": "sha512-yyVo6qRvTZ/8pd/+czJgaIXv6bi6tRPscXiZYn2AoIgwwuAvtS6gr9oQ+tJOqhd65idbeBYPvS8kI3CYE22IIQ==", "dependencies": { "@google-cloud/bigquery": "^4.7.0", "@google-cloud/resource-manager": "^3.0.0", @@ -1297,9 +1297,9 @@ } }, "node_modules/@grpc/proto-loader/node_modules/protobufjs": { - "version": "6.11.3", - "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.3.tgz", - "integrity": "sha512-xL96WDdCZYdU7Slin569tFX712BxsxslWwAfAhCYjQKGTq7dAU91Lomy6nLLhh/dyGhk/YH4TwTSRxTzhuHyZg==", + "version": "6.11.4", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.4.tgz", + "integrity": "sha512-5kQWPaJHi1WoCpjTGszzQ32PG2F4+wRY6BmAT4Vfw56Q2FZ4YZzK20xUYQH4YkfehY1e6QSICrJquM6xXZNcrw==", "hasInstallScript": true, "dependencies": { "@protobufjs/aspromise": "^1.1.2", @@ -8173,9 +8173,9 @@ } }, "node_modules/proto3-json-serializer/node_modules/protobufjs": { - "version": "6.11.3", - "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.3.tgz", - "integrity": "sha512-xL96WDdCZYdU7Slin569tFX712BxsxslWwAfAhCYjQKGTq7dAU91Lomy6nLLhh/dyGhk/YH4TwTSRxTzhuHyZg==", + "version": "6.11.4", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.4.tgz", + "integrity": "sha512-5kQWPaJHi1WoCpjTGszzQ32PG2F4+wRY6BmAT4Vfw56Q2FZ4YZzK20xUYQH4YkfehY1e6QSICrJquM6xXZNcrw==", "hasInstallScript": true, "dependencies": { "@protobufjs/aspromise": "^1.1.2", @@ -11218,9 +11218,9 @@ } }, "@firebaseextensions/firestore-bigquery-change-tracker": { - "version": "1.1.25", - "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.25.tgz", - "integrity": "sha512-8pY8X0i0sknjIszik3Zh2WJF/3tfGAhgusnxUFaQKsyVaMlfir090oOZswewRDGWCGjBR2rNw79l4xuccaQ9Rw==", + "version": "1.1.27", + "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.27.tgz", + "integrity": "sha512-yyVo6qRvTZ/8pd/+czJgaIXv6bi6tRPscXiZYn2AoIgwwuAvtS6gr9oQ+tJOqhd65idbeBYPvS8kI3CYE22IIQ==", "requires": { "@google-cloud/bigquery": "^4.7.0", "@google-cloud/resource-manager": "^3.0.0", @@ -11807,9 +11807,9 @@ }, "dependencies": { "protobufjs": { - "version": "6.11.3", - "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.3.tgz", - "integrity": "sha512-xL96WDdCZYdU7Slin569tFX712BxsxslWwAfAhCYjQKGTq7dAU91Lomy6nLLhh/dyGhk/YH4TwTSRxTzhuHyZg==", + "version": "6.11.4", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.4.tgz", + "integrity": "sha512-5kQWPaJHi1WoCpjTGszzQ32PG2F4+wRY6BmAT4Vfw56Q2FZ4YZzK20xUYQH4YkfehY1e6QSICrJquM6xXZNcrw==", "requires": { "@protobufjs/aspromise": "^1.1.2", "@protobufjs/base64": "^1.1.2", @@ -17253,9 +17253,9 @@ }, "dependencies": { "protobufjs": { - "version": "6.11.3", - "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.3.tgz", - "integrity": "sha512-xL96WDdCZYdU7Slin569tFX712BxsxslWwAfAhCYjQKGTq7dAU91Lomy6nLLhh/dyGhk/YH4TwTSRxTzhuHyZg==", + "version": "6.11.4", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.4.tgz", + "integrity": "sha512-5kQWPaJHi1WoCpjTGszzQ32PG2F4+wRY6BmAT4Vfw56Q2FZ4YZzK20xUYQH4YkfehY1e6QSICrJquM6xXZNcrw==", "requires": { "@protobufjs/aspromise": "^1.1.2", "@protobufjs/base64": "^1.1.2", diff --git a/firestore-bigquery-export/scripts/import/package-lock.json b/firestore-bigquery-export/scripts/import/package-lock.json index 1de2b563a..e5bfe97d8 100644 --- a/firestore-bigquery-export/scripts/import/package-lock.json +++ b/firestore-bigquery-export/scripts/import/package-lock.json @@ -668,9 +668,9 @@ } }, "node_modules/@firebaseextensions/firestore-bigquery-change-tracker": { - "version": "1.1.25", - "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.25.tgz", - "integrity": "sha512-8pY8X0i0sknjIszik3Zh2WJF/3tfGAhgusnxUFaQKsyVaMlfir090oOZswewRDGWCGjBR2rNw79l4xuccaQ9Rw==", + "version": "1.1.27", + "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.27.tgz", + "integrity": "sha512-yyVo6qRvTZ/8pd/+czJgaIXv6bi6tRPscXiZYn2AoIgwwuAvtS6gr9oQ+tJOqhd65idbeBYPvS8kI3CYE22IIQ==", "dependencies": { "@google-cloud/bigquery": "^4.7.0", "@google-cloud/resource-manager": "^3.0.0", @@ -8400,9 +8400,9 @@ } }, "@firebaseextensions/firestore-bigquery-change-tracker": { - "version": "1.1.25", - "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.25.tgz", - "integrity": "sha512-8pY8X0i0sknjIszik3Zh2WJF/3tfGAhgusnxUFaQKsyVaMlfir090oOZswewRDGWCGjBR2rNw79l4xuccaQ9Rw==", + "version": "1.1.27", + "resolved": "https://registry.npmjs.org/@firebaseextensions/firestore-bigquery-change-tracker/-/firestore-bigquery-change-tracker-1.1.27.tgz", + "integrity": "sha512-yyVo6qRvTZ/8pd/+czJgaIXv6bi6tRPscXiZYn2AoIgwwuAvtS6gr9oQ+tJOqhd65idbeBYPvS8kI3CYE22IIQ==", "requires": { "@google-cloud/bigquery": "^4.7.0", "@google-cloud/resource-manager": "^3.0.0", From 10685b69f25e02824fca7baab77454e5a98b9be2 Mon Sep 17 00:00:00 2001 From: Jacob Cable Date: Fri, 18 Aug 2023 09:50:53 +0100 Subject: [PATCH 2/5] fix(firestore-bigquery-export): add missing locations back in --- firestore-bigquery-export/scripts/import/src/config.ts | 2 ++ 1 file changed, 2 insertions(+) diff --git a/firestore-bigquery-export/scripts/import/src/config.ts b/firestore-bigquery-export/scripts/import/src/config.ts index d2f945bbe..e437a811b 100644 --- a/firestore-bigquery-export/scripts/import/src/config.ts +++ b/firestore-bigquery-export/scripts/import/src/config.ts @@ -24,6 +24,8 @@ const validateLocation = (value: string) => { "us-west3", "southamerica-east1", "us-east1", + "europe-central2", + "europe-north1", "europe-west1", "europe-north1", "europe-west3", From f038d0597b10fa272363147bec55e97197ca4810 Mon Sep 17 00:00:00 2001 From: Darren Ackers Date: Wed, 23 Aug 2023 15:50:49 +0100 Subject: [PATCH 3/5] chore(*): updated version, changelogs and readmes --- firestore-bigquery-export/CHANGELOG.md | 8 ++++++++ firestore-bigquery-export/README.md | 18 ++++++++++++++++++ firestore-bigquery-export/extension.yaml | 2 +- 3 files changed, 27 insertions(+), 1 deletion(-) diff --git a/firestore-bigquery-export/CHANGELOG.md b/firestore-bigquery-export/CHANGELOG.md index d926cd4c1..832ab6a95 100644 --- a/firestore-bigquery-export/CHANGELOG.md +++ b/firestore-bigquery-export/CHANGELOG.md @@ -1,3 +1,11 @@ +## Version 0.1.35 + +fixed - add missing locations back in + +fixed - use module instead namespace + +fixed - added e2e testing, upgraded dependencies + ## Version 0.1.34 feat - added failure policy diff --git a/firestore-bigquery-export/README.md b/firestore-bigquery-export/README.md index dbfb65b55..0f8759316 100644 --- a/firestore-bigquery-export/README.md +++ b/firestore-bigquery-export/README.md @@ -55,6 +55,24 @@ Prior to sending the document change to BigQuery, you have an opportunity to tra The response should be indentical in structure. +#### Using Customer Managed Encryption Keys + +By default, BigQuery encrypts your content stored at rest. BigQuery handles and manages this default encryption for you without any additional actions on your part. + +If you want to control encryption yourself, you can use customer-managed encryption keys (CMEK) for BigQuery. Instead of Google managing the key encryption keys that protect your data, you control and manage key encryption keys in Cloud KMS. + +For more general information on this, see [the docs](https://cloud.google.com/bigquery/docs/customer-managed-encryption). + +To use CMEK and the Key Management Service (KMS) with this extension +1. [Enable the KMS API in your Google Cloud Project](https://console.cloud.google.com/apis/enableflow?apiid=cloudkms.googleapis.com). +2. Create a keyring and keychain in the KMS. Note that the region of the keyring and key *must* match the region of your bigquery dataset +3. Grant the BigQuery service account permission to encrypt and decrypt using that key. The Cloud KMS CryptoKey Encrypter/Decrypter role grants this permission. +4. When installing this extension, enter the resource name of your key. It will look something like the following: +``` +projects//locations//keyRings//cryptoKeys/ +``` +If you follow these steps, your changelog table should be created using your customer-managed encryption. + #### Backfill your BigQuery dataset This extension only sends the content of documents that have been changed -- it does not export your full dataset of existing documents into BigQuery. So, to backfill your BigQuery dataset with all the documents in your collection, you can run the [import script](https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/IMPORT_EXISTING_DOCUMENTS.md) provided by this extension. diff --git a/firestore-bigquery-export/extension.yaml b/firestore-bigquery-export/extension.yaml index 64ad39b09..a7f18b43f 100644 --- a/firestore-bigquery-export/extension.yaml +++ b/firestore-bigquery-export/extension.yaml @@ -13,7 +13,7 @@ # limitations under the License. name: firestore-bigquery-export -version: 0.1.34 +version: 0.1.35 specVersion: v1beta displayName: Stream Firestore to BigQuery From 4be5c4f914ce91f3790abc305b2d03d80ac391fc Mon Sep 17 00:00:00 2001 From: Jacob Cable Date: Thu, 24 Aug 2023 10:10:01 +0100 Subject: [PATCH 4/5] docs(firestore-bigquery-export): add instructions for adding KMS role --- firestore-bigquery-export/PREINSTALL.md | 11 ++++++++++- firestore-bigquery-export/README.md | 11 ++++++++++- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/firestore-bigquery-export/PREINSTALL.md b/firestore-bigquery-export/PREINSTALL.md index b1fff3f5e..a0113b639 100644 --- a/firestore-bigquery-export/PREINSTALL.md +++ b/firestore-bigquery-export/PREINSTALL.md @@ -58,7 +58,16 @@ For more general information on this, see [the docs](https://cloud.google.com/bi To use CMEK and the Key Management Service (KMS) with this extension 1. [Enable the KMS API in your Google Cloud Project](https://console.cloud.google.com/apis/enableflow?apiid=cloudkms.googleapis.com). 2. Create a keyring and keychain in the KMS. Note that the region of the keyring and key *must* match the region of your bigquery dataset -3. Grant the BigQuery service account permission to encrypt and decrypt using that key. The Cloud KMS CryptoKey Encrypter/Decrypter role grants this permission. +3. Grant the BigQuery service account permission to encrypt and decrypt using that key. The Cloud KMS CryptoKey Encrypter/Decrypter role grants this permission. First find your project number. You can find this for example on the cloud console dashboard `https://console.cloud.google.com/home/dashboard?project={PROJECT_ID}`. The service account which needs the Encrypter/Decrypter role is then `bq-PROJECT_NUMBER@bigquery-encryption.iam.gserviceaccount.com`. You can grant this role through the credentials service in the console, or through the CLI: +``` +gcloud kms keys add-iam-policy-binding \ +--project=KMS_PROJECT_ID \ +--member serviceAccount:bq-PROJECT_NUMBER@bigquery-encryption.iam.gserviceaccount.com \ +--role roles/cloudkms.cryptoKeyEncrypterDecrypter \ +--location=KMS_KEY_LOCATION \ +--keyring=KMS_KEY_RING \ +KMS_KEY +``` 4. When installing this extension, enter the resource name of your key. It will look something like the following: ``` projects//locations//keyRings//cryptoKeys/ diff --git a/firestore-bigquery-export/README.md b/firestore-bigquery-export/README.md index 0f8759316..dabd321b9 100644 --- a/firestore-bigquery-export/README.md +++ b/firestore-bigquery-export/README.md @@ -66,7 +66,16 @@ For more general information on this, see [the docs](https://cloud.google.com/bi To use CMEK and the Key Management Service (KMS) with this extension 1. [Enable the KMS API in your Google Cloud Project](https://console.cloud.google.com/apis/enableflow?apiid=cloudkms.googleapis.com). 2. Create a keyring and keychain in the KMS. Note that the region of the keyring and key *must* match the region of your bigquery dataset -3. Grant the BigQuery service account permission to encrypt and decrypt using that key. The Cloud KMS CryptoKey Encrypter/Decrypter role grants this permission. +3. Grant the BigQuery service account permission to encrypt and decrypt using that key. The Cloud KMS CryptoKey Encrypter/Decrypter role grants this permission. First find your project number. You can find this for example on the cloud console dashboard `https://console.cloud.google.com/home/dashboard?project={PROJECT_ID}`. The service account which needs the Encrypter/Decrypter role is then `bq-PROJECT_NUMBER@bigquery-encryption.iam.gserviceaccount.com`. You can grant this role through the credentials service in the console, or through the CLI: +``` +gcloud kms keys add-iam-policy-binding \ +--project=KMS_PROJECT_ID \ +--member serviceAccount:bq-PROJECT_NUMBER@bigquery-encryption.iam.gserviceaccount.com \ +--role roles/cloudkms.cryptoKeyEncrypterDecrypter \ +--location=KMS_KEY_LOCATION \ +--keyring=KMS_KEY_RING \ +KMS_KEY +``` 4. When installing this extension, enter the resource name of your key. It will look something like the following: ``` projects//locations//keyRings//cryptoKeys/ From 7f97992afb3e79bf0e5a9b7c27874fe4d5ed014f Mon Sep 17 00:00:00 2001 From: Jacob Cable Date: Thu, 24 Aug 2023 10:21:41 +0100 Subject: [PATCH 5/5] feat(firestore-bigquery-export): add regex validation for KMS key --- firestore-bigquery-export/README.md | 2 +- firestore-bigquery-export/extension.yaml | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/firestore-bigquery-export/README.md b/firestore-bigquery-export/README.md index dabd321b9..02ce77c3f 100644 --- a/firestore-bigquery-export/README.md +++ b/firestore-bigquery-export/README.md @@ -135,7 +135,7 @@ To install an extension, your project must be on the [Blaze (pay as you go) plan * Use new query syntax for snapshots: If enabled, snapshots will be generated with the new query syntax, which should be more performant, and avoid potential resource limitations. -* Cloud KMS key name: Instead of Google managing the key encryption keys that protect your data, you control and manage key encryption keys in Cloud KMS. If this parameter is set, the extension will specify the KMS key name when creating the BQ table. +* Cloud KMS key name: Instead of Google managing the key encryption keys that protect your data, you control and manage key encryption keys in Cloud KMS. If this parameter is set, the extension will specify the KMS key name when creating the BQ table. See the PREINSTALL.md for more details. diff --git a/firestore-bigquery-export/extension.yaml b/firestore-bigquery-export/extension.yaml index a7f18b43f..181fee645 100644 --- a/firestore-bigquery-export/extension.yaml +++ b/firestore-bigquery-export/extension.yaml @@ -326,8 +326,10 @@ params: label: Cloud KMS key name description: >- Instead of Google managing the key encryption keys that protect your data, you control and manage key encryption keys in Cloud KMS. - If this parameter is set, the extension will specify the KMS key name when creating the BQ table. + If this parameter is set, the extension will specify the KMS key name when creating the BQ table. See the PREINSTALL.md for more details. type: string + validationRegex: 'projects/([^/]+)/locations/([^/]+)/keyRings/([^/]+)/cryptoKeys/([^/]+)' + validationErrorMessage: The key name must be of the format 'projects/PROJECT_NAME/locations/KEY_RING_LOCATION/keyRings/KEY_RING_ID/cryptoKeys/KEY_ID'. required: false events: