feat: Token encryption #2157

jachym-tousek-keboola · 2024-11-26T10:14:58Z

Jira: https://keboola.atlassian.net/browse/PSGO-911

Changes:

Encrypt keboola tokens in Stream API

provisioning/stream/local.sh

provisioning/stream/kubernetes/templates/api/deployment.yaml

internal/pkg/service/common/dependencies/encryption.go

internal/pkg/service/stream/components.go

internal/pkg/service/stream/encryption/config.go

internal/pkg/service/stream/encryption/encryption.go

internal/pkg/service/stream/sink/type/tablesink/keboola/bridge/slice.go

internal/pkg/service/stream/sink/type/tablesink/keboola/bridge/token.go

internal/pkg/service/stream/sink/type/tablesink/keboola/bridge/token_test.go

internal/pkg/service/stream/encryption/config.go

internal/pkg/service/stream/sink/type/tablesink/keboola/bridge/bridge.go

…911-stream-token-encryption

github-actions · 2025-01-02T09:49:20Z

Stream Kubernetes Diff [CI]

Between base b8421f8 ⬅️ head c80feac.

Expand

--- /tmp/artifacts/test-k8s-state.old.json.processed.kv	2025-01-02 09:49:10.729363784 +0000
+++ /tmp/artifacts/test-k8s-state.new.json.processed.kv	2025-01-02 09:49:11.245357913 +0000
@@ -13 +13 @@
-<ConfigMap/stream-config>.data["config.yaml"] = "###############################################################################################...
+<ConfigMap/stream-config>.data["config.yaml"] = "###############################################################################################...
@@ -200 +200 @@
-<Deployment/stream-api>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<Deployment/stream-api>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -364 +364 @@
-<Deployment/stream-http-source>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<Deployment/stream-http-source>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -525 +525 @@
-<Deployment/stream-storage-coordinator>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<Deployment/stream-storage-coordinator>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -609 +609 @@
-<Endpoints/stream-etcd-headless>.subsets[0].addresses[1].hostname = "stream-etcd-0";
+<Endpoints/stream-etcd-headless>.subsets[0].addresses[1].hostname = "stream-etcd-1";
@@ -613 +613 @@
-<Endpoints/stream-etcd-headless>.subsets[0].addresses[1].targetRef.name = "stream-etcd-0";
+<Endpoints/stream-etcd-headless>.subsets[0].addresses[1].targetRef.name = "stream-etcd-1";
@@ -616 +616 @@
-<Endpoints/stream-etcd-headless>.subsets[0].addresses[2].hostname = "stream-etcd-1";
+<Endpoints/stream-etcd-headless>.subsets[0].addresses[2].hostname = "stream-etcd-0";
@@ -620 +620 @@
-<Endpoints/stream-etcd-headless>.subsets[0].addresses[2].targetRef.name = "stream-etcd-1";
+<Endpoints/stream-etcd-headless>.subsets[0].addresses[2].targetRef.name = "stream-etcd-0";
@@ -659 +659 @@
-<Endpoints/stream-etcd>.subsets[0].addresses[1].targetRef.name = "stream-etcd-0";
+<Endpoints/stream-etcd>.subsets[0].addresses[1].targetRef.name = "stream-etcd-1";
@@ -665 +665 @@
-<Endpoints/stream-etcd>.subsets[0].addresses[2].targetRef.name = "stream-etcd-1";
+<Endpoints/stream-etcd>.subsets[0].addresses[2].targetRef.name = "stream-etcd-0";
@@ -717 +717 @@
-<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[0].hostname = "stream-storage-writer-reader-0";
+<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[0].hostname = "stream-storage-writer-reader-1";
@@ -721 +721 @@
-<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[0].targetRef.name = "stream-storage-writer-reader-0";
+<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[0].targetRef.name = "stream-storage-writer-reader-1";
@@ -724 +724 @@
-<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[1].hostname = "stream-storage-writer-reader-1";
+<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[1].hostname = "stream-storage-writer-reader-0";
@@ -728 +728 @@
-<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[1].targetRef.name = "stream-storage-writer-reader-1";
+<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[1].targetRef.name = "stream-storage-writer-reader-0";
@@ -1214,2 +1214,2 @@
-<Pod/stream-api-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
-<Pod/stream-api-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<Pod/stream-api-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
+<Pod/stream-api-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -1534 +1534 @@
-<Pod/stream-etcd-0>.spec.containers[0].env[21].value = "new";
+<Pod/stream-etcd-0>.spec.containers[0].env[21].value = "existing";
@@ -1780 +1780 @@
-<Pod/stream-etcd-1>.spec.containers[0].env[21].value = "new";
+<Pod/stream-etcd-1>.spec.containers[0].env[21].value = "existing";
@@ -2026 +2026 @@
-<Pod/stream-etcd-2>.spec.containers[0].env[21].value = "new";
+<Pod/stream-etcd-2>.spec.containers[0].env[21].value = "existing";
@@ -2350,2 +2350,2 @@
-<Pod/stream-http-source-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
-<Pod/stream-http-source-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<Pod/stream-http-source-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
+<Pod/stream-http-source-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -2742,2 +2742,2 @@
-<Pod/stream-storage-coordinator-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
-<Pod/stream-storage-coordinator-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<Pod/stream-storage-coordinator-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
+<Pod/stream-storage-coordinator-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -2978 +2978 @@
-<Pod/stream-storage-writer-reader-0>.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<Pod/stream-storage-writer-reader-0>.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -3061 +3061 @@
-<Pod/stream-storage-writer-reader-0>.spec.containers[1].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<Pod/stream-storage-writer-reader-0>.spec.containers[1].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -3231 +3231 @@
-<Pod/stream-storage-writer-reader-1>.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<Pod/stream-storage-writer-reader-1>.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -3314 +3314 @@
-<Pod/stream-storage-writer-reader-1>.spec.containers[1].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<Pod/stream-storage-writer-reader-1>.spec.containers[1].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -3552 +3552 @@
-<ReplicaSet/stream-api-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<ReplicaSet/stream-api-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -3723 +3723 @@
-<ReplicaSet/stream-http-source-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<ReplicaSet/stream-http-source-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -3891 +3891 @@
-<ReplicaSet/stream-storage-coordinator-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<ReplicaSet/stream-storage-coordinator-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -3932,0 +3933,12 @@
+<Secret/sh.helm.release.v1.stream-etcd.v2> = {};
+<Secret/sh.helm.release.v1.stream-etcd.v2>.apiVersion = "v1";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.data = {};
+<Secret/sh.helm.release.v1.stream-etcd.v2>.kind = "Secret";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata = {};
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.labels = {};
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.labels.name = "stream-etcd";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.labels.owner = "helm";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.labels.version = "2";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.name = "sh.helm.release.v1.stream-etcd.v2";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.namespace = "stream";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.type = "helm.sh/release.v1";
@@ -4228 +4240 @@
-<StatefulSet/stream-etcd>.spec.template.spec.containers[0].env[21].value = "new";
+<StatefulSet/stream-etcd>.spec.template.spec.containers[0].env[21].value = "existing";
@@ -4478 +4490 @@
-<StatefulSet/stream-storage-writer-reader>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<StatefulSet/stream-storage-writer-reader>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:c80feac-1735810915";
@@ -4558 +4570 @@
-<StatefulSet/stream-storage-writer-reader>.spec.template.spec.containers[1].image = "docker.io/keboola/stream-api:b8421f8-1735810311";
+<StatefulSet/stream-storage-writer-reader>.spec.template.spec.containers[1].image = "docker.io/keboola/stream-api:c80feac-1735810915";


(see artifacts in the Github Action for more information)

jachym-tousek-keboola changed the title ~~feat: Encryption configuration~~ feat: Token encryption Nov 26, 2024

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch 10 times, most recently from 81d70c2 to eaac77c Compare November 27, 2024 11:22

jachym-tousek-keboola commented Nov 27, 2024

View reviewed changes

provisioning/stream/local.sh Outdated Show resolved Hide resolved

jachym-tousek-keboola commented Nov 27, 2024

View reviewed changes

provisioning/stream/kubernetes/templates/api/deployment.yaml Outdated Show resolved Hide resolved

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch from d467867 to 2ecf2aa Compare November 27, 2024 14:53

jachym-tousek-keboola requested a review from Matovidlo November 29, 2024 12:19

Matovidlo requested changes Nov 29, 2024

View reviewed changes

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch 2 times, most recently from 745c984 to 1f01ea7 Compare December 2, 2024 08:25

jachym-tousek-keboola requested a review from Matovidlo December 2, 2024 11:51

Matovidlo reviewed Dec 4, 2024

View reviewed changes

internal/pkg/service/stream/encryption/config.go Show resolved Hide resolved

internal/pkg/service/stream/sink/type/tablesink/keboola/bridge/bridge.go Outdated Show resolved Hide resolved

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch from e90d3cb to 9f9056e Compare December 4, 2024 12:47

jachym-tousek-keboola requested a review from Matovidlo December 4, 2024 12:47

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch 6 times, most recently from 95b016b to 64bc260 Compare December 6, 2024 11:08

jachym-tousek-keboola marked this pull request as ready for review December 6, 2024 11:48

jachym-tousek-keboola requested a review from hosekpeter as a code owner December 6, 2024 11:48

jachym-tousek-keboola added 5 commits December 17, 2024 12:21

feat: Upload credentials encryption

8ddc201

fix: Secret key length

96c02ff

feat: Add byte slice support to configmap

e2ba68c

test: Fix stream tests

fa355ca

refactor: Add DecryptToken method

e307174

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch 2 times, most recently from 990cd92 to 3676be8 Compare December 17, 2024 16:38

fix: Add custom error

37438f7

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch from 3676be8 to 37438f7 Compare December 17, 2024 16:41

jachym-tousek-keboola added 3 commits December 18, 2024 13:24

feat: Log cloud encryption attempts

eca7757

fix: Prevent empty byte slices

194e200

debug: Log metadata

fc25599

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch from adb788f to fc25599 Compare December 18, 2024 14:43

debug: Test decrypt immediately

5a42cae

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch 2 times, most recently from 7802c81 to 2d37a0b Compare December 19, 2024 10:54

debug: Disable cache

3510844

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch from 2d37a0b to 3510844 Compare December 19, 2024 11:02

jachym-tousek-keboola added 4 commits December 19, 2024 13:36

debug: Log encrypted token

d347dd6

debug: Log encryption inputs

64529cb

debug: crc32

ce36c39

fix: Update go-cloud-encrypt

b62ae72

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch from ce36c39 to b62ae72 Compare December 20, 2024 12:04

Merge commit 'ce36c39028b4c2ed8cf045e9ec1c690eb014a138' into jt-psgo-…

41bab0e

…911-stream-token-encryption

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch from f2b4cfd to 41bab0e Compare December 20, 2024 13:17

debug: New encoding method

05fdca9

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch from 43259e0 to 193d49b Compare January 2, 2025 09:19

debug: Use json

c80feac

jachym-tousek-keboola force-pushed the jt-psgo-911-stream-token-encryption branch from 193d49b to c80feac Compare January 2, 2025 09:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Token encryption #2157

feat: Token encryption #2157

jachym-tousek-keboola commented Nov 26, 2024

github-actions bot commented Jan 2, 2025

feat: Token encryption #2157

Are you sure you want to change the base?

feat: Token encryption #2157

Conversation

jachym-tousek-keboola commented Nov 26, 2024

github-actions bot commented Jan 2, 2025

Stream Kubernetes Diff [CI]