Skip to content
This repository has been archived by the owner on Jul 31, 2024. It is now read-only.

Problème de déploiement #62

Open
skante93 opened this issue Apr 17, 2024 · 0 comments
Open

Problème de déploiement #62

skante93 opened this issue Apr 17, 2024 · 0 comments

Comments

@skante93
Copy link

skante93 commented Apr 17, 2024

English version below

Procédure de déploiement:

OS: Ubuntu 22.04
Kubernetes: v1.29.2
Helm : v3.14.2

  • Pour éviter les problèmes liées au load balancing, ingress et aux PVCs, j'ai choisi de tenter le déploiement des charts helm dans un envrionnement cloud: scaleway.
  • Après création du cluster kubernetes, et d'une VM de travail, je me logué sur le dernier puis installé un certain nombre d'outils:
    • kubectl pour interagir avec le cluster k8s
    • helm pour installer les charts afin d'installer les différents composantes d'un connecteur fiware
    • l'utilitaire "yq", utilisé par le script generate.sh (mentionné plus tard)
  • Ensuite j'ai cloné le "dépôt git du connecteur fiware":https://github.com/FIWARE-Ops/data-space-connector?tab=readme-ov-file

Essai 1

Initialement, j'ai suivi les indications du README, à savoir:

$ helm repo add dsc https://fiware-ops.github.io/data-space-connector/
$ helm repo update
$ helm install -n fc1 fc1 dsc/data-space-connector --create-namespace -f data-space-connector/values.yaml --set argoApplications=false

Note: j'appelle la release fc1 ainsi que le namespace dans lequel créer les composantes, puis avec --set, je specifie le déploiement se fait avec helm (pas avec argo).

J'ai l'erreur suivante:

Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * job fc1-keycloak-keycloak-config-cli failed: BackoffLimitExceeded

J'ai aussi changé les valeurs, me référant à l'exemple service-provider-ips, voici le résultat:

Code
$ helm install -n fc1 fc1 dsc/data-space-connector --create-namespace -f examples/service-provider-ips/values-dsc.yaml --set argoApplications=false

W0417 13:33:43.276530    8200 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
W0417 13:33:43.283632    8200 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
W0417 13:33:43.283651    8200 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
W0417 13:33:43.312711    8200 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
W0417 13:33:43.312751    8200 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * timed out waiting for the condition

J'ai attendu pendant un dizaine de minutes pensant que ce problème allait se résoudre tout seul (je me suis dit que peut être que les probes étaient trop optimistes), cependant rien n'a changé entre temps.

Essai 2

Quand cet exemple a echoué, j'ai exécuté le script generate.sh pour constituer les charts en local, mais pour ça j'ai du ajouter un certain nombre de dépôts helm:

Code
declare -A H_REPOS=(
[i4trust]=https://i4trust.github.io/helm-charts
[fiware]=https://fiware.github.io/helm-charts
[bitnami]=https://charts.bitnami.com/bitnami
[konghq]=https://charts.konghq.com
[bitname-full-index]=https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
)

for repo in ${!H_REPOS[@]}; do
        helm repo add $repo ${H_REPOS[$repo]}
done

helm repo update

sed 's/^version=.*/version="${1:-0.0.0}"/' -i generate.sh

./generate.sh

Ce script m'a généré une application helm dans charts/data-space-connector que j'ai tenté de déployer, voici le résultat:

Code
$ helm install -n fc1 fc1 charts/data-space-connector/ --set argoApplications=false --create-namespace

Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * job fc1-keycloak-keycloak-config-cli failed: BackoffLimitExceeded

Débogage

J'ai cru en premier que le problème était dû au fait que je n'ai pas changé les paramètres par default des charts, ce qui a provoqué le disfonctionnement observé. En d'autres termes, j'ai supposé que les manifestes kubernetes étaient correctes mais les containers plantaient après le démarrage car les valeurs par défaut ne correspondaient pas à mon environnement de travail (tel que l'adresse du did server dans la composante keycloak).

Mais après avoir investigué avec kubectl (describe, logs, get -o yaml), je me suis rendu compte que c'était dû à des mauvaises configurations kubernetes, en voici la liste :

Objet Raison
deploy/fc1-mongodb Could not write in mounted volume (pvc fc1-mongodb) DB directory
StatefulSet/postgresql-dsc password authentication failed for user "postgres"
deploy/fc1-activation-service for both volumes "ccs-init-volume" and "config-volume" : attempt to mount volume from CM that does not exist
deploy/fc1-dsba-pdp for volume "ishare-secret": secret used to create volume does not exist
StatefulSet/fc1-keycloak failed for multiple volumes (did-config, profiles, did-secret), because related CMs / secrets not found
Job/fc1-keycloak-keycloak-config-cli no idea
deploy/fc1-vcwaltid-certs for volume "certs": secret used to create volume does not exist
StatefulSet/keyrock-dsc secret "vcwaltid" not found
  • Pour la plupart de sont des volumes qui ne peuvent pas être montés car les secrets / configmaps dont ils dépendent n'existent pas.
  • Pour le StatefulSet postgres, étrangement c'est un problème d'identifiants.
  • Pour le déploiement mongo, c'est un problème de doit d'écriture dans une volume monté dedans.

Voici la liste de tous les pods :

Code
$ kubectl -n fc1 get po

NAME                                                            READY   STATUS                       RESTARTS        AGE
fc1-activation-service-7b8d4d47b5-n5p7n                         0/1     Init:CrashLoopBackOff        8 (3m45s ago)   20m
fc1-contract-management-7955db6958-87jn8                        0/1     CrashLoopBackOff             8 (3m37s ago)   20m
fc1-credentials-config-service-6954d58fbb-tl5rm                 1/1     Running                      3 (19m ago)     20m
fc1-dsba-pdp-8b5b57669-vtkhb                                    0/1     ContainerCreating            0               20m
fc1-keycloak-0                                                  0/1     Init:0/2                     0               20m
fc1-keycloak-keycloak-config-cli-9v7rl                          0/1     Error                        0               20m
fc1-keycloak-keycloak-config-cli-g2qdk                          0/1     Error                        0               19m
fc1-kong-54dd78df4c-dnhvt                                       1/1     Running                      0               20m
fc1-mongodb-594d5c4d5b-v4d5q                                    0/1     CrashLoopBackOff             8 (3m27s ago)   20m
fc1-orion-5486d78d87-7zwcn                                      0/1     CrashLoopBackOff             7 (5m6s ago)    20m
fc1-tm-forum-api-customer-bill-management-54ccf7c7df-k6g6g      1/1     Running                      0               20m
fc1-tm-forum-api-customer-management-68b46d8598-wt77t           1/1     Running                      0               20m
fc1-tm-forum-api-envoy-6c4d66c96b-7qgmq                         1/1     Running                      0               20m
fc1-tm-forum-api-party-catalog-65dbb9c489-558f9                 1/1     Running                      0               20m
fc1-tm-forum-api-product-catalog-54485899b9-54hgh               1/1     Running                      0               20m
fc1-tm-forum-api-product-inventory-64c898fbd6-9r5vn             1/1     Running                      0               20m
fc1-tm-forum-api-product-ordering-management-5cc4dd867c-h6t7h   1/1     Running                      0               20m
fc1-tm-forum-api-resource-catalog-5885997f9d-mxcb4              1/1     Running                      0               20m
fc1-tm-forum-api-resource-function-activation-9b8747467-z79j9   1/1     Running                      0               20m
fc1-tm-forum-api-resource-inventory-5bc7865889-crxkz            1/1     Running                      0               20m
fc1-tm-forum-api-service-catalog-5bcc8f7777-cmbwr               1/1     Running                      0               20m
fc1-trusted-issuers-list-5b58884b94-f2hvl                       1/1     Running                      2 (19m ago)     20m
fc1-vcverifier-649cdd6467-jzp5t                                 1/1     Running                      0               20m
fc1-vcwaltid-54db68b97f-4szwr                                   1/1     Running                      0               20m
fc1-vcwaltid-certs-d855c8b4d-gjjr2                              0/1     ContainerCreating            0               20m
keyrock-dsc-0                                                   0/1     CreateContainerConfigError   0               20m
mysql-dsc-0                                                     1/1     Running                      0               20m
postgresql-dsc-0                                                0/1     CrashLoopBackOff             8 (3m29s ago)   20m


$ kubectl get -n fc1 statefulset,deploy

NAME                              READY   AGE
statefulset.apps/fc1-keycloak     0/1     25m
statefulset.apps/keyrock-dsc      0/1     25m
statefulset.apps/mysql-dsc        1/1     25m
statefulset.apps/postgresql-dsc   0/1     25m

NAME                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/fc1-activation-service                          0/1     1            0           25m
deployment.apps/fc1-contract-management                         0/1     1            0           25m
deployment.apps/fc1-credentials-config-service                  1/1     1            1           25m
deployment.apps/fc1-dsba-pdp                                    0/1     1            0           25m
deployment.apps/fc1-kong                                        1/1     1            1           25m
deployment.apps/fc1-mongodb                                     0/1     1            0           25m
deployment.apps/fc1-orion                                       0/1     1            0           25m
deployment.apps/fc1-tm-forum-api-customer-bill-management       1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-customer-management            1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-envoy                          1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-party-catalog                  1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-product-catalog                1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-product-inventory              1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-product-ordering-management    1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-resource-catalog               1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-resource-function-activation   1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-resource-inventory             1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-service-catalog                1/1     1            1           25m
deployment.apps/fc1-trusted-issuers-list                        1/1     1            1           25m
deployment.apps/fc1-vcverifier                                  1/1     1            1           25m
deployment.apps/fc1-vcwaltid                                    1/1     1            1           25m
deployment.apps/fc1-vcwaltid-certs                              0/1     1            0           25m

Mes questions sont les suivantes:

  • comment dois-je procéder pour arriver à déployer le chart?
  • J'ai cru lire dans le dépôt qui documente le connecteur que des certificats EiDAS étaient nécessaire. Pourquoi donc y a-t-il les charts prévoient-ils cert-manager?
  • Qu'est ce qui manque pour faire marcher l'exemple?

Votre aide serait le bienvenu.


Deployment process:

OS: Ubuntu 22.04
Kubernetes: v1.29.2
Helm : v3.14.2

  • In order to avoid potential issues with load balancing, ingress and PVCs, I opted to deploy the helm charts in a cloud environment: scaleway.
  • After creating the Kubernetes cluster, plus a VM to work inside, I logged on the latter then installed a few tools that would be needed later:
    • kubectl to interact with the k8s cluster
    • helm to install the chart, therefore the components of the fiware dsc
    • utility "yq", used by the generate.sh script
  • Then, I cloned the git repo of the fiware connector

Attempt 1

Initially, I followed the indications of the README, i.e. :

$ helm repo add dsc https://fiware-ops.github.io/data-space-connector/
$ helm repo update
$ helm install -n fc1 fc1 dsc/data-space-connector --create-namespace -f data-space-connector/values.yaml --set argoApplications=false

Note: I named the realase as well as the namespace fc1, and with --set, je specify that helm ought to be used, not argo.

This is the error I observed:

Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * job fc1-keycloak-keycloak-config-cli failed: BackoffLimitExceeded

I also changed the values, using the service-provider-ips example, this is what I observed:

Code
$ helm install -n fc1 fc1 dsc/data-space-connector --create-namespace -f examples/service-provider-ips/values-dsc.yaml --set argoApplications=false

W0417 13:33:43.276530    8200 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
W0417 13:33:43.283632    8200 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
W0417 13:33:43.283651    8200 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
W0417 13:33:43.312711    8200 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
W0417 13:33:43.312751    8200 warnings.go:70] annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * timed out waiting for the condition

After waiting for a while (~10 mins), thinking the kubernetes probed might've been too optimistic and issues might just go away, nothing changed.

Attempt 2

When the example failed, I ran the script generate.sh to recreate the charts, and then see if the helm repo was somehow outdated. I had to pull some other repos for that to work, like the following:

Code
declare -A H_REPOS=(
[i4trust]=https://i4trust.github.io/helm-charts
[fiware]=https://fiware.github.io/helm-charts
[bitnami]=https://charts.bitnami.com/bitnami
[konghq]=https://charts.konghq.com
[bitname-full-index]=https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
)

for repo in ${!H_REPOS[@]}; do
        helm repo add $repo ${H_REPOS[$repo]}
done

helm repo update

sed 's/^version=.*/version="${1:-0.0.0}"/' -i generate.sh

./generate.sh

As a result, I managed to get a local helm chart in charts/data-space-connector, that I deployed leading to the following result:

Code
$ helm install -n fc1 fc1 charts/data-space-connector/ --set argoApplications=false --create-namespace

Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * job fc1-keycloak-keycloak-config-cli failed: BackoffLimitExceeded

Debugging

First, I thought the issue was related a poor configuration since I just rolled with the default settings in all the values files, not matching at all my work environment.

However, after investgating more with kubectl (describe, logs, get -o yaml), I found out that the problem laid to some misconfiguration of different kubernetes objects, here is a list:

Objet Raison
deploy/fc1-mongodb Could not write in mounted volume (pvc fc1-mongodb) DB directory
StatefulSet/postgresql-dsc password authentication failed for user "postgres"
deploy/fc1-activation-service for both volumes "ccs-init-volume" and "config-volume" : attempt to mount volume from CM that does not exist
deploy/fc1-dsba-pdp for volume "ishare-secret": secret used to create volume does not exist
StatefulSet/fc1-keycloak failed for multiple volumes (did-config, profiles, did-secret), because related CMs / secrets not found
Job/fc1-keycloak-keycloak-config-cli no idea
deploy/fc1-vcwaltid-certs for volume "certs": secret used to create volume does not exist
StatefulSet/keyrock-dsc secret "vcwaltid" not found
  • Mostly it is about volumes that cannot be mounted because the secrets / configmaps they originate from don't exist.
  • For the StatefulSet postgres, strangely, it is an issue of wrong credentials.
  • As for the deployment mongo, it is about permissions to write inside a mounted volume.

Check out the list of the pods :

Code
$ kubectl -n fc1 get po

NAME                                                            READY   STATUS                       RESTARTS        AGE
fc1-activation-service-7b8d4d47b5-n5p7n                         0/1     Init:CrashLoopBackOff        8 (3m45s ago)   20m
fc1-contract-management-7955db6958-87jn8                        0/1     CrashLoopBackOff             8 (3m37s ago)   20m
fc1-credentials-config-service-6954d58fbb-tl5rm                 1/1     Running                      3 (19m ago)     20m
fc1-dsba-pdp-8b5b57669-vtkhb                                    0/1     ContainerCreating            0               20m
fc1-keycloak-0                                                  0/1     Init:0/2                     0               20m
fc1-keycloak-keycloak-config-cli-9v7rl                          0/1     Error                        0               20m
fc1-keycloak-keycloak-config-cli-g2qdk                          0/1     Error                        0               19m
fc1-kong-54dd78df4c-dnhvt                                       1/1     Running                      0               20m
fc1-mongodb-594d5c4d5b-v4d5q                                    0/1     CrashLoopBackOff             8 (3m27s ago)   20m
fc1-orion-5486d78d87-7zwcn                                      0/1     CrashLoopBackOff             7 (5m6s ago)    20m
fc1-tm-forum-api-customer-bill-management-54ccf7c7df-k6g6g      1/1     Running                      0               20m
fc1-tm-forum-api-customer-management-68b46d8598-wt77t           1/1     Running                      0               20m
fc1-tm-forum-api-envoy-6c4d66c96b-7qgmq                         1/1     Running                      0               20m
fc1-tm-forum-api-party-catalog-65dbb9c489-558f9                 1/1     Running                      0               20m
fc1-tm-forum-api-product-catalog-54485899b9-54hgh               1/1     Running                      0               20m
fc1-tm-forum-api-product-inventory-64c898fbd6-9r5vn             1/1     Running                      0               20m
fc1-tm-forum-api-product-ordering-management-5cc4dd867c-h6t7h   1/1     Running                      0               20m
fc1-tm-forum-api-resource-catalog-5885997f9d-mxcb4              1/1     Running                      0               20m
fc1-tm-forum-api-resource-function-activation-9b8747467-z79j9   1/1     Running                      0               20m
fc1-tm-forum-api-resource-inventory-5bc7865889-crxkz            1/1     Running                      0               20m
fc1-tm-forum-api-service-catalog-5bcc8f7777-cmbwr               1/1     Running                      0               20m
fc1-trusted-issuers-list-5b58884b94-f2hvl                       1/1     Running                      2 (19m ago)     20m
fc1-vcverifier-649cdd6467-jzp5t                                 1/1     Running                      0               20m
fc1-vcwaltid-54db68b97f-4szwr                                   1/1     Running                      0               20m
fc1-vcwaltid-certs-d855c8b4d-gjjr2                              0/1     ContainerCreating            0               20m
keyrock-dsc-0                                                   0/1     CreateContainerConfigError   0               20m
mysql-dsc-0                                                     1/1     Running                      0               20m
postgresql-dsc-0                                                0/1     CrashLoopBackOff             8 (3m29s ago)   20m


$ kubectl get -n fc1 statefulset,deploy

NAME                              READY   AGE
statefulset.apps/fc1-keycloak     0/1     25m
statefulset.apps/keyrock-dsc      0/1     25m
statefulset.apps/mysql-dsc        1/1     25m
statefulset.apps/postgresql-dsc   0/1     25m

NAME                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/fc1-activation-service                          0/1     1            0           25m
deployment.apps/fc1-contract-management                         0/1     1            0           25m
deployment.apps/fc1-credentials-config-service                  1/1     1            1           25m
deployment.apps/fc1-dsba-pdp                                    0/1     1            0           25m
deployment.apps/fc1-kong                                        1/1     1            1           25m
deployment.apps/fc1-mongodb                                     0/1     1            0           25m
deployment.apps/fc1-orion                                       0/1     1            0           25m
deployment.apps/fc1-tm-forum-api-customer-bill-management       1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-customer-management            1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-envoy                          1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-party-catalog                  1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-product-catalog                1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-product-inventory              1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-product-ordering-management    1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-resource-catalog               1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-resource-function-activation   1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-resource-inventory             1/1     1            1           25m
deployment.apps/fc1-tm-forum-api-service-catalog                1/1     1            1           25m
deployment.apps/fc1-trusted-issuers-list                        1/1     1            1           25m
deployment.apps/fc1-vcverifier                                  1/1     1            1           25m
deployment.apps/fc1-vcwaltid                                    1/1     1            1           25m
deployment.apps/fc1-vcwaltid-certs                              0/1     1            0           25m

My questions are the following:

  • how to I proceed to successfully deploy the chart?
  • I remember reading in the repo documenting the connector that EiDAS certificates were required. If so, then why doe the charts include cert-manager?
  • How do I get the example to work?

Your help would be greatly appreciated.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant