Operator falsely errors and does not let to upgrade MongoDB cluster replicas #1613

KarooolisZi · 2024-09-04T09:43:37Z

Hello,

I was trying to adjust replicas number in the CR MongoDB yaml manifest.
Only change to current CR was changing 'replicas' from 2 to 3.

That is strange because according to operator I should be able to do this. My statefulset is not scaling. I checked last applied configurations, last success apply and no any differences were there. I compared these configuration to my VCS configuration. No changes were detected.

Using 6.0.4 MongoDB community version.
Using 0.7.8 operator version.

I tried removing annotations for existing CRD which states as 'failed' after apply and reapply again - no result. Nothing was changed so previous setup is still online and working. However, it prompts errors on operator without any reasons even after applying same configuration with 2 'replicas' again.

I have another environment with same specific operator and MongoDB versions. I was able to add replica and even arbiter to specs there. That was also the only change made to CR of MongoDB.

The error I get:
ERROR controllers/mongodb_status_options.go:104 Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet: error creating/updating StatefulSet: StatefulSet.apps “mongodb” is invalid: spec: Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘ordinals’, ‘template’, ‘updateStrategy’, ‘persistentVolumeClaimRetentionPolicy’ and ‘minReadySeconds’ are forbidden

What did you do to encounter the bug?
Steps to reproduce the behavior:

Changed my CR MongoDB cluster yaml manifest database.yaml spec.members: 2 to spec.members: 3
Use kubectl apply -f database.yaml
Checked MongoDB statefulset pods (no replica was added).
Checked MongoDB community operator logs. Observed the error.

What did you expect?
I expected operator to add an additional member to existing MongoDB cluster in statefulset. Making members number 3 instead of existing 2.

What happened instead?
Statefulset of MongoDB still had 2 members. MongoDB operator threw error (pasted in description).

Screenshots
If applicable, add screenshots to help explain your problem.

Operator Information

0.7.8
docker.io/mongo:6.0.4

Kubernetes Cluster Information

AWS EKS
1.28

If possible, please include:

The operator logs:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 2024-09-04T07:11:25.133Z INFO controllers/replica_set_controller.go:137 Reconciling MongoDB {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.134Z DEBUG controllers/replica_set_controller.go:139 Validating MongoDB.Spec {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.134Z DEBUG controllers/replica_set_controller.go:148 Ensuring the service exists {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.134Z DEBUG agent/replica_set_port_manager.go:122 No port change required {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.142Z INFO controllers/replica_set_controller.go:463 Create/Update operation succeeded {"ReplicaSet": "mongodb-<NAME>/mongodb", "operation": "updated"} 2024-09-04T07:11:25.142Z DEBUG controllers/replica_set_controller.go:409 Scaling up the ReplicaSet, the StatefulSet must be updated first {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.142Z INFO controllers/replica_set_controller.go:330 Creating/Updating StatefulSet {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.151Z ERROR controllers/mongodb_status_options.go:104 Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet: error creating/updating StatefulSet: StatefulSet.apps "mongodb" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden github.com/mongodb/mongodb-kubernetes-operator/controllers.messageOption.ApplyOption /workspace/controllers/mongodb_status_options.go:104 github.com/mongodb/mongodb-kubernetes-operator/pkg/util/status.Update /workspace/pkg/util/status/status.go:25 github.com/mongodb/mongodb-kubernetes-operator/controllers.ReplicaSetReconciler.Reconcile /workspace/controllers/replica_set_controller.go:200 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Below we assume that your replicaset database pods are named mongo-<>. For instance:

❯ k get pods
NAME      READY   STATUS    RESTARTS   AGE
mongo-0   2/2     Running   0          19h
mongo-1   2/2     Running   0          19h
                                                                                     
❯ k get mdbc
NAME    PHASE     VERSION
mongo   Running   4.4.0

yaml definitions of your MongoDB Deployment(s):
kubectl get mdbc -oyaml

apiVersion: v1
items:
- apiVersion: mongodbcommunity.mongodb.com/v1
  kind: MongoDBCommunity
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"mongodbcommunity.mongodb.com/v1","kind":"MongoDBCommunity","metadata":{"annotations":{},"name":"mongodb","namespace":"mongodb-<SENSITIVE>"},"spec":{"additionalMongodConfig":{"storage.wiredTiger.engineConfig.journalCompressor":"zlib"},"members":3,"security":{"authentication":{"modes":["SCRAM"]}},"statefulSet":{"spec":{"template":{"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"NodeGroup","operator":"In","values":["<SENSITIVE>"]}]}]}},"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"app","operator":"In","values":["mongodb"]}]},"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"containers":[{"name":"mongod","resources":{"limits":{"cpu":"1","memory":"2Gi"},"requests":{"cpu":"500m","memory":"1Gi"}}}]}},"volumeClaimTemplates":[{"metadata":{"name":"data-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"70G"}},"storageClassName":"ebs-sc"}},{"metadata":{"name":"logs-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10G"}},"storageClassName":"ebs-sc"}}]}},"type":"ReplicaSet","users":[{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"readWrite"},{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"readWrite"},{"db":"<SENSITIVE>","name":"readWrite"},{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"read"},{"db":"<SENSITIVE>","name":"read"},{"db":"<SENSITIVE>","name":"changestreamrole"}],"scramCredentialsSecretName":"<SENSITIVE>"}],"version":"6.0.4"}}
      mongodb.com/v1.lastAppliedMongoDBVersion: 6.0.4
    creationTimestamp: "2024-01-03T07:47:03Z"
    generation: 18
    labels:
      k8slens-edit-resource-version: v1
    name: mongodb
    namespace: mongodb-<SENSITIVE>
    resourceVersion: "386378973"
    uid: 8dbc92a1-061b-4ebb-a2be-d1b5dd6d696b
  spec:
    additionalMongodConfig:
      storage.wiredTiger.engineConfig.journalCompressor: zlib
    members: 3
    security:
      authentication:
        ignoreUnknownUsers: true
        modes:
        - SCRAM
    statefulSet:
      spec:
        template:
          spec:
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: NodeGroup
                      operator: In
                      values:
                      - <SENSITIVE>
              podAntiAffinity:
                preferredDuringSchedulingIgnoredDuringExecution:
                - podAffinityTerm:
                    labelSelector:
                      matchExpressions:
                      - key: app
                        operator: In
                        values:
                        - mongodb
                    topologyKey: kubernetes.io/hostname
                  weight: 100
            containers:
            - name: mongod
              resources:
                limits:
                  cpu: "1"
                  memory: 2Gi
                requests:
                  cpu: 500m
                  memory: 1Gi
        volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 70G
            storageClassName: ebs-sc
        - metadata:
            name: logs-volume
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 10G
            storageClassName: ebs-sc
    type: ReplicaSet
    users:
    - db: <SENSITIVE>
      name: <SENSITIVE>
      passwordSecretRef:
        name: <SENSITIVE>
      roles:
      - db: <SENSITIVE>
        name: clusterAdmin
      - db: <SENSITIVE>
        name: userAdminAnyDatabase
      scramCredentialsSecretName: <SENSITIVE>
    - db: <SENSITIVE>
      name: <SENSITIVE>
      passwordSecretRef:
        name: <SENSITIVE>
      roles:
      - db: <SENSITIVE>
        name: readWrite
      scramCredentialsSecretName: <SENSITIVE>
    - db: <SENSITIVE>
      name: <SENSITIVE>
      passwordSecretRef:
        name: <SENSITIVE>
      roles:
      - db: <SENSITIVE>
        name: readWrite
      scramCredentialsSecretName: <SENSITIVE>
    - db: <SENSITIVE>
      name: <SENSITIVE>
      passwordSecretRef:
        name: <SENSITIVE>
      roles:
      - db: <SENSITIVE>
        name: readWrite
      - db: <SENSITIVE>
        name: readWrite
      scramCredentialsSecretName: backup-user-scram
    - db: <SENSITIVE>
      name: <SENSITIVE>
      passwordSecretRef:
        name: <SENSITIVE>
      roles:
      - db: <SENSITIVE>
        name: dbAdmin
      - db: <SENSITIVE>
        name: userAdminAnyDatabase
      - db: <SENSITIVE>
        name: readWrite
      - db: <SENSITIVE>
        name: readWrite
      - db: <SENSITIVE>
        name: readWrite
      scramCredentialsSecretName: <SENSITIVE>
    - db: <SENSITIVE>
      name: <SENSITIVE>
      passwordSecretRef:
        name: <SENSITIVE>
      roles:
      - db: <SENSITIVE>
        name: read
      - db: <SENSITIVE>
        name: read
      - db: <SENSITIVE>
        name: changestreamrole
      scramCredentialsSecretName: <SENSITIVE>
    version: 6.0.4
  status:
    currentMongoDBMembers: 2
    currentStatefulSetReplicas: 2
    message: 'Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet:
      error creating/updating StatefulSet: StatefulSet.apps "mongodb" is invalid:
      spec: Forbidden: updates to statefulset spec for fields other than ''replicas'',
      ''ordinals'', ''template'', ''updateStrategy'', ''persistentVolumeClaimRetentionPolicy''
      and ''minReadySeconds'' are forbidden'
    mongoUri: <SENSITIVE>,<SENSITIVE>/?replicaSet=mongodb
    phase: Failed
    version: 6.0.4
kind: List
metadata:
  resourceVersion: ""

yaml definitions of your kubernetes objects like the statefulset(s), pods (we need to see the state of the containers):
kubectl get sts -oyaml

apiVersion: v1
items:
- apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    creationTimestamp: "2024-01-03T07:47:03Z"
    generation: 31
    labels:
      app: mongodb-<SENSITIVE>
    name: mongodb
    namespace: mongodb-<SENSITIVE>
    ownerReferences:
    - apiVersion: mongodbcommunity.mongodb.com/v1
      blockOwnerDeletion: true
      controller: true
      kind: MongoDBCommunity
      name: mongodb
      uid: <SENSITIVE>
    resourceVersion: "<SENSITIVE>"
    uid: <SENSITIVE>
  spec:
    persistentVolumeClaimRetentionPolicy:
      whenDeleted: Retain
      whenScaled: Retain
    podManagementPolicy: OrderedReady
    replicas: 2
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: mongodb-<SENSITIVE>
    serviceName: mongodb-<SENSITIVE>
    template:
      metadata:
        annotations:
          kubectl.kubernetes.io/restartedAt: "2024-09-02T08:55:35Z"
        creationTimestamp: null
        labels:
          app: mongodb-<SENSITIVE>
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: NodeGroup
                  operator: In
                  values:
                  - <SENSITIVE>
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - mongodb
                topologyKey: kubernetes.io/hostname
              weight: 100
        containers:
        - command:
          - /bin/sh
          - -c
          - |2+

            #run post-start hook to handle version changes
            /hooks/version-upgrade

            # wait for config and keyfile to be created by the agent
             while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

            # start mongod with this configuration
            exec mongod -f /data/automation-mongod.conf;

          env:
          - name: AGENT_STATUS_FILEPATH
            value: /healthstatus/agent-health-status.json
          image: docker.io/mongo:6.0.4
          imagePullPolicy: IfNotPresent
          name: mongod
          resources:
            limits:
              cpu: "1"
              memory: 2Gi
            requests:
              cpu: 500m
              memory: 1Gi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /data
            name: data-volume
          - mountPath: /healthstatus
            name: healthstatus
          - mountPath: /hooks
            name: hooks
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: mongodb-keyfile
          - mountPath: /tmp
            name: tmp
        - command:
          - /bin/bash
          - -c
          - |-
            current_uid=$(id -u)
            AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
            declare -r current_uid
            if ! grep -q "${current_uid}" /etc/passwd ; then
            sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
            echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
            export NSS_WRAPPER_PASSWD=/tmp/passwd
            export LD_PRELOAD=libnss_wrapper.so
            export NSS_WRAPPER_GROUP=/etc/group
            fi
            agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -maxLogFileDurationHrs 24 -logLevel DEBUG
          env:
          - name: AGENT_STATUS_FILEPATH
            value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
          - name: AUTOMATION_CONFIG_MAP
            value: mongodb-config
          - name: HEADLESS_AGENT
            value: "true"
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1
          imagePullPolicy: Always
          name: mongodb-agent
          readinessProbe:
            exec:
              command:
              - /opt/scripts/readinessprobe
            failureThreshold: 40
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
          - mountPath: /var/lib/automation/config
            name: automation-config
            readOnly: true
          - mountPath: /data
            name: data-volume
          - mountPath: /var/log/mongodb-mms-automation/healthstatus
            name: healthstatus
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: mongodb-keyfile
          - mountPath: /tmp
            name: tmp
        dnsPolicy: ClusterFirst
        initContainers:
        - command:
          - cp
          - version-upgrade-hook
          - /hooks/version-upgrade
          image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6
          imagePullPolicy: Always
          name: mongod-posthook
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /hooks
            name: hooks
        - command:
          - cp
          - /probes/readinessprobe
          - /opt/scripts/readinessprobe
          image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12
          imagePullPolicy: Always
          name: mongodb-agent-readinessprobe
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 2000
          runAsNonRoot: true
          runAsUser: 2000
        serviceAccount: mongodb-<SENSITIVE>
        serviceAccountName: mongodb-<SENSITIVE>
        terminationGracePeriodSeconds: 30
        volumes:
        - emptyDir: {}
          name: agent-scripts
        - name: automation-config
          secret:
            defaultMode: 416
            secretName: mongodb-<SENSITIVE>
        - emptyDir: {}
          name: healthstatus
        - emptyDir: {}
          name: hooks
        - emptyDir: {}
          name: mongodb-<SENSITIVE>
        - emptyDir: {}
          name: tmp
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50G
        storageClassName: ebs-sc
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10G
        storageClassName: ebs-sc
        volumeMode: Filesystem
      status:
        phase: Pending
  status:
    availableReplicas: 2
    collisionCount: 0
    currentReplicas: 2
    currentRevision: mongodb-<SENSITIVE>
    observedGeneration: 31
    readyReplicas: 2
    replicas: 2
    updateRevision: mongodb-<SENSITIVE>
    updatedReplicas: 2
kind: List
metadata:
  resourceVersion: ""

kubectl get pods -oyaml

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      agent.mongodb.com/version: "6"
      kubectl.kubernetes.io/restartedAt: "2024-09-02T08:55:35Z"
    creationTimestamp: "2024-09-02T08:56:31Z"
    generateName: mongodb-
    labels:
      app: mongodb-<SENSITIVE>
      apps.kubernetes.io/pod-index: "0"
      controller-revision-hash: mongodb-<SENSITIVE>
      statefulset.kubernetes.io/pod-name: <SENSITIVE>
    name: mongodb-0
    namespace: mongodb-<SENSITIVE>
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: StatefulSet
      name: mongodb
      uid: 934952c3-22f8-4b93-a6e3-34c85bc85fc0
    resourceVersion: "386287077"
    uid: 5c442610-203d-45ca-b280-dec81b9a923d
  spec:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: NodeGroup
              operator: In
              values:
              - <SENSITIVE>
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - mongodb
            topologyKey: kubernetes.io/hostname
          weight: 100
    containers:
    - command:
      - /bin/sh
      - -c
      - |2+

        #run post-start hook to handle version changes
        /hooks/version-upgrade

        # wait for config and keyfile to be created by the agent
         while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

        # start mongod with this configuration
        exec mongod -f /data/automation-mongod.conf;

      env:
      - name: AGENT_STATUS_FILEPATH
        value: /healthstatus/agent-health-status.json
      image: docker.io/mongo:6.0.4
      imagePullPolicy: IfNotPresent
      name: mongod
      resources:
        limits:
          cpu: "1"
          memory: 2Gi
        requests:
          cpu: 500m
          memory: 1Gi
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /data
        name: <SENSITIVE>
      - mountPath: /healthstatus
        name: healthstatus
      - mountPath: /hooks
        name: hooks
      - mountPath: /var/log/mongodb-mms-automation
        name: logs-volume
      - mountPath: /var/lib/mongodb-mms-automation/authentication
        name: <SENSITIVE>
      - mountPath: /tmp
        name: <SENSITIVE>
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-t6sf8
        readOnly: true
    - command:
      - /bin/bash
      - -c
      - |-
        current_uid=$(id -u)
        AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
        declare -r current_uid
        if ! grep -q "${current_uid}" /etc/passwd ; then
        sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
        echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
        export NSS_WRAPPER_PASSWD=/tmp/passwd
        export LD_PRELOAD=libnss_wrapper.so
        export NSS_WRAPPER_GROUP=/etc/group
        fi
        agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -maxLogFileDurationHrs 24 -logLevel DEBUG
      env:
      - name: AGENT_STATUS_FILEPATH
        value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
      - name: AUTOMATION_CONFIG_MAP
        value: mongodb-config
      - name: HEADLESS_AGENT
        value: "true"
      - name: POD_NAMESPACE
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.namespace
      image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1
      imagePullPolicy: Always
      name: mongodb-agent
      readinessProbe:
        exec:
          command:
          - /opt/scripts/readinessprobe
        failureThreshold: 40
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      resources:
        limits:
          cpu: "1"
          memory: 500M
        requests:
          cpu: 500m
          memory: 400M
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /opt/scripts
        name: agent-scripts
      - mountPath: /var/lib/automation/config
        name: automation-config
        readOnly: true
      - mountPath: /data
        name: data-volume
      - mountPath: /var/log/mongodb-mms-automation/healthstatus
        name: healthstatus
      - mountPath: /var/log/mongodb-mms-automation
        name: logs-volume
      - mountPath: /var/lib/mongodb-mms-automation/authentication
        name: mongodb-keyfile
      - mountPath: /tmp
        name: tmp
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-t6sf8
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    hostname: <SENSITIVE>
    initContainers:
    - command:
      - cp
      - version-upgrade-hook
      - /hooks/version-upgrade
      image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6
      imagePullPolicy: Always
      name: mongod-posthook
      resources: {}
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /hooks
        name: hooks
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-t6sf8
        readOnly: true
    - command:
      - cp
      - /probes/readinessprobe
      - /opt/scripts/readinessprobe
      image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12
      imagePullPolicy: Always
      name: mongodb-agent-readinessprobe
      resources: {}
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /opt/scripts
        name: agent-scripts
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-t6sf8
        readOnly: true
    nodeName: <SENSITIVE>
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
      fsGroup: <SENSITIVE>
      runAsNonRoot: true
      runAsUser: <SENSITIVE>
    serviceAccount: <SENSITIVE>
    serviceAccountName: <SENSITIVE>
    subdomain: <SENSITIVE>
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: data-volume
      persistentVolumeClaim:
        claimName: data-volume-mongodb-0
    - name: logs-volume
      persistentVolumeClaim:
        claimName: logs-volume-mongodb-0
    - emptyDir: {}
      name: agent-scripts
    - name: automation-config
      secret:
        defaultMode: 416
        secretName: <SENSITIVE>
    - emptyDir: {}
      name: healthstatus
    - emptyDir: {}
      name: hooks
    - emptyDir: {}
      name: <SENSITIVE>
    - emptyDir: {}
      name: tmp
    - name: kube-api-access-t6sf8
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: <SENSITIVE>
        - configMap:
            items:
            - key: <SENSITIVE>
              path: <SENSITIVE>
            name: <SENSITIVE>
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2024-09-02T08:56:38Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2024-09-02T08:56:55Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2024-09-02T08:56:55Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2024-09-02T08:56:31Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://5a01d0a5116ac5af31cee1f7b0fe6b516d22b5e223042adfb035ee3fcaf7d284
      image: docker.io/library/mongo:6.0.4
      imageID: docker.io/library/mongo@sha256:a4f2db6f54aeabba562cd07e5cb758b55d6192dcc6f36322a334ba0b0120aaf1
      lastState: {}
      name: mongod
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2024-09-02T08:56:38Z"
    - containerID: containerd://fab2952924c2028fab4117bb26aaf53715a1993792a5730226823925fb9146db
      image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1
      imageID: quay.io/mongodb/mongodb-agent@sha256:00dc01552af1eac8a457d7214ae4f913dd2a14409c0125dead25808e40cc0d62
      lastState: {}
      name: mongodb-agent
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2024-09-02T08:56:38Z"
    hostIP: <SENSITIVE>
    initContainerStatuses:
    - containerID: containerd://42c48fb8a9556f01b3ab8d61b985aec62c4e177dc53b2b65e0afecaa3991dbe0
      image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6
      imageID: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:5c1483fa22cfb772f186a30180fcfa4af91ff3a638b28a37cc9f997f8ac046f9
      lastState: {}
      name: mongod-posthook
      ready: true
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: containerd://42c48fb8a9556f01b3ab8d61b985aec62c4e177dc53b2b65e0afecaa3991dbe0
          exitCode: 0
          finishedAt: "2024-09-02T08:56:36Z"
          reason: Completed
          startedAt: "2024-09-02T08:56:36Z"
    - containerID: containerd://144d11be92c06da8c8c69faa841708f4caf73ebb9a9255cecfa499e8a05befa4
      image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12
      imageID: quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:ef6c144de962bc976161b41972cc85988189870e05ad4ada0798200c0a988547
      lastState: {}
      name: mongodb-agent-readinessprobe
      ready: true
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: containerd://144d11be92c06da8c8c69faa841708f4caf73ebb9a9255cecfa499e8a05befa4
          exitCode: 0
          finishedAt: "2024-09-02T08:56:37Z"
          reason: Completed
          startedAt: "2024-09-02T08:56:37Z"
    phase: Running
    podIP: <SENSITIVE>
    podIPs:
    - ip: <SENSITIVE>
    qosClass: Burstable
    startTime: "2024-09-02T08:56:31Z"
kind: List
metadata:
  resourceVersion: ""

The agent clusterconfig of the faulty members:
kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/lib/automation/config/cluster-config.json

{"version":6,"processes":[{"name":"mongodb-0","disabled":false,"hostname":"<SENSITIVE>","args2_6":{"net":{"port":27017},"repli
cation":{"replSetName":"mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilityVersion":"6.0","processTyp
e":"mongod","version":"6.0.4","authSchemaVersion":5},{"name":"mongodb-1","disabled":false,"hostname":"<SENSITIVE>","args2_6":{
"net":{"port":27017},"replication":{"replSetName":"mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilit
yVersion":"6.0","processType":"mongod","version":"6.0.4","authSchemaVersion":5}],"replicaSets":[{"_id":"mongodb","members":[{"_id":0,"host":"mongodb-0","priority":1,"arbi
terOnly":false,"votes":1},{"_id":1,"host":"mongodb-1","priority":1,"arbiterOnly":false,"votes":1}],"protocolVersion":"1","numberArbiters":0}],"auth":{"usersWanted":[{"mec
hanisms":[],"roles":[{"role":"clusterAdmin","<SENSITIVE>":"<SENSITIVE>"},{"role":"userAdminAnyDatabase","<SENSITIVE>":"<SENSITIVE>"}],"user":"<SENSITIVE>","<SENSITIVE>":"<SENSITIVE>","authenticationRestrictions":[],"
scramSha256Creds":{"iterationCount":15000,"salt":"WioeMJQXT8w9Coif5Fq3gqV1OfeDi64Bvq/maw==","serverKey":"iK8SJLwCzmk95+mUePC3wrpGw29Tfx9vN+ZCKSMPKMM=","storedKey":"Z33GU9
ix2W++nlnkFBIbYP7kEATZ/6sDVQqdhEd+tT0="},"scramSha1Creds":{"iterationCount":10000,"salt":"Q9mmbNXpyLDRtYmoln1xgA==","serverKey":"AmaNP+YmbrNf23l8URaZAZKKOz0=","storedKey"
:"0d8SscAfTMph+2aW416TXB1/UZw="}},{"mechanisms":[],"roles":[{"role":"readWrite","<SENSITIVE>":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions":[],"scramSha
256Creds":{"iterationCount":15000,"salt":"GOkTsWgdrct5KSSQtTHC20myEJM76v5OMEGXOA==","serverKey":"ksqF9YIWnI50+bQJhjl0/zA1a0H0UpcNnzxnFEjciV4=","storedKey":"GyxjpwCp9hTHsK
5CX2ObkIs73NP2zL1VrwbCQTLDvGE="},"scramSha1Creds":{"iterationCount":10000,"salt":"4RAcRyAxnRCQQhcHWRDA2w==","serverKey":"59KQ8PQV/rS4zxSuVca/tQbDNWw=","storedKey":"v68O4b
x8u7/RNIks1WBvmXIJ+H8="}},{"mechanisms":[],"roles":[{"role":"readWrite","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions":[],"scramSha256Creds
":{"iterationCount":15000,"salt":"vYX0jOTF0NvPPmpQm1oz/b7v1/sAnFOMWdm5Pg==","serverKey":"MWuSeUedUk33YD57g/pVw+kV89vQK8OmTibRLl2hR0U=","storedKey":"ffXuxQ5HTf0FcH2FdcNvKi
gWSO/TgPdF0elXk9iYX3E="},"scramSha1Creds":{"iterationCount":10000,"salt":"zV90H0Z2XJ8sCiupCSK3PQ==","serverKey":"IwdxN4BVrGSqyLPXDhrbZKFsbtc=","storedKey":"e5XxJTwdueUyUy
Jd3ioqQFuEKbc="}},{"mechanisms":[],"roles":[{"role":"readWrite","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions
":[],"scramSha256Creds":{"iterationCount":15000,"salt":"0HwrZJa9FIMCy5r4erCq7o2gb/RSaHofCV1XMw==","serverKey":"4FTg/fstci+8W6BZE4jfyXpLJr9/f4zsuDiKrLnBcgg=","storedKey":"
lThVu1E2tv14Q7H58DYYNK1jlqXaIZDCp/Omp44wR1A="},"scramSha1Creds":{"iterationCount":10000,"salt":"gDxOqOLC16/e/WvhWSGDdA==","serverKey":"q6SKd30cOY+PFQnqRFMpdgmNTFA=","stor
edKey":"9swpmpNkjLofRRRprZWGrCBoolk="}},{"mechanisms":[],"roles":[{"role":"dbAdmin","db":"<SENSITIVE>"},{"role":"userAdminAnyDatabase","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions":[],"scramSha256Creds":{"iterationC
ount":15000,"salt":"m40BqXb1jbVT4NIN9NjAdbTdQp84O6KtEbRjgA==","serverKey":"hztnbCDXJs0zBcwtaJsLquRtEgCHDykKj04SaQ3eLn8=","storedKey":"cftguOpTPNr4QYvL2XtrRlybPzi96CgzGoZ9
1EVZg2g="},"scramSha1Creds":{"iterationCount":10000,"salt":"SofiWm+P4s3RwvvIxflOOQ==","serverKey":"73knk0VrQPm6PWSYM5PFYwUK1lA=","storedKey":"wnYGbRIv2qPtcpv3j4r+lUX8x/4=
"}},{"mechanisms":[],"roles":[{"role":"read","db":"<SENSITIVE>"},{"role":"read","db":"<SENSITIVE>"},{"role":"changestreamrole","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authentica
tionRestrictions":[],"scramSha256Creds":{"iterationCount":15000,"salt":"wzzV+4JFqwIGw1mAzRucb1oiIcYYR/gdcwZyJw==","serverKey":"Ty105G/oxXhrv9UwgIqqXHO7ZxM5LYW9T/mta7uiQYo
=","storedKey":"63E/7kQy2g4O/MUd0a62q8pQNBtITkJ74dUagrRESO4="},"scramSha1Creds":{"iterationCount":10000,"salt":"R/PWGnO94tygxdNjIavdkQ==","serverKey":"Gr8vtcLPyb/pR/h8GRi
vOrY6/sE=","storedKey":"kwWqu/dnDzNTDOEu03TcgeDWxPY="}}],"disabled":false,"authoritativeSet":false,"autoAuthMechanisms":["SCRAM-SHA-256"],"autoAuthMechanism":"SCRAM-SHA-2
56","deploymentAuthMechanisms":["SCRAM-SHA-256"],"autoUser":"mms-automation","key":"8tQDoV1eZdKJvpc7cA8rtu939Glj1IgsL9CNE1nf7SuZMFw8Te47PmhA9Z1NPi27cRw5+bs16kenEAPP82V7v5
1Xcv5Q9xPZKUxltKlc3t9cfq2Q7Il42DJsrjhQUhne5lKNghWLRHPSFVb8IHbuImgPcvu7mPz6VsYClu6Lno5ewW3ziIvilIW/2xvpxqG0qz4jvz5/cmtTWeNn7VJzNOYwYurWdFfvdUDL+Z+kQqcbsa95SSYA8217h6aKE2gu
OwlVpK0VZBYCPACg+ID1dARawAHG7xCA92lFttymLfgu8kbUXeW6RxBsgwz5iuOXjiIrm8XpWhpWHLJNplf5YaGsqBMIbRlH3tAXGv6auqLaiGup3+kQXDJNwC7Juaa5F0FGXg+PdQPMOH4xv2SZy0zGHh988CaEtXhBVWiQ06
FhnNWyxziLCl8BGJpCbD2bsGiiUBcUHvxkCybARhguLYdnS60+tlJcMIr3rpt7MTgRuHhwki0gX1KcVmEe+tPeg57RdqcVcEEqqHYwc4Ghkk/PF/10BlsO0NiUZmJxZqow7ffSRZHtZ/VKW2og6CZp2V3BaYZmzYwHn5XFFRCD
NUu8mbwvtySQVSlVVY4GbKRkgepYsWrYGc20yPH7Hzni9b8N0zCmX8HPy5icn8+jf4z7BRw=","keyfile":"/var/lib/mongodb-mms-automation/authentication/keyfile","keyfileWindows":"%SystemDriv
e%\\MMSAutomation\\versions\\keyfile","autoPwd":"eX-rXNR2PB_ytwagyylk"},"tls":{"CAFilePath":"","clientCertificateMode":"OPTIONAL"},"mongoDbVersions":[{"name":"6.0.4","bui
lds":[{"platform":"linux","url":"","gitVersion":"","architecture":"amd64","flavor":"rhel","minOsVersion":"","maxOsVersion":"","modules":[]},{"platform":"linux","url":"","
gitVersion":"","architecture":"amd64","flavor":"ubuntu","minOsVersion":"","maxOsVersion":"","modules":[]}]}],"backupVersions":[],"monitoringVersions":[],"options":{"downl
oadBase":"/var/lib/mongodb-mms-automation"}}

The agent health status of the faulty members:
- kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json

{"statuses":{"mongodb-0":{"IsInGoalState":true,"LastMongoUpTime":1725441832,"ExpectedToBeUp":true,"ReplicationStatus":1}},"mmsStatus":{"mongodb-0":{"name":"mongodb-0","la
stGoalVersionAchieved":6,"plans":[{"automationConfigVersion":6,"started":"2024-09-02T08:56:39.289177326Z","completed":"2024-09-02T08:56:44.316657181Z","moves":[{"move":"S
tart","moveDoc":"Start the process","steps":[{"step":"StartFresh","stepDoc":"Start a mongo instance  (start fresh)","isWaitStep":false,"started":"2024-09-02T08:56:39.2891
92036Z","completed":"2024-09-02T08:56:43.2738461Z","result":"success"}]},{"move":"WaitAllRsMembersUp","moveDoc":"Wait until all members of this process' repl set are up",
"steps":[{"step":"WaitAllRsMembersUp","stepDoc":"Wait until all members of this process' repl set are up","isWaitStep":true,"started":"2024-09-02T08:56:43.273886773Z","co
mpleted":"2024-09-02T08:56:43.274018861Z","result":"success"}]},{"move":"RsInit","moveDoc":"Initialize a replica set including the current MongoDB process","steps":[{"ste
p":"RsInit","stepDoc":"Initialize a replica set","isWaitStep":false,"started":"2024-09-02T08:56:43.274046073Z","completed":null,"result":"error"}]},{"move":"WaitFeatureCo
mpatibilityVersionCorrect","moveDoc":"Wait for featureCompatibilityVersion to be right","steps":[{"step":"WaitFeatureCompatibilityVersionCorrect","stepDoc":"Wait for feat
ureCompatibilityVersion to be right","isWaitStep":true,"started":null,"completed":null,"result":""}]}]},{"automationConfigVersion":6,"started":"2024-09-03T12:29:37.736565
486Z","completed":"2024-09-03T12:29:42.80790282Z","moves":[{"move":"EnsureAutomationCredentials","moveDoc":"Ensure the automation user exists","steps":[{"step":"EnsureAut
omationCredentials","stepDoc":"Ensure the automation user exists","isWaitStep":false,"started":"2024-09-03T12:29:37.736594109Z","completed":"2024-09-03T12:29:42.806979734
Z","result":"success"}]}]}],"errorCode":0,"errorString":""}}}

The verbose agent logs of the faulty members:
- kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent-verbose.log

[2024-09-03T12:57:47.152+0000] [.debug] [src/mongoctl/processctl.go:ProcessType:1306] <mongodb-0> [12:57:47.152] Checking process type of mongodb-0.mongodb-svc.mongodb-su
rplus.svc.cluster.local:27017 (local=false)...
[2024-09-03T12:57:47.152+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.152] Starting RunCommand(dbName=admin, cmd=[{isMaste
r 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.153] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{isMa
ster 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":27},"signature":{"
hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"connectionId":13,"electionId":"7fffffff000000000000008d","hosts":["mongodb-0.mong
odb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","mongodb-1.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017"],"ismaster":true,"lastWrite":{"lastWriteDate":"2024-09-03
T12:57:47Z","majorityOpTime":{"t":141,"ts":{"T":1725368267,"I":27}},"majorityWriteDate":"2024-09-03T12:57:47Z","opTime":{"t":141,"ts":{"T":1725368267,"I":27}}},"localTime
":"2024-09-03T12:57:47.153Z","logicalSessionTimeoutMinutes":30,"maxBsonObjectSize":16777216,"maxMessageSizeBytes":48000000,"maxWireVersion":17,"maxWriteBatchSize":100000,
"me":"mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","minWireVersion":0,"ok":1,"operationTime":{"T":1725368267,"I":27},"primary":"mongodb-0.mongodb-svc.mo
ngodb-<SENSITIVE>.svc.cluster.local:27017","readOnly":false,"secondary":false,"setName":"mongodb","setVersion":1,"topologyVersion":{"counter":10,"processId":"66d57dc915c38471
0a9d334d"}}
[2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.153] Starting RunCommand(dbName=admin, cmd=[{getPara
meter 1} {featureCompatibilityVersion 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.153] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getP
arameter 1} {featureCompatibilityVersion 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"
T":1725368267,"I":27},"signature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"featureCompatibilityVersion":{"version":"6.0"
},"ok":1,"operationTime":{"T":1725368267,"I":27}}
[2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:FeatureCompatibilityVersion:3796] <mongodb-0> [12:57:47.153] FeatureCompatibilityVersion on 3.6+ is 6.
0
[2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:ParsedArgs:2414] <mongodb-0> [12:57:47.153] Getting parsed args
[2024-09-03T12:57:47.154+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.154] Starting RunCommand(dbName=admin, cmd=[{isMaste
r 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.154+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.154] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{isMa
ster 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":27},"signature":{"
hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"connectionId":13,"electionId":"7fffffff000000000000008d","hosts":["mongodb-0.mong
odb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","mongodb-1.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017"],"ismaster":true,"lastWrite":{"lastWriteDate":"2024-09-03
T12:57:47Z","majorityOpTime":{"t":141,"ts":{"T":1725368267,"I":27}},"majorityWriteDate":"2024-09-03T12:57:47Z","opTime":{"t":141,"ts":{"T":1725368267,"I":27}}},"localTime
":"2024-09-03T12:57:47.154Z","logicalSessionTimeoutMinutes":30,"maxBsonObjectSize":16777216,"maxMessageSizeBytes":48000000,"maxWireVersion":17,"maxWriteBatchSize":100000,
"me":"mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","minWireVersion":0,"ok":1,"operationTime":{"T":1725368267,"I":27},"primary":"mongodb-0.mongodb-svc.mo
ngodb-<SENSITIVE>.svc.cluster.local:27017","readOnly":false,"secondary":false,"setName":"mongodb","setVersion":1,"topologyVersion":{"counter":10,"processId":"66d57dc915c38471
0a9d334d"}}
[2024-09-03T12:57:47.154+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.154] Starting RunCommand(dbName=admin, cmd=[{getCmdL
ineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.155] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getC
mdLineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":27},"signatu
re":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"argv":["mongod","-f","/data/automation-mongod.conf"],"ok":1,"operationTime"
:{"T":1725368267,"I":27},"parsed":{"config":"/data/automation-mongod.conf","net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"au
thorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath"
:"/data","engine":"wiredTiger","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}}
[2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:overwriteWithRedactedArgsFromConfig:2495] <mongodb-0> [12:57:47.155] Attempting to overwrite redacted
args with real value of redacted args from config file [/data/automation-mongod.conf] for host [27017]
[2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:overwriteWithRedactedArgsFromConfig:2506] <mongodb-0> [12:57:47.155] Overwrote redacted args with real
 value of redacted args from config file [/data/automation-mongod.conf] for host [27017]
[2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.155] Starting RunCommand(dbName=admin, cmd=[{getPara
meter 1} {tlsMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.156+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.156] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getP
arameter 1} {tlsMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":28
},"signature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"ok":1,"operationTime":{"T":1725368267,"I":28},"tlsMode":"disabled
"}
[2024-09-03T12:57:47.156+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.156] Starting RunCommand(dbName=admin, cmd=[{getPara
meter 1} {clusterAuthMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.156+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.156] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getP
arameter 1} {clusterAuthMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":172536826
7,"I":28},"signature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"clusterAuthMode":"keyFile","ok":1,"operationTime":{"T":17
25368267,"I":28}}
[2024-09-03T12:57:47.157+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.157] Starting RunCommand(dbName=admin, cmd=[{getDefa
ultRWConcern 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.157+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.157] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getD
efaultRWConcern 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":28},"si
gnature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"defaultReadConcern":{"level":"local"},"defaultReadConcernSource":"impl
icit","defaultWriteConcern":{"w":"majority","wtimeout":0},"defaultWriteConcernSource":"implicit","localUpdateWallClockTime":"2024-09-03T12:57:47.157Z","ok":1,"operationTi
me":{"T":1725368267,"I":28}}
[2024-09-03T12:57:47.157+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.157] Starting RunCommand(dbName=admin, cmd=[{getCmdL
ineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.158+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.158] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getC
mdLineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":28},"signatu
re":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"argv":["mongod","-f","/data/automation-mongod.conf"],"ok":1,"operationTime"
:{"T":1725368267,"I":28},"parsed":{"config":"/data/automation-mongod.conf","net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"au
thorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath"
:"/data","engine":"wiredTiger","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}}
[2024-09-03T12:57:47.159+0000] [.info] [src/mongoclientservice/mongoclientservice.go:func1:1619] [12:57:47.159] Testing auth with username __system db=local to mongodb-0.
mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) connectMode=SingleConnect ipversion=0 tls=false
[2024-09-03T12:57:47.169+0000] [.info] [src/mongoctl/processctl.go:GetKeyHashes:2080] <mongodb-0> [12:57:47.169] Able to successfully auth to mongodb-0.mongodb-svc.mongod
b-<SENSITIVE>.svc.cluster.local:27017 (local=false) using desired auth key
[2024-09-03T12:57:47.169+0000] [.debug] [state/stateutil/fickleutil.go:CurrentFickle:51] <mongodb-0> [12:57:47.169] Starting CurrentFickle...
[2024-09-03T12:57:47.169+0000] [.debug] [src/mongoctl/replsetctl.go:ReplSetStatus:285] <mongodb-0> [12:57:47.169] Getting repl set status for process mongodb-0.mongodb-sv
c.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false)...
[2024-09-03T12:57:47.169+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.169] Starting RunCommand(dbName=admin, cmd=[{replSet
GetStatus 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.170+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.170] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{repl
SetGetStatus 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"members":[{"_id":0,"name":"mongodb-0.mongodb-svc.mongodb-su
rplus.svc.cluster.local:27017","health":1,"state":1,"self":true,"optime":{"ts":{"T":1725368267,"I":29},"t":141},"optimedate":"2024-09-03T12:57:47Z"},{"_id":1,"name":"mong
odb-1.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","health":1,"state":2,"self":false,"optime":{"ts":{"T":1725368266,"I":67},"t":141},"optimedate":"2024-09-03T12:5
7:46Z"}],"errmsg":"","myState":1,"optimes":{"lastCommittedOpTime":{"ts":{"T":1725368267,"I":28},"t":141},"appliedOpTime":{"ts":{"T":1725368267,"I":29},"t":141},"durableOp
Time":{"ts":{"T":1725368267,"I":29},"t":141}}}

The text was updated successfully, but these errors were encountered:

nammn · 2024-09-04T10:13:19Z

hey @KarooolisZi thanks for opening an Issue!

looking at your applied sts and applied cr one can see that you've increased the storage of the pvcClaim.

mdbc

        volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 70G <---- look at me!
            storageClassName: ebs-sc
        - metadata:
            name: logs-volume
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 10G

vs

sts

    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50G  <---- look at me!
        storageClassName: ebs-sc
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10G
        storageClassName: ebs-sc
        volumeMode: Filesystem

I suggest that you update your claim to be equal to the one in the sts and you should be fine

KarooolisZi · 2024-09-04T10:20:43Z

@nammn
Thank you for such a fast response. I might missed this one.
I can see my PVs are now 70GB yet sts config has 50GB.
I remember changing storage because of the urgent need. Is there any chance to make sts use these PVs already created by mdbc.
Or the only resolution is to either get back to old size in mdbc or recreate sts?

nammn · 2024-09-04T10:22:50Z

@KarooolisZi you can try the following to have the operator use the new pvc sizes, please note that this is a limitation of sts. You cannot resize the storage used by an sts. Read more here: kubernetes/enhancements@763b35f

steps:

kubectl delete sts --cascade=false (that ensures you only orphan your pods!)
the operator will re-create the sts with your new pvc values

KarooolisZi · 2024-09-04T10:25:19Z

@nammn Yes, I am aware of limitation just thinking about the workaround.
Would these steps guarantee sts works as previously and attaches required pvc?

nammn · 2024-09-04T10:30:04Z

yes, since the pvcs are having the same name and therefore the sts will re-attach them, all assuming you didn't change the name.

Closing this one, since the issue itself was a miss-configuration.

KarooolisZi · 2024-09-04T10:31:29Z

Yes, great misconfiguration. I have definitely spent too much time on it..
Thank you for your patience and swift response

KarooolisZi · 2024-09-06T10:28:21Z

@nammn After doing so, my sts shows as 0/0 with replicas 2 and does not recreate pods. I have one pod but it is not in sts somehow.

It looks like adding an arbiter crashes operator. Trying to do 2 members and 1 arbiter. 2 members work perfectly.

After getting it to work, I get 0/2 as mongo agent is not ready after adding arbiter. Strange behaviour

nammn · 2024-09-09T09:40:43Z

@KarooolisZi please open a new issue and provide the required information as given in the issue template

KarooolisZi · 2024-09-09T11:51:56Z

@nammn Did it: #1615

nammn self-assigned this Sep 4, 2024

nammn closed this as completed Sep 4, 2024

nammn mentioned this issue Sep 4, 2024

MongoDB Failed Status #1552

Closed

KarooolisZi mentioned this issue Sep 9, 2024

MongoDBCommunity not using the replicaSetHorizons values to generate Mongo Uri resulting in MongoNetworkError #1614

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Operator falsely errors and does not let to upgrade MongoDB cluster replicas #1613

Operator falsely errors and does not let to upgrade MongoDB cluster replicas #1613

KarooolisZi commented Sep 4, 2024

nammn commented Sep 4, 2024

KarooolisZi commented Sep 4, 2024

nammn commented Sep 4, 2024

KarooolisZi commented Sep 4, 2024

nammn commented Sep 4, 2024

KarooolisZi commented Sep 4, 2024

KarooolisZi commented Sep 6, 2024 •

edited

Loading

nammn commented Sep 9, 2024

KarooolisZi commented Sep 9, 2024

Operator falsely errors and does not let to upgrade MongoDB cluster replicas #1613

Operator falsely errors and does not let to upgrade MongoDB cluster replicas #1613

Comments

KarooolisZi commented Sep 4, 2024

nammn commented Sep 4, 2024

KarooolisZi commented Sep 4, 2024

nammn commented Sep 4, 2024

KarooolisZi commented Sep 4, 2024

nammn commented Sep 4, 2024

KarooolisZi commented Sep 4, 2024

KarooolisZi commented Sep 6, 2024 • edited Loading

nammn commented Sep 9, 2024

KarooolisZi commented Sep 9, 2024

KarooolisZi commented Sep 6, 2024 •

edited

Loading