PVC Missing Selector on Restore

[ndrafahl]

Hey!

I don't think this is an issue with Velero, I think it's more of a misunderstanding on my part on how to backup and recover a PV/PVC where I'm not actually restoring the contents of the PV, just the kubernetes resource that is associated to the PV/PVC.

In a helm chart, I have a PV and PVC defined:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: shared-pv-volume-xyz
  labels:
    type: nfs
    name: shared-pv-volume-xyz
spec:
  storageClassName: nfs
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: the_nfs_server
    path: /
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: shared-pv-claim
  namespace: my_namespace
spec:
  storageClassName: nfs
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      name: shared-pv-volume-xyz

I created a velero schedule with the following it, where the PV and PVC are both included:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: myschedulename
  namespace: velero
spec:
  paused: false

  schedule: 0 * * * *

  useOwnerReferencesInBackup: false

  template:
    csiSnapshotTimeout: 10m

    includedNamespaces:
    - 'my_namespace'
    
    includedResources:
    - '*'
    
    excludedResources:
    - storageclasses.storage.k8s.io
    - events
    - pods
    - backups.velero.io
    
    includeClusterResources: null
    
    orLabelSelectors:
      - matchLabels:
          mylabel: mylabelvalue

    snapshotVolumes: false
    
    ttl: 24h0m0s

    defaultVolumesToFsBackup: false

    snapshotMoveData: false

    datamover: velero

    uploaderConfig:
        parallelFilesUpload: 10

From here, I created a backup from the snapshot, and confirm both the PV and the PVC both show up within the backup by doing a velero backup describe <the backup name> --details.

To test out the recovery process, I then helm uninstall the release, and confirm both the PV and the PVC are deleted from the cluster.

Then, I do a recovery via velero and confirm that the PV and the PVCs are both restored and bound properly, and everything comes up OK. This includes restoring the secrets that helm was maintaining for that release (so I can do my helm commands against that release in the cluster again). Unfortunately though, I can no longer helm upgrade that release due to the PVC:

$ helm upgrade my-release my-chart -n my-namespace--reuse-values
Error: UPGRADE FAILED: cannot patch "shared-pv-claim" with kind PersistentVolumeClaim: PersistentVolumeClaim "shared-pv-claim" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
  core.PersistentVolumeClaimSpec{
        AccessModes: {"ReadWriteMany"},
-       Selector:    nil,
+       Selector:    s"&LabelSelector{MatchLabels:map[string]string{name: shared-pv-volume-xyz,},MatchExpressions:[]LabelSelectorRequirement{},}",
        Resources:   {Requests: {s"storage": {i: {...}, s: "1Gi", Format: "BinarySI"}}},
        VolumeName:  "shared-pv-volume-xyz",
        ... // 4 identical fields
  }

The problem I run into is I can no longer do a helm upgrade of that release. The PVC prior to removal/restoration had label selectors on there to select the PV:

spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      name: shared-pv-volume-xyz
  storageClassName: nfs
  volumeMode: Filesystem
  volumeName: shared-pv-volume-xyz

But the restored PVC no longer has those selectors:

spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs
  volumeMode: Filesystem
  volumeName: shared-pv-volume-xyz

I'm wondering if I'm missing something super obvious here? One option I would have is to just not include the pv/pvc in the backup, and just helm upgrade... the release to re-create the PV/PVC with the proper selector being applied, but I'm trying to avoid additional steps on restoration.

I tried doing some searching across issues/discussions and couldn't find anything obvious, so apologies if this has been answered already.

Thanks!!

[kaovilai]

Can you describe the generated backup? velero backup describe <backup-name> I am wondering if this is using CSI snapshot.

[ndrafahl]

Sorry for the late reply on this; definitely, here it is:

Name:         tenant-20241122141918
Namespace:    velero
Labels:       app.kubernetes.io/managed-by=Helm
              platform/instance=tenant
              platform/part-of=tenant
              platform/tenant=nick
              velero.io/schedule-name=tenant
              velero.io/storage-location=default
Annotations:  meta.helm.sh/release-name=tenant
              meta.helm.sh/release-namespace=kinetic
              velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.28.8
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=28

Phase:  Completed

Uploader config:
  Parallel files upload:  10


Namespaces:
  Included:  velero, kinetic, kinetic-tenant, ingress-nginx
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        storageclasses.storage.k8s.io, events, pods, backups.velero.io
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  platform/tenant=nick or name=tenant

Storage Location:  default

Velero-Native Snapshot PVs:  false
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  24h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-11-22 14:19:18 +0000 UTC
Completed:  2024-11-22 14:19:22 +0000 UTC

Expiration:  2024-11-23 14:19:18 +0000 UTC

Total items to be backed up:  40
Items backed up:              40

Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - schedules.velero.io
  apps/v1/Deployment:
    - kinetic-tenant/integrator-nick
    - kinetic-tenant/task-nick
  apps/v1/ReplicaSet:
    - kinetic-tenant/integrator-nick-5dfd4f7654
    - kinetic-tenant/integrator-nick-b6b885579
    - kinetic-tenant/task-nick-7ffcd49796
  discovery.k8s.io/v1/EndpointSlice:
    - ingress-nginx/ingress-nginx-internal-nick-stgnp
    - kinetic-tenant/integrator-nick-g5m72
    - kinetic-tenant/task-nick-twz2v
  networking.k8s.io/v1/Ingress:
    - kinetic-tenant/nick-ingress-integrator
    - kinetic-tenant/nick-ingress-task
    - kinetic/nick-ingress-core
    - kinetic/nick-ingress-core-pack
    - kinetic/nick-ingress-core-system-space-override
    - kinetic/nick-ingress-loghub
    - kinetic/nick-ingress-oas-console
    - kinetic/nick-ingress-space-console
  networking.k8s.io/v1/NetworkPolicy:
    - kinetic-tenant/kinetic-tenant-network-policy
  rbac.authorization.k8s.io/v1/RoleBinding:
    - kinetic-tenant/default
  v1/ConfigMap:
    - kinetic-tenant/integrator-config
    - kinetic-tenant/task-config
  v1/Endpoints:
    - ingress-nginx/ingress-nginx-internal-nick
    - kinetic-tenant/integrator-nick
    - kinetic-tenant/task-nick
  v1/Namespace:
    - ingress-nginx
    - kinetic
    - kinetic-tenant
    - velero
  v1/PersistentVolume:
    - shared-pv-volume-tenant
  v1/PersistentVolumeClaim:
    - kinetic-tenant/shared-pv-claim
  v1/Secret:
    - kinetic-tenant/integrator-adapter-secret
    - kinetic-tenant/internal-tls-secret
    - kinetic-tenant/nick-config-secret
    - kinetic-tenant/rabbit-default-user
    - kinetic/sh.helm.release.v1.tenant.v1
    - kinetic/sh.helm.release.v1.tenant.v2
  v1/Service:
    - ingress-nginx/ingress-nginx-internal-nick
    - kinetic-tenant/integrator-nick
    - kinetic-tenant/task-nick
  velero.io/v1/Schedule:
    - velero/tenant

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots: <none included>

  Pod Volume Backups: <none included>

HooksAttempted:  0
HooksFailed:     0

[kaovilai]

If you are referring to the section in red below being missing

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: shared-pv-claim
  namespace: my_namespace
spec:
  storageClassName: nfs
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  selector:
-   matchLabels:
-     name: shared-pv-volume-xyz

then it is expected.

Per below, label selector limits volumes that can be bound to the claim.

Selector

Claims can specify a label selector to further filter the set of volumes. Only the volumes whose labels match the selector can be bound to the claim. The selector can consist of two fields:

• matchLabels - the volume must have a label with this value
• matchExpressions - a list of requirements made by specifying key, list of values, and operator that relates the key and values. Valid operators include In, NotIn, Exists, and DoesNotExist.

The selector are being removed by this code

velero/pkg/backup/actions/backup_pv_action.go

Lines 71 to 77 in 9f0026d

	// When StorageClassName is set to "", it means no StorageClass is specified,
	// even the default StorageClass is not used. Only keep the Selector for this case.
	// https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reserving-a-persistentvolume
	if pvc.Spec.StorageClassName == nil || *pvc.Spec.StorageClassName != "" {
	// Clean the selector to make the PVC to dynamically allocate PV.
	pvc.Spec.Selector = nil
	}

came in via #6481
to resolve #6444

If there is a reason why selector should be there in the PVC after restore please open an issue title like CSI restore should not remove PVC selector and explain the motivation.