velero use "exclude-namespaces" to backup not worked #3259

liyan-su · 2020-09-29T10:13:16Z

liyan-su
Sep 29, 2020

What steps did you take and what happened:

create 1 cluster with 3masters and 500 work nodes.
create 2500 namespace, 5000pods(10000deployments), 3997 loadbalance service
velero backup create backup-exclude --exclude-namespaces velero,default,pks-system,kube-node-lease,kube-public,kube-system,nsx-system(failed)

What did you expect to happen:

I hope velero can backup succefully, but status is "InProgress" for long time, I check backup details

root@3694960b654b:~# velero backup describe backup-exclude --details
Name:         backup-exclude
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.18.8+vmware.1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=18

Phase:  InProgress

Errors:    0
Warnings:  0

Namespaces:
  Included:  *
  Excluded:  velero, default, pks-system, kube-node-lease, kube-public, kube-system, nsx-system

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-09-29 09:25:21 +0000 UTC
Completed:  <n/a>

Expiration:  2020-10-29 09:25:21 +0000 UTC

Resource List:  <backup resource list not found>

Velero-Native Snapshots: <none included>

and then check velero log, but didn't find any information about backup-exclude, I try many times, the result is the same.

root@3694960b654b: kubectl logs velero-6c8b8dc5db-sj42j -n velero

time="2020-09-29T09:25:46Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:177"
time="2020-09-29T09:25:46Z" level=info msg="Starting Velero server v1.4.2_vmware.1 (56a08a4d695d893f0863f697c2f926e27d70c0c5-dirty)" logSource="pkg/cmd/server/server.go:179"
time="2020-09-29T09:25:46Z" level=info msg="1 feature flags enabled []" logSource="pkg/cmd/server/server.go:181"
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-remap-version
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-pvc-node-selector
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-preserve-fields
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2020-09-29T09:25:47Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2020-09-29T09:25:47Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:361" namespace=velero
time="2020-09-29T09:25:47Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:367" namespace=velero
time="2020-09-29T09:25:50Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:396"
time="2020-09-29T09:25:50Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:430"
time="2020-09-29T09:25:50Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:437"
time="2020-09-29T09:25:50Z" level=warning msg="Velero restic daemonset not found; restic backups/restores will not work until it's created" logSource="pkg/cmd/server/server.go:509"
time="2020-09-29T09:25:50Z" level=info msg="Starting controllers" logSource="pkg/cmd/server/server.go:585"
time="2020-09-29T09:25:50Z" level=info msg="Starting metric server at address [:8085]" logSource="pkg/cmd/server/server.go:593"
time="2020-09-29T09:25:50Z" level=info msg="Backup sync period is 1m0s" logSource="pkg/controller/backup_sync_controller.go:75"
time="2020-09-29T09:25:50Z" level=info msg="Waiting for informer caches to sync" logSource="pkg/cmd/server/server.go:853"
time="2020-09-29T09:25:52Z" level=info msg="Done waiting for informer caches to sync" logSource="pkg/cmd/server/server.go:856"
time="2020-09-29T09:25:52Z" level=info msg="Informer cache synced" informer="*v1.PodVolumeBackup" logSource="pkg/cmd/server/server.go:867"
time="2020-09-29T09:25:52Z" level=info msg="Informer cache synced" informer="*v1.DeleteBackupRequest" logSource="pkg/cmd/server/server.go:867"
time="2020-09-29T09:25:52Z" level=info msg="Informer cache synced" informer="*v1.Backup" logSource="pkg/cmd/server/server.go:867"
time="2020-09-29T09:25:52Z" level=info msg="Informer cache synced" informer="*v1.Schedule" logSource="pkg/cmd/server/server.go:867"
time="2020-09-29T09:25:52Z" level=info msg="Informer cache synced" informer="*v1.ResticRepository" logSource="pkg/cmd/server/server.go:867"
time="2020-09-29T09:25:52Z" level=info msg="Informer cache synced" informer="*v1.ServerStatusRequest" logSource="pkg/cmd/server/server.go:867"
time="2020-09-29T09:25:52Z" level=info msg="Informer cache synced" informer="*v1.VolumeSnapshotLocation" logSource="pkg/cmd/server/server.go:867"
time="2020-09-29T09:25:52Z" level=info msg="Informer cache synced" informer="*v1.BackupStorageLocation" logSource="pkg/cmd/server/server.go:867"
time="2020-09-29T09:25:52Z" level=info msg="Informer cache synced" informer="*v1.DownloadRequest" logSource="pkg/cmd/server/server.go:867"
time="2020-09-29T09:25:52Z" level=info msg="Informer cache synced" informer="*v1.Restore" logSource="pkg/cmd/server/server.go:867"
time="2020-09-29T09:25:52Z" level=info msg="Server started successfully" logSource="pkg/cmd/server/server.go:881"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=serverstatusrequest logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=restic-repository logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=backup-sync logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=schedule logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=gc-controller logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=restore logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=gc-controller logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=restore logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=backup-deletion logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=backup logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Starting controller" controller=downloadrequest logSource="pkg/controller/generic_controller.go:76"
time="2020-09-29T09:25:52Z" level=info msg="Checking for expired DeleteBackupRequests" controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:551"
time="2020-09-29T09:25:52Z" level=info msg="Done checking for expired DeleteBackupRequests" controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:579"

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

kubectl logs deployment/velero -n velero
velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
velero backup logs <backupname>
velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
velero restore logs <restorename>

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

Velero version (use velero version): v1.4.2_vmware.1
Velero features (use velero client config get features):
Kubernetes version (use kubectl version):v1.18.8
Kubernetes installer & version:
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

👍 for "I would like to see this bug fixed as soon as possible"
👎 for "There are more important bugs to focus on right now"

zubron · 2020-09-29T22:00:26Z

zubron
Sep 29, 2020

Hi @liyan-su. I'd like to check - are you able to perform any backups without using --exclude-namespaces? The backup that you are trying to produce would likely be very large given the number of resources, so may take some time for Velero to process. I'm trying to determine if this is actually an issue with the exclude namespaces behaviour or is just due to the size of the backup. Thanks!

0 replies

liyan-su · 2020-09-30T07:26:11Z

liyan-su
Sep 30, 2020
Author

hi @zubron
I try to backup failed for many times beacause of use "exclude-namespaces", after that I use --include-namepaces with 2500 namespace string, it is complete and success.
Thanks!

0 replies

zubron · 2020-09-30T18:27:34Z

zubron
Sep 30, 2020

Thanks for the info, @liyan-su! Did the backups using "exclude-namespaces" explicitly fail, or are they not completing? (Your original message said they were stuck in progress for a long time).

0 replies

liyan-su · 2020-10-14T02:16:30Z

liyan-su
Oct 14, 2020
Author

hi @zubron
hi, backups using "exclude-namespaces" stuck in progress, and we use "kubectl logs velero-6c8b8dc5db-sj42j -n velero" to check log (above has attached) but not any backup information.

0 replies

mozz-lx · 2020-12-08T00:00:32Z

mozz-lx
Dec 8, 2020

I got the same behavior on my clusters. I detected that when using the --exclude-namespaces option the velero pod crashes and it is restarted then the backup gets stuck in "InProgress" but unfortunately the log does not display any error.

NAME                  STATUS       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
full-backup-exclude   InProgress   0        0          2020-12-07 23:29:49 +0000 UTC   29d       default            <none>
full-backup-include   Completed    0        0          2020-12-07 23:53:09 +0000 UTC   29d       default            <none>

Also, I detected a similar behavior with other cluster trying to backup some namespaces individually. The velero pod crashes when try to list the events on that namespace and there are 50522.

command: velero create backup ott --include-namespaces ott
Log

time="2020-12-08T15:46:16Z" level=info msg="Setting up backup log" backup=velero/ott controller=backup logSource="pkg/controller/backup_controller.go:512"
time="2020-12-08T15:46:16Z" level=info msg="Setting up backup temp file" backup=velero/ott logSource="pkg/controller/backup_controller.go:534"
time="2020-12-08T15:46:16Z" level=info msg="Setting up plugin manager" backup=velero/ott logSource="pkg/controller/backup_controller.go:541"
time="2020-12-08T15:46:16Z" level=info msg="Getting backup item actions" backup=velero/ott logSource="pkg/controller/backup_controller.go:545"
time="2020-12-08T15:46:16Z" level=info msg="Setting up backup store to check for backup existence" backup=velero/ott logSource="pkg/controller/backup_controller.go:551"
time="2020-12-08T15:46:16Z" level=info msg="Writing backup version file" backup=velero/ott logSource="pkg/backup/backup.go:236"
time="2020-12-08T15:46:16Z" level=info msg="Including namespaces: ott" backup=velero/ott logSource="pkg/backup/backup.go:242"
time="2020-12-08T15:46:16Z" level=info msg="Excluding namespaces: <none>" backup=velero/ott logSource="pkg/backup/backup.go:243"
time="2020-12-08T15:46:16Z" level=info msg="Including resources: *" backup=velero/ott logSource="pkg/backup/backup.go:246"
time="2020-12-08T15:46:16Z" level=info msg="Excluding resources: <none>" backup=velero/ott logSource="pkg/backup/backup.go:247"
time="2020-12-08T15:46:16Z" level=info msg="Backing up all pod volumes using restic: false" backup=velero/ott logSource="pkg/backup/backup.go:248"
time="2020-12-08T15:46:16Z" level=info msg="Getting items for group" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:76"
time="2020-12-08T15:46:16Z" level=info msg="Getting items for resource" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:165" resource=pods
time="2020-12-08T15:46:16Z" level=info msg="Listing items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:291" namespace=ott resource=pods
time="2020-12-08T15:46:16Z" level=info msg="Retrieved 496 items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:297" namespace=ott resource=pods
time="2020-12-08T15:46:16Z" level=info msg="Getting items for resource" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:165" resource=persistentvolumeclaims
time="2020-12-08T15:46:16Z" level=info msg="Listing items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:291" namespace=ott resource=persistentvolumeclaims
time="2020-12-08T15:46:16Z" level=info msg="Retrieved 1 items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:297" namespace=ott resource=persistentvolumeclaims
time="2020-12-08T15:46:16Z" level=info msg="Getting items for resource" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:165" resource=persistentvolumes
time="2020-12-08T15:46:16Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:192" resource=persistentvolumes
time="2020-12-08T15:46:16Z" level=info msg="Getting items for resource" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:165" resource=limitranges
time="2020-12-08T15:46:16Z" level=info msg="Listing items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:291" namespace=ott resource=limitranges
time="2020-12-08T15:46:16Z" level=info msg="Retrieved 0 items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:297" namespace=ott resource=limitranges
time="2020-12-08T15:46:16Z" level=info msg="Getting items for resource" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:165" resource=replicationcontrollers
time="2020-12-08T15:46:16Z" level=info msg="Listing items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:291" namespace=ott resource=replicationcontrollers
time="2020-12-08T15:46:16Z" level=info msg="Retrieved 0 items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:297" namespace=ott resource=replicationcontrollers
time="2020-12-08T15:46:16Z" level=info msg="Getting items for resource" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:165" resource=secrets
time="2020-12-08T15:46:16Z" level=info msg="Listing items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:291" namespace=ott resource=secrets
time="2020-12-08T15:46:16Z" level=info msg="Retrieved 2 items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:297" namespace=ott resource=secrets
time="2020-12-08T15:46:16Z" level=info msg="Getting items for resource" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:165" resource=resourcequotas
time="2020-12-08T15:46:16Z" level=info msg="Listing items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:291" namespace=ott resource=resourcequotas
time="2020-12-08T15:46:16Z" level=info msg="Retrieved 0 items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:297" namespace=ott resource=resourcequotas
time="2020-12-08T15:46:16Z" level=info msg="Getting items for resource" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:165" resource=events
time="2020-12-08T15:46:16Z" level=info msg="Listing items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:291" namespace=ott resource=events

the second namespace backup crashes in the same way when it is trying to list the configmaps and I have 1790 configmaps.

adding some extra information. I have enabled the debug mode on my velero installation. here is the log before the pod restarted.

time="2020-12-08T16:56:34Z" level=info msg="Listing items" backup=velero/ott group=v1 logSource="pkg/backup/item_collector.go:291" namespace=ott resource=events
time="2020-12-08T16:56:36Z" level=debug msg="plugin process exited" backup=velero/ott cmd=/velero error="signal: killed" logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/velero pid=173
time="2020-12-08T16:56:36Z" level=debug msg="plugin process exited" backup=velero/ott cmd=/plugins/velero-plugin-for-aws error="signal: killed" logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velero-plugin-for-aws pid=185

0 replies

mozz-lx · 2020-12-08T20:06:43Z

mozz-lx
Dec 8, 2020

hi @liyan-su

I found that my issue was the limit of memory configured for the velero pod, after increasing the memory all worked for me. how is your pod memory configured?

Regards.

0 replies

liyan-su · 2020-12-09T03:04:45Z

liyan-su
Dec 9, 2020
Author

hi @mozz-lx
thanks for your reply, I cann't remember clearly, may be cpu:2, memory:2G, after exclude-namespaces pending, I use include-namespace all worked for me.

0 replies

dsu-igeek · 2021-02-01T21:27:09Z

dsu-igeek
Feb 1, 2021

I've been trying to reproduce this with a test case but I can't so far. I'm able to backup/restore 5K namespaces. @liyan-su Would you post the YAML that you are using to create the namespaces please?

0 replies

FengLovesBach · 2024-11-21T17:40:00Z

FengLovesBach
Nov 21, 2024

@liyan-su: hi, has the issue been resolved. I am considering using exclude_namespaces, but if the issue hasn't been solved then I'd rather not use it. Thank you.

1 reply

blackpiglet Nov 22, 2024
Maintainer

This the exclude_namespaces function always works.
This discussion is related to performance, not the filtering function.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

velero use "exclude-namespaces" to backup not worked #3259

{{title}}

Replies: 9 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

velero use "exclude-namespaces" to backup not worked #3259

liyan-su Sep 29, 2020

Replies: 9 comments · 1 reply

zubron Sep 29, 2020

liyan-su Sep 30, 2020 Author

zubron Sep 30, 2020

liyan-su Oct 14, 2020 Author

mozz-lx Dec 8, 2020

mozz-lx Dec 8, 2020

liyan-su Dec 9, 2020 Author

dsu-igeek Feb 1, 2021

FengLovesBach Nov 21, 2024

blackpiglet Nov 22, 2024 Maintainer

liyan-su
Sep 29, 2020

Replies: 9 comments 1 reply

zubron
Sep 29, 2020

liyan-su
Sep 30, 2020
Author

zubron
Sep 30, 2020

liyan-su
Oct 14, 2020
Author

mozz-lx
Dec 8, 2020

mozz-lx
Dec 8, 2020

liyan-su
Dec 9, 2020
Author

dsu-igeek
Feb 1, 2021

FengLovesBach
Nov 21, 2024

blackpiglet Nov 22, 2024
Maintainer