velero backup logs or describe is not working though backup create is working fine

[akki-9922]

What steps did you take and what happened:

I have installed minio in GKE and using PVC to store backup.
Minio is running with cluster IP  type service under minio namesapce.   

After that I have created bucket using mc and its working fine. In the GKE cluster I can connect with minio bucket using s5cmd.
and its working fine.

Now I have installed the velero using belwo command -

velero install \
  --provider aws\
  --bucket velero \
  --backup-location-config region=us-east-1,s3ForcePathStyle="true",s3Url=http://<ClusterIP-Of-minio-svc>:9000 \
  --plugins velero/velero-plugin-for-aws:v1.0.0  \
  --namespace velero \
  --secret-file cloud-credentials-minio

velero gets installed in velero NS and I am able to take backup.
velero backup create pod-ubuntu --include-namespaces default --include-resources pods --wait
Backup request "pod-ubuntu" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.

Backup completed with status: Completed. You may check for more information using the commands `velero backup describe pod-ubuntu` and `velero backup logs pod-ubuntu`.

When I ran backup logs and backup describe, I am getting error so restore is also not working. 

**velero backup logs pod-ubuntu**
An error occurred: Get "http://34.118.238.97:9000/velero/backups/pod-ubuntu/pod-ubuntu-logs.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20241121%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241121T134241Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=b1bfb060e83607565ae368dcbcf12fa2ebdd90baec9e3e61e24360ca920017b5": dial tcp XX.XX.XX.XX:9000: i/o timeout

velero backup describe pod-ubuntu
Backup Volumes:
<error getting backup volume info: Get "http://34.118.238.97:9000/velero/backups/pod-ubuntu/pod-ubuntu-volumeinfo.json.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20241121%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241121T135556Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=4dc2e66215ba28e60be625bc9fcd7700659259558b72f7bd65187742e888e7f2": context deadline exceeded>

But is I do #velero backup get it is working fine.
velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
minio-bkp Completed 0 0 2024-11-21 13:29:58 +0000 UTC 29d default
pod Completed 0 0 2024-11-21 13:35:21 +0000 UTC 29d default
pod-ubuntu Completed 0 0 2024-11-21 13:42:33 +0000 UTC 29d default

What did you expect to happen:*
It should be able to describe and restore the backup.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version):
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[kaovilai]

",s3Url=http://<ClusterIP-Of-minio-svc>:9000

That's an internal network.

Velero cli you execute logs or describe will need to have access to that cluster network.

If s3url is not externally accessible you won't be able to use velero cli directly externally.

You can try kubectl exec -it -n velero deploy/velero -- /velero backup describe <>

Pardon me if typos as currently replying on mobile.

[kaovilai]

Backup create do not rely on client having s3url access. Log and describe does.

#6167 is planned to help you not have to do above kubectl workaround.

[kaovilai]

As this is expected behavior or requirements duplicate of #6167 closing. Feel free to keep commenting to troubleshoot.

[akki-9922]

Got it. Its working now.

Further I installed, velero with below command in my baremetal k8s cluster [NOT A CLOUD K8s]-

velero install
--provider aws
--bucket velero
--backup-location-config region=us-east-1,s3ForcePathStyle="true",s3Url=http://xxxxxxxx:9000
--plugins velero/velero-plugin-for-aws:v1.0.0,velero/velero-plugin-for-csi:latest
--namespace velero
--secret-file minio-acces-key \ --image=velero/velero:v1.11.0
--plugins velero/velero-plugin-for-aws:v1.0.0
--use-node-agent --use-volume-snapshots=true
--snapshot-location-config snapshotClass=rook-cephfs-snapshot -features EnableCSI

And created A volume snapshot class and updated VolumeSnapShotLocation as below -

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: rook-cephfs-snapshot
**driver: rook-ceph.cephfs.csi.ceph.com**
deletionPolicy: Delete

Spec:
Config:
Snapshot Class: rook-cephfs-snapshot
Provider: local

After this my backup is working fine though it was PartialFailed while backing PV and PVc due to Volumesnapshot was not configured.

I guess snapshot looks ok but whatever the data was updated into PVC filesystem, after install application, is not restored and set to default.

I don't see any details guide in Velero doc for baremetal setup. I am unable to fix to ensure no data loss placed in PVC/pv mounted inside stateful set pod .

[kaovilai]

Volumesnapshot was not configured.

I'm assuming you are referring to missing volumesnapshotclass label?

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: test-snapclass
+ labels:
+   velero.io/csi-volumesnapshot-class: "true"

per doc you need to label volumesnapshotclass for velero CSI to pick it up and use it to snapshot volumes.

The Velero CSI plugins, to backup CSI backed PVCs, will choose the VolumeSnapshotClass in the cluster based on the following logic:

1. Default Behavior: You can simply create a VolumeSnapshotClass for a particular driver and put a label on it to indicate that it is the default VolumeSnapshotClass for that driver. For example, if you want to create a VolumeSnapshotClass for the CSI driver disk.csi.cloud.com for taking snapshots of disks created with disk.csi.cloud.com based storage classes, you can create a VolumeSnapshotClass like this:

   <span style="color:#66d9ef">apiVersion</span>: snapshot.storage.k8s.io/v1
   <span style="color:#66d9ef">kind</span>: VolumeSnapshotClass
   <span style="color:#66d9ef">metadata</span>:
     <span style="color:#66d9ef">name</span>: test-snapclass
     <span style="color:#66d9ef">labels</span>:
       <span style="color:#66d9ef">velero.io/csi-volumesnapshot-class</span>: <span style="color:#e6db74">"true"</span>
   <span style="color:#66d9ef">driver</span>: disk.csi.cloud.com
   

   Note: For each driver type, there should only be 1 VolumeSnapshotClass with the label velero.io/csi-volumesnapshot-class: "true".

[kaovilai]

The Volume Snapshot Location (VSL) api is not used for CSI snapshots.

there are several types of velero volume backup methods. Please select one and review that doc's Installing Velero section and backup examples.

• CSI snapshot
• CSI snapshot with data movement
• File system backup
• native plugin snapshots (which is using velero-plugins-for-) and requires setting up VSL.

for bare metal you are limited to first three. There is no need to configure VSL.