Backup fails to complete with no useful errors in log

[tailsec]

Steps to reproduce

1. Attempt to run the backup through the management interface

Expected behavior

A backup should be created

Actual behavior

Borg appears to have created the backup but then deletes it saying it failed without providing more information

Host OS

Windows 11

Nextcloud AIO version

7.12.1

Current channel

Latest

Other valuable info

borgbackup.log

[szaimen]

Hi, if you run the backup anorher time, does the same happen again?

[tailsec]

That output was from the second time attempting to create a backup.

Since then I've added a new hard drive to my server and migrated the data dir per this article

Unfortunately attempting to create the backup again generated this error:

Performing backup...
Did not find borg.config file in the mastercontainer volume.
Cannot create a backup as this is wrong.

Looks like it might be related to this issue but I can't see what the resolution was to be able to verify it.

[szaimen]

I found your issue:

ud_aio_volumes/nextcloud_aio_nextcloud_data/System Volume Information: dir_open: [Errno 13] Permission denied: 'System Volume Information'

[tailsec]

I had to kill the borg backup container and it's completely broken:

Performing backup...
Starting the backup...
Inconsistency detected. Please run "borg check /mnt/borgbackup/borg" - although likely this is "beyond repair".
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5343, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5263, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 176, in wrapper
    kwargs['manifest'], kwargs['key'] = Manifest.load(repository, compatibility)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/helpers/manifest.py", line 184, in load
    cdata = repository.get(cls.MANIFEST_ID)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1207, in get
    self.index = self.open_index(self.get_transaction_id())
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/repository.py", line 415, in get_transaction_id
    self.check_transaction()
  File "/usr/lib/python3.11/site-packages/borg/repository.py", line 404, in check_transaction
    raise self.CheckNeeded(msg)
borg.repository.Repository.CheckNeeded: Inconsistency detected. Please run "borg check /mnt/borgbackup/borg" - although likely this is "beyond repair".

Platform: Linux 448e4ac168b9 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64
Linux: Unknown Linux  
Borg: 1.2.7  Python: CPython 3.11.6 msgpack: 1.0.6 fuse: llfuse 1.5.0 [pyfuse3,llfuse]
PID: 36  CWD: /
sys.argv: ['/usr/bin/borg', 'create', '-v', '--stats', '--compression', 'auto,zstd', '--exclude-caches', '--exclude', '/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/nextcloud.log*', '--exclude', '/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/audit.log', '/mnt/borgbackup/borg::20240222_074219-nextcloud-aio', '/nextcloud_aio_volumes/']
SSH_ORIGINAL_COMMAND: None

Deleting the failed backup archive...
Inconsistency detected. Please run "borg check /mnt/borgbackup/borg" - although likely this is "beyond repair".
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5343, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5263, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 189, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 1195, in do_delete
    return self._delete_archives(args, repository)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 1203, in _delete_archives
    manifest, key = Manifest.load(repository, (Manifest.Operation.DELETE,))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/helpers/manifest.py", line 184, in load
    cdata = repository.get(cls.MANIFEST_ID)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1207, in get
    self.index = self.open_index(self.get_transaction_id())
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg/repository.py", line 415, in get_transaction_id
    self.check_transaction()
  File "/usr/lib/python3.11/site-packages/borg/repository.py", line 404, in check_transaction
    raise self.CheckNeeded(msg)
borg.repository.Repository.CheckNeeded: Inconsistency detected. Please run "borg check /mnt/borgbackup/borg" - although likely this is "beyond repair".

Platform: Linux 448e4ac168b9 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64
Linux: Unknown Linux  
Borg: 1.2.7  Python: CPython 3.11.6 msgpack: 1.0.6 fuse: llfuse 1.5.0 [pyfuse3,llfuse]
PID: 37  CWD: /
sys.argv: ['/usr/bin/borg', 'delete', '--stats', '/mnt/borgbackup/borg::20240222_074219-nextcloud-aio']
SSH_ORIGINAL_COMMAND: None

Backup failed!
You might want to check the backup integrity via the AIO interface.

I ran the backup integrity check and the repair option (borg check repair), deleted the container and image and re-ran the backup, and the above error can't be resolved.

Any advice on how to recover from this?

[szaimen]

Any advice on how to recover from this?

You need to delete the backup archive, possibly the backup_cache docker volume and remove the borg.config file from the mastercontainer volume which should allow you to start over.

[tailsec]

I followed those steps and re-attempted to create the backup, but again - it ran for over 8 hours and created over 500GB in backup files and was still going.

The same file permission error appeared as soon as I initiated the backup process, but as before it didn't seem to cause any issues and continued compressing files to the borg\data\... location.

Looking at the logs it appears that it's recursively backing itself up - i.e. it's backing up the backup directory as well as the nextcloud files.

e.g. you can see the backup files in the borg data directory being rolled into the backup as though it's a nextcloud file - this goes in an infinite loop causing the backup to keep growing and never complete:

 O 56.54 GB C 56.10 GB D 12874 N nextcloud_aio_vo...1/$RZR0IYC/data/0/10
 O 56.54 GB C 56.10 GB D 12874 N nextcloud_aio_vo...1/$RZR0IYC/data/0/10
 O 56.56 GB C 56.11 GB D 12874 N nextcloud_aio_vo...1/$RZR0IYC/data/0/10
 O 56.57 GB C 56.12 GB D 12874 N nextcloud_aio_vo...1/$RZR0IYC/data/0/10
 O 56.58 GB C 56.13 GB D 12874 N nextcloud_aio_vo...1/$RZR0IYC/data/0/10
 O 56.59 GB C 56.14 GB D 12874 N nextcloud_aio_vo...1/$RZR0IYC/data/0/10
 O 56.59 GB C 56.15 GB D 12874 N nextcloud_aio_vo...1/$RZR0IYC/data/0/10
 O 56.60 GB C 56.16 GB D 12874 N nextcloud_aio_vo...1/$RZR0IYC/data/0/10
 O 56.61 GB C 56.16 GB D 12875 N nextcloud_aio_vo...1/$RZR0IYC/data/0/12
 O 56.62 GB C 56.17 GB D 12875 N nextcloud_aio_vo...1/$RZR0IYC/data/0/12
 O 56.63 GB C 56.18 GB D 12875 N nextcloud_aio_vo...1/$RZR0IYC/data/0/12
 O 56.64 GB C 56.20 GB D 12875 N nextcloud_aio_vo...1/$RZR0IYC/data/0/12
 O 56.65 GB C 56.21 GB D 12875 N nextcloud_aio_vo...1/$RZR0IYC/data/0/12
 O 56.66 GB C 56.21 GB D 12875 N nextcloud_aio_vo...1/$RZR0IYC/data/0/12
 O 56.67 GB C 56.22 GB D 12875 N nextcloud_aio_vo...1/$RZR0IYC/data/0/12

I haven't changed any settings that I can think of which would cause this - @szaimen have you encountered this in previous issues and been able to resolve it?

[szaimen]

Can you post the output of sudo docker inspect nextcloud-aio-borgbackup here?

[tailsec]

Sure thing, here you go:

[
    {
        "Id": "50fc7ee4c989f261ea333aaac6df7a9c46487da4efeeef5c35c57601c060bae8",
        "Created": "2024-02-28T07:06:58.10320643Z",
        "Path": "/start.sh",
        "Args": [],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 143,
            "Error": "",
            "StartedAt": "2024-02-28T07:06:58.549092264Z",
            "FinishedAt": "2024-02-28T08:16:41.29950544Z"
        },
        "Image": "sha256:36f6fb0efd1d2f1156f6ca8338a21023e1e45f40d9ef76cf20ceab50af2d5439",
        "ResolvConfPath": "/var/lib/docker/containers/50fc7ee4c989f261ea333aaac6df7a9c46487da4efeeef5c35c57601c060bae8/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/50fc7ee4c989f261ea333aaac6df7a9c46487da4efeeef5c35c57601c060bae8/hostname",
        "HostsPath": "/var/lib/docker/containers/50fc7ee4c989f261ea333aaac6df7a9c46487da4efeeef5c35c57601c060bae8/hosts",
        "LogPath": "/var/lib/docker/containers/50fc7ee4c989f261ea333aaac6df7a9c46487da4efeeef5c35c57601c060bae8/50fc7ee4c989f261ea333aaac6df7a9c46487da4efeeef5c35c57601c060bae8-json.log",
        "Name": "/nextcloud-aio-borgbackup",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "nextcloud_aio_backup_cache:/root:rw",
                "/run/desktop/mnt/host/x:/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data:rw",
                "nextcloud_aio_mastercontainer:/nextcloud_aio_volumes/nextcloud_aio_mastercontainer:rw",
                "/run/desktop/mnt/host/x/nc_backup:/mnt/borgbackup:rw",
                "nextcloud_aio_elasticsearch:/nextcloud_aio_volumes/nextcloud_aio_elasticsearch:rw",
                "nextcloud_aio_redis:/mnt/redis:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "nextcloud-aio",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": [
                "SYS_ADMIN"
            ],
            "CapDrop": [
                "NET_RAW"
            ],
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": true,
            "SecurityOpt": [
                "apparmor:unconfined",
                "label:disable"
            ],
            "Tmpfs": {
                "/nextcloud_aio_volumes": "",
                "/tmp": ""
            },
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [
                {
                    "PathOnHost": "/dev/fuse",
                    "PathInContainer": "/dev/fuse",
                    "CgroupPermissions": "rwm"
                }
            ],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "nextcloud_aio_nextcloud",
                    "Target": "/nextcloud_aio_volumes/nextcloud_aio_nextcloud"
                },
                {
                    "Type": "volume",
                    "Source": "nextcloud_aio_apache",
                    "Target": "/nextcloud_aio_volumes/nextcloud_aio_apache"
                },
                {
                    "Type": "volume",
                    "Source": "nextcloud_aio_database",
                    "Target": "/nextcloud_aio_volumes/nextcloud_aio_database"
                },
                {
                    "Type": "volume",
                    "Source": "nextcloud_aio_database_dump",
                    "Target": "/nextcloud_aio_volumes/nextcloud_aio_database_dump"
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ],
            "Init": true
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/89500001448e4d905a783fd5127b91799ed042229c6159135d0a923b707f6a8b-init/diff:/var/lib/docker/overlay2/3f6eb0a40f87f2af46404ab95af1c175ba0f7a4583e6391f1c35319845c93c17/diff:/var/lib/docker/overlay2/4634b3fc3e05198cd1823f5777e513bd795e37aaca9d737d95fc124fe74c5303/diff:/var/lib/docker/overlay2/1c1233a2425901cdaa1eeea34b3608623e1d3a00bafffd2cf82d1c0128881284/diff",
                "MergedDir": "/var/lib/docker/overlay2/89500001448e4d905a783fd5127b91799ed042229c6159135d0a923b707f6a8b/merged",
                "UpperDir": "/var/lib/docker/overlay2/89500001448e4d905a783fd5127b91799ed042229c6159135d0a923b707f6a8b/diff",
                "WorkDir": "/var/lib/docker/overlay2/89500001448e4d905a783fd5127b91799ed042229c6159135d0a923b707f6a8b/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "nextcloud_aio_backup_cache",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_backup_cache/_data",
                "Destination": "/root",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_mastercontainer",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_mastercontainer",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/run/desktop/mnt/host/x/nc_backup",
                "Destination": "/mnt/borgbackup",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_database_dump",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_database_dump/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_database_dump",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/run/desktop/mnt/host/x",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_elasticsearch",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_elasticsearch/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_elasticsearch",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_redis",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_redis/_data",
                "Destination": "/mnt/redis",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_nextcloud",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_nextcloud/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_nextcloud",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_apache",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_apache/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_apache",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_database",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_database/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_database",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "50fc7ee4c989",
            "Domainname": "",
            "User": "root",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "BORG_PASSWORD=4410bafe2c4ccf3a0cdde9841b0fa8bb2c3257107d6d8541",
                "BORG_MODE=backup",
                "SELECTED_RESTORE_TIME=",
                "BACKUP_RESTORE_PASSWORD=",
                "ADDITIONAL_DIRECTORIES_BACKUP=",
                "BORGBACKUP_HOST_LOCATION=/run/desktop/mnt/host/x/nc_backup",
                "BORG_HOST_ID=nextcloud-aio-borgbackup",
                "BORG_RETENTION_POLICY=--keep-within=7d --keep-weekly=4 --keep-monthly=6",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Image": "nextcloud/aio-borgbackup:latest",
            "Volumes": {
                "/root": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "/start.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.centurylinklabs.watchtower.enable": "false"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "bc5043c6ff7e9a48e788df989e9812eefb6a49331ec9a72b7739c4139b95c6ca",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/bc5043c6ff7e",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "nextcloud-aio": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "50fc7ee4c989"
                    ],
                    "MacAddress": "",
                    "NetworkID": "94058892b2cb539f471484a9a515293e0fdec0c69057d86466327ecbad8e93a8",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DriverOpts": null
                }
            }
        }
    }
]

[szaimen]

        {
            "Type": "bind",
            "Source": "/run/desktop/mnt/host/x/nc_backup",
            "Destination": "/mnt/borgbackup",
            "Mode": "rw",
            "RW": true,
            "Propagation": "rprivate"
        },

...

        {
            "Type": "bind",
            "Source": "/run/desktop/mnt/host/x",
            "Destination": "/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data",
            "Mode": "rw",
            "RW": true,
            "Propagation": "rprivate"
        },

Apparently you set NEXTCLOUD_DATADIR to /run/desktop/mnt/host/x and the backup dir to /run/desktop/mnt/host/x/nc_backup to a subdir of it. This is obviously not supported ad NEXTCLOUD_DATADIR is of course included in the backup and in your case the backup included in the backup as wel if it a subdir of the datadir...

[tailsec]

Thanks for pointing that out - at first I thought that was the cause of the issue so I changed the backup folder to a separate drive altogether, but the issue still occurs with the borg data folders still being backed up recursively:

B O 141.58 GB C 141.14 GB D 41797 N nextcloud_aio_v...T923/borg/data/0/3
B O 141.59 GB C 141.15 GB D 41797 N nextcloud_aio_v...T923/borg/data/0/3
B O 141.60 GB C 141.15 GB D 41797 N nextcloud_aio_v...T923/borg/data/0/3
B O 141.61 GB C 141.16 GB D 41797 N nextcloud_aio_v...T923/borg/data/0/3
B O 141.62 GB C 141.17 GB D 41797 N nextcloud_aio_v...T923/borg/data/0/3
B O 141.62 GB C 141.18 GB D 41798 N nextcloud_aio_v...T923/borg/data/0/4
B O 141.63 GB C 141.19 GB D 41798 N nextcloud_aio_v...T923/borg/data/0/4
B O 141.65 GB C 141.20 GB D 41798 N nextcloud_aio_v...T923/borg/data/0/4
B O 141.65 GB C 141.21 GB D 41798 N nextcloud_aio_v...T923/borg/data/0/4

I also made sure the old borg backup location on the x drive was deleted to make sure the old files weren't backed up when I ran again.

For reference, the directories are now on separate drives altogether:

            {
                "Type": "bind",
                "Source": "/run/desktop/mnt/host/q/nc_backup",
                "Destination": "/mnt/borgbackup",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }

...

            {
                "Type": "bind",
                "Source": "/run/desktop/mnt/host/x",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }

The full dump of the docker inspect command:

[
    {
        "Id": "b1d3e0828b7c7d51e2cf75748fb52a197852fd94464385f7a8a48ece0db80ce1",
        "Created": "2024-03-02T22:14:32.586033289Z",
        "Path": "/start.sh",
        "Args": [],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 143,
            "Error": "",
            "StartedAt": "2024-03-02T22:14:33.052912015Z",
            "FinishedAt": "2024-03-02T23:59:09.903881099Z"
        },
        "Image": "sha256:e0fc05eb61d5e8548cb7dc6c43b7ebb7ee6c5f8f563f72f58e6af3ee1eb5406f",
        "ResolvConfPath": "/var/lib/docker/containers/b1d3e0828b7c7d51e2cf75748fb52a197852fd94464385f7a8a48ece0db80ce1/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/b1d3e0828b7c7d51e2cf75748fb52a197852fd94464385f7a8a48ece0db80ce1/hostname",
        "HostsPath": "/var/lib/docker/containers/b1d3e0828b7c7d51e2cf75748fb52a197852fd94464385f7a8a48ece0db80ce1/hosts",
        "LogPath": "/var/lib/docker/containers/b1d3e0828b7c7d51e2cf75748fb52a197852fd94464385f7a8a48ece0db80ce1/b1d3e0828b7c7d51e2cf75748fb52a197852fd94464385f7a8a48ece0db80ce1-json.log",
        "Name": "/nextcloud-aio-borgbackup",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "nextcloud_aio_backup_cache:/root:rw",
                "/run/desktop/mnt/host/x:/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data:rw",
                "nextcloud_aio_mastercontainer:/nextcloud_aio_volumes/nextcloud_aio_mastercontainer:rw",
                "/run/desktop/mnt/host/q/nc_backup:/mnt/borgbackup:rw",
                "nextcloud_aio_elasticsearch:/nextcloud_aio_volumes/nextcloud_aio_elasticsearch:rw",
                "nextcloud_aio_redis:/mnt/redis:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "nextcloud-aio",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": [
                "SYS_ADMIN"
            ],
            "CapDrop": [
                "NET_RAW"
            ],
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": true,
            "SecurityOpt": [
                "apparmor:unconfined",
                "label:disable"
            ],
            "Tmpfs": {
                "/nextcloud_aio_volumes": "",
                "/tmp": ""
            },
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [
                {
                    "PathOnHost": "/dev/fuse",
                    "PathInContainer": "/dev/fuse",
                    "CgroupPermissions": "rwm"
                }
            ],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "nextcloud_aio_nextcloud",
                    "Target": "/nextcloud_aio_volumes/nextcloud_aio_nextcloud"
                },
                {
                    "Type": "volume",
                    "Source": "nextcloud_aio_apache",
                    "Target": "/nextcloud_aio_volumes/nextcloud_aio_apache"
                },
                {
                    "Type": "volume",
                    "Source": "nextcloud_aio_database",
                    "Target": "/nextcloud_aio_volumes/nextcloud_aio_database"
                },
                {
                    "Type": "volume",
                    "Source": "nextcloud_aio_database_dump",
                    "Target": "/nextcloud_aio_volumes/nextcloud_aio_database_dump"
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ],
            "Init": true
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/7de7d4f308bccfd41aa01a7bc9c4db6a8d90d3a74bbaae359076e9127e7eb661-init/diff:/var/lib/docker/overlay2/53800d1aaac699cae3a04c64a14d21d900ad3bb124c367ee5954f3ce0b1c53fe/diff:/var/lib/docker/overlay2/b1c570af671d768bcfaf045bb6376849d6cc606dcfb60f78aaf470152b63af79/diff:/var/lib/docker/overlay2/1c1233a2425901cdaa1eeea34b3608623e1d3a00bafffd2cf82d1c0128881284/diff",
                "MergedDir": "/var/lib/docker/overlay2/7de7d4f308bccfd41aa01a7bc9c4db6a8d90d3a74bbaae359076e9127e7eb661/merged",
                "UpperDir": "/var/lib/docker/overlay2/7de7d4f308bccfd41aa01a7bc9c4db6a8d90d3a74bbaae359076e9127e7eb661/diff",
                "WorkDir": "/var/lib/docker/overlay2/7de7d4f308bccfd41aa01a7bc9c4db6a8d90d3a74bbaae359076e9127e7eb661/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "nextcloud_aio_database",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_database/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_database",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_backup_cache",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_backup_cache/_data",
                "Destination": "/root",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/run/desktop/mnt/host/x",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_elasticsearch",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_elasticsearch/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_elasticsearch",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_nextcloud",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_nextcloud/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_nextcloud",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_database_dump",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_database_dump/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_database_dump",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_mastercontainer",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_mastercontainer",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/run/desktop/mnt/host/q/nc_backup",
                "Destination": "/mnt/borgbackup",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_redis",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_redis/_data",
                "Destination": "/mnt/redis",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_apache",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_apache/_data",
                "Destination": "/nextcloud_aio_volumes/nextcloud_aio_apache",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "b1d3e0828b7c",
            "Domainname": "",
            "User": "root",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "BORG_PASSWORD=4410bafe2c4ccf3a0cdde9841b0fa8bb2c3257107d6d8541",
                "BORG_MODE=backup",
                "SELECTED_RESTORE_TIME=",
                "BACKUP_RESTORE_PASSWORD=",
                "ADDITIONAL_DIRECTORIES_BACKUP=",
                "BORGBACKUP_HOST_LOCATION=/run/desktop/mnt/host/q/nc_backup",
                "BORG_HOST_ID=nextcloud-aio-borgbackup",
                "BORG_RETENTION_POLICY=--keep-within=7d --keep-weekly=4 --keep-monthly=6",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Image": "nextcloud/aio-borgbackup:latest",
            "Volumes": {
                "/root": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "/start.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.centurylinklabs.watchtower.enable": "false"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "61680ad58c87d230279f33c1c2c380aaf3041536e5a8277df7d548868eee8cb1",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/61680ad58c87",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "nextcloud-aio": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "b1d3e0828b7c"
                    ],
                    "MacAddress": "",
                    "NetworkID": "94058892b2cb539f471484a9a515293e0fdec0c69057d86466327ecbad8e93a8",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DriverOpts": null
                }
            }
        }
    }
]

[szaimen]

Can you set NEXTCLOUD_DATADIR to e.g. /run/desktop/mnt/host/x/nc (so a subfolder of the drive and not the root?) this should fix it. However you might need to move the files then first manhally to the subfolder...

[tailsec]

Nice one, that worked!

I don't know why that was the issue though - I attempted to use a subfolder early on when setting it up and it didn't like it so I partitioned a drive and changed the datadir to point to the root so it'd work.

Thanks for your help and sticking it out - I know this was a painful one but really glad to have figured it out in the end. Thanks!

[argoyal]

I understand that changing the data directory works well. But can you also tell if there is a way to exclude the System Volume Information and then attempt the backup? @szaimen

Basically what I am looking at is some sort of BORG_BACKUP_EXCLUDE_DIRECTORIES.