Solution to issue of "files changing during the backup process"

[mariomadproductions]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Question

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.4.0

Operating system (distribution) and version.

Linux Mint 20.3 Cinnamon

Hardware / network configuration, and filesystems used.

PC with ext4 SSD and ext4 USB backup HDD

How much data is handled by borg?

PC and backup disks are 2TB and 4TB respectively

Full borg commandline that lead to the problem (leave away excludes and passwords)

N/A

Describe the problem you're observing.

The quick start guide has a section called "Important note about files changing during the backup process" that seems to say that you should avoid running any programs that might change files if you want to have files in a consistent state (I assume you would want this?).

However operating systems might be changing files a lot of the time.

What is a sensible solution to this issue?

Running borgbackup from a linux installation on a USB device (same one used for backup?). However I'm not sure how that ties into "Important note about permissions" - i.e. can I just run it using sudo on the USB linux install, and its okay as long as I always do that? Also, I guess backing up the USB install won't really be important, as I'll have the passphrase/keys stored on multiple mediums anyway.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Include any warning/errors/backtraces from the system logs

N/A

[SpiritInAShell]

please counter-check everything yourself as I have been wrong often before. Take this as hints for your research:

What is a sensible solution to this issue?

I try to default my system's base filesystem to something (like) btrfs for snapshots.

However operating systems might be changing files a lot of the time.

Evaluate your borg create results. I backup my Linux Mint and the files reported as changed during backup seem very unimportant to me (like apt caches etc) (no experience in running restored systems... yet)

topic 1

IMO "Important note about permissions" is mostly about this:

1. you run borg create as userA
2. you run sudo -E borg create
3. you run borg create as userA and borg throws errors

• after step 2, the relevant files in /home/userA/{.config,.local,.cache}/borg will be "contaminated" with UID/GID from other user (root) (on creation/update of files)
• not an advice, only a report:
  • sudo chown -R $USER:%USER /home/userA/{.config,.local,.cache}/borg (that is in fish shell) may fix the problem in the home dir.
  • chown -R $USER:$USER $BORG_REPO may fix the problems in the repo

To avoid at least the latter problem, I can set BORG_REPO to something like ssh://userA@localhost/... so that accessing the repo will be done with my user's UID/GID.

• sudo -E to pass shell variables like $BORG_REPO to root shell
• when using keyfile and not repokey, you may need to copy the keyfile from ~/.config/borg/keys/ to root's home dir

topic 2 about backing up from a live USB distro or alike

TL;DR:

• when backing up from a USB/CD/live distro, for borg create and borg extract consider using --numeric-id to store/restore the INT value that the filesystem has stored.
• I believe borg seems to store everything available to the executing user automatically (unlike other archivers like GNU tar where you have to explicitely specify --acls etc)

You should be especially considering this: https://borgbackup.readthedocs.io/en/stable/quickstart.html#restoring-a-backup and https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html as they describe closely related problems

All AFAIK, all simplfied:

• borg create gathers every stat of a file it knows and saves it, that contains info about UID/GID stored either as STRING or as INT=numerical (right now can't find a source to quote)

• also flags like acls, xattrs (unless you use borg create --no{acl,xattrs,[...]})

• unlike borg create, borg extract requires more privileges (root) to apply many of those stored properties that are not user accessible.

• common filesystems do not store UID/GID as a string,

• a live system usually does not know the values of /etc/{passwd,group} from the "productive system". (few software like rsync have options for mapping UID/GID to their string representation)

• therefor the UID/GID of userA will be unknown to the live system

• therefor also a system related user from the live system like cdrom, sudo, dialout may have totally different numerical UID and GID

• this may be also true for chroot environments (see warning here: https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html#sshfs)

• therefor when backing up from a USB/CD/live distro, for borg create and borg extract consider using --numeric-id to store/restore the INT value that the filesystem has stored.

[mariomadproductions]

Evaluate your borg create results. I backup my Linux Mint and the files reported as changed during backup seem very unimportant to me (like apt caches etc)

I've tried this out now, and the printout doesn't list any changed files. Is there some argument I needed to use to make it do so? Or does it display that info default, and there happened to have been no changed files during the process? I did exclude /tmp and /run etc, so maybe stuff within /home/.temp just isn't relevant when you aren't running an application?

[mariomadproductions]

Thanks, that helps a lot. I'll look at the borg create results. But if it looks not so clear cut (which I guess it might do) if the "changed during backup" files will cause an issue, then I'll try it out with --numeric-id via a separate OS install - doing that looks reasonably straightforward. I guess the reason borg uses names instead of IDs by default is just so the backed up data is more portable/self-explanatory?

P.S. I wonder if doing a full disk image backup could let you skip the UID/GID issue entirely? Although maybe that is a lot slower than the normal backup mode?

[SpiritInAShell]

I guess the reason borg uses names instead of IDs by default is just so the backed up data is more portable/self-explanatory?

It appears to be the "natural" thing to do on a *NIX system, just like (GNU) tar also does.

In https://borgbackup.readthedocs.io/en/stable/quickstart.html#restoring-a-backup it is suggested:

For restoring, you usually want to work on the same machine as the same user that was also used to create the backups of the wanted files

I guess it depends very much on your situation:
a scenario:

• backup from /home/userA @ machineA with UID=1001, GID=1001
• the same person has account for userA @ machineB with UID=1009, GID=1009
• now you want to restore machineA://home/userA on machineB://home/userA
• here you would not want to use numerical IDs as the restore now would have the UID of either a non-existing or a totally different user.

P.S. I wonder if doing a full disk image backup could let you skip the UID/GID issue entirely? Although maybe that is a lot slower than the normal backup mode?

You might just shift the problem onto another level: if you restore the system, would /etc/passwd and /etc/group be part of the same backup and restored together? If yes, I guess, you would restore to the identical state.

Eg if you only restored the image of /home/ to a system with different mapping of userA to UID=xxx, you would have the same problems over again.

Considering that you are backing up a machine (like / etc), my simplified rules would be:

a) backing up the running system: do not use --numerical-ids but also backup/write down /etc/passwd and /etc/group. Read those files in case you need to adjust UID/GID of extracted files manually.

b) when backing up from a USB/CD/live boot, use --numerical-ids and especially backup those 2 files.

c) when restoring to the target system, have the users/groups from the previous system already added to the target system! That way borg will know how to map names to UID/GID numericals (not sure about this!)

Considering that you backup and restore a single user's home dir

... where all files are only his:

back it up either way, and run borg extract as that specific user. All files will have only his UID/GID.

[SpiritInAShell]

(sorry for all the rewrites and confusion... the lack of "theory of mind" makes it very hard to make precise statements that are true for others)

[ThomasWaldmann]

Converted this from issue to discussion.