Best way to deal with a repo with a *lot* of archives? (Relax-and-Recover Borg interaction)

[dannyob]

Due to a wayward automatic script, I have a borg repo that has not been pruned in a looong time: I have over a thousand archives in it. Almost every action -- prune, delete, etc -- using the defaults are taking hours if not days to complete.

What do folks recommend is the best way to clean this up? At the moment I'm just deleting a few hundred archives at a time, slowly, using borg prune --glob-archives. Should I just run prune, and wait longer? The machine that borg server is running on is not the most powerful, and the archives are on spinning drives, not SSD, so almost anything going on there is going to be pretty slow, but I'm okay to wait if it's going to complete its duties eventually.

Also, should I be running borg compact in between these stints of deleting? Or should I wait until the end of my labours?

Thank you so much for borg, this is the first time I've had any difficulties with it at all -- and it's not borg's fault!

[ThomasWaldmann]

You can slowly adjust borg prune parameters to converge against what you want to have in the end, maybe keeping the runtime down to a few hours per run (by not deleting too many archives in one go).

Be careful when adjusting parameters, first use --dry-run to see their effect.

If you use borg >= 1.2.0, you need to run borg compact to free space, but you can run it less frequently than prune (assuming that you have enough free space). e.g. prune daily, compact weekly.

compact has a --threshold parameter, which can also be adjusted converging to a goal (which could be the default value) to not compact too much at a time (maybe also keep it down to a few hours per run). You could try starting at 50 going down to 10. If the weekly run finishes too quickly (not doing much), lower the threshold.

Once you have fewer archives, some operations (e.g. borg check) will get significantly faster.

[dannyob]

Thank you! Yes, after a slow time deleting the first few hundred archives, everything is getting much quicker. Such a good, resilient program!

[dannyob]

(For those searching, the reason for this happening was that the current stable version of REAR (Relax-and-Recover) (v2.7) would stop entirely if the error code returned from borg was not zero. Because borg gives an error code of 1 for warnings, this would mean that, for example, if borg create warned that a file was changed while backing up, then REAR's subsequent "borg prune" command would not run. The backup would be created, but no subsequent pruning would take place. This behaviour is fixed in the latest REAR which should ship very soon.)

[ThomasWaldmann]

Oh. Good that they have fixed that.