Getting lots of error messages "[Errno 17] file exists"; borg -check didn't help #7727

sicherist · 2023-07-15T19:15:45Z

sicherist
Jul 15, 2023

Hello,

I'm not sure if this is a bug so before opening an issue, I would like to ask here.

All of a sudden I'm getting lots off error messages when running borgbackup on my external USB-HDD, also when pruning. The error messages for "borg create" look like this:

/home/xyz/.config/Signal/attachments.noindex/df/df5072fe64402dc1774995ceae2c3109b734682c93a902e88408d051684a8490: read: [Errno 17] File exists: '/media/xyz/BACKUP1500GB/Backup/xyz/data/3/3875'

It is always the .../3/3875 but different local files.

then this messages appear:

Local ExceptionGB C 0 B D 30127 N home/xyz/.local/lib/python3.9/site-packages/pandas/tests/arrays/boolean/test_logical.py                                                                                                                                                                                                                       
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5213, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5144, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 675, in do_create
    create_inner(archive, cache, fso)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 608, in create_inner
    self._rec_walk(path=path, parent_fd=parent_fd, name=name,
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 823, in _rec_walk
    self._rec_walk(
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 823, in _rec_walk
    self._rec_walk(
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 823, in _rec_walk
    self._rec_walk(
  [Previous line repeated 6 more times]
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 786, in _rec_walk
    status = self._process_any(path=path, parent_fd=parent_fd, name=name, st=st, fso=fso, cache=cache,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 688, in _process_any
    return fso.process_file(path=path, parent_fd=parent_fd, name=name, st=st, cache=cache)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1409, in process_file
    with self.create_helper(path, st, None) as (item, status, hardlinked, hardlink_master):  # no status yet
  File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1329, in create_helper
    self.add_item(item, stats=self.stats)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 602, in add_item
    self.items_buffer.add(item)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 357, in add
    self.flush()
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 385, in flush
    self.chunks.append(self.write_chunk(chunk))
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 401, in write_chunk
    id_, _, _ = self.cache.add_chunk(self.key.id_hash(chunk), chunk, self.stats, wait=False)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 951, in add_chunk
    self.repository.put(id, data, wait=wait)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1230, in put
    segment, offset = self.io.write_put(id, data)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1638, in write_put
    fd = self.get_write_fd(want_new=(id == Manifest.MANIFEST_ID), raise_full=raise_full)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1419, in get_write_fd
    self._write_fd = SyncFile(self.segment_filename(self.segment), binary=True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/platform/base.py", line 162, in __init__
    self.f = open(path, mode=mode)  # python file object
             ^^^^^^^^^^^^^^^^^^^^^
FileExistsError: [Errno 17] File exists: '/media/xyz/BACKUP1500GB/Backup/xyz/data/3/3875'

Platform: Linux xyz 6.2.0-25-generic #25-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 16 17:05:07 UTC 2023 x86_64
Linux: Unknown Linux  
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.3 fuse: pyfuse3 3.2.1 [pyfuse3,llfuse]
PID: 33526  CWD: /home/xyz
sys.argv: ['/usr/bin/borg', 'create', '--progress', '--stats', '/media/xyz/BACKUP1500GB/Backup/xyz::Privat-HomeRest-{now}', '/home/xyz', '--exclude-caches', '--exclude', '/home/xyz/.local/share/Trash', '--exclude', '/home/xyz/.thunderbird', '--exclude', '/home/xyz/.mozilla', '--exclude', '/home/xyz/Dokumente']
SSH_ORIGINAL_COMMAND: None

When I ran
borg check "/media/xyz/BACKUP1500GB/Backup/xyz" --repair
this happened:

This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!

Type 'YES' if you understand this and want to continue: YES
kf.i18n: KCatalog being used without a Q*Application instance. Some translations won't work
xxx-xxx-2021-09-26T23:11:12: home/xyz/Dokumente/xxx/xxx/xxx/xxx/xxx/xxx/xxx: New missing file chunk detected (Byte 0-290915, Chunk 25dc450906bdbcf244fefe09afb2196d087720a1adc16761317d4af08f76b3c1). Replacing with all-zero chunk.
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1852, in add_reference
    self.chunks.incref(id_)
  File "src/borg/hashindex.pyx", line 328, in borg.hashindex.ChunkIndex.incref
KeyError: b'U\x7f\xbf\xb6\xc1\xf1-)\xc5\x04\x1d\xeb#\xbevIcn\xc1\xf7\xf6\xf1\xef#\xfc\xe9?\xf0\xe8\xcd>w'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5213, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5144, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 347, in do_check
    if not args.repo_only and not ArchiveChecker().check(
                                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1647, in check
    self.rebuild_refcounts(archive=archive, first=first, last=last, sort_by=sort_by, glob=glob)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 2058, in rebuild_refcounts
    verify_file_chunks(info.name, item)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1894, in verify_file_chunks
    add_reference(chunk_id, size, csize, cdata)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1857, in add_reference
    self.repository.put(id_, cdata)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1230, in put
    segment, offset = self.io.write_put(id, data)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1638, in write_put
    fd = self.get_write_fd(want_new=(id == Manifest.MANIFEST_ID), raise_full=raise_full)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1419, in get_write_fd
    self._write_fd = SyncFile(self.segment_filename(self.segment), binary=True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/platform/base.py", line 162, in __init__
    self.f = open(path, mode=mode)  # python file object
             ^^^^^^^^^^^^^^^^^^^^^
FileExistsError: [Errno 17] File exists: '/media/xyz/BACKUP1500GB/Backup/xyz/data/3/3875'

Platform: Linux xyz 6.2.0-25-generic #25-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 16 17:05:07 UTC 2023 x86_64
Linux: Unknown Linux  
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.3 fuse: pyfuse3 3.2.1 [pyfuse3,llfuse]
PID: 45068  CWD: /home/xyz
sys.argv: ['/usr/bin/borg', 'check', '/media/xyz/BACKUP1500GB/Backup/xyz', '--repair']
SSH_ORIGINAL_COMMAND: None

Unfortunately I have no idea what I could try to solve the problem. Any help would be apreciated.

Answered by sicherist

Jul 16, 2023

Ok, i think I fixed it. I thought: The OS sees that "3875" exists even though it doesn't seem to exist. Maybe it is a file-system problem? The disk is NTFS-format so I tried "ntfsfix". It found problems but is unable to fix it. So I had to start a Windows-machine to run proper "chkdsk". This seemed to do the trick. Now borg check ran fine and found a lot of problems in the latest archive.

1809 orphaned objects found!                                                                                                                                                                                                   
Deleting 1809 orphaned and 1000 superseded objects...
Finished deleting orphaned/…

View full answer

ThomasWaldmann · 2023-07-15T22:37:54Z

ThomasWaldmann
Jul 15, 2023
Maintainer

kf.i18n: KCatalog being used without a Q*Application instance. Some translations won't work

Why is that showing up in between borg output? borg is not using Qt nor translations.

It looks like that repo is corrupted in a strange way that borg check --repair can't fix.

Can you do:

ls -l /media/xyz/BACKUP1500GB/Backup/xyz  # need to see all
ls -lR /media/xyz/BACKUP1500GB/Backup/xyz/data  # need to see only the last 100 files or so

0 replies

sicherist · 2023-07-16T08:03:18Z

sicherist
Jul 16, 2023
Author

Thank you for taking a look. The "kf.i18n"-Warning is probably unrelated. I think it comes because I get the passphrase from the KDE-Wallet like this

export BORG_PASSCOMMAND='kwallet-query -r xyz kdewallet'

Here is the output of "ls -l /media/xyz/BACKUP1500GB/Backup/xyz"

xyz@xyz:/media/xyz/BACKUP1500GB/Backup/xyz$ ls -l
insgesamt 29624
-rw------- 1 xyz xyz      964 Jul 14 00:22 config
drwxr-xr-x 1 xyz xyz        0 Okt 22  2022 data
-rw------- 1 xyz xyz    13571 Jul 14 00:22 hints.3874
-rw------- 1 xyz xyz 30304298 Jul 14 00:22 index.3874
-rw------- 1 xyz xyz      190 Jul 14 00:22 integrity.3874
-rw------- 1 xyz xyz       16 Jul 15 20:56 nonce
-rwxr-xr-x 1 xyz xyz       73 Nov 17  2019 README

and last lines of "ls -lR /media/xyz/BACKUP1500GB/Backup/xyz/data"

-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2763
-rwxr-xr-x 1 xyz xyz 281278898 Jul 25  2022 2764
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2765
-rwxr-xr-x 1 xyz xyz 224138127 Jul 25  2022 2766
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2767
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2768
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2770
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2772
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2774
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2776
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2778
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2780
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2782
-rwxr-xr-x 1 xyz xyz 524944925 Jul 25  2022 2783
-rwxr-xr-x 1 xyz xyz 524507048 Jul 25  2022 2784
-rwxr-xr-x 1 xyz xyz 527359327 Jul 25  2022 2785
-rwxr-xr-x 1 xyz xyz 524886218 Jul 25  2022 2786
-rwxr-xr-x 1 xyz xyz 527483342 Jul 25  2022 2787
-rwxr-xr-x 1 xyz xyz 368671143 Jul 25  2022 2788
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2789
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2791
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2793
-rwxr-xr-x 1 xyz xyz        17 Jul 25  2022 2795
-rwxr-xr-x 1 xyz xyz 403871305 Okt  8  2022 2796
-rwxr-xr-x 1 xyz xyz 125835245 Okt  8  2022 2800
-rwxr-xr-x 1 xyz xyz 525400913 Okt  8  2022 2804
-rwxr-xr-x 1 xyz xyz 525096709 Okt  8  2022 2805
-rwxr-xr-x 1 xyz xyz 529673897 Okt  8  2022 2806
-rwxr-xr-x 1 xyz xyz 524855340 Okt  8  2022 2807
-rwxr-xr-x 1 xyz xyz 525545577 Okt  8  2022 2808
-rwxr-xr-x 1 xyz xyz 524833698 Okt  8  2022 2809
-rwxr-xr-x 1 xyz xyz 524373902 Okt  8  2022 2810
-rwxr-xr-x 1 xyz xyz 524824906 Okt  8  2022 2812
-rwxr-xr-x 1 xyz xyz 226244836 Okt  8  2022 2813
-rwxr-xr-x 1 xyz xyz 156119965 Okt  8  2022 2821
-rwxr-xr-x 1 xyz xyz  24572670 Okt  8  2022 2831
-rwxr-xr-x 1 xyz xyz    190628 Okt 17  2022 2939
-rwxr-xr-x 1 xyz xyz 524863414 Okt 17  2022 2943
-rwxr-xr-x 1 xyz xyz 422760198 Okt 17  2022 2944
-rwxr-xr-x 1 xyz xyz  10146251 Okt 22  2022 2968
-rwxr-xr-x 1 xyz xyz     44137 Okt 22  2022 2976
-rwxr-xr-x 1 xyz xyz 524417446 Okt 22  2022 2984
-rwxr-xr-x 1 xyz xyz   5057927 Okt 22  2022 2985
-rwxr-xr-x 1 xyz xyz 528293398 Okt 22  2022 2993
-rwxr-xr-x 1 xyz xyz 525717829 Okt 22  2022 2994
-rwxr-xr-x 1 xyz xyz 526894288 Okt 22  2022 2995
-rwxr-xr-x 1 xyz xyz 524399448 Okt 22  2022 2996
-rwxr-xr-x 1 xyz xyz 526856216 Okt 22  2022 2997
-rwxr-xr-x 1 xyz xyz 525834807 Okt 22  2022 2998
-rwxr-xr-x 1 xyz xyz  25357430 Okt 22  2022 2999

./3:
insgesamt 34743736
-rwxr-xr-x 1 xyz xyz 532486710 Okt 22  2022 3015
-rwxr-xr-x 1 xyz xyz 441808323 Okt 22  2022 3016
-rwxr-xr-x 1 xyz xyz 525508536 Mär 29 14:36 3024
-rwxr-xr-x 1 xyz xyz  69908148 Mär 29 14:36 3025
-rwxr-xr-x 1 xyz xyz   2581270 Mär 29 14:36 3033
-rwxr-xr-x 1 xyz xyz 526158919 Mär 29 14:37 3041
-rwxr-xr-x 1 xyz xyz 527390342 Mär 29 14:37 3042
-rwxr-xr-x 1 xyz xyz 524714705 Mär 29 14:37 3043
-rwxr-xr-x 1 xyz xyz 525495248 Mär 29 14:38 3044
-rwxr-xr-x 1 xyz xyz 528081874 Mär 29 14:38 3045
-rwxr-xr-x 1 xyz xyz 525649936 Mär 29 14:38 3046
-rwxr-xr-x 1 xyz xyz 526719244 Mär 29 14:38 3047
-rwxr-xr-x 1 xyz xyz 527602990 Mär 29 14:41 3048
-rwxr-xr-x 1 xyz xyz 524836266 Mär 29 14:41 3049
-rwxr-xr-x 1 xyz xyz 525082891 Mär 29 14:42 3050
-rwxr-xr-x 1 xyz xyz 524610404 Mär 29 14:44 3051
-rwxr-xr-x 1 xyz xyz 524690377 Mär 29 14:45 3052
-rwxr-xr-x 1 xyz xyz 524827189 Mär 29 14:45 3053
-rwxr-xr-x 1 xyz xyz 527020611 Mär 29 14:45 3054
-rwxr-xr-x 1 xyz xyz 525805581 Mär 29 14:46 3055
-rwxr-xr-x 1 xyz xyz   1456593 Mär 29 14:46 3056
-rwxr-xr-x 1 xyz xyz 524627153 Mär 29 14:47 3064
-rwxr-xr-x 1 xyz xyz 524632473 Mär 29 14:47 3065
-rwxr-xr-x 1 xyz xyz 524456957 Mär 29 14:48 3066
-rwxr-xr-x 1 xyz xyz 526150883 Mär 29 14:48 3067
-rwxr-xr-x 1 xyz xyz 275881424 Mär 29 14:48 3068
-rwxr-xr-x 1 xyz xyz       719 Mär 29 14:48 3076
-rwxr-xr-x 1 xyz xyz 524406081 Mär 29 14:49 3084
-rwxr-xr-x 1 xyz xyz 525094994 Mär 29 14:49 3085
-rwxr-xr-x 1 xyz xyz 524347234 Mär 29 14:50 3086
-rwxr-xr-x 1 xyz xyz 525428357 Mär 29 14:50 3087
-rwxr-xr-x 1 xyz xyz 524328947 Mär 29 14:50 3088
-rwxr-xr-x 1 xyz xyz 188657840 Mär 29 14:50 3089
-rwxr-xr-x 1 xyz xyz 527142534 Mär 29 14:51 3100
-rwxr-xr-x 1 xyz xyz 524504935 Mär 29 14:51 3101
-rwxr-xr-x 1 xyz xyz 526889188 Mär 29 14:52 3102
-rwxr-xr-x 1 xyz xyz 524323085 Mär 29 14:52 3103
-rwxr-xr-x 1 xyz xyz 525221648 Mär 29 14:52 3104
-rwxr-xr-x 1 xyz xyz 524900552 Mär 29 14:52 3105
-rwxr-xr-x 1 xyz xyz 525091826 Mär 29 14:53 3106
-rwxr-xr-x 1 xyz xyz 528450056 Mär 29 14:53 3107
-rwxr-xr-x 1 xyz xyz 524828259 Mär 29 14:53 3108
-rwxr-xr-x 1 xyz xyz 526797498 Mär 29 14:53 3109
-rwxr-xr-x 1 xyz xyz 525748178 Mär 29 14:54 3110
-rwxr-xr-x 1 xyz xyz 526515222 Mär 29 14:54 3111
-rwxr-xr-x 1 xyz xyz 525507053 Mär 29 14:54 3112
-rwxr-xr-x 1 xyz xyz 528255043 Mär 29 14:55 3113
-rwxr-xr-x 1 xyz xyz 525049207 Mär 29 14:55 3114
-rwxr-xr-x 1 xyz xyz 524771033 Mär 29 14:55 3115
-rwxr-xr-x 1 xyz xyz 524315617 Mär 29 14:55 3116
-rwxr-xr-x 1 xyz xyz 524319923 Mär 29 14:56 3117
-rwxr-xr-x 1 xyz xyz 528621511 Mär 29 14:56 3118
-rwxr-xr-x 1 xyz xyz 528698982 Mär 29 14:56 3119
-rwxr-xr-x 1 xyz xyz 525606472 Mär 29 14:57 3120
-rwxr-xr-x 1 xyz xyz 524508794 Mär 29 14:57 3121
-rwxr-xr-x 1 xyz xyz 525181549 Mär 29 14:57 3122
-rwxr-xr-x 1 xyz xyz 526045650 Mär 29 14:58 3123
-rwxr-xr-x 1 xyz xyz 526518555 Mär 29 14:58 3124
-rwxr-xr-x 1 xyz xyz 526498104 Mär 29 14:59 3126
-rwxr-xr-x 1 xyz xyz 525944915 Mär 29 14:59 3127
-rwxr-xr-x 1 xyz xyz 524474790 Mär 29 15:00 3129
-rwxr-xr-x 1 xyz xyz 524660515 Mär 29 15:00 3130
-rwxr-xr-x 1 xyz xyz     41423 Apr  1 10:35 3139
-rwxr-xr-x 1 xyz xyz 117748249 Apr  1 10:37 3170
-rwxr-xr-x 1 xyz xyz 527986762 Apr  5 12:52 3261
-rwxr-xr-x 1 xyz xyz 525322959 Apr 11 20:38 3304
-rwxr-xr-x 1 xyz xyz 137937833 Apr 15 22:32 3323
-rwxr-xr-x 1 xyz xyz 524755822 Apr 26 13:35 3376
-rwxr-xr-x 1 xyz xyz 401109979 Apr 26 13:37 3402
-rwxr-xr-x 1 xyz xyz       716 Mai  7 22:24 3434
-rwxr-xr-x 1 xyz xyz       717 Mai 19 17:12 3474
-rwxr-xr-x 1 xyz xyz 527756425 Mai 25 19:36 3511
-rw------- 1 xyz xyz 526596592 Jun  3 17:07 3567
-rw------- 1 xyz xyz       716 Jun  3 17:08 3584
-rw------- 1 xyz xyz 133484538 Jun 10 21:47 3621
-rw------- 1 xyz xyz 524847567 Jun 16 13:52 3674
-rw------- 1 xyz xyz 524572028 Jun 19 22:15 3726
-rw------- 1 xyz xyz  70008700 Jun 23 22:26 3769
-rw------- 1 xyz xyz 173154734 Jun 29 22:46 3822
-rw------- 1 xyz xyz 183079191 Jun 29 22:47 3830
-rw------- 1 xyz xyz       717 Jun 29 22:47 3838
-rw------- 1 xyz xyz 435501878 Jun 29 22:47 3846
-rw------- 1 xyz xyz      9047 Jun 29 22:48 3855
-rw------- 1 xyz xyz 342022985 Jun 29 22:48 3857
-rw------- 1 xyz xyz        17 Jun 29 22:48 3858
-rw------- 1 xyz xyz   1536076 Jul  4 19:31 3859
-rw------- 1 xyz xyz        49 Jul  4 19:31 3860
-rw------- 1 xyz xyz      9102 Jul  4 19:31 3861
-rw------- 1 xyz xyz        17 Jul  4 19:31 3862
-rw------- 1 xyz xyz       500 Jul  4 19:31 3863
-rw------- 1 xyz xyz        49 Jul  4 19:31 3864
-rw------- 1 xyz xyz      9051 Jul  4 19:31 3865
-rw------- 1 xyz xyz        17 Jul  4 19:31 3866
-rw------- 1 xyz xyz       896 Jul  4 19:31 3867
-rw------- 1 xyz xyz        49 Jul  4 19:31 3868
-rw------- 1 xyz xyz      9121 Jul  4 19:31 3869
-rw------- 1 xyz xyz        17 Jul  4 19:31 3870
-rw------- 1 xyz xyz       717 Jul  4 19:33 3871
-rw------- 1 xyz xyz        49 Jul  4 19:33 3872
-rw------- 1 xyz xyz      9100 Jul  4 19:33 3873
-rw------- 1 xyz xyz        17 Jul  4 19:31 3874

2 replies

ThomasWaldmann Jul 16, 2023
Maintainer

FileExistsError: [Errno 17] File exists: '/media/xyz/BACKUP1500GB/Backup/xyz/data/3/3875'

That does not seem true. Did you delete that file?

sicherist Jul 16, 2023
Author

No, I didn't change anything.

ThomasWaldmann · 2023-07-16T13:40:35Z

ThomasWaldmann
Jul 16, 2023
Maintainer

What you could try:

just retry

Considering that the file it complained about being in the way is gone, just retry:

borg check --repair -v --progress repo

alternatively, kill all indexes/caches and repair

borg delete --cache-only repo
cd repo
rm *.3874
borg check --repair -v --progress repo

1 reply

sicherist Jul 16, 2023
Author

Thanks, I tried both methods and both times borg check quits with an error:

borg check --repair -v --progress /media/xyz/BACKUP1500GB/Backup/xyz 
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!

Type 'YES' if you understand this and want to continue: YES 
Starting repository check
finished segment check at segment 3874                                                                                                                                                                                                                                                                                                             
Starting repository index check                                                                                                                                                                                                                                                                                                                    
Finished full repository check, no problems found.
Starting archive consistency check...
Enter passphrase for key /media/xyz/BACKUP1500GB/Backup/xyz: 
Analyzing archive xyz-xyz-2019-11-30T20:01:00 (1/131)                                                                                                                                                                                                                                                                           
Analyzing archive xyz-xyz-2019-12-18T11:33:43 (2/131)                                                                                                                                                                                                                                                                           
Analyzing archive xyz-xyz-2020-01-26T20:21:43 (3/131)                                                                                                                                                                                                                                                                           
Analyzing archive xyz-xyz-2020-02-27T16:09:07 (4/131)                                                                                                                                                                                                                                                                           
Analyzing archive xyz-xyz-2020-07-16T14:25:11 (5/131)                                                                                                                                                                                                                                                                           
Analyzing archive xyz-xyz-2020-09-28T23:17:19 (6/131)                                                                                                                                                                                                                                                                           
Analyzing archive xyz-xyz-2020-10-29T16:50:40 (7/131)                                                                                                                                                                                                                                                                           
Analyzing archive xyz-xyz-2020-11-19T10:14:24 (8/131)                                                                                                                                                                                                                                                                           
Analyzing archive xyz-xyz-2021-02-27T22:02:55 (9/131)                                                                                                                                                                                                                                                                           
Analyzing archive xyz-xyz-2021-03-31T09:21:51 (10/131)                                                                                                                                                                                                                                                                          
Analyzing archive xyz-xyz-2021-04-28T13:56:13 (11/131)                                                                                                                                                                                                                                                                          
Analyzing archive xyz-xyz-2021-05-30T22:02:08 (12/131)                                                                                                                                                                                                                                                                          
Analyzing archive xyz-xyz-2021-06-14T23:19:36 (13/131)                                                                                                                                                                                                                                                                          
Analyzing archive nnn-mmm-2021-09-26T23:11:12 (14/131)                                                                                                                                                                                                                                                                       
nnn-mmm-2021-09-26T23:11:12: home/xyz/Dokumente/nnn/mmm/aaaaaaaaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbbbbbb/ccccccccccccccccc/dddddddddddddddddddddd/eeeeeeeeeeeeeeeeeee.pdf: New missing file chunk detected (Byte 0-290915, Chunk 25dc450906bdbcf244fefe09afb2196d087720a1adc16761317d4af08f76b3c1). Replacing with all-zero chunk.
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1852, in add_reference
    self.chunks.incref(id_)
  File "src/borg/hashindex.pyx", line 328, in borg.hashindex.ChunkIndex.incref
KeyError: b'U\x7f\xbf\xb6\xc1\xf1-)\xc5\x04\x1d\xeb#\xbevIcn\xc1\xf7\xf6\xf1\xef#\xfc\xe9?\xf0\xe8\xcd>w'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5213, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5144, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 347, in do_check
    if not args.repo_only and not ArchiveChecker().check(
                                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1647, in check
    self.rebuild_refcounts(archive=archive, first=first, last=last, sort_by=sort_by, glob=glob)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 2058, in rebuild_refcounts
    verify_file_chunks(info.name, item)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1894, in verify_file_chunks
    add_reference(chunk_id, size, csize, cdata)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1857, in add_reference
    self.repository.put(id_, cdata)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1230, in put
    segment, offset = self.io.write_put(id, data)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1638, in write_put
    fd = self.get_write_fd(want_new=(id == Manifest.MANIFEST_ID), raise_full=raise_full)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1419, in get_write_fd
    self._write_fd = SyncFile(self.segment_filename(self.segment), binary=True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/platform/base.py", line 162, in __init__
    self.f = open(path, mode=mode)  # python file object
             ^^^^^^^^^^^^^^^^^^^^^
FileExistsError: [Errno 17] File exists: '/media/xyz/BACKUP1500GB/Backup/xyz/data/3/3875'

Platform: Linux xyz 6.2.0-25-generic #25-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 16 17:05:07 UTC 2023 x86_64
Linux: Unknown Linux  
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.3 fuse: pyfuse3 3.2.1 [pyfuse3,llfuse]
PID: 16820  CWD: /media/xyz/BACKUP1500GB/Backup
sys.argv: ['/usr/bin/borg', 'check', '--repair', '-v', '--progress', '/media/xyz/BACKUP1500GB/Backup/xyz']
SSH_ORIGINAL_COMMAND: None

ThomasWaldmann · 2023-07-16T13:44:43Z

ThomasWaldmann
Jul 16, 2023
Maintainer

BTW, you seem to have many 17b files from borg 1.1.

After you repaired the repo, do this step from the changelog / upgrade notes once per repo for all repos that used borg < 1.2 before:

borg compact --cleanup-commits repo

1 reply

sicherist Jul 16, 2023
Author

Ok, i think I fixed it. I thought: The OS sees that "3875" exists even though it doesn't seem to exist. Maybe it is a file-system problem? The disk is NTFS-format so I tried "ntfsfix". It found problems but is unable to fix it. So I had to start a Windows-machine to run proper "chkdsk". This seemed to do the trick. Now borg check ran fine and found a lot of problems in the latest archive.

1809 orphaned objects found!                                                                                                                                                                                                   
Deleting 1809 orphaned and 1000 superseded objects...
Finished deleting orphaned/superseded objects.
Writing Manifest.
Committing repo.
Archive consistency check complete, problems found.

Answer selected by sicherist

ThomasWaldmann · 2023-07-16T20:19:43Z

ThomasWaldmann
Jul 16, 2023
Maintainer

OK. Orphaned/superseeded is no big issue, just stuff that is not referenced by an archive.

But in another check run it complained about a missing chunk and that it replaced it by an all-zero replacement chunk.
Due to the crash, maybe that was not committed though.

But maybe proceed as follows:

run backup(s) to this repo that create archive(s) of all data you usually put into that repo
run another borg check --repair afterwards - that might repair damaged chunks in case they reappeared.

1 reply

sicherist Jul 17, 2023
Author

Yes, there were also a lot of missing chunks replaced by all-zero chunks. This was "healed" like you described. Thank you.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting lots of error messages "[Errno 17] file exists"; borg -check didn't help #7727

{{title}}

Replies: 5 comments 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Getting lots of error messages "[Errno 17] file exists"; borg -check didn't help #7727

sicherist Jul 15, 2023

Replies: 5 comments · 5 replies

ThomasWaldmann Jul 15, 2023 Maintainer

sicherist Jul 16, 2023 Author

ThomasWaldmann Jul 16, 2023 Maintainer

sicherist Jul 16, 2023 Author

ThomasWaldmann Jul 16, 2023 Maintainer

just retry

alternatively, kill all indexes/caches and repair

sicherist Jul 16, 2023 Author

ThomasWaldmann Jul 16, 2023 Maintainer

sicherist Jul 16, 2023 Author

ThomasWaldmann Jul 16, 2023 Maintainer

sicherist Jul 17, 2023 Author

sicherist
Jul 15, 2023

Replies: 5 comments 5 replies

ThomasWaldmann
Jul 15, 2023
Maintainer

sicherist
Jul 16, 2023
Author

ThomasWaldmann Jul 16, 2023
Maintainer

sicherist Jul 16, 2023
Author

ThomasWaldmann
Jul 16, 2023
Maintainer

sicherist Jul 16, 2023
Author

ThomasWaldmann
Jul 16, 2023
Maintainer

sicherist Jul 16, 2023
Author

ThomasWaldmann
Jul 16, 2023
Maintainer

sicherist Jul 17, 2023
Author