Why is moe copying extra files a second time, from the destination folder back to the destination folder?

[ali-ramadhan]

I'm not sure if this is actually a bug so I haven't opened a bug report (also not sure how to reproduce without the files, dummy files maybe?).

I'm trying to organize some extra files when I noticed that moe is copying the extra from the source directory to the destination directory in the library correctly, but then it moves it from the destination folder to the destination folder during which the name gets changed incorrectly.

I think the main issue is that I have a custom extra_path which seems to be evaluated twice?

Here's my moe config: https://github.com/ali-ramadhan/moe-plugins/blob/main/moe_config.toml
Here's my organize_extras plugin: https://github.com/ali-ramadhan/moe-plugins/blob/main/organize_extras.py

I'll try to describe the issue in more detail below. Apologies for the long post!

---

Here's the source directory structure:

some_album
├── 01.flac
├── 02.flac
├── 03.flac
├── album.cue
├── album.log
├── cover.jpg
└── scans
   ├── 1.jpg
   ├── 2.jpg
   ├── 3.jpg
   └── cd.jpg

so I expect it to preserve the scans directory but here's what I end up with:

some_album
├── 01.flac
├── 02.flac
├── 03.flac
├── album.cue
├── album.log
├── album.m3u
├── cover.jpg
├── scans_1.jpg
├── scans_2.jpg
├── scans_3.jpg
└── scans_cd.jpg

Here's the log with moe -v add and I've added a print statement to show the result of organize_extras(extra):

INFO: Copied item. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/01.flac, item=Some Artist - Track 1]
INFO: Copied item. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/02.flac, item=Some Artist - Track 2]
INFO: Copied item. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/03.flac, item=Some Artist - Track 3]
Processing extra: ../source/Some Album/Some Artist - Some Album.log -> Some Album.log
INFO: Copied item. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/Some Album.log, item=Some Artist - Some Album (2024): Some Album.log]
Processing extra: ../source/Some Album/Some Album.cue -> Some Album.cue
INFO: Copied item. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/Some Album.cue, item=Some Artist - Some Album (2024): Some Album.cue]
Processing extra: ../source/Some Album/Some Artist - Some Album.jpg -> Some Artist - Some Album.jpg
INFO: Copied item. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/Some Artist - Some Album.jpg, item=Some Artist - Some Album (2024): Some Artist - Some Album.jpg]
Processing extra: ../source/Some Album/scans/1.jpg -> scans/1.jpg
INFO: Copied item. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/scans_1.jpg, item=Some Artist - Some Album (2024): scans_1.jpg]
Processing extra: ../source/Some Album/scans/2.jpg -> scans/2.jpg
INFO: Copied item. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/scans_2.jpg, item=Some Artist - Some Album (2024): scans_2.jpg]
Processing extra: ../source/Some Album/scans/3.jpg -> scans/3.jpg
INFO: Copied item. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/scans_3.jpg, item=Some Artist - Some Album (2024): scans_3.jpg]
Processing extra: ../source/Some Album/scans/cd.jpg -> scans/cd.jpg
INFO: Copied item. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/scans_cd.jpg, item=Some Artist - Some Album (2024): scans_cd.jpg]
INFO: Copied album. [dest=/home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024), album=Some Artist - Some Album (2024)]
Processing extra: /home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/Some Album.log -> Some Album.log
Processing extra: /home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/Some Artist - Some Album.m3u -> Some Artist - Some Album.m3u
Processing extra: /home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/Some Album.cue -> Some Album.cue
Processing extra: /home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/Some Artist - Some Album.jpg -> Some Artist - Some Album.jpg
Processing extra: /home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/scans_1.jpg -> scans_1.jpg
Processing extra: /home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/scans_2.jpg -> scans_2.jpg
Processing extra: /home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/scans_3.jpg -> scans_3.jpg
Processing extra: /home/alir/music-tagging-moe/library-tagged/Some Artist - Some Album (2024)/scans_cd.jpg -> scans_cd.jpg
INFO: Wrote tags to track. [track=Some Artist - Track 1]
INFO: Wrote tags to track. [track=Some Artist - Track 2]
INFO: Wrote tags to track. [track=Some Artist - Track 3]
INFO: Added item to the library. [item=Some Artist - Some Album (2024)]

So I notice that the function organize_extras(extra) is being called twice on each extra. Once when copying each track, then again when copying the album.

And indeed when I print a stacktrace in organize_extras I see that in the second call the stacktrace is coming from copy_album.

Stacktrace from the first time organize_extras is called:

  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/bin/moe", line 8, in <module>
    sys.exit(main())
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/cli.py", line 167, in main
    _parse_args(args)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/cli.py", line 113, in _parse_args
    parsed_args.func(session=session, args=parsed_args)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/add/add_cli.py", line 90, in _parse_args
    _add_path(session, path, album)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/add/add_cli.py", line 126, in _add_path
    moe.add.add_item(session, Album.from_dir(path))
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/add/add_core.py", line 79, in add_item
    session.flush()
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4352, in flush
    self._flush(objects)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4380, in _flush
    self.dispatch.before_flush(self, flush_context, objects)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/sqlalchemy/event/attr.py", line 378, in __call__
    fn(*args, **kw)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/library/lib_item.py", line 168, in _edit_before_flush
    config.CONFIG.pm.hook.edit_new_items(session=session, items=new_items)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 72, in edit_new_items
    copy_item(item)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 231, in copy_item
    _copy_file_item(item)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 254, in _copy_file_item
    dest = fmt_item_path(item)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 113, in fmt_item_path
    item_path = _eval_path_template(config.CONFIG.settings.move.extra_path, item)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 142, in _eval_path_template
    path_part = _lazy_fstr_item(template_part, lib_item)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 190, in _lazy_fstr_item
    return eval(f'f"""{template}"""')  # noqa: B907
  File "<string>", line 1, in <module>
  File "/home/alir/.config/moe/plugins/organize_extras.py", line 51, in organize_extras
    traceback.print_stack()

Second time:

  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/bin/moe", line 8, in <module>
    sys.exit(main())
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/cli.py", line 167, in main
    _parse_args(args)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/cli.py", line 113, in _parse_args
    parsed_args.func(session=session, args=parsed_args)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/add/add_cli.py", line 90, in _parse_args
    _add_path(session, path, album)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/add/add_cli.py", line 126, in _add_path
    moe.add.add_item(session, Album.from_dir(path))
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/add/add_core.py", line 79, in add_item
    session.flush()
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4352, in flush
    self._flush(objects)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4380, in _flush
    self.dispatch.before_flush(self, flush_context, objects)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/sqlalchemy/event/attr.py", line 378, in __call__
    fn(*args, **kw)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/library/lib_item.py", line 168, in _edit_before_flush
    config.CONFIG.pm.hook.edit_new_items(session=session, items=new_items)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 72, in edit_new_items
    copy_item(item)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 229, in copy_item
    _copy_album(item)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 247, in _copy_album
    _copy_file_item(extra)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 254, in _copy_file_item
    dest = fmt_item_path(item)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 113, in fmt_item_path
    item_path = _eval_path_template(config.CONFIG.settings.move.extra_path, item)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 142, in _eval_path_template
    path_part = _lazy_fstr_item(template_part, lib_item)
  File "/home/alir/.cache/pypoetry/virtualenvs/moe-plugins-RYSGbTNP-py3.11/lib/python3.11/site-packages/moe/move/move_core.py", line 190, in _lazy_fstr_item
    return eval(f'f"""{template}"""')  # noqa: B907
  File "<string>", line 1, in <module>
  File "/home/alir/.config/moe/plugins/organize_extras.py", line 51, in organize_extras
    traceback.print_stack()

[jtpavlock]

What's happening is that moe is copying all items (albums, tracks, and extras) being added to the library, but when albums are copied, so are its underlying tracks and extras. So the added items list might look like:

1. album1
2. album1/track1.mp3
3. album1/extra1.mp3

First, it will copy album1, which includes track1 and extra1 as well as they belong to that album. Moe will then reach track1, and check if it's already been copied or not. In order to do this, it generates the final path it would copy the item to, and then checks if it's already there. If so, no need to copy the file again (this logic exists in _copy_file_item).

So your custom function gets called a second time when moe reaches the extra for a second time and generates the final path it would copy the extra to, to check if it already exists there.

The problem seems to be that your custom organize_extras function changes based on the extra's path, which is a case I hadn't considered. Moe currently operates on the assumption that an item's new path is independent of its current one.

I do think this could be considered a bug, as we should allow custom path functions as you're attempting without having to worry about this edge case, so I'd like to see a fix within Moe itself.

@moe.hookimpl(trylast=True)
def edit_new_items(items: list[LibItem]):
    """Copies and formats the path of an item after it has been added to the library."""
    for item in items:
        copy_item(item)

If you're willing to give it a shot, I assume the fix would lie in the above function, to not call copy_item on tracks or extras if their album also exists within items (as they would get copied when their album gets copied).

A workaround in the meantime for you may be to check if the album already exists at its final destination (call fmt_item_path on the extra's album within your organize_extras function), and if so, don't edit the extra's path. I'm not entirely sure if this would still work though if Moe tries copying the extra before copying the entire album, which seems to be happening in your example.

[ali-ramadhan]

Thanks for digging into this and explaining what's going on! What I'm seeing makes sense now. Not sure if there's a better way to organize/rename extras based on their extension, but it doesn't seem hard to fix this behavior.

If you're willing to give it a shot, I assume the fix would lie in the above function, to not call copy_item on tracks or extras if their album also exists within items (as they would get copied when their album gets copied).

Maybe I can sort items to copy albums first, then iterate over any tracks and extras in items and only copy them if they were not part of an album that was already copied.