[Feature Request] Skip past the music pauses in music videos

[KulenFlulen]

Quick summary

Some music videos contain sections where the song pauses for a scene which only makes sense in the music video. The extra noise isn't a problem, being muted, but the pause desyncs the video from the song.
I would like to be able to configure sections of the video which get skipped while playing the map, be it configurable in-game or config-only.

Details

Quite a few music videos have one break, but some might have multiple breaks. Just in case, I think there should be a list of start and end times for the multiple sections to skip, even if only one is used most of the time. It should probably reference the video timestamp instead of the map's, remaining consistent with the endVideoAt option. Also similar to endVideoAt, decimal places would be helpful with the time stamps.

My inspiration was this music video which follows the song fairly tightly, except for the silent section between 0:54 and 1:06.
https://www.youtube.com/watch?v=3nlSDxvt6JU

[Kevga]

Thank you for the suggestion. I thought about something similar to this before, basically a way to create multiple sync points to modify the timeline of the video. This would allow certain parts of the video to be repeated, other parts to be skipped or whatever else suits the video. Main issue I see with this is that the video player has a variable amount of delay when seeking to other parts in the video, which can make it difficult to keep the video in sync after a jump on the timeline. I'll work on this when I find some time for it.

[Fusseldieb]

Chiming in, this would be an awesome feature.

Probably most useful in remixes, where the remixes don't have a video, but the original does.

But also, in the meantime someone could just edit it manually using a video editor and replace the mp4 in the music folder.

[b-rad15]

I've got the code already to do this in my version, would have to port it over. It uses Aurio to sync the video, determine when sync changes, then cuts the video over the split. Worked very well for the songs I tested with. I could try doing it myself but I've been out of the modding game for a bit so it could take some time to get back into the swing of things

[Kevga]

Hey @b-rad15!

I have looked into Aurio and specifically your implementation a while back. I don't remember all the specifics of why I didn't port it, but here are some things that I still remember:

1. I would like to use it as a library and implement the logic in the mod itself, instead of having to package another executable that interfaces with Aurio
2. I couldn't get Aurio to read directly from Ogg Vorbis files, and I believe you ran into the same issue, seeing as your implementation converts the audio files to wav first. That's something I would like to avoid doing. I believe that would mean having to make some modifications to Aurio to make use of NAudio/Vorbis.
3. It bothers me a bit that Aurio depends on the ffmpeg libraries. I think in your release, you packaged 32- and 64-bit libraries of ffmpeg for use with Aurio and additionally the ffmpeg executable for youtube-dl, so three instances of ffmpeg in total. Though I'm not sure if it is at all possible to avoid this.

Regarding this issue in particular, I don't know if it's worth having to do the audio fingerprinting, cutting and re-encoding over the implementation I suggested above. Those seem like tasks that take quite a while to complete, especially on lower end hardware. And I assume you probably can't play maps while this runs due to risk of stuttering.

In general though, I think using audio fingerprinting to find the offset would absolutely be great to have. I just would like to avoid having to re-encode the video.

[b-rad15]

I think I've got 1, 2, and 3 fixed. 1 was only ever a convenience thing never a code limitation, 2 was just to avoid 3 I got it built to where it should be able to use FFmpeg to read any file natively, 3 I should be able to be fixed by using a shared version of ffmpeg (so the exe and dlls are separate, size should be roughly the same though) so that Aurio can read the dlls and the yt-dlp can use the exe. Last point is valid but I did do some testing and when I tried there was always too much seek delay.

The last part thought really is the question for how to implement it. IMO the best way is to make the file once, that way it'll work perfectly every time but either way, may as well get this working (since I hate setting offsets manually anyway lol) then go from there for this issue specifically.

[b-rad15]

Finally got some more time and worked on this a bit more. I thought I replied to this again but I guess not. I didn't implement the code back then because it wasn't working nearly well enough, the audio from the video sections and the audio source didn't match up the way it needed to, so I was trying to figure out why. I just thought to test it with each section separately instead of in one video now (for the few tests I did) and it works perfectly (and much faster than before). Using a different way to concatenate the sections in FFmpeg is all it took to fix

[Kevga]

Do you plan on opening a PR for this feature? If so, we should discuss some of the implementation details

[b-rad15]

Gotta work on some other stuff first but once I get the chance and very simple working implementation then yeah