Split track selection to video and audio selection

[Arnei]

In the "Tracks" view, instead of de-/selecting the whole track, the video and audio streams can be de-/selected individually. At least one video stream must remain selected. All audio streams can be deselected. If the audio stream is marked as unavailable, it cannot be de-/selected.

Shows the waveform from the timeline for a graphical representation of the audio stream. However, there will only be one waveform generated for the timeline, so in case of additional tracks a placeholder image will be shown instead. Any suggestions/submissions for a better placeholder graphic are very welcome.

Resolves #1009. #1009 also mentions potential issues in the backend, while this PR only addresses the frontend. From my limited testing the backend seems fine when using the default community workflows (no errors, the correct streams get published). If there are any users around that are still using the track selection feature in the old editor, and that are aware of any backend issues, please do tell me about them.

Courtesy of the Opencast 2023 Crowdfunding.

[github-actions]

This pull request is deployed at test.editor.opencast.org/1230/2023-12-14_10-06-39/ .
It might take a few minutes for it to become available.

[lkiesow]

There are definitely issues in the backend. But I also see usability issues in the frontend.

For example, what will be the result of a selection like this?

As a user, my assumption would be to get either a single video with mixed audio streams from the other tracks or three tracks, two with audio and one with video.

However, the muxing command being executed is:

[ffmpeg, -nostdin, -nostats, -i, ...fc9.mp4, -i, ...ad4.mp4, -shortest, -c, copy, -map, 0:v:0, -map, 1:a:0, ...b5-copy.mp4]

This completely discards one of the selected audio streams. As a user, I can just describe that as a bug.

And then, what would be the expected result of something like this?

Will all audio streams be mixed into a single audio stream and applied to A? Or will just C be transferred to A? As a user, you have no idea.

[Arnei]

My thoughts on how to go about the usability issues you highlighted:

1. Leave it as is, people will "somehow just know" how it works at their place. Arguably workable, but not a great solution.

2. Introduce restrictions on what combination of tracks can and cannot be selected. Would need to be configurable, because there will always be admins that want to allow combinations that others would restrict, like the ones you mentioned. But then we again run into the issue of having to "just know" what certain combinations will do.

3. Add more buttons/checkboxes/ui that describes what should happen with the tracks, e.g. Mux all remaining audio streams onto A/B/C or Keep audio on tracks, Allow tracks with only audio. Arguable the way to go even if it muddles up the UI somewhat, but also requires an editor workflow to support all those new flags.

[lkiesow]

The original goal of the editor was to be simple since it is meant for non-technical end-users. I would really like to keep that goal since we expose the editor to such users.

Therefor, a lot of implicit knowledge as a requirement isn't a great solution and neither is having a lot of boxes and buttons with technical terms.

As an alternative, we could introduce one simple way to handle the result. My suggestion would be: If you remove any audio stream, all other audio streams will be mixed and attached to all videos. This rule is simple and works with as many videos tracks as you want. It also works well with other parts of Opencast like the Paella Player, which will just play back a single stream.

Running with that idea, we would need a change in the backend workflow operation and a hint about how this works in the UI of this pull request. The hint should also include that the mixing will happen if a track is already missing an audio stream.

Furthermore, since users should be able to deal with individual audio streams, we also need to make sure the editor can playback all streams, and individual streams. A simple solution may be playback controls on the tracks page where you then disable the audio streams of the tracks were audio has been removed?

[github-actions]

This pull request has conflicts ☹
Please resolve those so we can review the pull request.
Thanks.

[github-actions]

This pull request is deployed at test.editor.opencast.org/1230/2024-01-18_15-31-01/ .
It might take a few minutes for it to become available.

[KatrinIhler]

Finally getting around to commenting on this... I will not chime in with the ongoing discussion, but just say that

At least one video stream must remain selected

is something I would like to see either being configurable in the backend, or simply not enforced (which I think you already knew I was gonna say ;)), because it doesn't take into account audio-only use cases. Consider the following:

1. An institution with custom workflows for publication that can handle audio-only input (either by rendering a static image into it, or by having a player that can handle audio-only), and
2. A recording where the video is either black or otherwise damaged or unusable, but the audio is fine, and I want to deselect the video and publish the audio only with my custom workflow, or
3. A recording that doesn't come with a video stream in the first place that I want to cut and publish.

Both of these cases should be possible with the new editor. It doesn't have to be the case out of the box, but I shouldn't need to patch the code to make this work, just do some configuration. So basically the same arguments I made here, except that for the new editor, making this configurable with an "allow audio only in editor" or something would actually be worth it.

[github-actions]

This pull request has conflicts ☹
Please resolve those so we can review the pull request.
Thanks.

[KatrinIhler]

See my comment above, also conflicts. :(

[luniki]

Lars sent me this "oneliner" to create a triple event:

curl -u admin:opencast "${SERVER}/ingest/addMediaPackage/fast" \
        -F 'flavor=presenter/source' \
        -F mediaUri=https://example.com/opencast/test-media/goat.mp4 \
        -F 'flavor=presentation/source' \
        -F mediaUri=https://example.com/opencast/test-media/goat.mp4 \
        -F 'flavor=experiment/source' \
        -F mediaUri=https://example.com/opencast/test-media/goat.mp4 \
        -F identifier=three \
        -F title="Three Video Streams" \
        -F creator="$(shuf -n 1 /usr/share/dict/words)"

[Arnei]

Results of todays discussion between Sascha, Katrin, Lars, Arne and Marcus:

• Should be able to select 0, 1 or more videos
• In the same vein, should be able to select 0, 1 or more audios
• At least one track (video or audio) needs to remain
• The same audio is applied to all videos: None, one or multiple muxed
  • Have a note in the UI that informs the user about muxing
• In case no video remains selected, the remaining audios are muxed into one audio track
• Default selection when opening up the tracks view: All videos are selected and only the first audio track is selected
  • ...why did we want this again?
• There should be a big confirm button
  • This idea here is to avoid a user making an accidental change to their event

Key-Image

[luniki]

I talked to @Arnei about this and created a first version of it. Here are some screenshots.

Initial state after opening the "Tracks" tab

After de-selecting a video track

After de-selecting all video tracks

After de-selecting all but one track

[luniki]

The "Confirm selection" buttons seems to be useless. Last we talked we defined:

There should be a big confirm button
* This idea here is to avoid a user making an accidental change to their event

The initial state is that all tracks are selected. This can also be easily determined in the code. So if we never de-select any tracks, we don't change the event?!

[snoesberger]

The "Confirm selection" buttons seems to be useless. Last we talked we defined:

There should be a big confirm button

• This idea here is to avoid a user making an accidental change to their event

The initial state is that all tracks are selected. This can also be easily determined in the code. So if we never de-select any tracks, we don't change the event?!

IIRC, the idea was not to have a confirm button, but to have a checkbox to activate the track selection. So if a user wants to de-select tracks, they first have to activate this "modify track selection" checkbox. As long as the checkbox is inactive, video and audio selection is grayed-out, nothing can be changed and the track selection will be completely ignored. As soon as the checkbox is activated, the user will be able to de-select video and/or audio tracks, and Opencast will process video and audio accordingly (muxing audio, etc.) as soon as the user starts the workflow. @KatrinIhler and @lkiesow please correct me if I'm wrong.

[KatrinIhler]

Agreed with @snoesberger completely . The difference between an activated and deactivated track selection, if the user does not do anything else, is whether the audio tracks are kept as-is or are muxed.

So no to the confirm button (bc this doesn't persist this decision in a visible way later), yes to a checkbox or something to activate the whole selection thingy (as shown on the initial sketch).

[luniki]

I have pushed my changes to this PR to my own branch: https://github.com/luniki/opencast-editor/commits/tracks-select-audio/

[luniki]

Here are some screenshot of my current state of track selection.

Entering the track selection

Enabled track selection

De-selected all but one track

[luniki]

The checkbox labelled "Customize track selection" is a new part of the state of the editor which has to be transferred to the Opencast EditorService endpoint. At the moment there is an additional boolean key in the JSON object.

This requires changes in the EditorRestEndpointBase class (and the GSON classes).

Or do I get this wrong? How should that checkbox be encoded?

[luniki]

I pushed my changes to my forked branch of opencast-editor: https://github.com/luniki/opencast-editor/commits/tracks-select-audio/

In the backend I edited the EditingData class: https://gitlab.elan-ev.de/mlunzena/opencast-bern/-/commits/feature/track-selection

[github-actions]

This pull request has conflicts ☹
Please resolve those so we can review the pull request.
Thanks.

[github-actions]

This pull request is deployed at test.editor.opencast.org/1230/2024-06-26_15-03-28/ .
It might take a few minutes for it to become available.

[github-actions]

This pull request is deployed at test.editor.opencast.org/1230/2024-06-27_09-04-50/ .
It might take a few minutes for it to become available.

[github-actions]

This pull request is deployed at test.editor.opencast.org/1230/2024-06-27_09-10-39/ .
It might take a few minutes for it to become available.

[github-actions]

This pull request is deployed at test.editor.opencast.org/1230/2024-06-27_09-18-32/ .
It might take a few minutes for it to become available.

[Arnei]

I handled the ToDos from my last comment by going for the mentioned alternatives.

This means that

• Whether audio only should be allowed is now configurable (NOT ALLOWED by default)
• Whether the track selection should be shown for events with +3 videos (NOT SHOWN by default)
• Info text has been changed in the following ways:
  • User is not told anymore that multiple audio tracks will be mixed.
  • User is now told that audio track will be duplicated onto all videos, if exactly one audio track is selected.

While these "solutions" are not what exactly what we wanted, they do accurately reflect the current state of the Opencast backend and workflows. Furthermore, solving these ToDos "properly" (make OC workflows handle audio only, make select-tracks WOH handle +3 videos, implement audio muxing) is a lot of work that likely will not happen anytime soon, and I would hate to see this PR get blocked because of that. Even if this PR does not achieve all the things we set out to do, it still constitutes a considerable improvement. I would therefore like to see it merged, and create follow-up issues for the "proper" solutions.

[snoesberger]

While these "solutions" are not what exactly what we wanted, they do accurately reflect the current state of the Opencast backend and workflows. Furthermore, solving these ToDos "properly" (make OC workflows handle audio only, make select-tracks WOH handle +3 videos, implement audio muxing) is a lot of work that likely will not happen anytime soon, and I would hate to see this PR get blocked because of that. Even if this PR does not achieve all the things we set out to do, it still constitutes a considerable improvement. I would therefore like to see it merged, and create follow-up issues for the "proper" solutions.

I completely agree with you. I also prefer a first version with the basic functionalities that we can already use. The rest of the customisation can be done in separate PRs.

[snoesberger]

@lkiesow do you also agree with the procedure described by @Arnei in this comment?
I'd love to see this PR merged, because then we could add the translations to Crowdin and use this feature properly in production.

[snoesberger]

We have now been running this PR in production for almost two months without any problems. From our user point of view (without being able to judge the code quality), this PR could be merged.

[github-actions]

This pull request has conflicts ☹
Please resolve those so we can review the pull request.
Thanks.

[github-actions]

This pull request is deployed at test.editor.opencast.org/1230/2024-12-06_08-06-22/ .
It might take a few minutes for it to become available.

[github-actions]

This pull request has conflicts ☹
Please resolve those so we can review the pull request.
Thanks.

[JulianKniephoff]

CI fails. This should make it work.

cf. 60939d9#diff-1a831a072af0a35e8057bc0f12c7427b0eb10fdf4bc4eff1ed3f308e23af7a7bL38-R43

[github-actions]

This pull request is deployed at test.editor.opencast.org/1230/2024-12-12_16-32-25/ .
It might take a few minutes for it to become available.

Addressed