v3.27.0

Release v3.27.0 (2022-03-31)

The v3.27.0 release has now been published to npm.
It is a big release which includes many features, fixes and improvements, among which:

• A new option, maxVideoBufferSize, allows to indicate to the RxPlayer that the current device has limited available memory.
  The RxPlayer will then use that value to prevent overflowing it by loading video segments adaptively

• A new audioTrackSwitchingMode, "reload", allows to work-around some compatibility issues seen on some recent Chrome versions, where audio could be lost after switching audio track in the "direct" mode

• Safari support has been improved for some advanced usages

• DRM: the keySystems[].singleLicensePer option now accepts a third license requesting mode: "periods". It enables optimizations on use cases like key rotation and support of targets with a very limited number of key slots available.

• DRM: the "content" mode of the keySystems[].singleLicensePer option is now more optimized: persistent and non-persistent caches should now lead to more cache hit than before under this mode.

• Many other bug fixes and improvements, you can refer to the changelog for more information

Configuring a maximum video buffer size

Issues with low-end devices

At Canal+, we use the RxPlayer in production on a large panel of devices. Some of them being real powerhouses without any performance or memory issue, others being more limited on processing speed and memory.
When playing high-bitrate media contents - such as with ultra-high video definition (UHD) - on some devices of that second category, we sometimes entered a situation where the memory of the device would quickly become full as video content was buffered by the RxPlayer, leading to all kinds of performance and media issues.

The situation was particularly problematic on some older smart TVs, where we had temporarily advised application developpers that they should set for now a lower wantedBufferAhead.

This configuration option configures the amount of media content the RxPlayer will pre-load in advance. Setting a lower value thus reduces the amount of loaded media data and thus lowers memory usage, but it also raises the risk of rebuffering.

Image: Difference between a low wantedBufferAhead (left) and a high one (right), we can see that we load more segments in the buffer when that value is high (visible both in the buffered grey part of the progress bar or in the visual representation of the audio and video buffer's content at the bottom - don't mind the color used there, they are randomized)

Moreover, it was a general setting: lower-bitrate video or even audio would also respect this value - even if only high-bitrate video was problematic.
We thus decided to provide a proper answer to this specific problem.

The maxVideoBufferSize solution

The solution we came with was to add a maxVideoBufferSize option to the RxPlayer.

The idea is that the application developers might know in some cases much better than the RxPlayer the limitations of a specific device their application runs on.
Setting a maxVideoBufferSize will thus serve as a hint telling the RxPlayer how much video data, in kilobytes, it should buffer at maximum to prevent most of those memory-related issues.

The RxPlayer will then use this hint to fill the video buffer adaptively:

• while video data is still below this maxVideoBufferSize limit, the RxPlayer will try to reach the wantedBufferAhead setting, as usual
• as soon as this memory limit is encountered, the RxPlayer will in most cases stop loading new video data until that limit can be respected again - this happens for example once old video data is garbage-collected.

To implement this feature, the RxPlayer now estimates the storage size used by video media segments present in the browser's buffer as well as of future video segments it wants to download. Additionning those allows to check if the maxVideoBufferSize limit is still respected.

Image: Here we're playing the same content with equal wantedBufferAhead and maxVideoBufferSize (10 Megabytes) but different qualities. To the left is a video of a lower quality, to the right a higher one. We can see that the buffer size constructed on the right is much lower. This is here because the maxVideoBufferSize limit was reached. We can also see that on the left, the constructed buffer is roughly equal between audio and video, this indicates that the maxVideoBufferSize was probably never reached before the wantedBufferAhead.

However that type of estimate is not an exact science, the real memory reserved by the browser to handle those segments might actually be higher or lower. The RxPlayer may also decide by itself to still load some video data once that limit is reached if it judges that the buffer built is too low for smooth playback.
As such, this maxVideoBufferSize option only acts as a hint and not a true hard limit.

The maxVideoBufferSize API

This maxVideoBufferSize limit can be set at either one of two places.

Either in the constructor, documented in the constructor options page:

const rxPlayer = new RxPlayer({
  videoElement,
  /**
   * This is expressed in kilobytes, so here this corresponds to 50MB or
   * ~47.68 MiB.
   */
  maxVideoBufferSize: 50000,
});

Or at any time (even as content plays) through a method:

rxPlayer.setMaxVideoBufferSize(50000);

This method's documentation is under this link.

The last set maxVideoBufferSize can be retrieved through the following getter:

rxPlayer.getMaxVideoBufferSize();

This method's documentation is under this link.

Note that by default this value is set to Infinity, which is also the value to set it to to disable this limit.

DRM: A "periods" mode for the keySystems[].singleLicensePer loadVideo option

The singleLicensePer: "content" mode

Last year (v3.24.0), we added the singleLicensePer property to our DRM-related options.
It was added to optimize the playback of contents with multiple decryption keys: by only performing a single license request for all decryption keys instead of the usual one per key, we were able to reduce the time and resources needed to play those contents both on the client and on the server-side.

Image: By default, the RxPlayer will load a single license per new encryption metadata encountered. This is adapted when the license server emits only the key asked, like illustrated here.

Image: Through the singleLicensePer: "content" mode we implemented at the time, the RxPlayer will load a single license for the whole content instead. This only works if the license server returns all keys linked to the content at once, and not just the asked one.

Technically, this option changes the RxPlayer's behavior on two aspects:

1. Only a single HTTP(S) request is performed - even when multiple keys are needed concurrently
2. It also communicates to the RxPlayer the fact that any key that is not found in the license - yet should have been in it - are keys voluntarily witheld by the license server.
   Thus the Representation (a.k.a. profile or quality) corresponding to those witheld keys should not be played (and be fallbacked from if already buffered).

Image: In this example, the license request for the video 1080p did not return a key for the 4K video quality. Because we're in a singleLicensePer: "content" mode and thus should have received all available keys, the RxPlayer deduce that the 4K video quality is not decipherable. It will thus avoid this quality and remove it from the video buffer if it was already pushed in it.

This option is used a lot at Canal+, where the majority of customer-facing applications use that "content" mode, so we can play contents faster and put less stress on our back-end.

Inconvenients of this mode

However, performing only a single license request for the whole content has some drawbacks, the main ones being:

• All keys needed to decrypt any part of the content has to be known in advance, even decryption keys for future sub-contents, because no new license request will be performed in the future (all those keys we didn't know of in the first license request will be assumed to have been voluntarily witheld).

• Some contents could need a lot of decryption keys and some peculiar devices have a relatively low limit of "key slots" available.
  In that case, it might be problematic to push a single license containing more keys than what the device can handle, leading to all kinds of playback errors.

  

  Image: Case of a device with only 3 "key slots" available. Here, if receiving a license containing 4 keys, the device would not be able to use all of them. This can lead to all sorts of issues.

  If those keys were in different licenses, the RxPlayer could easily decide to remove keys from the previous license as a new one is considered. With a single license however, the player cannot perform this kind of tricks.

To better handle those cases while still staying optimized, we needed to come up with a third way.

singleLicensePer: "periods"

With this release comes a third singleLicensePer mode (beside the default "init-data" mode and the "content" mode), called "periods".

The idea is to use the concept of a Period to provide a middle-road between a license for a single key ("init-data") and a license for the whole content ("content").
In the singleLicensePer: "periods" mode, a single license is linked to one or multiple Periods (e.g. one or multiple TV programs), allowing to still obtain multiple keys per license yet still allow other licenses to be fetched in the future (e.g. for a future TV program).

Image: Illustration of the player's behavior under singleLicensePer: "periods" mode. In this example, the license request for a hypothetical 1080p video quality of a Period "1", also returns keys for a Period "2". Then the player encounter a Period "3" for which it doesn't have any key, it thus performs the license request for it and get here just keys for the Period "3" (the number of Periods a license apply to in this mode can be variable from one to many).

In that mode, it is still possible for the license server to voluntarily withold some keys from the returned license. The main example being witholding the most sensitive keys when the device asking for the license is not trusted enough to properly handle them.
Moreover, in the "periods" mode, the number of Periods a license will apply to is not explicitely communicated, it only indicates that the license will apply to one or multiple Periods.

This creates an issue: The RxPlayer still needs to know at some point which keys have been witheld so it can fallback from the corresponding qualities, how to know them if it doesn't know how many Period the license is implicitely linked to?

Image: Example of a license in the singleLicensePer: "periods" mode. What about the key for 720p in the "Period 2", was it voluntarily not included or not ? More importantly, what about keys for an hypothetical Period 3, is that Period not decipherable or just not included?

To answer those problems, we decided on a behavior that should always be respected by the license server in a singleLicensePer: "periods" mode.
A license returned by a license server under that mode:

1. should contain all the compatible keys for the Period of the Representation for which the license request was done.

   That is, if the license request was done for a Representation in the second Period, the license fetched will be assumed to contain all compatible keys linked to the second Period.
   This means that all expected keys which are absent will be considered as not compatible - thus their corresponding Representation will be fallbacked from).

2. may contain either all compatible keys for any other Period(s) or none of them.

   The rule here is that as long as the license contain at least one decryption key linked to a Representation of any other Period, the RxPlayer will assume that the license server returned all compatible keys for that Period. Any other key linked to that Period but absent from the license will considered as not compatible - and thus their corresponding Representation will be fallbacked from.

Under those rules, the RxPlayer will always be able to tell which decryption keys are usable from which are not, allowing to still allow it to automatically fallback from undecipherable qualities.

Image: To take back the previous example, here it will mean that 720p from the Period 2 is not decipherable (because we have one other key from that Period), and knowing which Representations can be decrypted in the Period 3 might require a new license request, unless this license was returned when already asking for keys from that Period, in which case no encrypted Representation from that Period is decipherable.

If you're interested by this new license requesting mode, it is documented in the corresponding decryption option documentation page.

A "reload" audioTrackSwitchingMode

The audioTrackSwitchingMode option

We added the audioTrackSwitchingMode loadVideo option in the v3.22.0 (2020-11-17) to let applications choose the RxPlayer's behavior when switching the current audio track:

• by default, you would have the "seamless" mode, which will not cause any interruption but under which you might still hear the old audio track for a few seconds:

  seamless.mp4

  Video: behavior of the default "seamless" audioTrackSwitchingMode - don't forget to unmute!

• you could alternatively choose the "direct" mode, leading to a much direct audible switch but which may lead to a temporary interruption while the new audio track is loaded.

  direct.mp4

  Video: behavior of the "direct" audioTrackSwitchingMode

How "direct" is implemented

The first mode was the simplest one to technically implement. The second one however is more tricky, as we want to be able to remove the audio data that is currently being played yet browser support on doing just that is very iffy: playback might be stuck, sound might be lost etc.

We however found a strategy that worked in most situations in this "direct" mode:

1. We begin to remove all data from the previous audio track
2. We perform a very small seek of some milliseconds.
   This will have generally the effect of "flushing" lower-level buffers: the browser will now try to re-construct its buffer by reading again all audio data available.
3. Because audio data has been removed in the first step, the browser will realize that it has no audio data to play, it will thus pause and we'll enter a "BUFFERING" state.
4. The RxPlayer loads the media data corresponding to the new audio track. After it is loaded and pushed to the buffer, the browser is able to play again, we will thus play the new audio track and go out of the "BUFFERING" state.

This is generally what still happens behind the hood if you chose the "direct" audioTrackSwitchingMode when switching the currently-played audio track, and it works pretty well.

Solution for an issue with the "direct" mode: the "reload" mode

However we encountered recently some cases, with specific contents and only on Chrome, where the audio track could be lost in this operation. We're still trying to find out how to reproduce it and avoid this issue.

Yet, to still provide a last-resort solution if you encounter it, we decided to add a new "reload" audioTrackSwitchingMode, which is technically simpler and should cause less browser and content compatibility issues.
The downside is that instead of leading to a simple "BUFFERING" state when switching the audio track, you might encounter a "RELOADING" state instead, under which the video will become temporarily black and some API will not be usable.

What actually happens in that mode after switching the audio track is just:

1. We remove all buffers linked to the media element and create new ones: this is the "RELOADING" state
2. Once the new buffers are filled with media data again (with the new audio track), we can play and go out of the "RELOADING" state.

recording.mp4

Video: What happens visually in the new "reload" audioTrackSwitchingMode.

This option should be rarely needed. Only use it if you encounter issues in the "direct" mode (such as losing audio).

This mode is documented here, alongside the other audioTrackSwitchingMode modes.

Improved Safari support

During the development of the v3.27.0 release, we worked a lot toward improving Safari support both on web and mobile.

When talking media streaming, Safari is often the hardest of the main browsers to support, due to API missing and behaviors different than what we were used to.
Moreover, because many streaming actors usually prefer to play HLS contents on this platform, which the RxPlayer supports only indirectly (through the directfile transport), supporting more advanced features on it was not the priority for some time.

However this changed recently.
For example in several applications at Canal+, the RxPlayer is now the main player used on Safari even for complex contents such as encrypted HLS contents with multiple tracks and decryption keys.

This move made us realize what didn't work and encouraged us to fix it more quickly, for this release.

It includes multiple improvements and fixes specific to Safari, among which:

• We improved support of encrypted contents on Safari mobile, by relying on the vendored yet better-known behavior or the WebKitMediaKeys interface instead of the standard but less reliable MediaKeys interface

• When switching the audio track in directfile mode, all other audio tracks are disabled beforehand to fix audio issues encountered since MacOS Monterey

• The "wall-clock time", here corresponding to the live-offseted time for live contents, is now correctly calculated for "directfile" contents through the getWallClockTime method, the positionUpdate event and the optional wallClockTime property given to the seekTo method.

  This is mostly useful when playing HLS live contents in directfile mode on Safari.

Added TypeScript types

This release adds several TypeScript types to the exported types accessible through the rx-player/types path when importing the RxPlayer.

First it fixes the exported IManifestLoader and ISegmentLoader interfaces by adding to the first the url property and to the second the progress callback, both were forgotten when declaring those exported types.

But we also added:

• the IBitrateEstimate interface, corresponding to the payload of bitrateEstimationChange events
• the IPositionUpdate interface, for the positionUpdate events
• the IPlayerState type, which defines the state that can be obtained through the getPlayerState method and the playerStateChange event

All of those are documented in the exported types documentation.

DASH: priorization of the selectionPriority attribute over all others

When starting playback of a DASH content, the RxPlayer has to choose which initial AdaptationSets (for example which audio, video and text tracks) it will start with.
Because that type of choice is generally linked to user preference (like the preferred audio language), applications tend to rely on APIs such as the ones defined for track-preferences to guide the RxPlayer toward the right AdaptationSets to start with.

But this doesn't completely erase the problem: what should the player choose if no preference is set, or if several AdaptationSets are compatible with the given preferences?

The DASH-IF IOP, a document trying to standardize DASH packagers and players for better interoperability, tried to solve that question by providing several means to prioritize AdaptationSets between one another.
As such, an AdaptationSet can be anounced as being the "main" one and/or can have a selectionPriority attribute indicating that it should be preferred to another one when equal in other matters.

So far so good, yet all that complexity brought forward an ambiguity:
Recently an application asked to the RxPlayer to choose between a "main" audio AdaptationSet with a low selectionPriority and a "dubbed" (non-main) audio AdaptationSet with a higher selectionPriority.
What should it choose in that situation?

Image: Both a selectionPriority attribute and a "main" Role indication are present here, which should we select in priority?

The RxPlayer's previous behavior was to choose the "main" AdaptationSet first and the selectionPriority second, as it was our interpretation of the DASH-IF IOP on the subject.
However, this part of this document is very ambiguous and it wasn't the interpretation of the application developpers (those that needed to play the corresponding contents).

We thus tested the same stream on several media players to see what was the general behavior here. Sadly, different players also had different interpretations!

In the end, we chose to implement what seems to be more logical, albeit not what we understood when reading the DASH-IF IOP: Now, if multiple AdaptationSets are equally preferred, it will be selected by selectionPriority first.
The "role" (the attribute which may be set to "main"), is still used but is now only secondary, if the selectionPriority attribute still indicates a tie between AdaptationSets.

Before this release, "main" AdaptationSet were put first and selectionPriority second. This new release may thus change the behavior on content having both of those attributes.

dev and canal pre-releases

In last release's release note, we wrote that we were working toward improving the stability of the RxPlayer, especially to avoid regressions before a new release. We were at the time doing that by testing each new version for a time before the day of the official release.

We went further on this path while developping the v3.27.0 by generating multiple "pre-releases", which can be seen as beta releases with less stability guarantees.
Those are regularly published to npm through two tags:

• dev: Those are simple alpha/beta releases, which may also only include an incomplete feature set when compared to the official release
  It can be used by anyone, to either perform tests before the official release or to obtain new features without waiting for it.

• canal: Exactly like dev releases but also includes some work-arounds that should only be useful to Canal+.
  In particular, it contains non-standard-compliant safeguards in our subtitles parsing code, to avoid some issues we know we have with some contents played by Canal+ applications.
  It normally should only be useful to Canal+, though everybody can rely on canal releases if wanted.

Those two types of pre-releases are for the moment done on-demand, either:

• when a new feature is quickly needed by an application. We will there generate a dev or canal pre-release as soon as this feature is ready.

• or when an application wish to help us toward testing the RxPlayer before an official release.
  This is a win-win situation, as we'll detect and fix issues encountered quickly, more than if it was only seen just after the official release.

We're quite happy with this new strategy, yet we're still listening (and studying) about ways of improving the RxPlayer's releasing process.
If you want to provide inputs, you're very welcome to do it (through GitHub issues, for example)!

Changelog

Features

• Add a maxVideoBufferSize constructor option and {get,set}MaxVideoBufferSize methods to limit the size of loaded video data buffered at the same time [#1041, #1054]
• DRM: Add a "periods" mode to the keySystems[].singleLicensePer loadVideo option, allowing to obtain decryption license for groups of Periods allowing a compromise between optimization, features and compatibility [#1028, #1061]
• Add a "reload" audioTrackSwitchingMode to work-around rare compatibility issues when switching audio tracks [#1089]

Bug fixes

• subtitles: Fix rare issue where subtitles could be skipped due to a rounding error [#1064]
• DASH: fix issue where the wrong segments would be requested on $Number$-based MPD with a SegmentTimeline older than the timeShiftBufferDepth [#1052, #1060]
• directfile: disable all audio tracks before enabling one to work-around Safari issue on MacOS Monterey [#1067]
• avoid performing a small seek when changing the audio track [#1080]
• api: allow switching to RELOADING state synchronously after LOADED [#1083]
• Safari Mobile: Improve decryption support on Safari mobile by relying on the vendored WebKitMediaKeys API [#1072]
• DASH: Fix issue which prevented the integrity check of most MP4 DASH segments when transportOptions.checkMediaSegmentIntegrity was set to true
• avoid unnecessary warning logs when loading some initialization segments [#1049]
• TypeScript: Add forgotten TypeScript types in the exposed segment and manifest loader APIs [#1057]
• DRM: Avoid decryption issues when a license is persisted in a singleLicensePer "init-data" mode but loaded in a "content" mode [#1031, #1042]
• DRM: Totally avoid the theoretical possibility of leaking MediaKeySessions when a generateRequest or load call takes multiple seconds [#1093]

Other improvements

• DASH: always consider that the non-last Period is finished when it contains SegmentTimeline elements [#1047]
• add better buffer cleaning logic on a browser's QuotaExceededError to better handle memory limitations [#1065]
• DASH: Prioritize selectionPriority attribute over a "main" Role when ordering AdaptationSets [#1082]
• directfile/Safari: use the getStartDate method in getWallClockTime, seekTo and the positionUpdate event when available to obtain true offseted "wall-clock" times when playing HLS contents on Safari [#1055]
• DRM: Improve DRM Session caches performance when singleLicensePer is set to "content"
• DRM: Stop retrying closing MediaKeySessions multiple times when it fails, instead doing it only once when it should work [#1093]
• TypeScript: Add IBitrateEstimate, IPositionUpdate and IPlayerState types to the exported types [#1084]
• Remove dependency on pinkie's promise ponyfill [#1058, #1090]
• tests: add performance tests, to better catch and avoid performance regressions [#1053, #1062]
• DRM: Refactor DRM logic for better maintainability. DRM-linked logs are now prefixed by DRM: instead of EME: like previously [#1042]