v3.26.1

Release v3.26.1 (2021-09-14)

The v3.26.1 release has now been published.

It is a relatively small release which provides minor improvements and works around rare browser issues:

• We encountered very rare situations where the player would indefinitely freeze due to some contents and browser issues.
  We now work-around those by automatically "unfreezing" the player when it froze for a given time.
• Likewise a subtle browser issue could in rare scenario lead to media data being requested multiple times in a loop.
  The player now detects this situation and prevent that loop from happening
• We greatly improved the DASH_WASM experimental feature (which accelerates and improves the memory situation when parsing large DASH MPD by relying on a WebAssembly parser).
  Mostly its size and the time to compile it have been drastically reduced.
• More types are exported through the rx-player/types path.
  Those were types needed from already exported APIs, such as types about track preferences.
• Infinite rebuffering issues that could happen when some Periods in a multi-Period contents had no video content should now be fixed
• We continued our efforts to improve on initial loading time, by requesting what is called the initialization segment parallely to the first media segments when it makes sense.

As those are mostly very technical improvements, we will try to explain and dive into the details of what this release brings in the next chapters.
If this is too much details, you can jump as always to the changelog at the end of this release note!

Better handling of "frozen" playback

This release works around a very rare issue we infrequently encountered where the content appears to be indefinitely rebuffering.
This case is internally called "freezing".

How content playback basically works with the RxPlayer

To introduce what we call "frozen" playback, we need to introduce the basics of how content playback with the RxPlayer works.

First, the RxPlayer does not decode audio and video data itself, it relies on the browser (and underlying codecs) to do so.
What happens is that the player fetches the right audio and video "segments" (small decodable parts of the content) and indicates to the browser the position that should be played.
The browser and the codecs it integrates then ensure that the audio and video content corresponding to that position is decoded.

To ensure smooth playback, the player has to make sure that audio and video segments are "pushed" to the browser before the decoding operation reached their corresponding starting positions (or said another way: before we need one of them).
If the player couldn't load the segment before that deadline, the browser won't have anything to decode and will sort-of "freeze" until that data is pushed.

That behavior is also called "rebuffering" or more colloquially just "buffering".

Example of rebuffering. We can see, below the video, a Buffer content chart. It indicates that the current position (the red line in both buffers) is close to the end of the currently buffered audio and video data, we're thus awaiting new data before being able to play

"Freezing"

Yet in rare circumstances, that freezing situation can happen even when the right segments have been "pushed".
There can be legitimate reasons, like when a needed decryption key has not been obtained yet. Though there are times when there seems to be no apparent cause, at least from the point of view of the player.

In the RxPlayer team, we call that problematic case "freezing" (whereas regular rebuffering cases are just internally called "rebuffering").
Thankfully, this happens in very rare circumstances. We usually encounters unexplained stucked playback only in the following cases:

• poorly-packaged contents
• playing with a very high playback rate (i.e. at a high speed)
• when the player is doing risky behavior in some particular platforms - for example pushing a video segment containing data for the already-buffered current position on some old samsung TVs could lead to that situation.
  We however worked hard on this last category to avoid them in most cases.

Here we can see, that the content seems to be rebuffering despite having both audio and video data in front of the current position (the red lines in the Buffer content chart). Here this actually happened after I set a playback rate of 10 (meaning I'm playing the content at 10 times its regular speed) and waiting.

Even if very rare, it can happen and having a logic in last-resort would not be superfluous.

Going out of that frozen state back into regular playback is usually pretty easy.
In most scenarios, a simple little "seek" of 1 millisecond (which means going 1 millisecond in the past or in the future) is sufficient to "unfreeze" playback.
This is probably the case because seeking re-triggers some lower-level code which was linked to the source of freezing, e.g. low-level audio and video buffers could be emptied and filled again and the lower-level playback logic could be restarted from the new "seeked" position.

The new logic

So in definitive, the resolution of that issue is simple: if the player detects that "freezing" is happening, we just have to perform a little seek to unfreeze.

However, detecting when the player is abnormally frozen is not really straightforward.

The naive way would be just to look at two different point in time when playing with enough data, and see if the current position has evolved. If it did not, it must mean that we're stuck and - because we have data - playback must be freezing.

Yet there are multiple issues with doing just that:

1. Poor performance could lead to false positives.
   For example, we noticed that after a seek, some low-end devices can take up to multiple seconds to restart playback, even when enough data is available to play.
   By resolving freezes the naive way, the player thus might think it is frozen because a seek operation is taking a long time to finish. Here it might perform another seek, which will lengthen even more playback (we will also risk to be in an infinite seeking loop !).
2. There are sometimes good reasons for playback not restarting yet. The main one being that the decryption key for the incoming data is not yet obtained.

When considering all possibilities, this "freezing" state thus becomes very complicated to detect.
We ended-up relying on an imperfect yet sufficient solution:

• The player only performs the seek if playback is abnormally stuck after a very high delay (between 6 and 7 seconds).
  Being frozen for around 7 seconds sounds like a lot, but this logic should only run as a last resort anyway.
• A lot more playback characteristics than just the current position are looked at to deduce if we should be able to play.
• Already-existing and less risky "unfreezing" strategies will run before it.

On the same case of freeze while playing at a very high playback rate, we can see that this logic is finally able to un-stuck us.
Here is a case with that logic in action (with added logs):
https://user-images.githubusercontent.com/8694124/124958710-bf687e80-e01a-11eb-94a8-3c2ebaad7301.mp4

Removal of the cached segment detector

Web browsers maintain a somewhat complicated HTTP cache to avoid requesting multiple times already-fetched resources.

Thanks to it, a request made normally from JavaScript (through either an XMLHttpRequest object or a fetch call) will sometimes avoid the round-trip to the server. The main advantage of doing that is that it reduces latency.
All this happens without the knowing of the RxPlayer, for which the request still appear as if it was performed through the network.

Yet detecting when a response comes from the cache is sometimes needed by the player.
For example, the internal logic calculating the network bandwidth (to select the optimal video and audio qualities) relies heavily on the measure of the time it takes to download data from the network. When such data comes from the cache instead, it appears that the user has an exceptionally high bandwidth.

So until this release, the RxPlayer had what we called a Cached Segment Detector, whose job was to deduce heuristically whether data came from the browser cache or the network. It basically compared times to download such data and ignored all metrics which indicated an abnormally high bandwidth.

  -----------------
 | network request | 1. A network request metric is generated
 |     metric      |    after a request
  -----------------
         |                         2. "does it come from
         |                            the browser's cache?"   +----------------+
         +-> +------------------+ --------------------------> | Cached Segment |
             | network listener |               3. "yes"/"no" |    detector    |
             +------------------+ <-------------------------- +----------------+
                     | 4. If it doesn't seem to come from the cache
                     |    (and other rules)
                     |          +-------------------+
                     +--------> | Network bandwidth |
                                |     calculator    |
                                +-------------------+
                                        | 5. gives bandwidth
                                        ...

v3.26.0

Release v3.26.0 (2021-06-10)

The v3.26.0 release has now been published.

You might ask: where is the v3.25.0?
We had several issues while trying to publish it. Instead of releasing officially a v3.25.2 (as we had two issues while publishing), we guessed it might be more logical for an user as a new minor version and abandoning/deprecating the 3.25.x versions.

This release includes among other improvements:

• DASH trickmode tracks can now be handled in two ways:
  1. for trick play (e.g. playing at a faster rate a video track with a lesser framerate)
  2. for thumbnail generation (i.e. "preview thumbnails" backed by a video element, rendering frames taken from that trickmode track).

• HDR information is now parsed for DASH contents and is communicated through the corresponding video tracks APIs

• A new "experimental" (meaning stable but the API may change in future versions) DASH MPD parser, relying on WebAssembly.
  It has been seen to be 4 to 5 times faster than the regular JS MPD parser when handling very large MPDs (of several megabytes in size) based on large <SegmentTimeline> elements.

• The audioTrackSwitchingMode option of loadVideo, which allows to configure the player's behavior when changing the audio track, should now be able to handle the "direct" mode mostly without needing to reload the content.
  As it seems to provide much nicer and faster audio transitions than the default, we suggest that most applications set it (we do not set it by default because of a possibly API-breaking behavior change).

• The size of the cache of encryption sessions (call "MediaKeySession") is now completely configurable for when you know the limitations of the current device

• plain-text subtitles files (i.e. that are not embedded in a container file) are now better handled. In previous versions, relying on them could lead to the player keeping re-fetching them

DASH Trickmode handling

This new release allows to display trickmode tracks when available, anytime the application wants to.

What are trickmode tracks?

"Trickmode" tracks are specific video tracks which only show a subset of the frames of another video track.
You can see it as a copy of a video track, at a much lower frame rate.

The idea behind those is that a player can select those instead of the original one when playing in fast-forward/backward mode:

• trickmode tracks are generally in a much lower bitrate, which means less bandwidth is taken by playing those (even more when considering that the playback speed multiply that value)
• it is generally much easier to decode (even more when considering that only intra-frames are typically present), leading to less CPU usage, which could be a problem at higher speed
• it provides a different aesthetic, which might be preferred when fast-forwarding, for example

DASH contents allows to define trickmode track though its Manifest document, the MPD (as far as we know, HLS also supports those type of "I-Frames only" contents).

Preview

We made a quick video to show the result of enabling/disabling trickmode tracks on our demo page:
https://user-images.githubusercontent.com/8694124/121335185-a769e600-c91a-11eb-9bee-7bb81d727971.mp4

Here, we update the playback rate to x10 and switch to trickmode tracks, then we return to a regular speed and disable trickmode tracks.
More information on the API and behavior below.

You can notice a visible black screen when enabling/disabling the trickmode track. This is because the RxPlayer is currently switching to the "RELOADING" state when this happens.
This can be a annoyance. We plan however to remove that step in future versions.

How it is handled by the RxPlayer

As described in the previous chapter, trickmode tracks are most-of-all useful when playing a content at faster speeds.

We began its implementation by introducing new APIs and events specific to this use-case, but finally found out that it was too complex and too hard for an application to manage.
We thus ended up just relying on already-existing APIs.

Enabling / Disabling

To enable or disable trickmode tracks, a new optional object argument has been added to setPlaybackRate:

// Play at x5 the regular speed and enable trick mode tracks if available in the
// current content/Period
rxPlayer.setPlaybackRate(5, { preferTrickModeTracks: true });

// Not setting `preferTrickModeTracks` doesn't change the value it was set to
// previously.
// Here we stay with trickmode tracks enabled and just update the speed.
rxPlayer.setPlaybackRate(3);

// Play at regular speed and disable trick mode tracks
rxPlayer.setPlaybackRate(1, { preferTrickModeTracks: false });

As you can see, setting the preferTrickModeTracks property on that object allows to enable or disable trickmode tracks.
Moreover, this setting can be set when no content is playing and is persisted to the current instance of the RxPlayer (exactly like the playback rate is), so if you want to disable it when switching content, you will have to call stPlaybackRate again, with preferTrickModeTracks set to false.

You can also know at any time if that option is active through a new RxPlayer method, areTrickModeTracksEnabled:

const areEnabled = rxPlayer.areTrickModeTracksEnabled();
if (areEnabled) {
  console.log("Trickmode tracks will be used when available.");
} else {
  console.log("Trickmode tracks are not enabled.");
}

Note that for now, enabling or disabling trickmode tracks may temporarily show a black screen while the player is in a RELOADING state. We might improve on that in future releases however, to avoid reloading at all in most cases.

Events

To know WHEN a video trickmode track is enabled and to know when it is disabled, we again chose to rely on the already-existing videoTrackChange event.

On trickmode video tracks a new isTrickModeTrack property will be set to true. You can rely on it to know if the currently chosen video track is a trickmode track:

rxPlayer.addEventListener("videoTrackChange", (videoTrack) => {
  if (videoTrack.isTrickModeTrack === true) {
    console.log("Now switching to a trickmode track");
  } else {
    console.log("Now switching to a regular video track");
  }
});

Likewise, getVideoTrack will also have the same property (getAvailableVideoTracks, however, won't list trickmode tracks, as those cannot be switched to through the setVideoTrack method).

On all APIs returning video tracks (videoTrackChange/getVideoTrack/getAvailableVideoTracks) another optional property has also been added, trickModeTracks.
If set to an array, it contains the trickmode track(s) linked to this "regular" video track. You can use this for example to know when a track is linked to one or several trickmode tracks.

If no trickmode tracks are available

We chose the word prefer in preferTrickModeTracks on purpose because the played content might not have any trickmode track.

In that case, the regular video track will continue to play. This is also valid in DASH multi-Period contents where some Period do have trickmode and others do not.
Here, the trickmode track will be used when available and if not, the regular video track will be played instead.

Documentation

As usual, the documentation of the corresponding API have been updated:

• documentation for setPlaybackRate
• documentation for areTrickModeTracksEnabled
• documentation for the videoTrackChange event
• documentation for getVideoTrack
• documentation for getAvailableVideoTracks

Generating thumbnails from DASH trickmode tracks

Another possible use-case for trickmode tracks, is to use them for generating "preview thumbnails", for example while hovering the seek bar.

This release adds the experimental[1] tool, the VideoThumbnailLoader.

Here is an example of how you can use it:

import RxPlayer from "rx-player";

// import the VideoThumbnailLoader and the `DASH_LOADER` to handle DASH
// contents.
import VideoThumbnailLoader, {
  DASH_LOADER
} from "rx-player/experimental/tools/VideoThumbnailLoader";

const player = new RxPlayer({ /* some options */ });

// Link logic to handle DASH segments
VideoThumbnailLoader.addLoader(DASH_LOADER);

// Video element used to display thumbnails.
const thumbnailVideoElement = document.createElement("video");

// Link VideoThumbnailLoader to the RxPlayer instance
const videoThumbnailLoader = new VideoThumbnailLoader(
  thumbnailVideoElement,
  player
);

player.loadVideo({ /* some options */ });

// Ask for the VideoThumbnailLoader to fetch a thumbnail for the current
// content that should be displayed at presentation time = 200 seconds.
videoThumbnailLoader.setTime(200);

Because the VideoThumbnailLoader uses video segments to display thumbnails, it has to rely on an HTML video element, and not an <image> element like one would probably expect.

We tried hard to make the API as easy to use as possible, you can see its documentation here.

To show you a preview of how it can look like, here is an example now available in our demo page:
https://user-images.githubusercontent.com...

v3.24.0

Release v3.24.0 (2021-04-01)

(No, it's not an april fools joke, unless it doesn't work, in which case we will pretend it was)

A new release is here! Amongst the new features and improvements:

• a new inbandEvent event is now triggered when event metadata is encountered in a segment (for example through "emsg" boxes in an ISOBMFF segment), just before that segment is pushed

• a new singleLicensePer option now allows to optimize the loading of contents with multiple decryption keys, to be able to perform only a single license request.

• we improved a lot the compatibility of the RxPlayer with several devices, most importantly many Tizen versions (e.g. Samsung TVs) and the PlayStation 4 (as to the PlayStation 5, it should already be handled).

• multiple optimizations have been added to allow lower content loading times

• many other bug fixes and improvements (you can see the changelog at the bottom of this release note)

A new player event: inbandEvent

"Inband" events

Streaming formats specify different ways to emit events through a content.
Those events can be sent at any time when streaming the content, and can be used to signal generic or format-specific information to a player or the application (e.g. signaling about parental ratings on the current content or about the Manifest's validity).

The RxPlayer already supported one kind of event: the MPEG-DASH manifest EventStream node, allowing an event to be declared directly in the Manifest.
But instead of only having events in the manifest, those can also be found inside media segments.

MPEG-DASH uses such format, by exploiting the "emsg" box of the ISOBMFF standard. This box is included in the metadata part of ISOBMFF segments (as in: it is not included in the media data itself) and is usually related to the media presentation time.

Those type of events, which are included directly in the segments, are usually called "inband events".

In the RxPlayer: inbandEvents event

The RxPlayer now parses these boxes (only the version 0 of it for now) and :

• Either the event has a specific meaning for a player (generally to give information
 about the manifest validity, allowing the player to consider when it may refresh the manifest) in which case it will handled it by itself.
• Either, the event is generic, or specific but not yet handled by the player. In which case it will be parsed and emitted 
by the RxPlayer through an event.

This new event is called "inbandEvents".
It emits one or more parsed inband events, as several "emsg" boxes may be found when parsing a unique media segment.

The "inbandEvents" event is emitted just after the corresponding segment is parsed, unlike the already-existing "streamEvent", which is sent at the presentation time of the concerned event.

And as an "emsg" box is associated with a specific segment, the player send it at the moment it gets it, before that segment is actually pushed.
This is important as it may be used to signal that the current segment contains content that may be subject to rating restrictions.
In that case, the client may need to interrupt playback before the segment's data is buffered.

This new event and its format is documented here.

singleLicensePer option: single license request for contents with multiple keys

Previous situation

When playing encrypted contents, the RxPlayer has a simple logic to know when it needs to perform a licence request:

1. Let's say that the player switches to a new Representation/quality
2. It looks at that quality's linked encryption initialization data if it exists:
   • if it was already encountered, it does nothing in particular with it
   • if it wasn't yet encountered, it goes to the next step
3. The player creates a new decryption session (called MediaKeySession) associated to that data and ask that session to generate a license request with it
4. The browser sends the player an event back with a "challenge" allowing it to perform a license request by calling the getLicense callback
5. Your application fetches the license and give it to the RxPlayer, which then pass it down through the MediaKeySession.update API
6. The RxPlayer then looks at which keys were contained in that license and reacts accordingly (it might fallback for unsupported qualities etc.)

This works pretty well, but it generally leads to multiple license requests when a content has different keys depending on the quality chosen.

This is because Representations linked to different keys will generally have different initialization data. The RxPlayer will thus create a new MediaKeySession and perform a new license request for each of them.

Optimization: performing only a single license request

We've looked at possible ways to optimize this logic. The most evident solution is to only perform a single license request.

Yet that would mean that the license returned contains all needed keys.
As such, this optimization relies on two things:

1. the license server has to be aware of this "trick" and return all keys even when a single one is asked for
2. the RxPlayer has to only perform a single license request.
   It could still technically perform multiple ones but this would be pointless as the first one already gave everything needed.

For that second part, we added the new singleLicensePer properties to the keySystems option of the loadVideo API.
By setting it to "content" (more possible values might come in the future), you can signal that there's only a single license available for the whole content.

It can be used as such:

rxPlayer.loadVideo({
  // ...
  keySystems: [{
    // ...
    singleLicensePer: "content",
  }]
}

This option is documented inside the keySystems documentation.

Remaining problem: what about missing keys?

But if the RxPlayer only decided to perform a single license request and did not change anything else in its behavior, it might lead to an other issue: what should happen if one of the keys is not in the license?

This might happen when the CDM, the module plugged in the browser allowing content decryption, is not trusted enough by the license server, resulting in the latter withholding some keys (the more restrictive ones).

The decision we've made was that, in the case an awaited decryption key isn't present in the pushed license, the player will automatically decide to mark the corresponding Representations as "undecipherable".

In this situation, those qualities won't be played. If segments corresponding to that quality were already pushed (we load media segments in parallel of the license request as an optimization measure), the RxPlayer will first try to remove them and might in some very rare conditions switch to the "RELOADING"state while doing so.

Improved compatibility with several platforms

At Canal+ Group, we recently had a wave of new platforms on which the RxPlayer was ported (I'll just name-drop the PlayStation 5, PlayStation 4 and Samsung TVs but there's others).
This was the occasion to improve our records on portability.

The main issues we encounter when porting the RxPlayer are subtle browser behavior differences, sometimes still specification-compliant and sometimes not, most of them being related to their respective MSE/EME implementations.

There was a lot of changes on our side (and multiple times on the implementer's side! We don't always want to fix another application's mess :p).

Most notably, they're fixing:

• A mysterious infinite loading situation when beginning a live content on some platforms (amongst which, Samsung TVs), most likely due to an overflow on the browser-side after setting the duration (#914)
• A situation where both the browser and the RxPlayer wanted to seek at different times, leading to a conflict (#922)
• decoding issues we encountered when pushing segments on top of the current position (#925)
• A leak of MediaKeySession on platforms where the generateRequest API is especially long (#920)
• Persistent MediaKeySessions being thrown away because too much time passed before the player was aware of its linked keys (#928, #926)

Loading time improvements

In this release, we also began (and still continuing) trying to heavily optimize the initial content loading time:

• One of the main improvements is that now when calling loadVideo when a content was already playing, the new content's manifest will be loaded in parallel of the old content being stopped (implementation details: #894)

• Another one, only for encrypted contents, is to depend only on the encryption initialization data in the Manifest if it already contains the right one (instead of the default behavior, which waits for a segment to be fetched first).

  This optimization was a complex one to implement because it implied recognizing which encryption initialization data is needed, whereas the previous implementation just used all available data - a solution which always work (implementation details: #911, #919).

  If you rely on persistent licenses, this optimization won't be available to you by default. You will need to set the disableRetroCompatibility property to true in the licenseStorage option (itself part of the keySystems `loadVi...

v3.23.1

Release v3.23.1 (2021-02-01)

This very small release only fixes a single issue, brought with the v3.23.0 release published this same day, which made it impossible to play encrypted contents on Safari.

Sorry about that, we normally test that specific use case, but we didn't after a quick code update* we did just before releasing the v3.23.0.

We will try to be more careful from now on and always test that use case before new releases, no matter how small the update!

* For the more curious readers, the incriminating update we're speaking of here was about throwing better errors when playing encrypted contents on platforms which do not support decryption : ed282ee. While writing that, I didn't realize that the else condition I was replacing was for a much more general case than what I replaced it with.

Changelog

Bug fixes

• Fix support of encrypted contents on Safari (v3.23.0 regression)

v3.23.0

Release v3.23.0 (2021-02-01)

This release adds multiple new methods and options to the RxPlayer, but also many other improvements.

Most importantly:

• A reload method has been added, to be able to manually quickly re-load a content after something when wrong (or even when nothing went wrong at all)

• a whole new discontinuity management system has been written, which allows the player to automatically detect and step over most "holes" in a content, even when those were not declared as such in the Manifest.
  Those type of holes are most often encountered between Periods in multi-Period DASH contents but can also be encountered in poorly encoded/packaged contents.

• A onCodecSwitch loadVideo option has been added, so you can ask the player to reload automatically when switching between incompatible video codecs or audio codecs, which you might want to do on some devices/browsers.

• multiple methods and options have been added to indicate a minimum audio and/or video bitrate to which the player should switch to when adaptive streaming is enabled

• Fallbacking to another Representation due to an in-exploitable decryption key (only when the keySystems[].fallbackOn loadVideo option is set) could be non-functional on some devices since the v3.21.1. This is now fixed.

A new reload method


We’ve noticed that there are some cases where an application wants to load again the last loaded content.
Some of these reasons could be:

• A fatal error occurred due to exceptional reasons, and thus may not reproduce after reloading the content.
• A decoding error occurred at a given playback time (e.g. the corresponding media chunk is defective), and provoked a crash. Here we can re-load while "skipping" the faulty segment.

For now, if an application developer wants to reload the exact same content, even with the same options than before, he/she needs to call the loadVideo API again. Then, the loading process will be decomposed into several steps, some of which are time-consuming: loading the manifest, communicating with the media browser APIs, etc.

This implies that after re-loading a content, all these steps will be taken for a second time. In some cases this is completely unnecessary, as fetched or parsed resources may not change from the first loadVideo to the next.

For now, we have identified one major step that can be avoided: fetching the manifest and turn it into our internal manifest format.
This operation can take a certain amount of time, which could be removed here by using the previously loaded manifest.

Thus, we’ve introduced a new API, the reload method.

It can be called at any moment, after the LOADED state for the last loadVideo has been received, even if the content is still playing.
The aim of the API is to propose to reload the last loaded content as fast as possible, by default at the last playback position it was set to.

The API options allow setting an absolute or relative (to the last playback position) position to reload playback:

// Reload the content at the last position it was set to
rxPlayer.reload();

// Reload video at a specific position
rxPlayer.reload({ reloadAt: { position: 55 } });

// Reload 5 seconds AFTER the last position it was set to
rxPlayer.reload({ reloadAt: { relative: +5 } });

// Reload 5 seconds BEFORE the last position it was set to
rxPlayer.reload({ reloadAt: { relative: -5 } });

The reload method is documented here.

Improved discontinuity management

This release contains a completely revamped discontinuity management logic.

A "discontinuity" is the name we give to holes in a media content.
For example, we could have a live content with an imperfect transition between ads and the show coming just after, where a few seconds of video would be missing.
Here the player has to do something - most likely step over that hole - to avoid getting stuck on it.

Most discontinuities were already handled in previous versions, but there were still issues we had - most often in contents concatenating multiple sub-contents (multi-Period DASH contents, "ad-switched" contents, MetaPlaylists...).

Previous discontinuity management logic

In previous versions of the RxPlayer, two types of discontinuities where handled, through a very simple logic.

When the player was stuck, it basically asked itself two questions:

• is there a very small (less than 1 second) hole at the current position? -> If yes, seek over it.
• does the Manifest tells us that there's no segment there? -> If yes, seek at the start of the next segment.

Those worked with most type of discontinuities, which are often either very small or clearly announced in the Manifest.
Yet it still had important drawbacks:

1. It was possible to have false positives: we could be stepping over what we thought to be small holes but which were just yet-to-be-downloaded segments

2. For discontinuities not announced in the Manifest specifically, the size of the "hole" had to be known before being able to seek over it.
   This meant we had first to load and push the first segment coming after that discontinuity.

   This could lead to a deadlock when loading the next segment is not yet possible.

3. Discontinuities defined in the Manifest might not reflect the exact reality once the segments are pushed: discontinuities might actually be bigger or smaller than initially announced

Solution: changing the whole logic

So we completely scrapped what we did before and thought about what the optimal way of handling discontinuities should be.

The answer we came with was the following:

1. The RxPlayer can reliably know which "holes" in the audio and/or video buffers won't ever be filled by any segment. When such a situation is encountered, it is signaled internally.

2. When one of this hole is finally encountered, playback may be stuck.
   When that happens, the player will cross-reference the position at which it is stuck with the currently signaled "holes". When a hole corresponds to that position, it automatically seeks over it.

This should completely eliminate the risk of false positives and handle almost all discontinuities whether they were clearly defined in the Manifest or not.

That new logic also comes with a smarter handling of boundaries between multiple Periods, which is where most discontinuities are encountered.

A new DISCONTINUITY_ENCOUNTERED event

Because an application might want to be be notified when discontinuities are seeked over, we also added a new warning event, emitted when this happens.
Emitted with that warning event, will be a MEDIA_ERROR instance with the "DISCONTINUITY_ENCOUNTERED" code.

You can thus know when a discontinuity has been seeked over by writing:

rxPlayer.addEventListener("warning", function (warning) {
  if (warning.code === "DISCONTINUITY_ENCOUNTERED") {
    console.log("A discontinuity has been encountered:", warning);
  }
});

Codec switching behavior

It is sometimes possible to encounter multiple different audio and/or video codecs in the same content.

For example, you might have a high-quality HEVC video track and a lower-quality AVC video track. You might also have a multi-Period DASH contents, where some Periods contain a Dolby Digital Plus audio track but others do not.

As a consequence, the RxPlayer could need to transition between two completely different codecs.
For example, the application could chose to select an HEVC video track while it was playing an AVC one, or a codec switch could just occur while entering a new Period.

The previous strategy when doing that was to just concatenate both type of segments on the same buffer while using the changeType API if available.
We found that strategy to be unreliable: depending on the device and browser, switching the codec that way could either work or totally fail. We have multiple examples of problem we encountered when doing just that on some devices: decoding error, error while appending and infinite rebuffering issues.

Another strategy would be to just reload the content any time an audio or video codec switch is encountered.
This should work for all platforms and browsers, but it means that for a small period of time, the player will go through the RELOADING state, during which the screen becomes black and most APIs are not usable. This leads to a much less pleasant experience.

From those elements, we thought that the best possible solution was to choose the first strategy (just continue pushing) when it worked and use the second one (reload) when it didn't.
We first planned to automatically detect the right strategy to take in the player (e.g. based on the presence or not of the changeType API) but we found out that doing this is very difficult and may not even be possible.

In the end, the solution we propose in this release is to let the application select the correct strategy through a new option.
An application should have a better idea of supported platforms and browsers, and the behavior when switching codecs can generally easily be checked.

This takes the form of a new loadVideo option, called onCodecSwitch, which is documented here.
It can take two values: either "continue" which is the default non-disruptive strategy (which may not work for all targets) we already have, or "reload", the disruptive strategy known to work for all targets:

// Reload every time a new codec is enc...

v3.22.0

Release v3.22.0 (2020-11-17)

Overview

This release adds multiple features and improvements:

• add an audioTrackSwitchingMode property to loadVideo to select the RxPlayer's behavior when switching the audio track (to choose between smoothness or instantaneity)
• add the initialManifest loadVideo option to avoid loading the Manifest when not necessary
• add an enableFastSwitching loadVideo option to enable or disable optimizations doing segment replacement in the browser's buffer
• the TextTrackRenderer tool is not experimental anymore. It can be used without fearing that its API will change
• A new version of the experimental local Manifest which allows much more advanced behavior, such as playing a content which is still downloading.
• A fix for an issue which made it impossible to play encrypted contents when in directfile mode
• saner defaults for TTML subtitles (white, centered text)

audioTrackSwitchingMode: behavior when switching between audio tracks

Until now when switching to a different audio track, the RxPlayer tried to make the transition as smooth as possible.

It doesn't interrupt playback of the old track, but load in parallel the new track and once done, replace the old by the new - most of the time without needing to pause playback.
Because this means that the user would still play in the old track for some time, a time limit (now of 2.5 seconds) is set from the point when the user chose a different track. If he/she has played until that limit, the player will enter a BUFFERING phase until the new track is available.

This behavior allows a very smooth experience but has two drawbacks:

1. It means that the switch won't happen right away.
   For example when switching between languages, you might prefer to interrupt the track in the previous language to play the right one at the same position you were in when you switched, even if it means interrupting playback for a while.

2. On some peculiar devices, multiple issues could arise when replacing those tracks smoothly. Sometimes it is not taken into account, sometime it doesn't even work.
   Note that we're not speaking of the usual web targets here but only of a minority of embedded devices.

Thus, we now let the possibility to adopt different strategies depending on how the audioTrackSwitchingMode API is set:

• Either, "seamless", which is the default behavior, where the transition will appear smooth and will not interrupt playback but could take time to be effective.
• Or, "direct", it will be much more instantaneous but it is possible that the player will go in "RELOADING" state in the meantime and so, may interrupt playback during a brief time.
  Note that we try to avoid going into the "RELOADING" state. We only do it when it is really necessary.

This property is a loadVideo option documented here.

For example to go into a more "direct" track switching, you can do:

rxPlayer.loadVideo({
  // ...
  audioTrackSwitchingMode: "direct",
});

initialManifest: provide the Manifest file if already downloaded

At Canal+, we encountered situations where the application calling the RxPlayer needed to process data from the Manifest before loading a video.

As such, the Manifest would be loaded twice. Once by the application beforehand and then by the RxPlayer.
This mean that we would perform an unnecessary request which would be both a waste of time and resources.

To load faster when the Manifest is already available, we consequently just added the initialManifest option to the transportOptions of loadVideo:

rxPlayer.loadVideo({
  // ...
  transportOptions: {
    initialManifest: myManifest,
    // ...
  }
});

As using it has some implications for live contents, we advise to first read the corresponding API documentation, in the transportOptions before using it.

enableFastSwitching: avoid segment replacement on some targets

The RxPlayer always try to play the best possible quality by default.

For example, when you begin to play a content at a low quality (because of poor bandwidth or because this is the initial content) but the bandwidth indicates that we can now download segments of a much higher quality, the player will try to directly replace the low-quality segments it has in its buffer by the higher quality segments it can download.

This lead to a much faster visible change in quality than if the player only loaded and pushed segments coming after those already loaded.

This optimization - replacing low-quality segments with higher quality-segments for raising faster in quality - has been integrated in the player for multiple years, but was given a name only recently: "fast-switching" (name admittedly taken from another player, dash.js as they developped a similar optimization).

By default, "fast-switching" is enabled and has been for a long time. But it's now possible to disable it through a new loadVideo option: enableFastSwitching.
By setting it to false, the player will just push new segments at the end of the buffer, regardless of the quality they are in.

The main reason you would want to disable that feature, and the main reason behind making that option configurable, is that some specific environments react very poorly when segments already in the buffer are replaced.
For example, we noted that some rare environments do not consider the new segments (making that optimization useless). We also worked recently on an environment which reacted very poorly to this optimization, leading sometimes to the video "freezing".
This was for a very specific environment which was still in heavy development and they since fixed that issue, but it was the main trigger behind making this behavior configurable, as anyone could encounter the same situation on a new device.

So you should probably not put it to false (even if there's no problem in doing so beside slower change in quality) but it can be useful when you know what you're doing.
You can consider this as an "expert" option!

This option is documented here.
It is set to true by default. To disable fast-switching, you can thus do:

rxPlayer.loadVideo({
  // ...
  enableFastSwitching: false,
})

The TextTrackRenderer is not experimental anymore

The TextTrackRenderer tool, which was added in the v3.18.0 version of the RxPlayer, allows to show any subtitles in the supported formats on top of a media element (it both parses and synchronizes them), even when the content is not played through the RxPlayer.

Until now, the API was marked as "experimental" which meant that its API could change at any time.

It now has a stable API, meaning it's not experimental anymore!

Note that if you already used it, you now have an action on your side to be compatible to the v3.22.0.
When before you imported the TextTrackRenderer as such:

import TextTrackRenderer, {
  TTML_PARSER,
  VTT_PARSER,
  SRT_PARSER,
  SAMI_PARSER,
} from "rx-player/experimental/tools/TextTrackRenderer";

You now have to get rid of the /experimental tp instead write:

import TextTrackRenderer, {
  TTML_PARSER,
  VTT_PARSER,
  SRT_PARSER,
  SAMI_PARSER,
} from "rx-player/tools/TextTrackRenderer";

As usual, its API documentation is available here.

Default TTML style

We encountered an issue where a user of the RxPlayer library basically told us that we badly formatted TTML subtitles, he couldn't read them well as they were in a black color and they were displayed on the top left of the screen.

After some investigations, we found out that we had no big problem with TTML formatting but that the real issue is that we do not make styling decisions if no style is present in the TTML file itself (and generally, when you are using TTML subtitles, the style is included!).
This means that by default, only the already-present styles of the concerned HTML elements are considered.
On most cases, this means black and on the top left of the screen!

Resolving that issue was not straightforward, because no default style seems to be defined in the TTML specification.
So we weren't sure of what to do here.

After looking at what other players were doing and at what the most sensible thing to do in that situation would be, we now consider that if no style at all is declared in a TTML file, we should apply a default style at its place.
Therefore, if no styles are defined for regions in the TTML, but other styles are (e.g. on HTML elements), we decide not to apply the default style. Indeed, we consider the style to have been defined on purpose, and we may interfere with the overall look of the cue if we apply a different style.

That default style objective is to display text cues in the clearer and cleaner possible way: at the bottom, centered and with white text

serverCertificate optimizations

The serverCertificate property of the keySystems option (in loadVideo) allows to communicate a server certificate.
This is DRM-specific data usually needed to encrypt message between a Content Decryption Module (or CDM) and a license server.

It appears that setting that certificate can take some time: up to 1 second on some embedded devices.
We thus decided to allow a new optimization: when we know a serverCertificate is alr...

v3.21.1

Release v3.21.1 (2020-09-21)

Overview

The v3.21.1 adds many small bug fixes and improvements:

• we fixed an issue that made it difficult to switch between key systems (e.g. from Widevine to PlayReady) on devices and browsers that support it
• we improved the support of DASH MPDs, in particular we now support contents with multiple levels of SegmentTemplate declaration
• we fixed some issues with the manifestUpdateUrl API, which allows to optimize the performance when playing very long Manifest files.
• we brought several small improvements to reduce the delay at which a track is switched and seek are performed and to improve the experience when switching quality
• we updated our adaptive logic so it favours even more smooth playback over maximizing the quality
• many other things, see our changelog on the bottom of this release note for more information (now with links on related PRs and issues!)

Switching between key systems

To be able to play encrypted contents on the browser, a CDM - or Content Decryption Module is used.

That module is usually an external software, plugged to the current browser, whose task will be to do what's needed to decrypt the content.
They are provided by DRM technologies whose names you have probably heard of if you play encrypted contents: Widevine, PlayReady, FairPlay etc.

 How encrypted contents are decrypted via the RxPlayer (simplification):

     +===================+
     |    Application    | Provide wanted DRM configuration(s)
     +===================+ and the logic to fetch a licence
              |
              V
        +----------+
        | RxPlayer | Try to open a session with a CDM having the
        +----------+ wanted configuration.
              |      Once opened, fetch and give the licence - allowing
              |      to play the content - to the CDM.
              V
  +-------------------------+
  |       Web browser       | Provide APIs to interact with CDM(s)
  +-------------------------+
              |
              V
           +-----+
           | CDM | Decrypt the content
           +-----+

Because multiple CDMs exist and because they might have different capabilities depending on the device on which the content plays (for example, devices made for video streaming might have more copy-protection mechanisms), not all browsers or devices use the same ones.

For the most part, we're stuck with only one possibility. For example most Chrome and Firefox builds on a desktop/laptop computer rely on the Widevine CDM and no other.
But other situations exist where multiple CDM can be available at the same time. For example the new Edge browser on desktop will usually allow us to interact either with PlayReady or Widevine. That exact situation is also found on other environments, like on recent Samsung TVs.

  +-------------------------+
  |       Edge Browser      |
  +-------------------------+
       /            \
      /              \
     |                |
     V                V
 +--------+      +---------+
 |  CDM   |      |   CDM   |
 |Widevine|      |PlayReady|
 +--------+      +---------+

The RxPlayer API let you choose which CDM you want by specifying what it calls a "key system": you can play a content with a Widevine CDM by claiming that you want the "widevine" key system.

All that work well as long as the corresponding key system is available. However, there was one situation where the RxPlayer was not so much reliable: when switching from one key system to another one, with the same RxPlayer instance.

Some browsers unexpectedly failed when "switching" the key system from one to another. After some initial tests, we thought that the situation was unsolvable: you were stuck with a key system once you've chosen one on some devices.

                 +-------------------------+
                 |         Browser         |
                 +-------------------------+
                      /            x
                     /              x
                    |                x
                    V                V
                +--------+      +---------+
                |  CDM   |      |   CDM   |
 CDM chosen  -->|Widevine|      |PlayReady|<-- We can't use that
 to play the    +--------+      +---------+    one anymore
 first encrypted
 content

However, we finally found out that by re-ordering some API calls, such error disappeared.
This is not the first time where we find that re-ordering DRM-related API calls unexpectedly unblock some situations on some devices. Because of this, we now follow a very particular sequence of calls that we know to work on all our tested platforms.

The result is that you can now reliably choose a different key system depending on the content played.

An in-depth explanation of the solution might rely on multiple technical terms (most of all MediaSource and MediaKeys) so I wont explain it in that release note. If you're still curious about it (and know what those terms roughly mean), you can look at the corresponding issue (#766) and PR (#744).

Reducing the delay of track switching and some seek operations

One of the main task of the RxPlayer is to schedule the right media data requests at the right time.

The basic behavior behind it is to download in parallel the next most needed audio, video and text (if subtitles are enabled) media "segments" (which are basically small decodable chunks of the whole media data).
When a segment is loaded, it is stored in a buffer so it can be decoded later (as soon as we need it). At the same time, we begin the request for the next most needed audio, video or text segment so we can maximize the bandwidth usage and provide smooth playback with the best possible quality.

 Queues of segment scheduled for download

     AUDIO QUEUE           VIDEO QUEUE          TEXT QUEUE
 +-----------------+   +-----------------+  +-----------------+
 | audio segment 1 |   | video segment 1 |  |  text segment 1 |
 +-----------------+   +-----------------+  +-----------------+
          |                     |                    |
          V                     V                    V
 +-----------------+   +-----------------+  +-----------------+
 | audio segment 2 |   | video segment 2 |  |  text segment 2 |
 +-----------------+   +-----------------+  +-----------------+
          |                     |                    |
          V                     V                    V
 +-----------------+   +-----------------+  +-----------------+
 | audio segment 3 |   | video segment 3 |  |  text segment 3 |
 +-----------------+   +-----------------+  +-----------------+
          |                     |                    |
          V                     V                    V
         ...                   ...                  ...          

However if we did just that, we could quickly encounter a problem. For example, audio segments are generally much smaller in size than video segments, and as such they are loaded much more rapidly. If we loaded the next audio segment as soon as the previous one's request ended, we would be limiting the bandwidth for the more urgent video segment request happening in parallel.

                      AUDIO QUEUE           VIDEO QUEUE     
                  +-----------------+   +-----------------+
distant, not  --> | audio segment 7 |   | video segment 1 | <-- close, really
urgent segment    +-----------------+   +-----------------+     urgent segment
                           |                     |            
                           V                     V            
                  +-----------------+   +-----------------+
                  | audio segment 8 |   | video segment 2 |
                  +-----------------+   +-----------------+
                           |                     |            
                           V                     V            
                  +-----------------+   +-----------------+
                  | audio segment 9 |   | video segment 3 |
                  +-----------------+   +-----------------+  
                           |                     |            
                           V                     V            
                          ...                   ...                                     

To solve this issue while still making the best use of the user's bandwidth, the RxPlayer make a compromise.

First, it assigns to each segment a priority.
When a media segment is loaded, the RxPlayer will first check the priority of the next most needed segment for that type (e.g. "audio"). It will only start a request for that segment if/when its priority is at least on par with the priority of the most needed segments for the other types. This sounds complicated, but it allows to download urgent segments with all available bandwidth while still being able to parallelize segment requests once urgent segments are all downloaded.

    AUDIO QUEUE           VIDEO QUEUE     
                      +-----------------+
                      | video segment 1 |
                      | (high priority) |
   Waiting until      +-----------------+
  higher priority              |            
  video segments               V            
    are loaded        +-----------------+
                      | video segment 2 |
       ...            | (high priority) |
                      +-----------------+
                               |          
                               V          
+-----------------+   +-----------------+
| audio segment 7 |   | video segment 3 |
| (lower priority)|   | (lower priority)|
+-----------------+   +-----------------+
         |                     |         
         V                     V         
        ...           ...

v3.21.0

Release v3.21.0 (2020-06-17)

Overview

The v3.21.0 brings multiple features amongst which:

• the RxPlayer now handle DASH EventStream's Event elements to emit timed events defined in a DASH MPD
• it is now possible to apply track preferences to the content(s) / Period(s) that are already playing.
• WebVTT "settings" are now considered when parsing subtitles in the html textTrackMode, thanks to @mattiaspalmgren for the contribution! Before, we only relied on WebVTT's CSS extensions to allow stylization of subtitles in this mode.
• the RxPlayer will now emit "warning" events when it detects minor errors in a DASH MPD, such as optional data in the wrong format.
• and a lot of other bug fixes and improvements - you can look at the changelog for more information.

Support of DASH EventStream's Event elements

What stream events are for

What we call "stream events" here is metadata synchronized with the content being played. There are several reasons why you might want to include those.

At CANAL+ for example, we started to study the implementation of ad-insertion and tracking solutions for live contents. Ad-insertion can work on the server-side by generating an MPD which will replace media segments by ads, for example by adding Period elements.

On the client-side (the media player), we then may want to know where an ad can be encountered and when the final user begins to see it (or when he/she skip it) for tracking purposes.

Here, DASH EventStream's <Event> elements can be used.
Such elements define timed events associated with the current content. When the user begins to play at a position within the time boundaries of that event, the corresponding Event element will be considered.
This element can contain specific user-defined information, such as the URL of an ad-tracking server.

But DASH <Event> elements are not restricted to advertising use cases, you could use one anytime you want to be notified when an user is at a particular position in the stream.

How the RxPlayer handle DASH EventStream elements

DASH's <EventStream> are elements of the MPD, inside the corresponding <Period> element.
Each EventStream can carry one or more <Event> elements, whose format and inner data depends on an attribute of the EventStream element called the schemeIdUri:

    <Period id="1">
      <EventStream schemeIdUri="urn:uuid:XYZY" timescale="1000" value="call">
        <Event presentationTime="20000" duration="10000" id="1">
          Some data
        </Event>
        <Event presentationTime="40000" duration="10000" id="2">
          Some other data
        </Event>
        <Event presentationTime="60000" duration="10000" id="3" />
      </EventStream>
      <!-- ... -->
    </Period>

Each Event element generally has an assiociated "presentation time" (the initial time the event should be triggered at), some specific data and, sometimes, a duration.

The RxPlayer acts like a passthrough with the Event's data.
Its only role here is to send "streamEvent" events when a user goes into the time boundaries of such events. By catching that event, an application can also be notified when the user "exits" the same event's time boundaries:

// Do something when the user enters a time area linked to an Event element
rxPlayer.addEventListener("streamEvent", (evt) => {
  console.log("Beginning of an event:", evt);

  // Catch the moment when we are exiting the event.
  // Note that only Event elements with a duration (and thus an end) can
  // have an "exiting" notion.
  if (evt.end !== undefined) {
    evt.onExit = () => {
      console.log("End of the event:", evt);
    };
  }
});

The user could also "skip" an event, like when seeking after it. As an application, you may want to know when that happens.

For this reason, the RxPlayer also reports when an event has been "skipped" through a new "streamEventSkip" event:

rxPlayer.addEventListener("streamEventSkip", (evt) => {
   console.log("An event has been skipped.", evt);
});

Other types of events

For now, only DASH EventStream are handled by the RxPlayer but several types of events exist.
DASH also has in-band events that are in the media containers themselves like for example in "emsg" ISOBMFF boxes.
Other streaming protocol have their own event format(s) as well.

We want to support more of them in future versions, the available APIs here were also written with those in mind.
The management of "emsg" boxes, for example, is something we are today considering to add in our next versions.

More information on those APIs

Because the concept and APIs could seem complex we have added both those APIs to our API documentation and added a new tutorial about it.

We advise people wanting to integrate stream event management to first read the tutorial and to then come back to the API documentation for checking what are the arguments awaited.

The tutorial is here.

And here is the documentation of the new APIs:

• the streamEvent event for when an event is encountered
• the streamEventSkip event for when an event is "skipped".

We also export two new types (which can be imported via "rx-player/types"):

• IStreamEvent: the object sent with a streamEvent or streamEventSkip event
• IStreamEventData: the value of the data attribute of an IStreamEvent.

Those are documented here.

Applying track preferences to the current content

The two APIs for track selection

When it comes to track selection in the RxPlayer, we have two set of APIs:

• the "classic" track selection APIs - like getAudioTrack, setVideoTrack, getAvailableTextTracks - will tell you which tracks are available and allow you to choose a specific one from that list

• the track preferences APIs - like setPreferredAudioTrack - allow to define a general wanted choice. For example, we could indicate that the RxPlayer should choose by default a french audio track and/or english subtitles if those are available.

The APIs in the first set list the available tracks, so the application can then choose a precize one.

The APIs in the second set let the RxPlayer do the choice itself based on criterias. They are also used in some internal RxPlayer optimizations, like when pre-loading a future content - where the right track to download is for now "guessed" by the RxPlayer.

The problem with track preferences

The track preferences APIs have thus many advantages, but they have a limit: they only apply preferences for future contents and periods.

The content which is currently played and the Periods (for DASH) or sub-contents (for MetaPlaylist) that we played previously will still remain with their last set track configuration.

This is by design, we didn't want to change the current tracks after a setPreferred... call because this could seem unexpected.
So we are left in what we could refer as a "hole" in our API:

• We can define the track for the current Period (or sub-content) but this won't affect any other Periods
• We can define a preference for new Periods but this won't affect the current one or the one that have already been played.

What if we just want to set a new global preference, that we want to apply retroactively (to the current and already-played Periods)?

We could call both APIs, but we would still be left with the previous Periods in their previous track configurations.

This problem can be seen for example when you want to enable an "audio-only" mode (by disabling the video track) for multi-period contents:

// The old way

// disable the current track
rxPlayer.disableVideoTrack();

// Update the preferences for the next content / periods
rxPlayer.setPreferredVideoTracks([null]);

But here seeking back to a previous period is going to display the video track again. To work-around that, we have to call disableVideoTrack() again when seeking back to them.

A solution

We did many drafts on how we could improve our track selection APIs so that all possible use-cases are handled.

Among those, we thought of giving "more power" to the classic track selection APIs, so they could also change the track for other Periods.
However, we found that that approach would necessitate a very complex API. For features as simple as applying a global preference, we thought that the API should stay simple.

In the end, we decided to add an optional boolean as a second argument to those track preferences methods:

• setPreferredAudioTracks
• setPreferredTextTracks
• setPreferredVideoTracks

When set to true that preference will be applied retroactively.

For example, disabling the video track globally now can simply be done this way:

rxPlayer.setPreferredVideoTracks([null], true);

A new tutorial on track selection

As you can see, the RxPlayer allows applications to have a complex track selection management. However, this can come at the cost of ...

v3.20.1

Release v3.20.1 (2020-05-06)

Overview

This is a small release which mainly contains bug fixes.

It most-of-all fixes a small regression brought in our v3.20.0 release.
This issue could lead users to not be able to play multiple encrypted contents. It was though relatively rare to encounter, as it was only triggered in the following situations:

• when using Internet Explorer 11 and playing multiple encrypted contents

• when linking more than RxPlayer instance to the same media element - after calling rxPlayer.dispose() to dispose the last one - and playing encrypted contents in more than one of such instance

For applications which did not dispose and re-instanciate the RxPlayer on the same media element and did not target IE11, this issue should have had no impact. Likewise, applications not playing encrypted contents are not concerned by this issue.

We also added other fixes to this release not linked to this (or any) regression.

Regression when disposing of a MediaKeys instance

The main issue here (summarized in the overview) was due to the way we changed our DRM-related code to better support Safari.

When doing that, we updated the code attaching MediaKeys instances (the main interface involved in content decryption) to the media element to be adapted to both the regular and Safari way of doing so.

However, we poorly considered and tested the possiblity of setting a null MediaKeys, which is usually what you want to set when disposing of the current instance attached to the element.
When wanting to set a null MediaKeys, the RxPlayer just ignored it instead.

This was done in two situations:

• between each content when playing in Internet Explorer 11, to work-around an issue on this platform
• after calling the rxPlayer.dispose() method, to dispose of that resource (the MediaKeys instance)

This wrong behavior only became an issue when loading a new encrypted content on the same media element, where the same MediaKeys logic would unexpectedly throw.

Fixing this issue was just a matter of better handling MediaKeys disposal, which is now done.
We do not detect an issue on IE11 or on applications calling dispose anymore.

DRM issue for contents with multiple keys

This is a rare and subtle issue that could have impacted very sporadically some applications playing encrypted contents - with several keys per content.

As such, I'll try to be very descriptive on what the problem was and how it could have had an impact.

What's a MediaKeySession?

A MediaKeySession is the browser API allowing to communicate decryption keys to the Content Decryption Module integrated in the browser, which tasks is to decrypt the content.

We create a single MediaKeySession per initialization data received, which roughly means one per license needed to decrypt the content.
In most encountered cases where we only have one key per license, we can thus simplify this concept by saying that we could need to create as much MediaKeySession for a single content as the number of keys in that content.

The MediaKeySession cache

We try when possible to keep old MediaKeySessions in a cache even when other unrelated key(s) are needed. Doing so has several advantages, like speeding-up the loading of an already-played content, the ability to avoid license requests we already have done etc.

This cache will store a maximum of 50 MediaKeySessions at the same time.

The problem

It all becomes more complicated when we consider another cache maintained by the RxPlayer, listing which encryption information has already been encountered in the current content.

That second cache is needed for a similar (but different enough) reason than the MediaKeySession cache: to only trigger some logic when new encryption data is received, and avoid duplicating such logic in other cases, for example when the same data is received two times in a row.

Where the problem lies is that the MediaKeySession cache has a logic to evict old stored data but the second cache - let's call it the InitializationData cache - has none, and there is some dependency between them due to how the RxPlayer consider both.

To explain this, let's consider an example:
We're playing a content with 51 keys, each one associated to a different license. Each time we're encountering a media segment needing a new key - and only in that case - we would be creating a new MediaKeySession, then loading its license and so on.
When reaching data needing the 51st key, we will remove the MediaKeySession linked to the 1st key to make place for the one linked to the 51st.
If then we encounter data related to the 1st key again, the InitializationData cache would ignore it, as it has been already encountered. But here because its linked MediaKeySession is now closed, we're not able to play that data. We will be left waiting for a decryption key which will never come.

This case could be considered extremely rare because encountering that much encryption data on a single content is in itself extremely rare.
But this issue could arise in more subtle ways, such as when zapping between multiple contents and playing them again, as those 50 MediaKeySession do not need to be from the same content.

Note that this problem was only an issue when playing a content needing several licenses. In any other cases, even when switching a lot between contents, everything should have worked fine.

The solution

The behavior for both cache stays the same, we do not limit the InitializationData cache as it wouldn't make sense when considered alone.

To resolve that problem in an other way, the RxPlayer core logic now removes manually entries from the InitializationData cache when the corresponding MediaKeySession cache entries are evicted. As both caches are not handled in the same place, this is done thanks to an event.

Avoid hash collision in DRM logic

Another issue was close to the one we just described: some encryption data could be ignored because the RxPlayer wrongly thought it has already been encountered.

This is because we do not compare the data itself, we first hash it (through a custom hashing function) and then use that hash for comparison. If two different encryption data give the same hash, the RxPlayer will wrongly consider both to be the same.

This is however limited by the fact that such collision should be very rare, as far as we know this case has never been encountered. Still, the cost of comparing the encryption data itself is very low, so low that we considered this "optimization" of only checking the hash as an unnecessary risk.

Now the initialization data is still hashed, but when the same hash is encountered, we still compare the encryption data linked to that hash.
That way we still profit from the speed at which a hash can be used for lookup while still protecting against collisions with a very small cost.

Switching back to the preferred audio, video or text track in a Directfile content

There was a minor issue with directfile contents and track management where an audio, video or text track set by the user could be replaced by the most "preferred" one (according to the preferred{Audio,Text,Video}Tracks APIs).

This was because on directfile contents, we run a logic setting the optimal track everytime the list of available ones change. The definition of "optimal" there was just based on the preferred tracks, without considering that the user could already have selected a track through the set{Audio,Text,Video}Track APIs.

In the case where that track is still available, it would be a weird behavior to reset the choice because a new unrelated track has been removed or added in the process.

This is now fixed. Now when a track was manually selected through the set{Audio,Text,Video}Track APIs, we stay on that track as long as it stays available.

Changelog

Bug fixes

• eme: fix OTHER_ERROR issue arising when loading a new encrypted media when a previous since-disposed instance of the RxPlayer-played encrypted contents on the same media element
• eme: fix OTHER_ERROR issue arising on Internet Explorer 11 when playing more than one encrypted content
• eme: fix issue where more than 50 active MediaKeySessions at the same time could lead to infinite rebuffering when playing back a previous content with multiple encryption keys
• directfile: for directfile contents, don't reset to the preferred audio, text or video track when the track list changes
• eme: remove any possibility of collision in the storage of EME initialization data. The previous behavior could theorically lead some initialization data to be ignored.
• eme: fix typo which conflated an EME "internal-error" key status and an "output-restricted" one.

v3.20.0

Release v3.20.0 (2020-04-22)

Overview

The v3.20.0 brings new features and improvements:

• With the addition of the disableVideoTrack method, it's now possible to play in an "audio-only" mode

• We added a preferredVideoTracks RxPlayer option, as well as setPreferredVideoTracks and getPreferredVideoTracks methods to specify which video tracks you would prefer based on its codecs and accessibility status. It's also possible to use those APIs to begin playing without any video track active.

• We added (thanks to @mattiaspalmgren) a signInterpreted property to video track APIs which signal when a video track contain a sign language interpretation (according to the DASH-IF IOP).

• We improved the audio preferences APIs to here also be able to specify a preferred audio codec.

• We added new optimizations for supporting very large MPDs (DASH's manifest)

• The Safari-specific EME implementation (for playing contents with DRM) should now be finished. Previously, you might not have been able to play DRM-ized contents with Safari.

disableVideoTrack

The ability to play in an "audio-only" mode was something we wanted to provide for some time now.

However, removing or adding a video buffer during playback is not something well supported on web browsers. In more exact terms, all SourceBuffers had to be created before we can start to push media contents to either of them. For this reason adding the possibility to disable and re-enable the video track while playing is usually not doable.

Thankfully, we already added the ability to "reload" the content (and its associated "RELOADING" state) in the v3.6.0 (august 2018) release. By simply doing that when the user disable or re-enable the video track - and some other code changes - we were able to provide this feature (those "code changes" being actually a little more complicated than how it sounds as it's the reason why we waited so long :p).

Disabling the current video track is now possible by calling the disableVideoTrack method:

function enableAudioOnly() {
  player.disableVideoTrack();

  // If you don't want subtitles to display, which may or may
  // not be something you want, you can also do
  // player.disableTextTrack();
}

Please note that doing so might lead the RxPlayer to reload the content for a very short time. While doing so, the video element will usually be reset to black and multiple APIs won't be usable. You will know when the player is reloading or not by checking when it goes in and out of the "RELOADING" state.

You can then re-enable the video track by using the usual track switching method: getAvailableVideoTracks, getVideoTrack and setVideoTrack:

function quitAudioOnly() {
  // example by just setting the first track
  const tracks = player.getAvailableVideoTracks();
  if (tracks.length !== 0) {
    player.setVideoTrack(tracks[0].id);
  }
}

It is also possible to begin without a video track enabled through the new preferredVideoTracks option, which is described later in this release note.

On a related note, you have to consider that, just like disableTextTrack, disableVideoTrack will only disable the video track for the current Period.
If you're playing a multi-Period content and if you want to disable the video track for every Period, you could either call disableVideoTrack at each periodChange event, or you could use the new preferredVideoTracks APIs (which are documented in this same release note) before loading the content.

We added the possiblity to disable the video track in our demo page if you want a quick peek of how it can look like.

signInterpreted accessibility info

Before writing about the new video track preference APIs, we need to add a note on the new signInterpreted boolean property added on video track APIs.

This new property is set to true when the corresponding video track contains a sign language interpretation.
We can extract this information from a manifest, thanks to accessibility features available in some DASH contents (you can refer to the DASH-IF IOP for more info).

This boolean was added to the object returned by getAvailableVideoTracks and getVideoTrack:

/**
 * Returns only video tracks containing a sign language interpretation.
 */
function getAvailableSignInterpretedVideoTracks() {
  return player.getAvailableVideoTracks
    .filter(track => track.signInterpreted === true);
}

/**
 * Returns true if the current video track contains a sign language
 * interpretation.
 */
function isCurrentTrackSignInterpreted() {
  const videoTrack = player.getVideoTrack();

  // videoTrack can be `undefined` or `null` for different reasons (see doc)
  if (typeof videoTrack === "object") {
    // Note: signInterpreted can still be `undefined` - when we don't know
    return videoTrack.signInterpreted === true;
  }
  return false;
}

Big thanks to @mattiaspalmgren for implementing this feature.

preferredVideoTracks

It is now possible to set a video track preference through the following new APIs:

• the preferredVideoTracks player option, when instantiating the RxPlayer
• the setPreferredVideoTracks method, at any time after instantiation

Exactly like the related audio and text APIs, those two allow to tell the RxPlayer which video track it should initially choose, according to various constraints of your choosing.

For now, you can only set those constraints in function of:

• the wanted video codecs
• whether the track should contain a sign language interpretation or not

The syntax is well explained in those two method's documentation, but let's still get through simple examples so you can quickly grasp what they can do.

Let's say that you would prefer video tracks with sign language interpretation. To spice things up, we will also prefer if the track contains only Representation in the vp9 codec. Here is what we could do:

const player = new RxPlayer({
  // ...
  preferredVideoTracks : [
    // at best, have both sign language and only the VP9 codec
    {
      signInterpreted: true,
      codec: { test: /^vp9/, all: true /* Representations should all be vp9 */ },
    },

    // If not found or supported, just sign language is fine
    { signInterpreted: true },

    // If not available either, fallback on track without sign language, but
    // still with VP9
    { codec: { all: true, test: /^vp9/ } },

    // Note: If the last one was still not available, we will here still have a
    // video track, which do not respect any of the constraints set here.
  ],
})

You could also do it at any time after instantiation, thanks to the setPreferredVideoTracks method:

player.setPreferredVideoTracks([
  { signInterpreted: true, codec: { all: true, test: /^vp9/ } },
  { signInterpreted: true },
  { codec: { all: true, test: /^vp9/ } },
]);

Last but not least, it is also possible to indicate that you wish that no video track is active, by adding a null in this list of preference.

You could add it as the only element to that array, meaning you want to play in audio-only (without a video track):

// As usual either through the constructor
const player = new RxPlayer({
  // ...
  preferredVideoTracks: [null],
});

// or through the setPreferredVideoTracks method
player.setPreferredVideoTracks([null]);

You could also fallback to no video track if none your constraints are supported, by putting null at the end of that same array. For example, if you want the vp9 codec or no video track, you can do:

// Through the constructor
const player = new RxPlayer({
  // ...
  preferredVideoTracks: [
    { codec: { all: true, test: /^vp9/ } },
    null // if not found play without a video track
  ],
});

// or through the setPreferredVideoTracks method
player.setPreferredVideoTracks([
  { codec: { all: true, test: /^vp9/ } },
  null // if not found play without a video track
]);

You can retrieve the last set list of preferrences at any time through the new getPreferredVideoTracks method.

improved preferredAudioTracks

We also improved the audio track preferences APIs by allowing you to set constraints on the wanted codecs.

Both setPreferredAudioTracks and the preferredAudioTracks constructor option now can take a new codec property.

When set, this property allows to declare that you would prefer that an audio track contains either:

• only Representation with a given codec
• at least one Representation with a given codec

As you might only want to choose a preference based on the codec and not on the language anymore, we also made the previous properties (both language and audioDescription) optional.

Let's look at examples. Let's say that you would prefer audio tracks containing Dolby Digital Plus only (also known as "ec-3"), you could do:

player.setPreferredAudioTracks([
  {
    codec: {
      test: /ec-3/, // Check for Dolby Digital Plus
  ...