v3.20.0

Release v3.20.0 (2020-04-22)

Overview

The v3.20.0 brings new features and improvements:

• With the addition of the disableVideoTrack method, it's now possible to play in an "audio-only" mode

• We added a preferredVideoTracks RxPlayer option, as well as setPreferredVideoTracks and getPreferredVideoTracks methods to specify which video tracks you would prefer based on its codecs and accessibility status. It's also possible to use those APIs to begin playing without any video track active.

• We added (thanks to @mattiaspalmgren) a signInterpreted property to video track APIs which signal when a video track contain a sign language interpretation (according to the DASH-IF IOP).

• We improved the audio preferences APIs to here also be able to specify a preferred audio codec.

• We added new optimizations for supporting very large MPDs (DASH's manifest)

• The Safari-specific EME implementation (for playing contents with DRM) should now be finished. Previously, you might not have been able to play DRM-ized contents with Safari.

disableVideoTrack

The ability to play in an "audio-only" mode was something we wanted to provide for some time now.

However, removing or adding a video buffer during playback is not something well supported on web browsers. In more exact terms, all SourceBuffers had to be created before we can start to push media contents to either of them. For this reason adding the possibility to disable and re-enable the video track while playing is usually not doable.

Thankfully, we already added the ability to "reload" the content (and its associated "RELOADING" state) in the v3.6.0 (august 2018) release. By simply doing that when the user disable or re-enable the video track - and some other code changes - we were able to provide this feature (those "code changes" being actually a little more complicated than how it sounds as it's the reason why we waited so long :p).

Disabling the current video track is now possible by calling the disableVideoTrack method:

function enableAudioOnly() {
  player.disableVideoTrack();

  // If you don't want subtitles to display, which may or may
  // not be something you want, you can also do
  // player.disableTextTrack();
}

Please note that doing so might lead the RxPlayer to reload the content for a very short time. While doing so, the video element will usually be reset to black and multiple APIs won't be usable. You will know when the player is reloading or not by checking when it goes in and out of the "RELOADING" state.

You can then re-enable the video track by using the usual track switching method: getAvailableVideoTracks, getVideoTrack and setVideoTrack:

function quitAudioOnly() {
  // example by just setting the first track
  const tracks = player.getAvailableVideoTracks();
  if (tracks.length !== 0) {
    player.setVideoTrack(tracks[0].id);
  }
}

It is also possible to begin without a video track enabled through the new preferredVideoTracks option, which is described later in this release note.

On a related note, you have to consider that, just like disableTextTrack, disableVideoTrack will only disable the video track for the current Period.
If you're playing a multi-Period content and if you want to disable the video track for every Period, you could either call disableVideoTrack at each periodChange event, or you could use the new preferredVideoTracks APIs (which are documented in this same release note) before loading the content.

We added the possiblity to disable the video track in our demo page if you want a quick peek of how it can look like.

signInterpreted accessibility info

Before writing about the new video track preference APIs, we need to add a note on the new signInterpreted boolean property added on video track APIs.

This new property is set to true when the corresponding video track contains a sign language interpretation.
We can extract this information from a manifest, thanks to accessibility features available in some DASH contents (you can refer to the DASH-IF IOP for more info).

This boolean was added to the object returned by getAvailableVideoTracks and getVideoTrack:

/**
 * Returns only video tracks containing a sign language interpretation.
 */
function getAvailableSignInterpretedVideoTracks() {
  return player.getAvailableVideoTracks
    .filter(track => track.signInterpreted === true);
}

/**
 * Returns true if the current video track contains a sign language
 * interpretation.
 */
function isCurrentTrackSignInterpreted() {
  const videoTrack = player.getVideoTrack();

  // videoTrack can be `undefined` or `null` for different reasons (see doc)
  if (typeof videoTrack === "object") {
    // Note: signInterpreted can still be `undefined` - when we don't know
    return videoTrack.signInterpreted === true;
  }
  return false;
}

Big thanks to @mattiaspalmgren for implementing this feature.

preferredVideoTracks

It is now possible to set a video track preference through the following new APIs:

• the preferredVideoTracks player option, when instantiating the RxPlayer
• the setPreferredVideoTracks method, at any time after instantiation

Exactly like the related audio and text APIs, those two allow to tell the RxPlayer which video track it should initially choose, according to various constraints of your choosing.

For now, you can only set those constraints in function of:

• the wanted video codecs
• whether the track should contain a sign language interpretation or not

The syntax is well explained in those two method's documentation, but let's still get through simple examples so you can quickly grasp what they can do.

Let's say that you would prefer video tracks with sign language interpretation. To spice things up, we will also prefer if the track contains only Representation in the vp9 codec. Here is what we could do:

const player = new RxPlayer({
  // ...
  preferredVideoTracks : [
    // at best, have both sign language and only the VP9 codec
    {
      signInterpreted: true,
      codec: { test: /^vp9/, all: true /* Representations should all be vp9 */ },
    },

    // If not found or supported, just sign language is fine
    { signInterpreted: true },

    // If not available either, fallback on track without sign language, but
    // still with VP9
    { codec: { all: true, test: /^vp9/ } },

    // Note: If the last one was still not available, we will here still have a
    // video track, which do not respect any of the constraints set here.
  ],
})

You could also do it at any time after instantiation, thanks to the setPreferredVideoTracks method:

player.setPreferredVideoTracks([
  { signInterpreted: true, codec: { all: true, test: /^vp9/ } },
  { signInterpreted: true },
  { codec: { all: true, test: /^vp9/ } },
]);

Last but not least, it is also possible to indicate that you wish that no video track is active, by adding a null in this list of preference.

You could add it as the only element to that array, meaning you want to play in audio-only (without a video track):

// As usual either through the constructor
const player = new RxPlayer({
  // ...
  preferredVideoTracks: [null],
});

// or through the setPreferredVideoTracks method
player.setPreferredVideoTracks([null]);

You could also fallback to no video track if none your constraints are supported, by putting null at the end of that same array. For example, if you want the vp9 codec or no video track, you can do:

// Through the constructor
const player = new RxPlayer({
  // ...
  preferredVideoTracks: [
    { codec: { all: true, test: /^vp9/ } },
    null // if not found play without a video track
  ],
});

// or through the setPreferredVideoTracks method
player.setPreferredVideoTracks([
  { codec: { all: true, test: /^vp9/ } },
  null // if not found play without a video track
]);

You can retrieve the last set list of preferrences at any time through the new getPreferredVideoTracks method.

improved preferredAudioTracks

We also improved the audio track preferences APIs by allowing you to set constraints on the wanted codecs.

Both setPreferredAudioTracks and the preferredAudioTracks constructor option now can take a new codec property.

When set, this property allows to declare that you would prefer that an audio track contains either:

• only Representation with a given codec
• at least one Representation with a given codec

As you might only want to choose a preference based on the codec and not on the language anymore, we also made the previous properties (both language and audioDescription) optional.

Let's look at examples. Let's say that you would prefer audio tracks containing Dolby Digital Plus only (also known as "ec-3"), you could do:

player.setPreferredAudioTracks([
  {
    codec: {
      test: /ec-3/, // Check for Dolby Digital Plus
      all: true // All Representation must have this codec (`false` would mean "at least one")
    }
  }
]);

If you would prefer a track that contains at least one Dolby Digital Plus Representation (but don't care if they all are), you could set the all property to false that way:

player.setPreferredAudioTracks([
  { codec: { all: false, test: /ec-3/ } }
]);

Of course, it's also possible to mix language preferences with codec preferences:

// In english and in preference in Dolby Digital Plus
player.setPreferredAudioTracks([
  { language: "eng", audioDescription: false,  codec: { all: false, test: /ec-3/ } },
  { language: "eng", audioDescription: false },
  { codec: { all: false, test: /ec-3/ } },
]);

yet other DASH optimizations

We continued our efforts from the last release to optimize DASH's MPD parsing.

Why optimizations were needed

Performance problems arised when you wanted to play live contents with short segments and multiple hours of timeshift on low-end devices (STB, ChromeCast etc). For example, at Canal+ Group we often see contents with 8 hours of timeshift window and a new segment every two seconds. On some devices, playing those could lead to a jittery playback.

This is because those contents had a huge manifest file - usually several MB in size - which had to be refreshed often. Parsing those was a process that was both time consuming and that put a lot of pressure on the garbage collector. After some time, the rythm could be unbearable even on regular computers. As an example, my laptop would crash after just an hour of playback on the most heavy manifests (around 3.5MB, refreshed every 2 or 3 seconds).

What we already did

We came with several optimizations in the v3.19.0 release, each saving a little more time and memory.

The "best" optimization we found was the ability to provide a secondary URL to a shorter version of the manifest which will be specifically used for manifest updates. This new feature completely resolved the problem we were seeing with our 8 hours streams.

In most cases, this is still overall the best optimization in terms of memory usage and of parsing time but it comes with a big downside: you have to package a second manifest.

An efficient but "riskier" solution

The new solution is based on the observation that when we refresh the manifest, segment information won't generally change a lot:

• new segments will generally be added at the end
• old segments, at the beginning, might be removed

By using this notion, we can thus take out a huge chunk of manifest parsing by just skipping information about the segments right in "the middle".

This would save both a lot of time doing the parsing, and avoid unnecessary memory allocations, as both were mainly due to the large amount of segment information present in the manifest.

So that's what we did. The results are very statisfying and with this new behavior, we report no problem with our bigger contents anymore. Actually, when profiling the time spent parsing the manifest, 90% of that time is now spent through the DOMParser.prototype.parseFromString function, which is a browser API we use to parse XML data more rapidly!

However, this optimization has an issue which made it kind of "risky": we could technically not register a change concerning segments declared in what we previously called "the middle".

How this risk is managed

We called this situation - where the MPD once parsed is different than what was actually downloaded - a "de-synchronization".

This de-synchronization is something we want to avoid as much as possible, so we took several measures to not only avoid it, but also detect it and fix it when it happens.

First, this optimization is only enabled when the following conditions are met:

• we're playing a dynamic MPD
• parsing the MPD takes a lot of time
• there's an unusually long list of segment information in the MPD

Then, at a regular interval, "full" MPD refresh (without this optimization) are still triggered.

At last, when a segment URL that should be available returns a 404 when playing such contents, we schedule a "full" MPD refresh, also without that optimization on.

How to profit from this optimization

After brainstorming whether an option should be provided to enable or disable this optimization, we found that the gain was so big and the associated risk so low that we could, at least for now, enable it by default, without an option to disable it.
This can sound scary but you shouldn't see a difference. If you see one and would like us to provide an API to disable it, you can open an issue.

Safari EME implementation

The ability to play contents with DRMs on the web is done thanks to a set of browser APIs, called the Encrypted Media Extensions (or "EME" for short).

The majority of web browsers in the wild support those APIs, allowing us to have a mostly unified codebase when it comes to handling content DRM. But there was still one browser that had a totally different implementation: Safari.

Safari 12.1 actually implements the standard EME APIs, but playing content with FairPlay - which simply-put is the DRM system implemented by Apple we typically find in Safari - does not seem to be compatible with them.

Therefore, custom safari-specific APIs have to be used instead for decyphering FairPlay media content. We didn't yet took the time to properly implement all of them until now.

From now on, you should be able to play FairPlay ciphered content on Safari by using the keySystems loadVideo option, like you would do for any other browser.

Note that this also means that we should now support HLS FairPlay contents on Safari, as Safari can already play HLS contents directly (you can play it through our "directfile" transport). This feature associated to the possibility to manage audio and video tracks for directfile contents by using the same APIs you would for any other content (that we introduced in the v3.18.0) is another step in the direction of being able to handle rich "directfile contents" - most notably HLS on Safari.

Ignore segments from cache from our adaptive algorithms

As an optimization, web browsers usually maintain a local cache which they can use to avoid doing http/https requests they already have done in the past. This allows such resources to be directly fetched from that cache which avoid the costly loading time needed to perform the request again, with as the nice side-effect a reduction of the charge on the server.

This is mainly a good thing, but it can impact the RxPlayer negatively.
The RxPlayer adaptative bitrate logic relies, amongst other elements, on network bandwidth estimations that are made by monitoring the fetching time and dividing it by the fetched resource's size.
If we don't set a special treatment for requests that just hit the cache, the RxPlayer might just think - after doing some requests hitting the cache - that the user has an extraordinary large bandwidth. It will then happily consider a higher video quality than what should normally be possible, ultimately leading to buffering.

Thankfully, this case is relatively rare. The RxPlayer usually does not perform requests for segments it has already requested. But in some rare cases, such as when it detects that an already-downloaded segment has been removed from the buffer to free some memory (which is something that the browsers often do), we can find ourselves in that situation.

So, we should just ignore segments request hitting the cache in our calculation, right?
Sadly this is not as easy as it sounds as no browser API seem to reliably provide this information. There is also a second edge case, what if all segments were hitting the cache? Should we just ignore every requests done?

We have decided to go with the following strategy:

• we put in place some kind time threshold to estimate if a chunk might have hit the cache (the loading time is under the threshold) or not (the loading time went over the threshold)
• if we found that each segment request done since the beginning of the content are below the threshold (i.e. seem to hit the cache), the adaptive logic still work exactly like before.
• when the first segment request over that threshold (so, that we think did not hit the cache) arrive, the bitrate estimation is reseted, and chunks below the threshold are ignored from now on.

This allows to still increase in quality if every chunk received seem ("seem" is important here) to be from the cache, and still correctly calculate the right bandwidth when new received chunks take much more time to load.

The drawback of using a threshold is that we could be in a case where some cached segments still take some time to be loaded. In the case where that time goes over that threshold, they will be incorrectly considered as non-cached.
Conversely, some network segments may take less time than the threshold to be fetched, in which case they might be incorrectly ignored. As these false positives and false negatives cases are limited, it may almost not influence the bitrate estimation. Moreover, the latter relies on a mean of several samples. As such, one sample may not be sufficient to influence the chosen playback quality.

Deprecated methods

We decided to deprecate the following methods:

• getManifest
• getCurrentAdaptations
• getCurrentRepresentations

This is because those three APIs do not seem to be used and become difficult to maintain. More specifically, they exposes some internal manifest structure, and as such limits us in how we can update this structure without breaking the API contract.

We could still provide some kind of translation layer between that API and our internal structure, but maintaining this seems to be not worth the candle.

As usual, you can find the list of every deprecated API, in the corresponding documentation page.

Changelog

Features

• api: add disableVideoTrack method
• api: add the preferredVideoTrack constructor option and setPreferredVideoTracks / getPreferredVideoTracks methods to set a video track preference (or to start with the video track disabled).
• api: add optional codec property to preferred audio tracks APIs, allowing applications to communicate a codec preference.
• api: make the language and audioDescription properties in preferredAudioTracks' objects optional.
• api: add signInterpreted to getVideoTrack and getAvailableVideoTracks return objects to know when a track contains sign language interpretation

Deprecated

• api: deprecate the getManifest() method
• api: deprecate the getCurrentAdaptations() method
• api: deprecate the getCurrentRepresentations() method

Bug fixes

• compat/eme: Set proper EME Safari implementation, to play contents with DRM on it without issues
• compat/directfile/iOS: On Safari iOS, fix auto-play warnings when a directfile content is played with the playsinline attribute set.
• directfile: In Directfile mode, always disable the current text track when a null is encountered in the preferredTextTracks array

Other improvements

• abr: ignore requests that may have directly hit the cache in our adaptive logic
• dash/perf: improve parsing efficiency for very large MPDs, at the expense of a very small risk of de-synchronization. Mechanisms still allow for regular re-synchronization.