v3.26.2

Release v3.26.2 (2022-01-11)

The v3.26.2 release has now been published.

It includes multiple bug fixes and improvements:

• The RxPlayer has now a better adaptive logic when playing DASH low-latency contents. This is explained with details below.
• Low-latency contents using a $Time$ indexing scheme, are now better handled
• The RxPlayer's documentation pages have been updated in depth, now (finally) including search capabilities!
• The player is now more resilient when parsing an MPD with unreachable optional resources, such as the ones behind <UTCTiming> elements
• With DASH contents, <ContentProtection> elements can now be declared at the <Representation> level
• many other small bug fixes and improvements (see the changelog at the bottom of this release note).

We'll now go in more details on some of those improvements.

New adaptive algorithm for DASH Low Latency contents

The low-latency adaptive status

It has now been more than two years since the RxPlayer introduced the lowLatencyMode option, allowing to play efficiently DASH Low Latency contents (often called DASH-LL or LL-DASH).

output.mp4

video: Example of playing the same DASH-LL content without lowLatencyMode (to the left) and with it (to the right). We can see that the right one plays almost ten seconds forward.

However there was one area where we knew we could do much better: the selection of the quality to play a.k.a. the adaptive logic.

The adaptive logic is notoriously an important issue when it comes to playing low-latency contents, due to specificies explained below.
When we started to implement low-latency playback, we tried to devise an algorithm specifically for it, but we didn't get good-enough result for us to go on with it.

What we decided instead was to keep our current less-than-ideal adaptive algorithm (which just risk to play a lower quality than what could be expected) and mostly wait for a more suitable adaptive algorithm to make its apparition. We felt that it was now the right time!

Problems with the current adaptive algorithms

The current adaptive algorithms

Previously, the RxPlayer relied on two kinds of algorithms for its adaptive logic:

• bandwidth-based algorithms: where we measure the current user's bandwidth by observing how fast media segments are downloading and choose a media quality with the highest bitrate compatible with that bandwidth.

  
  Depiction of the content of the video buffer when running the RxPlayer's bandwidth-based algorithm only.
  The blue color corresponds to the initial quality (which is lower than ideal), yellow is the ideal one and the red bar corresponds to the current playing position. Here we can see it reaches the ideal quality after some time loading the initial one. This is because it first had to perform bandwidth calculation with that initial quality before settling on the yellow.

• buffer-based algorithms: where we choose a quality depending on how much data has already been buffered in advance.
  If we have not much data in the buffer, we switch to a low quality to fill it, if however we have a lot of data in advance in the buffer, we can switch to a higher quality without much risks of rebuffering.

  
  Depiction of the content of the video buffer when running the RxPlayer's buffer-based algorithm only.
  The blue color corresponds to the initial quality (which is lower than ideal), yellow is the ideal one, green is better than the ideal one and the red bar still corresponds to the current playing position. Here we can see it reaches the ideal quality after more time than with the bandwidth-based approach but also reaches a better-than ideal quality at the end of the buffer, because the buffer has been filled.

Both are widely used type of algorithms which have advantages and disadvantages:

• bandwidth-based algorithms are often very fast to settle on a good-enough approximate of the ideal quality, yet in its current form are often set to be a little too pessimistic to make-up for potential network latencies and bandwidth variations.

• buffer-based algorithms allows to reach a higher quality as long as enough data is buffered. Sometimes even better quality than what the current network connection would allow (when enough buffer has been built).
  Yet it is slow to start (as data has to be buffered initially) and needs the ability to pre-buffer data, which is problematic for live contents because there the player plays too close to the live edge to be able to pre-load much of it.

Which is why in normal playback (non-low-latency ones), the RxPlayer will use a mix of the two: in most cases it will consider the best quality between what both kinds of algorithm estimate.

Issues with DASH low-latency contents

Sadly both buffer-based and network-based approachs break down when playing DASH Low Latency contents:

One of the principles behind DASH Low Latency is that small chunks of media data, called segments, can be requested while it is still generated in the back-end.
When that is the case, the request for a segment stays open until it is fully generated and transfered. In that situation, it's difficult to rely on the time it took to perform the request (as we do to perform bandwidth calculations), because a large part of that time might just have been spent "waiting" for the segment to be generated.

Also, the main point of low-latency being to reduce the delay between video/audio capture and its presentation to the user, the RxPlayer will play too close to the live edge to build enough buffer for buffer-based algorithms to be useful.

Depiction of a video buffer playing too close to the live edge (the red bar is the current position, the end of the container represents the current live position) to be able to build much buffer for the buffer-based approach

So none of the two previous approaches really worked, leading to a poorer quality thans what could be expected.

Algorithm specific for low-latency contents

As we were not in a hurry, our initial strategy was consequently to wait and see until the technology was more mature.
Some months ago, we noticed that multiple new published algorithms could finally be used for low-latency content specifically, some even having already an implementation in the open-source dash.js media player.

We first tried to implement one of those algorithms, called Learn2Adapt. Yet the implementation was less than ideal due to the difficulty to provide one of its input. When we checked dash.js, which can also rely on this algorithm for low-latency contents, that algorithm was also not satisfying.

After this failure, we redoubled our efforts and we worked on multiple fronts:

• We tried to implement another algorithm from scratch.
  It was written as a temporary solution which was better than what we had today, but could be less optimal than another, low-latency specific, one.

• we worked on a separate tool, whose goal was to test our adaptive algorithm in various network conditions (poor bandwidth, high latency, variable conditions etc.) and compare it to other media players (mostly dash.js and the shaka-player).
  

  Example of a graph outputed by this tool. Between other things, it allows to quickly check the evolution of the video bitrate and of the size of the buffer

Thanks to this approach, we were able to develop a new algorithm with confidence.

That new algorithm was called the "guess-based [quality] chooser".
It may sound scary to have the word "guess" in there, but it translates well what this algorithm is doing: trying, gradually, to guess what the ideal quality is.

The guess-based chooser

The GuessedBasedChooser is actually a class in the RxPlayer that implements the corresponding "guess-based" algorithm.

It relies, among other inputs, on a core "maintainability score" concept linked to each quality.
This score indicates if we seem to be loading that quality faster or slower that the time it takes to play it:

• if we load it faster, this quality can be maintained without the RxPlayer rebuffering. In that case the quality will have a high maintainability score.
• if we load it slower than the time it takes to play it, the RxPlayer has a high risk of rebuffering (at some point, we'll have to wait for data to load). Here the quality will have a low maintainability score.

The idea is then to use that score (and other inputs), to know if a quality is maintainable or not, and take the corresponding actions depending on that:

• if it seems maintainable enough for a long time, we may try the immediately superior quality, we could say the RxPlayer is "guessing" that the immediately-superior quality might be maintainable.
• it it doesn't seem maintainable (even for a very short time), we might go the other way: switch to the immediately inferior quality. We call that situation "taking a wrong guess", as the current quality was not a "good" (maintainable) guess.

  
  Example of a wrong guess. The RxPlayer was initially playing the maintainable "black" video quality. It tries to play the better "blue" video quality but noticed that it was not maintainable. So it quickly went back on the black one.

The algorithm is a little more complex, to avoid oscillating between qualities:

• before "guessing" a superior quality, a large panel of conditions have to be met.
  Those conditions, which include some minimal buffer size and a high-enough maintainability score on the current quality, are there to limit the possibility to rebuffer while a potentially "wrong guess" is taken.

• While "guessing" a quality, if anything don't go as fast as it would like, the RxPlayer will consider that guess as "wrong" and fallback on the previous quality.

  recording.mp4

video (click to play): Here the RxPlayer, playing the maintainable "green" video quality, tries to play an unmaintainable "yellow" one. However it realized that it was not maintainable before the corresponding first segment was even completely loaded, thus it is replaced by the green quality when we switch back.

• After a "wrong guess" has been taken, the RxPlayer forbid any new guess during a small amount of time.
  That time increases exponentially each time a consecutive wrong guess is taken.
  This rule prevent the RxPlayer from oscillating too much between qualities when a lot of wrong (not maintainable) guesses are made.

Results

We tested that algorithm a lot in different situations and are very pleased with the result when playing low-latency contents.

output.mp4

video: Left: new "guess-based" adaptive logic, right: previous adaptive logic.
We can see that we're raising the quality (first to 2 Mbps then to 3 Mbps) with the guess-based one, but stay on the low 1.1 Mbps video quality with the previous logic (the video quality can be checked on the top right corner of each video).

Though it has many disadvantages when compared to the bandwidth-based and buffer-based ones:

• it is much slower to reach the optimal quality, as guesses are only made step by step and after enough data has been loaded for each quality - so we can classify it as "maintainable".
  On tested streams with 7 video qualities, it could take more than one minute before reaching the top quality

• it is more "risky" (i.e. there are more rebuffering risks) than the other approachs, as the player may be attempting to play a quality which is too high for the current network conditions while having a small buffer size to cushion the risks of rebuffering.

Due to these reasons, the GuessBasedChooser is only used when the lowLatencyMode option is set to true AND when the RxPlayer is close enough to the live edge. If any of those conditions is not met, regular approachs are used instead.

New documentation pages now with search capability

We also recently worked a lot to improve on our API documentation pages.

The RxPlayer is a library with a lot of features, some which might be complex and only used for advanced needs.
Consequently, its documentation pages have to be complete enough, yet easy-enough to navigate for the most used features.

We were already happy with our previous documentation pages, though there was some areas we knew we could do better. One of this area, which has also been an asked feature for a long time, was the presence of a search bar, allowing to search particular options or keywords without having to browse the whole documentation.

First attempt: Docusaurus

Our previous documentation pages were generated by an homemade documentation generator, by reading markdown files written inside the ./doc directory.
We initially wanted to use the Docusaurus library instead, as it also relied on markdown base files and seemed to have a lot of features including search capability.

We worked on a pretty big pull request to perform the migration, but we encountered several problems, among which: it was opinionated in a way that didn't suit us, installing it was not stable (at least at the time when we tested it), it was quite heavy (doubling the size of our node_modules, even if we're just talking about development dependencies) and the new markdown syntax was not as standard as we would have liked.

We thus went into a second direction: improving the generator we already have.

Final result

In the end, we took a lot of inspiration from multiple recent documentation generators, mostly docusaurus and mdBook, to improve on what we had.
The result looks like that:

The key improvements are:

• The documentation is now splitted into more pages, divided between three categories: Getting Started (which contains tutorials and is useful when beginning), API (which is the complete API documentation describing in every little details the behavior of the player) and the Reference (which shows all player options and methods in a single page, allowing a quick look-up into anything you can do with the player).
• A search feature. Here we opted for client-side research, relying on the elasticlunr library.
• Decryption options are all now in its own page instead of being included in a loadVideo options page.
  We felt that this was needed, due to how difficult-to-navigate the previous loadVideo options page was
• The demo and github pages are made accessible directly on any documentation page
• The API documentation for previous RxPlayer versions is also easily accessible, by clicking on the "version" link

This is a new improvement but you might already have noticed it. We were very happy with the result and thus already deployed the new documentation pages, applied to the previous v3.26.1 version, in late december!

DASH: Handling of <ContentProtection> at the <Representation> level

Overview

In one of the applications using the RxPlayer, it was reported that the advanced keySystems[].fallbackOn RxPlayer options - allowing to fallback from an undecipherable quality to a decipherable one when playing contents encrypted with multiple keys for different qualities - did not work.

After some investigation, we found out that this was because of the way their contents were packaged, more specifically about where the <ContentProtection> elements were found in the MPD, the DASH's Manifest file.
The location it was found at was right if following the DASH standard but wasn't when following the stricter DASH-IF interoperability points.

Although we most often try to mainly follow this second standard, we still chose here to add support for both.
This subject is somewhat complex to explain from scratch but because we think it's interesting enough, we'll dive into those explanations right now in this release note!

Initialization data and key ids

When playing encrypted contents, two important values may be first needed by the RxPlayer:

• The "initialization data", which is some binary data that will be used to construct and perform the right license request.
  Without it, no license request will be done which means encrypted content won't be able to play.
  

• The "key ids" (also spelt KID), which are identifiers for the wanted decryption keys. They will be used by the RxPlayer so that it can identify which key can be used and which cannot (for example, because the device is not secure enough).

  Without it, the RxPlayer won't be able to perform advanced features such as automatically fallbacking from qualities linked to undecipherable keys to ones linked to a decipherable one.

  

For all this to work, the RxPlayer thus needs to obtain both metadata.
Thankfully initialization data, which is the most important of the two, is generally found in multiple places, at least when speaking about DASH contents:

• on the MPD
• on initialization segments - which are the first segments loaded by the RxPlayer

This means that failure to recuperate it from the MPD for example, is not a fatality for the RxPlayer's ability to play an encrypted content. It can still rely on the initialization data found on the initialization segment.

Key ids however are often harder to obtain: the main place where they can be found is in the MPD, and although they might also generally be found inside the initialization data binary, a media player is not supposed to read it from there.
This means that if we cannot obtain the key id on the MPD when playing contents with multiple keys, we most probably won't be able to fallback from an undecipherable quality to a decipherable one, at least without some help from the application.

The <ContentProtection> element

In the MPD, both the initialization data and the key id are found in a <ContentProtection> element.
If following the strict DASH-IF specification, that element is present inside the <AdaptationSet> element, which typically specifies a track.

Example of a <ContentProtection> element inside an <AdaptationSet> element.

The fact that it is present at the track-level (i.e. in the <AdaptationSet> element) can however cause problem when multiple qualities (i.e. multiple DASH <Representation> elements) from the same track rely on different decryption keys.

In that case, multiple <AdaptationSet> elements have to be defined: one per key.
This create a big issue: most players will consider different <AdaptationSet> as different tracks, and will thus never try to switch between multiple ones. Here it means that even if network conditions are good enough (or bad enough) to switch to another quality, the player will only automatically change the quality if the new one share the same decryption key than the old one.

In this image the lower 360p-to-720p video qualities share the same decryption key and thus can be part of the same <AdaptationSet>. 1080p and 4k video qualities however are linked to another key and have to be declared as another <AdaptationSet>. As such, they are considered as different tracks by the player, which is thus not able to freely switch between those two quality sets.

Thankfully, a solution exist in the DASH-IF specification! It is possible to indicate that multiple <AdaptationSet> can be switched between one another, by adding a special "descriptor" (another XML element) inside the corresponding <AdaptationSet>.

That element indicates with which other <AdaptationSet> one can be switched to and looks like that:

Problems with that complexity: the reality

As you've seen, all could theoretically work fine by following the DASH-IF specification to the letter, but it is very cumbersome and unnecessarily complex.

Perhaps in consequence, some DASH packagers have taken the simpler route of announcing <ContentProtection> elements one level below, at the <Representation> level (i.e. at the quality-level and not at the track-level), which is also authorized by the more flexible DASH specification (only not by the stricter DASH-IF IOP).
And because the RxPlayer only followed the strict DASH-IF interoperability points until now on that subject, those contents could not profit from the RxPlayer's advanced keySystem[].fallbackOn features.

We've finally decided to now support the parsing of <ContentProtection> elements inside <Representation> elements.
This means that from the v3.26.2 version, <ContentProtection> can either be declared at the <AdaptationSet> level, as per the DASH-IF recommendation (in which case, you might want to add elements to the MPD to indicate switchability) or at the <Representation> level, as authorized by the DASH specification. In that second scenario, no supplementary element will be needed.

DASH: Better resilience for UTCTiming elements handling

When parsing DASH's Manifest, the MPD, the RxPlayer might need to perform HTTP(S) requests for supplementary resources.

Those can take different forms:

• There are some who are required to be able to play the content. I'm thinking here of xlink elements, which can be summarized as sub-parts of the MPD which are stored elsewhere.
• Some other can be seen more as optional improvements, for example an <UTCTiming> element might provide an URL allowing the player to synchronize itself with the current time (which might not originally be aligned between the client and the server).

Previously, the RxPlayer failed with a MANIFEST_PARSE_ERROR code if any of those external resources could not be fetched.
This caused some problems recently, due to a DASH-IF revision for low-latency content which included the following recommendation for the <UTCTiming> element:

It might not be evident just by looking at it, but one big issue with that recommendation is that it relies on an HTTP url, and not an HTTPS one.
On recent browsers, an application cannot requests HTTP resources on HTTPS pages by default. Because most pages are served using HTTPS nowadays, this led to low-latency contents following this DASH-IF revision to not be playable by the RxPlayer.

Because we felt the player could be more resilient here, the RxPlayer now makes a distinction between mandatory resources (such as xlink elements) and resources it could do without (for example if the <UTCTiming> element's URL is not reachable, we could just rely on the user's system clock instead).
In the latter case, the RxPlayer won't stop with an error anymore. It will still send the corresponding MANIFEST_PARSE_ERROR error object, but as a "warning" event, so you can still be notified when something goes wrong.

Even if the RxPlayer is now more resilient, we still recommend that all resource URLs use HTTPS, at least if you want them to be reachable by the player.

Longer testing periods for more stability guarantees

Stability is one of the key focus of the RxPlayer.
Under that principle, we previously performed a lot of manual tests (in addition to integration and unit tests) on different platforms before each release.

As new features arrive and more things need testing, we felt that what we do currently could be improved on. We thus are currently trying new ways to test new RxPlayer versions before they are released.
One of the solution put in place for this release, was to initially test the new RxPlayer internally at Canal+:
Some Canal+'s developers already rely on the v3.26.2 since january 3rd (2022).

This longer and wider testing period might allow to find more possible issues before releases. Although this is a good start - it even helped us find an issue before the official release, we're still studying new ways to improve on that testing phase. Stay tuned for more change in that regard!

Code readability and maintainability improvements

In the background, we also worked a lot toward improving the RxPlayer's maintainability.
As new generations of developers arrive on the project, and as we also want to improve the welcomeness of the code to outside contributors, we're recently considerably updated both the inside of the code and the way the RxPlayer is built:

• First, we continued our efforts (there's still a long way to go!) toward simplifying most of our RxJS-based code... generally by removing the usage of the RxJS library from it!
  RxJS is very useful for specific matters such as event handling, but we felt that its behavior can be hard to manage and understand on code that has a lot of side-effects.
  As a funny consequence, one of the nice unintended by-product of removing most RxJS code is much more readable call stack traces, which was a pleasant surprise after being used to inexploitable ones.
• We also simplified a lot the way the modular player (the one imported when importing from "rx-player/minimal") is built
• For developers, we also added a development dependency to esbuild which allows to test new code much faster than our previous webpack-based demo bundling logic.

Changelog

Bug fixes

• API: re-switch to SEEKING state instead of BUFFERING when seeking to already-buffered data [#1015]
• DASH: provide default startNumber attribute for number-based SegmentTemplate indexes with a SegmentTimeline [#1009]
• TTML (subtitles): interpret percentages as relative to the computed cell size and not as the percentage of the inherited font size in the page [#1013]
• subtitles: Work-around recent Chrome issue where the content of a native <track> element would still be visible despite being removed from the DOM (issue only reproducible in the "native" textTrackMode) [#1039]
• API: Fix rare issue happening when switching rapidly between Representations, which led to multiple APIs such as getAvailableVideoBitrate or getAvailableAudioTracks returning either incorrect or empty results [#1018]
• Improve prevention of rare segment-loading loops by fixing an issue with the clean-up of the short-term buffer history we maintain [#1045]

Other improvements

• DASH-LL: Improve adaptive bitrate logic on low-latency contents by implementing a specific algorithm for those [#1025, #1036]
• DASH-LL: Improve handling of $Time$-based DASH-LL contents [#1020]
• DASH: Support UTCTiming element with the urn:mpeg:dash:utc:http-xsdate:2014 scheme [#1021]
• DOC: Important refactoring of the RxPlayer API documentation to improve readability, discoverability and to add search capability to it [#1016]
• DASH: handle ContentProtection elements that have been defined at the Representation-level (and not at the AdaptationSet-level, as defined by the DASH-IF IOP) [#1027]
• DASH: Be resilient when the resource behind an UTCTiming element leads to an error (usually due to an HTTP-related issue) - instead of failing with an error like now [#1026]
• Better estimate the duration of ISOBMFF segments with multiple moof boxes [#1037]
• EME: Add hex-encoded key id to the KEY_STATUS_CHANGE_ERROR error message so we can know which key we're talking about when debugging [#1033]
• dev/scripts: for the "modular" (a.k.a. minimal) RxPlayer build now rely on TypeScript's const enums, instead of uglily using sed, to replace compile-time constants. [#1014]
• dev/scripts: remove reliance on environment variables when running the RxPlayer build scripts [#1004]
• dev/scripts: add esbuild devdependency and add "s" script to allow faster checks for RxPlayer developpers [#1003]
• CI: Rely on Github actions instead of Travis for most CI-related matters [#1046]
• code/refacto: replace central Clock concept (Observable bringing media-related updates to the RxPlayer at a regular pace) by a more flexible PlaybackObserver class [#1002]