Video support

[jimjam-slam]

I've added the first part of this work in 998617c and 485f235: videos that autoplay and loop. Really it's just a class to ensure that they run full-bleed; the looping and autoplay is done by the browser.

But as #132 discusses, we might want video to hold off starting until it's visible. So the next part is to trigger that manually wth JS.

I think it probably makes sense to add a shortcut, analogous to Quarto's video shortcode, to do this while inserting the necessary attributes (eg. preload) on the video tag. (In fact, we may end up borrowing their chortcode and tweaking it!)

The last part of this is the most complicated: image sequences that progress as you scroll through the container (although I'd love to know if you can do this with a traditional video!).

Most of the implementations of this I've seen (eg. this one on dev.to — it uses React, but the principle should apply with vanilla JS) use a canvas element: you preload the images by calling a function ASAP to download them, then call requestAnimationFrame() on scroll to update which image is showing.

A user might either want to use regenerated images or ones generated by a code block in the doc — but it isn't clear to me whether there's some special treatment we can give, for example, an R code block to say "use the images emitted from this code block for a scroll video".

That said, perhaps you could have:

a. a sticky block where the image glob or image path is specified as an attribute, and then
b. a trigger attribute that specifies how far through the sequence. Or perhaps a progress block... or perhaps both are viable options?

::::{.cr-section}

<!-- or maybe a shortcode makes sense for this sticky? -->
:::{#cr-images video-scroll-first="images/pic001.png" video-scroll-last="images/pic135.png"}
:::

Check out this awesome video [@cr-images]{video-scroll=1}

It's pretty rad [@cr-images]{video-scroll=20}

Now we're going quite fast [@cr-images]{video-scroll=80}

We've paused for a bit [@cr-images]{video-scroll=80}

Now we're finished [@cr-images]{video-scroll=120}

::::

Or maybe with a progress block (although no option to vary the speed here):

<!-- can we reuse focus-on and detect that it's scrolling? -->
:::{.progress-block focus-on="cr-images"}

Step 1

Step 2

Step 3

Step 4
:::

What do you think, @andrewpbray?

[andrewpbray]

Thanks for getting this started! Lots of fun stuff here.

Feature 1: Play videos with trigger

In terms of API, what do you think of:

Checkout this lovely video. [@cr-vid]{play-video="true"}

I'm inclined to avoid any mention of autoplay since that's less well-defined in this setting since the user is in control of when the video would become un-transparent. Not sure what the best attribute name is: play, play-vid, play-video? I can take a crack at the necessary JS for this later on today.

Feature 2: Control video progression with scroll

Seems simpler to use a progress block! Controlling the play speed could always be done on the video file itself in pre-processing, right? (i.e. removing frames).

I'm not quite sure what the best API for this is. I like what you have. There's also this:

:::{.progress-block scroll-video="cr-vid"}

Step 1

Step 2

Step 3

Step 4
:::

A few things:

• While we could infer what to do if a progress block is focusing on a video, might be better to make that explicit in the attribute name.
• Seems like we'll be able to do the png sequence and less clear about setting the frame of the video file. That'd be good to know for deciding if we should use separate attribute names for those two or combine them.
• Seeing scroll-video here makes me wonder if start-video is clearer for feature 1 (since scrolling is also a form a playing a video).

[jimjam-slam]

@andrewpbray Agree with you on Feature 1! autoplay is too confusing (and it's a browser attribute, so it ought to be avoided).

I agree that a progress block is a nice interface! While I do also agree that you can change the number of frames to tweak the output speed, I think at minimum it would be good for users to be able to pause the progression between a number of steps where they want to stop and discuss something in more detail. You could technically run duplicate frames off, but it'd be a big waste of bandwidth. Maybe something like:

:::{.progress-block scroll-video="cr-vid"}

Step 1

Step 2 [@cr-vid]{pause-video}

Step 3 [@cr-vid]{pause-video}

<!-- video resumes as you pass this step -->
Step 4

Step 5
:::

These sorts of pauses would be helpful for videos that sweep over an area (eg. as events unfold over time).

[andrewpbray]

Good point - that'd be some very useful functionality.

One thing I'm realizing: we have crProgressTrigger and crProgressBlock. Do we want an interface that would, in concept, work for both?

Scroll a video through a single narrative block

:::{#cr-myvid}
{{< video myvid.mov >}}
:::

Step 1. [@cr-myvid]{.scroll-video}

Scroll a video through a progress block

:::{#cr-myvid}
{{< video myvid.mov >}}
:::

:::{.scroll-video focus-on="cr-myvid"}

Step 1.

Step 2.

Step 3.
:::

This raises a few interface questions:

1. Should focus effects always be attributes or can they be classes in situations where this is no arguments/values that we need to pass?
2. How should we get the trigger and progress block version aligned? For the former, .scroll-video or scroll-video="true" seem to make sense since the ref to the sticky is right there. But then how should we pass the ref to the sticky in the progress block version? focus-on is one option but would actually mess things up: I think it'd put all three steps as separate Paras in the same narrative block without the padding. .progress-block scroll-video="cr-vid" would work but then we have a different interface than the single trigger scroll.
3. If we use .progress-block scroll-video="cr-vid", do we even need .progress-block? Seems like no. Seems to make sense to keep that one narrowly scoped to exposing that ojs variable.

[andrewpbray]

Maybe we wade into getting a working example of an image sequence and see how it turns out. That'll probably help clarify which interface makes the most sense. Looking back, my stuff above is all based on the general notion of a "video" but image sequences are a slightly different beast; there's not just one file to point to.

I love the {{< image-sequence start="file1.png" end="file15.png" >}} idea by the way! (or something in that neighborhood.

[andrewpbray]

@jimjam-slam Here's a first pass at implementing feature 1.

It's implemented generically so that play-video="true" and pause-video="true" both work, but really any available video method will work when passed as ___-video="true".

One thing to decide is question 1 above. Should these be classes instead of attributes? This should be determined I think just based on what seems like the best interface. The implementation is straightforward - change this JS function to reference classes instead of attributes and then modify the lua filter so that a trigger will pass both classes and attributes to its enclosing block (wrap_block currently only passes attributes).

While testing this out, I did bump into this: https://developer.chrome.com/blog/autoplay/. Essentially, if the user does nothing but scroll straight down, it might not execute play().

[andrewpbray]

I've been tinkering this morning with tying progressBlock.progress to video.currentTime; effectively skipping the video to an appropriate frame as the progressBlock increments. I haven't quite gotten it working yet but just discovered this:

https://scrollyvideo.js.org/

In order words, I've been trying the "traditional" method 3. What are your thoughts on using a dedicated library for this sort of thing? It looks like the author has been working is within the last 3 mo, which is a good sign. If these video standards change over time, sure would be nice if we could take advantage of his keeping things up to date.

[jimjam-slam]

I've gotta come back to your comments properly, sorry @andrewpbray! I had half an hour and banged out some rayshader code to generate a sample video.

[jimjam-slam]

One thing to decide is question 1 above. Should these be classes instead of attributes? This should be determined I think just based on what seems like the best interface. The implementation is straightforward - change this JS function to reference classes instead of attributes and then modify the lua filter so that a trigger will pass both classes and attributes to its enclosing block (wrap_block currently only passes attributes).

Usually boolean attributes in HTML work without the ="true" part, so a user could just add play-video or pause-video. That doesn't presently work, but I'm guessing that the current implementation explicitly checks in Lua for a value of true (rather than just the presence of the attribute key).

Okay, I've just deleted a big wall of text — turns out that apart from specific HTML boolean attributes (like the ones that actually go on video elements), where the mere presence of the boolean attribute makes it true, attributes are always supposed to have values. So maybe {play-video} is a bad idea and it should be a class (in fact, Pandoc aborts processing on a span or div when it encounters an attribute with no value, so we'd need to implement a Lua filter to make it work anyway!).

[jimjam-slam]

I've been tinkering this morning with tying progressBlock.progress to video.currentTime; effectively skipping the video to an appropriate frame as the progressBlock increments. I haven't quite gotten it working yet but just discovered this:

https://scrollyvideo.js.org/

In order words, I've been trying the "traditional" method 3. What are your thoughts on using a dedicated library for this sort of thing? It looks like the author has been working is within the last 3 mo, which is a good sign. If these video standards change over time, sure would be nice if we could take advantage of his keeping things up to date.

I'll try hooking this up now — if we can do it with an external lib, I'm down for that! It seems like it's handling a few tasks (ie. it's not just setting the progress as we probably would), so it probably makes sense to have a dependency handle it.

On a semi-related note (which I'll spin out to a separate issue later), it might be worth us incorporating a web bundler to track our external dependencies...

[jimjam-slam]

One other thing: I realised that when I set up the demo with the ship and tea videos, I wrote them inline:

![](tea.mp4){#cr-tea .scale-to-cover alt="One person pours tea into another person's tea cup in front of an open fire." autoplay="true" loop="true"}

As opposed to:

:::{#cr-tea}
![](tea.mp4){.scale-to-cover alt="One person pours tea into another person's tea cup in front of an open fire." autoplay="true" loop="true"}
:::

or:

:::{#cr-tea .scale-to-cover}
![](tea.mp4){alt="One person pours tea into another person's tea cup in front of an open fire." autoplay="true" loop="true"}
:::

The commit I've just pushed generalises the CSS a bit more so that it should work regardless of whether you put .scale-to-cover and the Closeread ID on the fenced div or directly on the video (or the ID on the div and the class on the element). It probably still needs some testing with respect to the figure problems we've encountered in the past.

[jimjam-slam]

I've managed to load the video with ScrollyVideo, but the default trackscroll: true doesn't work — I think because the demos on their page have the sticky video as a direct sibling of the scrolling cards, whereas we have them split up. I think we probably need to use the provided method .setVideoPercentage() in the relevant OJS scroller (keeping in mind that, as you proposed, we may want a progress block-based option!). Will try to keep at this tomorrow!

[jimjam-slam]

a40df27 has a single scrollytelling video working (I've hardcoded it straight into the .js file, so it'll break the rest of the site). So it seems like it works conceptually!

This demo just ties things to trigger progress, so when you scroll from one trigger to another, it goes from 100% to 0% and the video essentially rewinds quickly. I think for that case you'd probably want no interpolation at all (if, indeed, it is require usually!).

Generalising this means working out what the JS code has to track in order to potentially advance several videos. I think it's probably worth us talking through the API in person @andrewpbray before I jump into that!

[jimjam-slam]

Just summarising our discussion @andrewpbray — this was a candidate API we were looking at (but still discussing if either of us is unhappy with it!):

Non-scrolling videos

::::{.cr-section layout="overlay-left"}

You can tell all sorts of stories with videos.

Maybe you've gone on a journey.  @cr-ship

:::{#cr-ship .scale-to-cover}
![](ship.mp4){alt="A ship sails in a polar region" loop="true"}
:::

You can trigger a video to play using `[@cr-ship]{play-video="true"}`. [@cr-ship]{.play-video}

You can also pause it using `[@cr-ship]{pause-video="true"}`. [@cr-ship]{.pause-video}

Play the video again [@cr-ship]{.play-video}

The video keeps playing

One thing I'd noticed but might spin into a separate issue is that we never show a sticky until the first trigger hits. Videos are a case where you'd probably want the first sticky to be visible without having to hit a sticky first, but there might be others (in fact it might even be an appropriate default).

Scrolling videos

:::::{.cr-section}

::::{.progress-block scroll-video="cr-rayshader"}
Here's our rayshader video!

We can keep talking about it for a while... [@cr-rayshader]{.pause-video}

... and a while longer!
::::

:::{#cr-rayshader}
:::
:::::

Some considerations for scrolling video:

Right now this scrolling video is hardcoded in to the JS code, but in real use it needs to be specified in the QMD. Two ways we could do that:

1. It's specified in the progress block: `scroll-video="cr-some_id"` becomes `scroll-video="rayshader.mp4"` and the video sticky block is added programmatically by Lua instead of being supplied by the user
2. a video is supplied in the sticky just as with a non-scrolling video. If we detect `scroll-video` attribute and a video inside the corresponding ID in the Lua, we remove the video and decorate the `cr-section` with the video URL so that the JS code can hook it up.

We might want to consider how our choice in API impacts people's ability to transition from a scrolling video to some other type of sticky (is there any impact?).