FPS Throttling

[lucasdinonolte]

@bravomartin @runemadsen @fdoflorenzano leaving this here to start a discussion.

When working on credits for a movie we noticed the need to export at a different framerate than 60. We solved this for now, by still exporting at 60 and then let ffmpeg do the conversion afterwards.

However there is the possibility to add a frameRate to the webm-writer. Which is currently hardcoded at 60 (see #124 ).

What this PR does so far is two things:

• Allows to set frameRate in your functions settings. This setting is then given to the WebM writer and also fed back to the functions handler in the mechanic object. This will then export your function at a target framerate.
• Adds drawLoop utils that allow to also preview your function at your target frameRate. This is mostly inspired by how p5.js does frame rate throttling. This also up for discussion how tightly we want to couple these helpers with the mechanic logic. For now I added them as "stupid" functions that users would still have to connect to their functions themselves.

Let's discuss if this the right approach and if we even want something like this in mechanic, or if we just live with the ffmpeg conversion solution if you want a different framerate.

[fdoflorenzano]

Hey @lnolte! Thanks for looking into this! I'm not super familiar with video creation so this is huge help!

I guess one thing I wonder by looking at everything is how would a design function use this different pieces. Can you share an example of such a design function? Just as a code block here is fine.

[lucasdinonolte]

Sure. Here is a simple example using the React renderer

import React, { useEffect, useRef } from "react";

import { useDrawLoop } from "@mechanic-design/engine-react";

export const handler = ({ inputs, mechanic }) => {
  const { width, height, duration } = inputs;
  const { frame, done, frameRate } = mechanic;

  const maxFrames = duration * frameRate;

  const isPlaying = useRef(true);

  // User is responsible of setting up everything
  const frameCount = useDrawLoop(isPlaying.current, frameRate);

  // User is also responsible of tearing everything down
  useEffect(() => {
    if (frameCount <= maxFrames) {
      frame();
    } else if (isPlaying.current) {
      isPlaying.current = false;
      done();
    }
  }, [frameCount]);

  return (
    <svg width={width} height={height}>
      <rect x="0" y="0" width={width} height={height} fill="white" />
      <text x={width / 2} y={height / 2}>
        {frameCount}
      </text>
    </svg>
  );
};

export const inputs = {
  width: {
    type: "number",
    default: 640,
  },
  height: {
    type: "number",
    default: 480,
  },
  duration: {
    type: "number",
    default: 6,
  },
};

export const settings = {
  engine: require("@mechanic-design/engine-react"),
  animated: true,
  frameRate: 24,
};

So my initial idea was to keep things simple and just provide helpers for users to get the desired behavior. For the react renderer it's the useDrawLoop hook. For the other renderers it's the drawLoop utils as shown below:

import { drawLoop } from "@mechanic-design/core";

export const handler = ({ inputs, mechanic }) => {
  const { width, height, duration } = inputs;
  const maxFrames = duration * mechanic.frameRate;

  const canvas = document.createElement("canvas");
  canvas.width = width;
  canvas.height = height;
  const ctx = canvas.getContext("2d");

  drawLoop(({ frameCount, stop }) => {
    console.log(frameCount, mechanic);
    ctx.save();
    ctx.clearRect(0, 0, width, height);
    ctx.fillStyle = "white";
    ctx.fillRect(0, 0, width, height);
    ctx.fillStyle = "red";
    ctx.font = "48px serif";
    ctx.fillText(frameCount, width / 2, height / 2);
    ctx.restore();

    if (frameCount <= maxFrames) {
      mechanic.frame(canvas);
    } else {
      stop();
      mechanic.done(canvas);
    }
  }, mechanic.frameRate);
};

export const inputs = {
  width: {
    type: "number",
    default: 300,
    min: 100,
  },
  height: {
    type: "number",
    default: 300,
    min: 100,
  },
  duration: {
    type: "number",
    default: 2,
  },
};

export const settings = {
  engine: require("@mechanic-design/engine-canvas"),
  animated: true,
  frameRate: 40,
};

This really is just a first draft. We could also think about integrating these helper functions tighter with the mechanic instance – because the instance would know the frameRate… But I think it depends on how "smart" we want mechanic to become.

Let me know if this makes sense. We could also see if we can schedule a quick call about this in the coming days 😊

[fdoflorenzano]

Thanks for these examples! It's all making sense. I think it's a good angle, I can't think of why not doing it like this.

I would want to test this examples before merging, and right now I have a little bit of hell with other pending branch. Is this urgent to get to main and release it?

In the meantime, if it all works, I would consider adapting some of the create-mechanic video examples to use this utilities too. Maybe if you have time you can check that out sometime while I test your base examples.

And sorry for taking this long to respond. This is great.

[lucasdinonolte]

@fdoflorenzano thanks, I'll adapt the create-mechanic examples. I also think there is no hurry in merging this. It's just something I wanted to explore while working on movie credits using mechanic.

Also yesterday @bravomartin had some cool ideas about the possibilities of a frame-based (rather than time-based) animation-approach in Mechanic. So I'd like him to have a look at this as well, to see if this could be a starting point for his ideas.

[bravomartin]

Love this! I would use the push to also add the ability to override the quality setting of the export?

[lucasdinonolte]

Closing this in favor of #152