From 83e4657e1c5dd7949c36948a10926c051a67b73a Mon Sep 17 00:00:00 2001 From: james7132 Date: Fri, 11 Feb 2022 13:15:03 -0800 Subject: [PATCH 01/14] Start #49: Animation Primitives --- rfcs/48-animation-primitives.md | 154 ++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 rfcs/48-animation-primitives.md diff --git a/rfcs/48-animation-primitives.md b/rfcs/48-animation-primitives.md new file mode 100644 index 00000000..9f3b3fd0 --- /dev/null +++ b/rfcs/48-animation-primitives.md @@ -0,0 +1,154 @@ +# Feature Name: `animation-primitives` + +## Summary + +Animation is a particularly complex, with many stateful and intersecting +elements. This RFC aims to detail a set of lowest-level APIs for authoring and +playing animations within Bevy. + +## Motivation + +Animation is at the heart of modern game development. A game engine without an +animation system is generally considered not yet production ready. + +This RFC aims to detail the absolute lowest level APIs to unblock ecosystem +level experimentation with building more complex animation systems (i.e. inverse +kinematics, animation state machine, humanoid retargetting, etc.) + +## Scope + +Animation is a huge area that spans multiple problem domains: + + 1. **Storage**: this generally covers on-disk storage in the form of assets as + well as the in-memory representation of static animation data. + 2. **Sampling**: this is how we sample the animation assets over time and + transform it into what is applied to the animated entities. + 3. **Application**: this is how we applied the sampled values to animated + entities. + 4. **Composition**: this is how we compose simple clips to make more complex + animated behaviors. + 4. **Authoring**: this is how we create the animation assets used by the engine. + +This RFC specifically aims to resolve only problems within the domains of storage +and sampling. Application can be distinctly decoupled from these earlier two +stages, treating the sampled values as a black box output, and composition and +authoring can be built separately upon the primitives provided by this RFC and +thus are explicit non-goals here. + +## User-facing explanation + +The core of this system is a trait called `Sample` which allows sampling +values of `T` from an underlying type at a provided time. `T` here can be +anything considered animatable. A few examples of high priority types to be +supported here are: + + - `f32`/`f64` + - `Vec2`/`Vec3`/`Vec3A`/`Vec4` (and their `f64` variants) + - `Color` + - `Option` where `T` can also be sampled + - `bool` for toggling on and off features. + - `Range` for a range for randomly sampling from (think particle systems) + - `Handle` for sprite animation, though can be generically used for any asset + swapping. + - etc. + +Built on top of this trait is the concept of a **animation graph**, a runtime +mutable directed acyclic graph of nodes for altering the sampling behavior for +a given property. There is always one root level node that is directly sampled +from the app world. It can either be a terminal node, or be a composition node +that samples it's children for values and combines the outputs before passing it +upstream. Some examples include: + + - `MixerNode` - a stateful multi-input composition node that outputs a + weighted sum of it's inputs. Can be used to make arbitrary blending of it's + inputs. + - `SelectorNode` - a multi-input composition node that outputs only the + currently selected input as it's output. All other inputs are not evaluated. + Like a MixerNode with a one-hot weight blend, but more computationally + efficient. + - `ConstantNode` - only outputs a constant value all times. + - `RepeatNode` - a single input node that loops it's initial input over time. + - `PingPongNode` - a single input node that loops it's initial input over + time, will + - `Arc>`/`Box>` - Anything that can be sampled can + be used as an input to the graph. + +The final lowest level and the concrete implementors of Sample are implemetors +of the trait `Curve`. Curves are meant to store the raw data for time-sampled +values. There may be multiple implementations of this trait, and they're +typically what is serialized and stored in assets. + +Finally the last major introduction is the `AnimationClip` asset, which bundles a +set of curves to the asoociated `Reflect` path they're bound to. This is the main +metadata source for actually binding sampled outputs to the fields they're +animating. + +## Implementation strategy + +Protoytpe implementation: https://github.com/HouraiTeahouse/bevy_prototype_animation + +TODO: Complete this section. + +## Drawbacks + +The main drawback to this approach is code complexity. There are a lot of `dyn +Trait` or `impl Trait` in these APIs and it might get a bit difficult to follow +without external 1000ft views of the entire system. However, this is meant to be +a flexible low-level API so this might be easier to gloss over once a more higher +level solution built on top of it is made. + +## Rationale and alternatives + +Bevy absolutely needs some form of a first-party animation system, no modern game +engine can be called production ready without one. Having this exist solely as a +third-party ecosystem crate is definitely unacceptable as it would promote a +facturing of the ecosystem with multiple incompatible baseline animation system +implementations. + +The design chosen here was explicitly to allow for maximum flexibility for both +engine and game developers alike. The main alternative is to completely remove +`Sample`, the animation graph, and it's nodes, and let developers directly +hook up animation curves from clips to entities. However, this lacks flexibility +and does not allow for users of the API to inject their own alterations to the +animation stream. + +The main potential issue with the current implementation is the very heavy use of +`Arc>` which has CPU cache, lifetime, and performance implications +on the stored data. Atomic operations disrupt the CPU cache; however, they're +only used when cloning or dropping an `Arc`. The structure of the animation +graphs are, for the most part, static. Likewise, the trait object use is likely +unavoidable so long as we rely on traits as a point of abstraction within the +graph. An alternative mgiht be to we want to transfer ownership of the curve, and +just make multiple copies of a potentially large and immutable animation data +buffer, but that comes with a signfigant memory and CPU cache performance cost. + +## Prior art + +This proposal is largely inspired by Unity's [Playable][playable] API, which has +a similar goal of building composable time-sequenced graphs for animation, audio, +and game logic. Several other game engines have very similar APIs and features: + + - Unreal has [AnimGraph][animgraph] for creating dynamic animations in + Blueprints. + - Godot has [animation trees][animation-trees] for creating dynamic animations in + Blueprints. + +The proposed API here doesn't purport or aim to directly replicate the features +seen in these other engines, but provide the absolute bare minimum API so that +engine developers or game developers can build them if they need to. + +Currently nothing like this exists in the entire Rust ecosystem. + +Note that while precedent set by other engines is some motivation, it does not on its own motivate an RFC. + +[playable]: https://docs.unity3d.com/Manual/Playables.html +[animgraph]: https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/SkeletalMeshAnimation/AnimBlueprints/AnimGraph/ +[animation-trees]: https://docs.godotengine.org/en/stable/tutorials/animation/animation_tree.html + +## Unresolved questions + +TODO: Complete + +## Future possibilities + +TODO: Complete From 718ffafae0955d85b780aa603a002b5da57d7252 Mon Sep 17 00:00:00 2001 From: james7132 Date: Fri, 11 Feb 2022 13:17:12 -0800 Subject: [PATCH 02/14] Fix RFC number --- rfcs/{48-animation-primitives.md => 49-animation-primitives.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename rfcs/{48-animation-primitives.md => 49-animation-primitives.md} (100%) diff --git a/rfcs/48-animation-primitives.md b/rfcs/49-animation-primitives.md similarity index 100% rename from rfcs/48-animation-primitives.md rename to rfcs/49-animation-primitives.md From 929f99a6f74cd8707f013cb70707b0d0c23111e7 Mon Sep 17 00:00:00 2001 From: James Liu Date: Fri, 11 Feb 2022 13:39:44 -0800 Subject: [PATCH 03/14] Apply suggested fix Co-authored-by: Alice Cecile --- rfcs/49-animation-primitives.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index 9f3b3fd0..bad1e018 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -9,7 +9,7 @@ playing animations within Bevy. ## Motivation Animation is at the heart of modern game development. A game engine without an -animation system is generally considered not yet production ready. +animation system is generally not considered production-ready. This RFC aims to detail the absolute lowest level APIs to unblock ecosystem level experimentation with building more complex animation systems (i.e. inverse From c79dd500c75965df361f0f3f1aa8e94342d0d8e7 Mon Sep 17 00:00:00 2001 From: James Liu Date: Fri, 11 Feb 2022 13:39:54 -0800 Subject: [PATCH 04/14] Apply suggested fix Co-authored-by: Alice Cecile --- rfcs/49-animation-primitives.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index bad1e018..c2426690 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -11,7 +11,7 @@ playing animations within Bevy. Animation is at the heart of modern game development. A game engine without an animation system is generally not considered production-ready. -This RFC aims to detail the absolute lowest level APIs to unblock ecosystem +This RFC aims to detail the absolute lowest-level APIs to unblock ecosystem level experimentation with building more complex animation systems (i.e. inverse kinematics, animation state machine, humanoid retargetting, etc.) From 4dbdc3ca88e914bf1f2367ff46308c6920d792eb Mon Sep 17 00:00:00 2001 From: james7132 Date: Fri, 11 Feb 2022 13:32:54 -0800 Subject: [PATCH 05/14] Fix grammar --- rfcs/49-animation-primitives.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index c2426690..e1bdaa1c 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -2,7 +2,7 @@ ## Summary -Animation is a particularly complex, with many stateful and intersecting +Animation is particularly complex, with many stateful and intersecting elements. This RFC aims to detail a set of lowest-level APIs for authoring and playing animations within Bevy. @@ -118,9 +118,10 @@ on the stored data. Atomic operations disrupt the CPU cache; however, they're only used when cloning or dropping an `Arc`. The structure of the animation graphs are, for the most part, static. Likewise, the trait object use is likely unavoidable so long as we rely on traits as a point of abstraction within the -graph. An alternative mgiht be to we want to transfer ownership of the curve, and -just make multiple copies of a potentially large and immutable animation data -buffer, but that comes with a signfigant memory and CPU cache performance cost. +graph. One alternative would be to transfer ownership of the Sample implementation, +and/or just make multiple copies of a potentially large and immutable animation +data buffer, but that comes with a signfigant memory and CPU cache performance +cost. ## Prior art From 3863ca0575c24400e1f7995717682309b93f7cfe Mon Sep 17 00:00:00 2001 From: James Liu Date: Fri, 11 Feb 2022 13:55:41 -0800 Subject: [PATCH 06/14] Apply typo fix Co-authored-by: Alice Cecile --- rfcs/49-animation-primitives.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index e1bdaa1c..47ffe3b4 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -85,7 +85,7 @@ animating. ## Implementation strategy -Protoytpe implementation: https://github.com/HouraiTeahouse/bevy_prototype_animation +Prototype implementation: https://github.com/HouraiTeahouse/bevy_prototype_animation TODO: Complete this section. From 92e0f08e7548d4430e7e4da86d882e4a65a35556 Mon Sep 17 00:00:00 2001 From: James Liu Date: Fri, 11 Feb 2022 13:55:57 -0800 Subject: [PATCH 07/14] Remove definitely Co-authored-by: Alice Cecile --- rfcs/49-animation-primitives.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index 47ffe3b4..31e3afd3 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -101,7 +101,7 @@ level solution built on top of it is made. Bevy absolutely needs some form of a first-party animation system, no modern game engine can be called production ready without one. Having this exist solely as a -third-party ecosystem crate is definitely unacceptable as it would promote a +third-party ecosystem crate is unacceptable as it would promote a facturing of the ecosystem with multiple incompatible baseline animation system implementations. From cd55bc29060caf7dc40b4a77af331555cbaf0518 Mon Sep 17 00:00:00 2001 From: James Liu Date: Fri, 11 Feb 2022 14:00:57 -0800 Subject: [PATCH 08/14] Remove template piece. Co-authored-by: Alice Cecile --- rfcs/49-animation-primitives.md | 1 - 1 file changed, 1 deletion(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index 31e3afd3..773204ed 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -140,7 +140,6 @@ engine developers or game developers can build them if they need to. Currently nothing like this exists in the entire Rust ecosystem. -Note that while precedent set by other engines is some motivation, it does not on its own motivate an RFC. [playable]: https://docs.unity3d.com/Manual/Playables.html [animgraph]: https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/SkeletalMeshAnimation/AnimBlueprints/AnimGraph/ From 7bcd08fe3922a06cf6585200d77d8cafe13bfe74 Mon Sep 17 00:00:00 2001 From: james7132 Date: Sat, 19 Feb 2022 23:33:21 -0800 Subject: [PATCH 09/14] Reorient for storage and sampling --- rfcs/49-animation-primitives.md | 258 ++++++++++++++++++++++++-------- 1 file changed, 195 insertions(+), 63 deletions(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index 773204ed..f2edb1a4 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -29,73 +29,223 @@ Animation is a huge area that spans multiple problem domains: animated behaviors. 4. **Authoring**: this is how we create the animation assets used by the engine. -This RFC specifically aims to resolve only problems within the domains of storage -and sampling. Application can be distinctly decoupled from these earlier two -stages, treating the sampled values as a black box output, and composition and +This RFC primarily aims to resolve only problems within the domains of storage +and sampling. Composition will only be touched briefly upon for the purposes of +animated type selection. Application can be distinctly decoupled from these +earlier two stages, treating the sampled values as a black box output, and authoring can be built separately upon the primitives provided by this RFC and thus are explicit non-goals here. ## User-facing explanation -The core of this system is a trait called `Sample` which allows sampling -values of `T` from an underlying type at a provided time. `T` here can be -anything considered animatable. A few examples of high priority types to be -supported here are: +The core of this system is a trait called `Curve` which defines a serializable +set of time-sequenced values that a user can opaquely sample values of type `T` +at a provided time. `T` here can be anything considered animatable. A few +examples of high priority types to be supported here are: - `f32`/`f64` - `Vec2`/`Vec3`/`Vec3A`/`Vec4` (and their `f64` variants) - `Color` - - `Option` where `T` can also be sampled - `bool` for toggling on and off features. - `Range` for a range for randomly sampling from (think particle systems) - `Handle` for sprite animation, though can be generically used for any asset swapping. - etc. -Built on top of this trait is the concept of a **animation graph**, a runtime -mutable directed acyclic graph of nodes for altering the sampling behavior for -a given property. There is always one root level node that is directly sampled -from the app world. It can either be a terminal node, or be a composition node -that samples it's children for values and combines the outputs before passing it -upstream. Some examples include: - - - `MixerNode` - a stateful multi-input composition node that outputs a - weighted sum of it's inputs. Can be used to make arbitrary blending of it's - inputs. - - `SelectorNode` - a multi-input composition node that outputs only the - currently selected input as it's output. All other inputs are not evaluated. - Like a MixerNode with a one-hot weight blend, but more computationally - efficient. - - `ConstantNode` - only outputs a constant value all times. - - `RepeatNode` - a single input node that loops it's initial input over time. - - `PingPongNode` - a single input node that loops it's initial input over - time, will - - `Arc>`/`Box>` - Anything that can be sampled can - be used as an input to the graph. - -The final lowest level and the concrete implementors of Sample are implemetors -of the trait `Curve`. Curves are meant to store the raw data for time-sampled -values. There may be multiple implementations of this trait, and they're -typically what is serialized and stored in assets. - -Finally the last major introduction is the `AnimationClip` asset, which bundles a -set of curves to the asoociated `Reflect` path they're bound to. This is the main -metadata source for actually binding sampled outputs to the fields they're -animating. +These curves can be used by themselves in dedicated contexts (i.e. particle +system sampling), or used as a part of a `AnimationClip` asset. +AnimationClips bundle multiple curves together in a cohesive manner to allow +sampling all curves in the clip together. Curves are keyed by the associated +`Reflect` path that is animated by the curve. AnimationClips are meant to sample +most, if not all, of the curves at the same time. ## Implementation strategy Prototype implementation: https://github.com/HouraiTeahouse/bevy_prototype_animation +### `Curve` Trait + +TODO: Detail why this exists. + +```rust +trait Curve { + fn duration(&self) -> f32; + fn sample(&self, time: f32) -> T; +} +``` + +#### `Animatable` Trait + +The sampled values of `Curve` need to implement `Animatable`, a trait that +allows interpolating and blending the stored values in curves. The general trait +may look like the following: + +```rust +struct BlendInput { + weight: f32, + value: T, +} + +trait Animatable { + fn interpolate(a: &Self, b: &Self, time: f32) -> Self; + fn blend(inputs: impl Iterator>) -> Self; +} +``` + +`interpolate` implements interpolation between two values of a given type given a +time. This typically will be a [linear interpolation][lerp], and have the `time` +parameter clamped to the domain of [0, 1]. However, this may not necessarily be +strictly be a continuous interpolation for discrete types like the integral +types or `Handle`. This may also be implemented as [spherical linear +interpolation][slerp] for quaternions. This will typically be required to +provide smooth sampling from the variety of curve implementations. If it is +desirable to "override" the default lerp behavior, newtype'ing an underlying +`Animatable` type and implementing `Animatable` on the newtype instead. + +`blend` expands upon this and provides a way to blend a collection of weighted +inputs into one output. This can be used as the base primitive implementation for +building more complex compositional systems. For typical numerical types, this +will often come out to just be a weighted sum. For non-continuous discrete types +like `Handle`, it may select the highest weighted input. + +A blanket implementation could be done on types that implement `Add + +Mul`, though this might conflict with a need for specialized +implementations for the following types: + - `Vec3` - needed to take advantage of SIMD instructions via `Vec3A`. + - `Handle` - need to properly use `clone_weak`. + +[lerp]: https://en.wikipedia.org/wiki/Linear_interpolation +[slerp]: https://en.wikipedia.org/wiki/Slerp + +#### Concrete Implementation: `CurveFixed` + +`CurveFixed` is the typical initial attempt at implementing an animation curve. +It assumes a fixed frame rate and stores each frame as individual elements in a +`Vec` or `Box<[T]>`. Sampling simply requires finding the corresponding +indexes in the buffer and calling `interpolate` on with the between the two +frames. This is very fast, typically incuring only the cost of one cache miss per +curve sampled. The main downside to this is the memory usage: a keyframe must be +stored every for `1 / framerate` seconds, even if the value does not change, +which may lead to storing a large amount of redundant data. + +#### Concrete Implementation: `CurveVariable` + +`CurveVariable`, like `CurveFixed`, stores its keyframe values in a contiguous +buffer like `Vec`. However, it stores the exact keyframe times in a parallel +`Vec`, allowing for the time between keyframes to be variable. This comes at a +cost: sampling the curve requires performing a binary search to find the +surroudning keyframe before sampling a value from the curve itself. This can +result in multiple cache misses just to sample from one curve. + +This cost can be mitigated by using cursor based sampling, where sampling returns +a keyframe index alongside the sampled value, which can be reused when sampling +again to minimize the time spent searching for the correct value. This notably +complicates sampling from these curves + +#### Concrete Implementation: `CurveVariableLinear` + +TODO: Complete this section. + +### Special Case: Mesh Deformation Animation (Skeletal/Morph) + +The heaviest use case for this animation system is for 3D mesh deformation +animation. This includes [skeletal animation][skeletal_animation] and [morph +target animation][morph_target_animation]. This section will not get into the +requisite rendering techniques to display the results of said deformations and +only focus on how to store curves for animation. + +Both `AnimationClip` and surrounding types that utilize it may need a specialized +shortcut for accessing and sampling `Transform` based poses from the clip. + +[skeletal_animation]: https://en.wikipedia.org/wiki/Skeletal_animation +[morph_target_animation]: https://en.wikipedia.org/wiki/Morph_target_animation + +### Curve Compression and Quantization + +Curves can be quite big. Each buffer can be quite large depending on the number +of keyframes and the representation of said keyframes, and a typical game with +multiple character animations may several hundreds or thousands of curves to +sample from. To both help cut down on memory usage and minimize cache misses, +it's useful to compress the most usecases seen by a + +`f32` is by far the most commonly animated type, and is foundational for building +more complex types (i.e. Vec2, Vec3, Vec4, Quat, Transform), which makes it a +prime target for compression. An effective method of compression is use of +quantization, where the upper and lower bounds of a curve are computed, and all +intermediate values are stored as discrete `u16`. This can cut memory usage of +`f32` curves by up to 50%. As `u16` can be converted back to `f32` losslessly, +the only loss in accuracy is when a source curve is quantized. An example of how +an implementation of how a struct might look can be seen below. + +```rust +struct QuantizedFloatCurve { + frame_rate: f32, + start_time: f32, + min_value: f32, + increment: f32, + frames: Vec, +} +``` + +One optional optimization for `Curve` is to use a bitvector to store values +instead a `Vec`, which would allow storing up to 8 fixed-framerate keyframes in +one byte. + +Another common low hanging fruit for compression is static curves: curves with a +singular value throughout the full duration of the curve itself. This drops the +need to store a full `Vec` to store only one value. This can be easily +represented as a enum. Extending the above example implementation, a resultant +implemntation might look like the following: + +```rust +enum CompressedFloatCurve { + Static { + start_time: f32, + duration: f32, + }, + Quantized { + start_time: f32, + frame_rate: f32, + min_value: f32, + increment: f32, + frames: Vec, + }, +} +``` + +TODO: Add note about [quaternion compression][quat-compression] + +We can then use this compose these compressed structs together to construct +more complex curves: + +```rust +struct CompressedVec3Curve { + x: CompressedVec3Curve, + y: CompressedVec3Curve, + z: CompressedVec3Curve, +} + +struct CompressedTransformCurve { + translation: CompressedVec3Curve, + scale: CompressedVec3Curve, + rotation: CompressedQuatCurve, +} +``` + +[quat_compression]: https://technology.riotgames.com/news/compressing-skeletal-animation-data + +### `AnimationClip` + TODO: Complete this section. ## Drawbacks -The main drawback to this approach is code complexity. There are a lot of `dyn -Trait` or `impl Trait` in these APIs and it might get a bit difficult to follow -without external 1000ft views of the entire system. However, this is meant to be -a flexible low-level API so this might be easier to gloss over once a more higher -level solution built on top of it is made. +The main drawback to this approach is the complexity. There are many different +implementors of `Curve`, required largely out of necessity for performance. It +may be difficult to know which one to use for when and will require signfigant +documentation effort. The core also centering around `Curve` as a trait also +encourages use of trait objects over concrete types, which may have runtime costs +associated with its use. ## Rationale and alternatives @@ -105,24 +255,6 @@ third-party ecosystem crate is unacceptable as it would promote a facturing of the ecosystem with multiple incompatible baseline animation system implementations. -The design chosen here was explicitly to allow for maximum flexibility for both -engine and game developers alike. The main alternative is to completely remove -`Sample`, the animation graph, and it's nodes, and let developers directly -hook up animation curves from clips to entities. However, this lacks flexibility -and does not allow for users of the API to inject their own alterations to the -animation stream. - -The main potential issue with the current implementation is the very heavy use of -`Arc>` which has CPU cache, lifetime, and performance implications -on the stored data. Atomic operations disrupt the CPU cache; however, they're -only used when cloning or dropping an `Arc`. The structure of the animation -graphs are, for the most part, static. Likewise, the trait object use is likely -unavoidable so long as we rely on traits as a point of abstraction within the -graph. One alternative would be to transfer ownership of the Sample implementation, -and/or just make multiple copies of a potentially large and immutable animation -data buffer, but that comes with a signfigant memory and CPU cache performance -cost. - ## Prior art This proposal is largely inspired by Unity's [Playable][playable] API, which has @@ -140,14 +272,14 @@ engine developers or game developers can build them if they need to. Currently nothing like this exists in the entire Rust ecosystem. - [playable]: https://docs.unity3d.com/Manual/Playables.html [animgraph]: https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/SkeletalMeshAnimation/AnimBlueprints/AnimGraph/ [animation-trees]: https://docs.godotengine.org/en/stable/tutorials/animation/animation_tree.html ## Unresolved questions -TODO: Complete + - Can we safely interleave multiple curves together so that we do not incur + mulitple cache misses when sampling from animation clips? ## Future possibilities From 6b85d6e2e0db7b90be634ac5b86cdd8f7a699a9a Mon Sep 17 00:00:00 2001 From: james7132 Date: Sat, 19 Feb 2022 23:58:17 -0800 Subject: [PATCH 10/14] Address more review comments --- rfcs/49-animation-primitives.md | 53 +++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index f2edb1a4..11116da5 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -38,13 +38,28 @@ thus are explicit non-goals here. ## User-facing explanation -The core of this system is a trait called `Curve` which defines a serializable -set of time-sequenced values that a user can opaquely sample values of type `T` -at a provided time. `T` here can be anything considered animatable. A few -examples of high priority types to be supported here are: +The end goal of this, and subsequent RFCs, is to deliver on a end-to-end +property-based animation system. Animation here specifically aims to provide +asset based, not-code based, workflows for altering *any* visibly mutable +property of a entity. This may include skeletal animation, where the bones deform +the verticies of a mesh, to make a 3D character run, but also may include +3D material animation, swapping sprites in a cycle for a 2D game, +enabling/disabling gameplay elements based on time and character movements +(i.e. hitboxes), etc. + +To oversimplify the entire system, this involves a pipeline that samples values, +based on time, optionally alters or blends multiple samples together, then +applies the final value to a component in the ECS World. + +The core of this system's storage is a trait called `Curve` which defines a +a serializable set of time-sequenced values that a user can opaquely sample +values of type `T` at a provided time. `T` here can be anything considered +animatable. A few examples of high priority types to be supported here are: - `f32`/`f64` - `Vec2`/`Vec3`/`Vec3A`/`Vec4` (and their `f64` variants) + - Any integer type, usually as a quantization of `f32` or `f64`. + - `Transform` (see "Special Case: Mesh Deformation Animation" below) - `Color` - `bool` for toggling on and off features. - `Range` for a range for randomly sampling from (think particle systems) @@ -59,6 +74,9 @@ sampling all curves in the clip together. Curves are keyed by the associated `Reflect` path that is animated by the curve. AnimationClips are meant to sample most, if not all, of the curves at the same time. +These base primitives can then be combined and composed together to build the +rest of the system, which will be detailed in one or more followup RFCs. + ## Implementation strategy Prototype implementation: https://github.com/HouraiTeahouse/bevy_prototype_animation @@ -96,7 +114,7 @@ trait Animatable { time. This typically will be a [linear interpolation][lerp], and have the `time` parameter clamped to the domain of [0, 1]. However, this may not necessarily be strictly be a continuous interpolation for discrete types like the integral -types or `Handle`. This may also be implemented as [spherical linear +types, `bool`, or `Handle`. This may also be implemented as [spherical linear interpolation][slerp] for quaternions. This will typically be required to provide smooth sampling from the variety of curve implementations. If it is desirable to "override" the default lerp behavior, newtype'ing an underlying @@ -242,7 +260,7 @@ TODO: Complete this section. The main drawback to this approach is the complexity. There are many different implementors of `Curve`, required largely out of necessity for performance. It -may be difficult to know which one to use for when and will require signfigant +may be difficult to know which one to use for when and will require signifigant documentation effort. The core also centering around `Curve` as a trait also encourages use of trait objects over concrete types, which may have runtime costs associated with its use. @@ -257,24 +275,13 @@ implementations. ## Prior art -This proposal is largely inspired by Unity's [Playable][playable] API, which has -a similar goal of building composable time-sequenced graphs for animation, audio, -and game logic. Several other game engines have very similar APIs and features: - - - Unreal has [AnimGraph][animgraph] for creating dynamic animations in - Blueprints. - - Godot has [animation trees][animation-trees] for creating dynamic animations in - Blueprints. - -The proposed API here doesn't purport or aim to directly replicate the features -seen in these other engines, but provide the absolute bare minimum API so that -engine developers or game developers can build them if they need to. - -Currently nothing like this exists in the entire Rust ecosystem. +On the subject of animation data compression, [ACL][acl] aims to provide a set of +animation compression algorithms for general use in game engines. It is +unfortunately written in C++, which may make it difficult to directly integrate +with Bevy, but it's open sourced under MIT, which would make a reimplementation +of its algorithms in Rust compatible with Bevy. -[playable]: https://docs.unity3d.com/Manual/Playables.html -[animgraph]: https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/SkeletalMeshAnimation/AnimBlueprints/AnimGraph/ -[animation-trees]: https://docs.godotengine.org/en/stable/tutorials/animation/animation_tree.html +[acl]: https://technology.riotgames.com/news/compressing-skeletal-animation-data ## Unresolved questions From 927842ed474372c31c1ff68ccba69b231c2ae861 Mon Sep 17 00:00:00 2001 From: james7132 Date: Mon, 21 Feb 2022 12:43:20 -0500 Subject: [PATCH 11/14] Flesh out more details about Animatable and Curve --- rfcs/49-animation-primitives.md | 146 ++++++++++++++++++++------------ 1 file changed, 91 insertions(+), 55 deletions(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index 11116da5..6180d504 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -1,13 +1,10 @@ # Feature Name: `animation-primitives` - ## Summary - Animation is particularly complex, with many stateful and intersecting elements. This RFC aims to detail a set of lowest-level APIs for authoring and playing animations within Bevy. ## Motivation - Animation is at the heart of modern game development. A game engine without an animation system is generally not considered production-ready. @@ -16,7 +13,6 @@ level experimentation with building more complex animation systems (i.e. inverse kinematics, animation state machine, humanoid retargetting, etc.) ## Scope - Animation is a huge area that spans multiple problem domains: 1. **Storage**: this generally covers on-disk storage in the form of assets as @@ -37,7 +33,6 @@ authoring can be built separately upon the primitives provided by this RFC and thus are explicit non-goals here. ## User-facing explanation - The end goal of this, and subsequent RFCs, is to deliver on a end-to-end property-based animation system. Animation here specifically aims to provide asset based, not-code based, workflows for altering *any* visibly mutable @@ -78,25 +73,12 @@ These base primitives can then be combined and composed together to build the rest of the system, which will be detailed in one or more followup RFCs. ## Implementation strategy - Prototype implementation: https://github.com/HouraiTeahouse/bevy_prototype_animation -### `Curve` Trait - -TODO: Detail why this exists. - -```rust -trait Curve { - fn duration(&self) -> f32; - fn sample(&self, time: f32) -> T; -} -``` - -#### `Animatable` Trait - -The sampled values of `Curve` need to implement `Animatable`, a trait that -allows interpolating and blending the stored values in curves. The general trait -may look like the following: +### `Animatable` Trait +To define values that can be properly smoothly sampled and composed together, a +trait is needed to determine the behavior when interpolating and blending values +of the type together. The general trait may look like the following: ```rust struct BlendInput { @@ -106,7 +88,7 @@ struct BlendInput { trait Animatable { fn interpolate(a: &Self, b: &Self, time: f32) -> Self; - fn blend(inputs: impl Iterator>) -> Self; + fn blend(inputs: impl Iterator>) -> Option; } ``` @@ -124,7 +106,10 @@ desirable to "override" the default lerp behavior, newtype'ing an underlying inputs into one output. This can be used as the base primitive implementation for building more complex compositional systems. For typical numerical types, this will often come out to just be a weighted sum. For non-continuous discrete types -like `Handle`, it may select the highest weighted input. +like `Handle`, it may select the highest weighted input. Even though a +iterator is inherently ordered in some way, the result provided by `blend` must +be order invariant for all types. If the provided iterator is empty, `None` +should be returned to signal that there were no values to blend. A blanket implementation could be done on types that implement `Add + Mul`, though this might conflict with a need for specialized @@ -135,19 +120,68 @@ implementations for the following types: [lerp]: https://en.wikipedia.org/wiki/Linear_interpolation [slerp]: https://en.wikipedia.org/wiki/Slerp -#### Concrete Implementation: `CurveFixed` +### `Curve` Trait +To store the raw values that can be sampled from, we introduce the concept of an +animation curve. An animation curve defines the function over which a value +evolves over a set period of time. These curves typically define a set of timed +keyframes that paramterize how the value changes over the established time +period. +We may have multiple ways an animation curve may store it's underlying keyframes, +so a generic trait `Curve` can be defined as follows: + +```rust +trait Curve : Serialize, DeserializeOwned { + fn duration(&self) -> f32; + fn sample(&self, time: f32) -> T; +} +``` + +A curve always covers the time range of `[0, duration]`, but must return valid +values when sampled at any non-negative time. What a curve does beyond this range +is implementation specifc. This typically will just return the value that would +be sampled at time 0 or `duration`, whichever is closer, but it doesn't have to +be. + +As shown in the above trait definition, curves must be serializable in some form, +either directly implementing `serde` traits, or via `Reflect`. This serialization +must be accessible even when turned into a trait object. This is required for +`AnimationClip` asset serialization/deserialization to work. + +#### `Animatable` Newtype Pattern +Since `Curve::sample` returns `T` and `T` is not a associated type but rather +a generic parameter, a type can implement multiple variants of `Curve`. One +possible common pattern is to define a newtype around `T` that defines new +interpolation or blending behavior when implementing `Animatable`. A blanket impl +can then be made for curves of the newtype as follows: + +```rust +impl> Curve for C { + fn duration(&self) -> f32 { + >::duration() + } + fn sample(&self, time: f32) -> T { + >::sample(time) + .map(|value| value.0) + } +} +``` + +One very common example of this might be quaternions, where either normal linear +interpolation can be used, or spherical linear interpolation can be used instead. +The underlying storage would not change, but the interpolation behavior would. + +#### Concrete Implementation: `CurveFixed` `CurveFixed` is the typical initial attempt at implementing an animation curve. It assumes a fixed frame rate and stores each frame as individual elements in a -`Vec` or `Box<[T]>`. Sampling simply requires finding the corresponding -indexes in the buffer and calling `interpolate` on with the between the two -frames. This is very fast, typically incuring only the cost of one cache miss per -curve sampled. The main downside to this is the memory usage: a keyframe must be -stored every for `1 / framerate` seconds, even if the value does not change, -which may lead to storing a large amount of redundant data. +`Vec` or `Box<[T]>`. Sampling simply requires finding the corresponding indexes +in the buffer and calling `interpolate` on with the between the two frames. This +is very fast, typically incuring only the cost of one cache miss per curve +sampled. The main downside to this is the memory usage: a keyframe must be stored +for every `1 / framerate` seconds, even if the value does not change, which may +lead to storing a large amount of redundant data. #### Concrete Implementation: `CurveVariable` - `CurveVariable`, like `CurveFixed`, stores its keyframe values in a contiguous buffer like `Vec`. However, it stores the exact keyframe times in a parallel `Vec`, allowing for the time between keyframes to be variable. This comes at a @@ -161,25 +195,9 @@ again to minimize the time spent searching for the correct value. This notably complicates sampling from these curves #### Concrete Implementation: `CurveVariableLinear` - TODO: Complete this section. -### Special Case: Mesh Deformation Animation (Skeletal/Morph) - -The heaviest use case for this animation system is for 3D mesh deformation -animation. This includes [skeletal animation][skeletal_animation] and [morph -target animation][morph_target_animation]. This section will not get into the -requisite rendering techniques to display the results of said deformations and -only focus on how to store curves for animation. - -Both `AnimationClip` and surrounding types that utilize it may need a specialized -shortcut for accessing and sampling `Transform` based poses from the clip. - -[skeletal_animation]: https://en.wikipedia.org/wiki/Skeletal_animation -[morph_target_animation]: https://en.wikipedia.org/wiki/Morph_target_animation - -### Curve Compression and Quantization - +#### Curve Compression and Quantization Curves can be quite big. Each buffer can be quite large depending on the number of keyframes and the representation of said keyframes, and a typical game with multiple character animations may several hundreds or thousands of curves to @@ -253,11 +271,33 @@ struct CompressedTransformCurve { [quat_compression]: https://technology.riotgames.com/news/compressing-skeletal-animation-data ### `AnimationClip` +A `AnimationClip` is a serializable asset type that encompasses a mapping of +property paths to curves. It effectively acts as a `HashMap>`, +but will require a bit of type erasure to ensure that the -TODO: Complete this section. +TODO: Devise a strategy for serializing curves here. -## Drawbacks +#### Special Case: Mesh Deformation Animation (Skeletal/Morph) +The heaviest use case for this animation system is for 3D mesh deformation +animation. This includes [skeletal animation][skeletal_animation] and [morph +target animation][morph_target_animation]. This section will not get into the +requisite rendering techniques to display the results of said deformations and +only focus on how to store curves for animation. + +Both `AnimationClip` and surrounding types that utilize it may need a specialized +shortcut for accessing and sampling `Transform` based poses from the clip. + +Instead of using string-based property paths, individual integer based bone IDs +should be used here to minimize curve lookup cost, and curves here should not be +trait objects but rather the aformentioned compressed tranform curves as concrete +types to reduce pointer indirections and cut out vtable lookups. This will likely +require dedicated skeletal animation sampling and application systems, as well as +specialized APIs for fetching and composing `Transform` values from these curves. +[skeletal_animation]: https://en.wikipedia.org/wiki/Skeletal_animation +[morph_target_animation]: https://en.wikipedia.org/wiki/Morph_target_animation + +## Drawbacks The main drawback to this approach is the complexity. There are many different implementors of `Curve`, required largely out of necessity for performance. It may be difficult to know which one to use for when and will require signifigant @@ -266,7 +306,6 @@ encourages use of trait objects over concrete types, which may have runtime cost associated with its use. ## Rationale and alternatives - Bevy absolutely needs some form of a first-party animation system, no modern game engine can be called production ready without one. Having this exist solely as a third-party ecosystem crate is unacceptable as it would promote a @@ -274,7 +313,6 @@ facturing of the ecosystem with multiple incompatible baseline animation system implementations. ## Prior art - On the subject of animation data compression, [ACL][acl] aims to provide a set of animation compression algorithms for general use in game engines. It is unfortunately written in C++, which may make it difficult to directly integrate @@ -284,10 +322,8 @@ of its algorithms in Rust compatible with Bevy. [acl]: https://technology.riotgames.com/news/compressing-skeletal-animation-data ## Unresolved questions - - Can we safely interleave multiple curves together so that we do not incur mulitple cache misses when sampling from animation clips? ## Future possibilities - TODO: Complete From cba9d69502290d54794b61b7e2c26f362d27f1e4 Mon Sep 17 00:00:00 2001 From: james7132 Date: Thu, 3 Mar 2022 16:05:44 -0800 Subject: [PATCH 12/14] Add note on PropertyPath --- rfcs/49-animation-primitives.md | 85 +++++++++++++++++++++++++++------ 1 file changed, 70 insertions(+), 15 deletions(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index 6180d504..f488b0e3 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -26,10 +26,16 @@ Animation is a huge area that spans multiple problem domains: 4. **Authoring**: this is how we create the animation assets used by the engine. This RFC primarily aims to resolve only problems within the domains of storage -and sampling. Composition will only be touched briefly upon for the purposes of -animated type selection. Application can be distinctly decoupled from these -earlier two stages, treating the sampled values as a black box output, and -authoring can be built separately upon the primitives provided by this RFC and +and sampling. The following will only be touched upon briefly: + + - Composition will only be touched briefly upon for the purposes of animated + type selection. + - Application is only touched upon in relation to how to select which property + is to be animated by which asset, as well as sampled value post-processing. + The rest can be distinctly decoupled from these earlier two stages, treating + the sampled values as a black box output. + +Authoring can be built separately upon the primitives provided by this RFC and thus are explicit non-goals here. ## User-facing explanation @@ -270,12 +276,65 @@ struct CompressedTransformCurve { [quat_compression]: https://technology.riotgames.com/news/compressing-skeletal-animation-data +### PropertyPath +To reduce reliance on expensive runtime parsing and raw stringly-typed +operations, the path to which a property is animated can be parsed into a +`PropertyPath` struct, which contains the metadata for fetching the correct +property in an component to animate during application. Property paths roughly +take the form of `path/to/entity@crate::module::Component.field.within.component`. + +The first part of the path is the named path in the Entity hierarchy. This relies +on the `Name` component to be populated at every Entity along a Transform +hierarchy to be present. If there are multiple children at any level with the +same name, the first child in `Children` with the matching name will be used. + +The second part of path is the component within the entity that is being +animated. This is the fullly qualified type name of the component being animated. +Dynamic components are not supported by this initial design. + +The third and final part of the path is a field path within the given component. +This should be a vailid path usable with the `GetPath` trait functions. This is +likely to be the most costly operation when implementing application, so a +pre-parsed subpath struct `FieldPath` should be used here if possible. + +A rough outline of how such a `PropertyPath` struct might look like: + +```rust +#[derive(Debug, Clone, PartialEq, Eq, Hash, Ord, ParitialOrd)] +pub struct EntityPath { + parts: Box<[Name]>, +} + +#[derive(Debug, Clone, PartialEq, Eq, Hash, Ord, ParitialOrd)] +pub struct PropertyPath { + entity: EntityPath, + component_type_id: TypeId, + component_name: String, + field: FieldPath, +} +``` + +A component TypeId will be stored alongside the component name. The TypeId is +required to remove string based component name lookups during application. +Using the TypeId also accelerates `Eq`, `Ord`, and `Hash` implementations as +these paths are commonly used as HashMap and BTreeMap keys. The raw component +name is only used when converting back into a raw string. + +As a result of needing runtime type information, serialization and deserialzation +cannot be implemented with serde derive macros, but will require a `TypeRegistry` +to handle the TypeId lookup. For simpler file representations, upon serialization, +PropertyPaths are written out as raw strings, and parsed on deserialization. + ### `AnimationClip` A `AnimationClip` is a serializable asset type that encompasses a mapping of -property paths to curves. It effectively acts as a `HashMap>`, -but will require a bit of type erasure to ensure that the +property paths to curves. It effectively acts as a `HashMap>`, but will require a bit of type erasure to ensure that the generics of +each curve is not visible in the concrete type definition. -TODO: Devise a strategy for serializing curves here. +As PropertyPath cannot be serialized without a TypeRegistry, AnimationClips will +also require one to be present during both serialization and deserialization. The +type erasure of each curve will also likely require `typetag`-like serialization +using the `TypeRegistry`. #### Special Case: Mesh Deformation Animation (Skeletal/Morph) The heaviest use case for this animation system is for 3D mesh deformation @@ -285,14 +344,10 @@ requisite rendering techniques to display the results of said deformations and only focus on how to store curves for animation. Both `AnimationClip` and surrounding types that utilize it may need a specialized -shortcut for accessing and sampling `Transform` based poses from the clip. - -Instead of using string-based property paths, individual integer based bone IDs -should be used here to minimize curve lookup cost, and curves here should not be -trait objects but rather the aformentioned compressed tranform curves as concrete -types to reduce pointer indirections and cut out vtable lookups. This will likely -require dedicated skeletal animation sampling and application systems, as well as -specialized APIs for fetching and composing `Transform` values from these curves. +shortcut for accessing and sampling `Transform` based poses from the clip. This +will likely take the form of a separate internal map from EntityPath to a +concrete curve type for Transform to avoid any dynamic dispatch overhead, and +dedicated lookup and sampling functions explicitly for this special case. [skeletal_animation]: https://en.wikipedia.org/wiki/Skeletal_animation [morph_target_animation]: https://en.wikipedia.org/wiki/Morph_target_animation From 8512c927f6f9fbe6cdea8bdf8204fd4f262f905a Mon Sep 17 00:00:00 2001 From: james7132 Date: Thu, 3 Mar 2022 16:16:57 -0800 Subject: [PATCH 13/14] Update with information about post_processing --- rfcs/49-animation-primitives.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index f488b0e3..89ecdd34 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -95,6 +95,7 @@ struct BlendInput { trait Animatable { fn interpolate(a: &Self, b: &Self, time: f32) -> Self; fn blend(inputs: impl Iterator>) -> Option; + unsafe fn post_process(&mut self, world: &World) {} } ``` @@ -123,6 +124,19 @@ implementations for the following types: - `Vec3` - needed to take advantage of SIMD instructions via `Vec3A`. - `Handle` - need to properly use `clone_weak`. +An unsafe `post_process` trait function is going to be required to build values +that are dependent on the state of the World. An example of this is `Handle`, +which requires strong handles to be used properly: a `Curve` can +implement `Curve>` by postprocessing the `HandleId` by reading the +associated `Assets` resource to make a strong handle. This is applied only +after blending is applied so post processing is only applied once per sampled +value. This function is unsafe by default as it may be unsafe to read any +non-Resource or NonSend resource from the World if application is run over +multiple threads, which may cause aliasing errors if read. Other unsafe +operations that mutate the World from a read-only reference is also unsound. The +default implementation here is a no-op, as most implementations do not need this +functionality, and will be optimized out via monomorphization. + [lerp]: https://en.wikipedia.org/wiki/Linear_interpolation [slerp]: https://en.wikipedia.org/wiki/Slerp From d61987809f62dc0a71f5b4f596c3aa4770c3dc41 Mon Sep 17 00:00:00 2001 From: james7132 Date: Sun, 20 Feb 2022 00:50:47 -0800 Subject: [PATCH 14/14] add note about quaternion compression --- rfcs/49-animation-primitives.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/rfcs/49-animation-primitives.md b/rfcs/49-animation-primitives.md index 89ecdd34..e9e84131 100644 --- a/rfcs/49-animation-primitives.md +++ b/rfcs/49-animation-primitives.md @@ -222,7 +222,7 @@ Curves can be quite big. Each buffer can be quite large depending on the number of keyframes and the representation of said keyframes, and a typical game with multiple character animations may several hundreds or thousands of curves to sample from. To both help cut down on memory usage and minimize cache misses, -it's useful to compress the most usecases seen by a +it'd be useful to compress the most common usecases. `f32` is by far the most commonly animated type, and is foundational for building more complex types (i.e. Vec2, Vec3, Vec4, Quat, Transform), which makes it a @@ -269,7 +269,13 @@ enum CompressedFloatCurve { } ``` -TODO: Add note about [quaternion compression][quat-compression] +A more complex, but crucial optimization is quaternion compression. In skeletal +animation, both scale and translation are typically unchanged throughout the +entire animation, relying primarily on rotation to convey motion +instead. Using the mathematical properties of quaternions, it's possible to +shrink a 128 bit Quaternion (four 4-byte f32s) into 48 bits with minimal +numerical error, a compression ratio of 0.375. Riot Games has detailed the +general approach in [this article](https://technology.riotgames.com/news/compressing-skeletal-animation-data). We can then use this compose these compressed structs together to construct more complex curves: @@ -288,7 +294,7 @@ struct CompressedTransformCurve { } ``` -[quat_compression]: https://technology.riotgames.com/news/compressing-skeletal-animation-data + ### PropertyPath To reduce reliance on expensive runtime parsing and raw stringly-typed