From 5117837651c44edc061b8b050b1731cca82e4d8f Mon Sep 17 00:00:00 2001 From: Nathan Graule Date: Wed, 6 Mar 2024 11:27:08 +0100 Subject: [PATCH 1/8] doc(bevy_audio): first pass at writing contribution guidelines --- crates/bevy_audio/src/lib.rs | 89 ++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/crates/bevy_audio/src/lib.rs b/crates/bevy_audio/src/lib.rs index c096877598247..6a2688d933194 100644 --- a/crates/bevy_audio/src/lib.rs +++ b/crates/bevy_audio/src/lib.rs @@ -19,6 +19,95 @@ //! }); //! } //! ``` +//! +//! # Contributing +//! +//! ## Under the trunk +//! +//! Some parts of the audio engine run on a high-priority thread, synchronized with the audio driver +//! as it requests audio data (or, in the case of input, notifies of new capture data). +//! +//! How often these callbacks are run by the driver depends on the audio stream settings; namely the +//! **sample rate** and the **buffer size**. These parameters are passed in as configuration when +//! creating a stream. +//! +//! Typical values for buffer size and sample rate are 512 samples at 48 kHz. This means that every 512 +//! samples of audio the driver is going to send to the sound card the output callback function is run +//! in this high-priority audio thread. Every second, as dictated by the sample rate, the sound card +//! needs 48 000 samples of audio data. This means that we can expect the callback function to be run +//! every `512/(48000 Hz)` or 10.666... ms. +//! +//! This figure is also the latency of the audio engine, that is, how much time it takes between a +//! user interaction and hearing the effects out the speakers. Therefore, there is a "tug of war" +//! between decreasing the buffer size for latency reasons, and increasing it for performance reasons. +//! The threshold for instantaneity in audio is around 15 ms, which is why 512 is a good value for +//! interactive applications. +//! +//! ## Real-time programming +//! +//! The parts of the code running in the audio thread have exactly `buffer_size/samplerate` seconds to +//! complete, beyond which the audio driver outputs silence (or worse, the previous buffer output, or +//! garbage data), which the user perceives as a glitch and severely deteriorates the quality of the +//! audio output of the engine. It is therefore critical to work with code that is guaranteed to finish +//! in that time. +//! +//! One step to achieving this is making sure that all machines across the spectrum of supported CPUs can +//! reliably perform the computations needed for the game in that amount of time, and play around with the +//! buffer size to find the best compromise between latency and performance. Another is to conditionally +//! enable certain effects for more powerful CPUs, when that is possible. +//! +//! But the main step is to write code run in the audio thread following real-time programming guidelines. +//! Real-time programming is a set of constraints on code and structures that guarantees the code completes +//! at some point, ie. it cannot be stuck in an infinite loop nor can it trigger a deadlock situation. +//! +//! Practically, the main components of real-time programming are about using wait-free and lock-free +//! structures. Examples of things that are *not* correct in real-time programming are: +//! +//! - Allocating anything on the heap (that is, no direct or indirect creation of a `Vec`, `Box`, or any +//! standard collection, as they are not designed with real-time programming in mind) +//! - Locking a mutex +//! - Generally, any kind of system call gives the OS the opportunity to pause the thread, which is an +//! unbounded operation as we don't know how long the thread is going to be paused for +//! - Waiting by looping until some condition is met (also called a spinloop or a spinlock) +//! +//! Writing wait-free and lock-free structures is a hard task, and difficult to get correct; however many +//! structures already exists, and can be directly used. There are crates for most replacements of standard +//! collections. +//! +//! ## Communication with the audio thread +//! +//! To be able to to anything useful with audio, the thread has to be able to communicate with the rest of +//! the system, ie. update parameters, send/receive audio data, etc., and all of that needs to be done within +//! the constraints of real-time programming, of course. +//! +//! ### Audio parameters +//! +//! In most cases, audio parameters can be represented by an atomic floating point value, where the game loop +//! updates the parameter, and it gets picked up when processing the next buffer. The downside to this approach +//! is that the audio only changes once per audio callback, and results in a noticeable "stair-step " motion +//! of the parameter. The latter can be mitigated by "smoothing" the change over time, using a tween or +//! linear/exponential smoothing. +//! +//! Precise timing for non-interactive events (ie. on the beat) need to be setup using a clock backed by the +//! audio driver -- that is, counting the number of samples processed, and deriving the time elapsed by diving +//! by the sample rate to get the number of seconds elapsed. The precise sample at which the parameter needs to +//! be changed can then be computed. +//! +//! Both interactive and precise events are hard to do, and need very low latency (ie. 64 or 128 samples for ~2 +//! ms of latency). It is fundamentally impossible to react to user event the very moment it is registered. +//! +//! ### Audio data +//! +//! Audio data is generally transferred between threads with circular buffers, as they are simple to implement, +//! fast enough for 99% of use-cases, and are both wait-free and lock-free. The only difficulty in using +//! circular buffers is how big they should be; however even going for 1 s of audio costs ~50 kB of memory, +//! which is small enough to not be noticeable even with potentially 100s of those buffers. +//! +//! ## Additional resources for audio programming +//! +//! More in-depth article about audio programming: +//! +//! Awesome Audio DSP: #![forbid(unsafe_code)] From 3cb9e1bf6829a40b9fa1d9c253535eebc2d61a53 Mon Sep 17 00:00:00 2001 From: Nathan Graule Date: Wed, 6 Mar 2024 11:35:10 +0100 Subject: [PATCH 2/8] doc(bevy_audio): add section on where real-time safety should be enforced --- crates/bevy_audio/src/lib.rs | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/crates/bevy_audio/src/lib.rs b/crates/bevy_audio/src/lib.rs index 6a2688d933194..ee0e8d5743a5b 100644 --- a/crates/bevy_audio/src/lib.rs +++ b/crates/bevy_audio/src/lib.rs @@ -103,6 +103,19 @@ //! circular buffers is how big they should be; however even going for 1 s of audio costs ~50 kB of memory, //! which is small enough to not be noticeable even with potentially 100s of those buffers. //! +//! ## Where in the code should real-time programming principles be applied? +//! +//! Any code that is directly or indirectly called by audio threads, needs to be real-time safe. +//! +//! For the Bevy engine, that is: +//! +//! - In the callback of `cpal::Stream::build_input_stream` and `cpal::Stream::build_output_stream`, and all +//! functions called from them +//! - In implementations of the [`rodio::source::Source`] trait, and all functions called from it +//! +//! Code that is run in Bevy systems do not need to be real-time safe, as they are not run in the audio thread, +//! but in the main game loop thread. +//! //! ## Additional resources for audio programming //! //! More in-depth article about audio programming: From 4838302aae100a6d4042eb069273453788b94eab Mon Sep 17 00:00:00 2001 From: Nathan Graule Date: Wed, 6 Mar 2024 11:55:25 +0100 Subject: [PATCH 3/8] doc(bevy_audio): use local Source in doc link --- crates/bevy_audio/src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/bevy_audio/src/lib.rs b/crates/bevy_audio/src/lib.rs index ee0e8d5743a5b..d6f7e5531d007 100644 --- a/crates/bevy_audio/src/lib.rs +++ b/crates/bevy_audio/src/lib.rs @@ -111,7 +111,7 @@ //! //! - In the callback of `cpal::Stream::build_input_stream` and `cpal::Stream::build_output_stream`, and all //! functions called from them -//! - In implementations of the [`rodio::source::Source`] trait, and all functions called from it +//! - In implementations of the [`Source`] trait, and all functions called from it //! //! Code that is run in Bevy systems do not need to be real-time safe, as they are not run in the audio thread, //! but in the main game loop thread. From 6de03d990702266df1c31ed84bfffaeb19cccb30 Mon Sep 17 00:00:00 2001 From: Nathan Graule Date: Wed, 6 Mar 2024 14:14:31 +0100 Subject: [PATCH 4/8] chore(bevy_audio): run cargo fmt --- crates/bevy_audio/src/lib.rs | 60 ++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/crates/bevy_audio/src/lib.rs b/crates/bevy_audio/src/lib.rs index d6f7e5531d007..6b3e122ca3d29 100644 --- a/crates/bevy_audio/src/lib.rs +++ b/crates/bevy_audio/src/lib.rs @@ -19,107 +19,107 @@ //! }); //! } //! ``` -//! +//! //! # Contributing -//! +//! //! ## Under the trunk -//! +//! //! Some parts of the audio engine run on a high-priority thread, synchronized with the audio driver //! as it requests audio data (or, in the case of input, notifies of new capture data). -//! +//! //! How often these callbacks are run by the driver depends on the audio stream settings; namely the //! **sample rate** and the **buffer size**. These parameters are passed in as configuration when //! creating a stream. -//! +//! //! Typical values for buffer size and sample rate are 512 samples at 48 kHz. This means that every 512 //! samples of audio the driver is going to send to the sound card the output callback function is run //! in this high-priority audio thread. Every second, as dictated by the sample rate, the sound card //! needs 48 000 samples of audio data. This means that we can expect the callback function to be run //! every `512/(48000 Hz)` or 10.666... ms. -//! +//! //! This figure is also the latency of the audio engine, that is, how much time it takes between a //! user interaction and hearing the effects out the speakers. Therefore, there is a "tug of war" //! between decreasing the buffer size for latency reasons, and increasing it for performance reasons. //! The threshold for instantaneity in audio is around 15 ms, which is why 512 is a good value for //! interactive applications. -//! +//! //! ## Real-time programming -//! +//! //! The parts of the code running in the audio thread have exactly `buffer_size/samplerate` seconds to //! complete, beyond which the audio driver outputs silence (or worse, the previous buffer output, or //! garbage data), which the user perceives as a glitch and severely deteriorates the quality of the //! audio output of the engine. It is therefore critical to work with code that is guaranteed to finish //! in that time. -//! +//! //! One step to achieving this is making sure that all machines across the spectrum of supported CPUs can //! reliably perform the computations needed for the game in that amount of time, and play around with the //! buffer size to find the best compromise between latency and performance. Another is to conditionally //! enable certain effects for more powerful CPUs, when that is possible. -//! +//! //! But the main step is to write code run in the audio thread following real-time programming guidelines. //! Real-time programming is a set of constraints on code and structures that guarantees the code completes //! at some point, ie. it cannot be stuck in an infinite loop nor can it trigger a deadlock situation. -//! +//! //! Practically, the main components of real-time programming are about using wait-free and lock-free //! structures. Examples of things that are *not* correct in real-time programming are: -//! +//! //! - Allocating anything on the heap (that is, no direct or indirect creation of a `Vec`, `Box`, or any //! standard collection, as they are not designed with real-time programming in mind) //! - Locking a mutex -//! - Generally, any kind of system call gives the OS the opportunity to pause the thread, which is an +//! - Generally, any kind of system call gives the OS the opportunity to pause the thread, which is an //! unbounded operation as we don't know how long the thread is going to be paused for //! - Waiting by looping until some condition is met (also called a spinloop or a spinlock) -//! +//! //! Writing wait-free and lock-free structures is a hard task, and difficult to get correct; however many //! structures already exists, and can be directly used. There are crates for most replacements of standard //! collections. -//! +//! //! ## Communication with the audio thread -//! +//! //! To be able to to anything useful with audio, the thread has to be able to communicate with the rest of //! the system, ie. update parameters, send/receive audio data, etc., and all of that needs to be done within //! the constraints of real-time programming, of course. -//! +//! //! ### Audio parameters -//! +//! //! In most cases, audio parameters can be represented by an atomic floating point value, where the game loop //! updates the parameter, and it gets picked up when processing the next buffer. The downside to this approach //! is that the audio only changes once per audio callback, and results in a noticeable "stair-step " motion //! of the parameter. The latter can be mitigated by "smoothing" the change over time, using a tween or //! linear/exponential smoothing. -//! +//! //! Precise timing for non-interactive events (ie. on the beat) need to be setup using a clock backed by the //! audio driver -- that is, counting the number of samples processed, and deriving the time elapsed by diving //! by the sample rate to get the number of seconds elapsed. The precise sample at which the parameter needs to //! be changed can then be computed. -//! +//! //! Both interactive and precise events are hard to do, and need very low latency (ie. 64 or 128 samples for ~2 //! ms of latency). It is fundamentally impossible to react to user event the very moment it is registered. -//! +//! //! ### Audio data -//! +//! //! Audio data is generally transferred between threads with circular buffers, as they are simple to implement, //! fast enough for 99% of use-cases, and are both wait-free and lock-free. The only difficulty in using //! circular buffers is how big they should be; however even going for 1 s of audio costs ~50 kB of memory, //! which is small enough to not be noticeable even with potentially 100s of those buffers. -//! +//! //! ## Where in the code should real-time programming principles be applied? -//! +//! //! Any code that is directly or indirectly called by audio threads, needs to be real-time safe. -//! +//! //! For the Bevy engine, that is: -//! +//! //! - In the callback of `cpal::Stream::build_input_stream` and `cpal::Stream::build_output_stream`, and all //! functions called from them //! - In implementations of the [`Source`] trait, and all functions called from it -//! +//! //! Code that is run in Bevy systems do not need to be real-time safe, as they are not run in the audio thread, //! but in the main game loop thread. -//! +//! //! ## Additional resources for audio programming -//! +//! //! More in-depth article about audio programming: -//! +//! //! Awesome Audio DSP: #![forbid(unsafe_code)] From 4a8e088abf2975caff4165cb324710707e0d61e5 Mon Sep 17 00:00:00 2001 From: Nathan Graule Date: Wed, 6 Mar 2024 18:57:33 +0100 Subject: [PATCH 5/8] doc(bevy_audio): rename heading + add "foreword" --- crates/bevy_audio/src/lib.rs | 40 +++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/crates/bevy_audio/src/lib.rs b/crates/bevy_audio/src/lib.rs index 6b3e122ca3d29..cb22a2d78e8ae 100644 --- a/crates/bevy_audio/src/lib.rs +++ b/crates/bevy_audio/src/lib.rs @@ -20,7 +20,19 @@ //! } //! ``` //! -//! # Contributing +//! # Fundamentals of working with audio +//! +//! This section is of interest to anybody working with the lowest levels of the audio engine. +//! +//! If you're looking for using effects that bevy already provides, or are working within Bevy's +//! ECS, then these guidelines do not apply. This concerns code that is going to run in tandem +//! with the audio driver, that is code directly talking to audio I/O or while implementing audio +//! effects. +//! +//! This section applies to the equivalent in abstraction level to working with nodes in the render +//! graph, and not manipulating entities with meshes and materials. +//! +//! Note that these guidelines are general to any audio programming application, and not just Bevy. //! //! ## Under the trunk //! @@ -74,6 +86,19 @@ //! structures already exists, and can be directly used. There are crates for most replacements of standard //! collections. //! +//! ## Where in the code should real-time programming principles be applied? +//! +//! Any code that is directly or indirectly called by audio threads, needs to be real-time safe. +//! +//! For the Bevy engine, that is: +//! +//! - In the callback of `cpal::Stream::build_input_stream` and `cpal::Stream::build_output_stream`, and all +//! functions called from them +//! - In implementations of the [`Source`] trait, and all functions called from it +//! +//! Code that is run in Bevy systems do not need to be real-time safe, as they are not run in the audio thread, +//! but in the main game loop thread. +//! //! ## Communication with the audio thread //! //! To be able to to anything useful with audio, the thread has to be able to communicate with the rest of @@ -103,19 +128,6 @@ //! circular buffers is how big they should be; however even going for 1 s of audio costs ~50 kB of memory, //! which is small enough to not be noticeable even with potentially 100s of those buffers. //! -//! ## Where in the code should real-time programming principles be applied? -//! -//! Any code that is directly or indirectly called by audio threads, needs to be real-time safe. -//! -//! For the Bevy engine, that is: -//! -//! - In the callback of `cpal::Stream::build_input_stream` and `cpal::Stream::build_output_stream`, and all -//! functions called from them -//! - In implementations of the [`Source`] trait, and all functions called from it -//! -//! Code that is run in Bevy systems do not need to be real-time safe, as they are not run in the audio thread, -//! but in the main game loop thread. -//! //! ## Additional resources for audio programming //! //! More in-depth article about audio programming: From 5f8c504dcbb0407c51cc81f91b5cdb5a406ad167 Mon Sep 17 00:00:00 2001 From: Nathan Graule Date: Fri, 8 Mar 2024 10:51:40 +0100 Subject: [PATCH 6/8] doc(bevy_audio): move contribution guidelins to CONTRIBUTING.md and reword a bit --- crates/bevy_audio/CONTRIBUTING.md | 152 ++++++++++++++++++++++++++++++ crates/bevy_audio/src/lib.rs | 115 ---------------------- 2 files changed, 152 insertions(+), 115 deletions(-) create mode 100644 crates/bevy_audio/CONTRIBUTING.md diff --git a/crates/bevy_audio/CONTRIBUTING.md b/crates/bevy_audio/CONTRIBUTING.md new file mode 100644 index 0000000000000..7346655d604fb --- /dev/null +++ b/crates/bevy_audio/CONTRIBUTING.md @@ -0,0 +1,152 @@ +# Contributing to `bevy_audio` + +This document highlights documents some general explanation and guidelines for +contributing code to this crate. It assumes knowledge of programming, but not +necessarily of audio programming specifically. It lays out rules to follow, on +top of the general programming and contribution guidelines of Bevy, that are of +particular interest for performance reasons. + +This section applies to the equivalent in abstraction level to working with +nodes in the render graph, and not manipulating entities with meshes and +materials. + +Note that these guidelines are general to any audio programming application, and +not just Bevy. + +# Fundamentals of working with audio + +## A brief introduction to digital audio signals + +Audio signals, when working within a computer, are digital streams of audio +samples (historically with different types, but nowadays the values are 32-bit +floats), taken at regular intervals of each other. + +How often this sampling is done is determined by the **sample rate** parameter. +This parameter is available to the users in OS settings, as well as some +applications. + +The sample rate directly determines the spectrum of audio frequencies that will +be representable by the system. That limit sits at half the sample rate, meaning +that any sound with frequencies higher than that will introduce artifacts. + +If you want to learn more, read about the **Nyquist sampling theorem** and +**Frequency aliasing**. + +## How the computer interfaces with the sound card + +When requesting for audio input or output, the OS creates a special +high-priority thread whose task it is to take in the input audio stream, and/or +produce the output stream. The audio driver passes an audio buffer that you read +from (for input) or write to (for output). The size of that buffer is also a +parameter that is configured when opening an audio stream with the sound card, +and is sometimes reflected in application settings. + +Typical values for buffer size and sample rate are 512 samples at a sample rate +of 48 kHz. This means that for every 512 samples of audio the driver is going to +send to the sound card the output callback function is run in this high-priority +audio thread. Every second, as dictated by the sample rate, the sound card +needs 48 000 samples of audio data. This means that we can expect the callback +function to be run every `512/(48000 Hz)` or 10.666... ms. + +This figure is also the latency of the audio engine, that is, how much time it +takes between a user interaction and hearing the effects out the speakers. +Therefore, there is a "tug of war" between decreasing the buffer size for +latency reasons, and increasing it for performance reasons. The threshold for +instantaneity in audio is around 15 ms, which is why 512 is a good value for +interactive applications. + +## Real-time programming + +The parts of the code running in the audio thread have exactly +`buffer_size/samplerate` seconds to complete, beyond which the audio driver +outputs silence (or worse, the previous buffer output, or garbage data), which +the user perceives as a glitch and severely deteriorates the quality of the +audio output of the engine. It is therefore critical to work with code that is +guaranteed to finish in that time. + +One step to achieving this is making sure that all machines across the spectrum +of supported CPUs can reliably perform the computations needed for the game in +that amount of time, and play around with the buffer size to find the best +compromise between latency and performance. Another is to conditionally enable +certain effects for more powerful CPUs, when that is possible. + +But the main step is to write code run in the audio thread following real-time +programming guidelines. Real-time programming is a set of constraints on code +and structures that guarantees the code completes at some point, ie. it cannot +be stuck in an infinite loop nor can it trigger a deadlock situation. + +Practically, the main components of real-time programming are about using +wait-free and lock-free structures. Examples of things that are *not* correct in +real-time programming are: + +- Allocating anything on the heap (that is, no direct or indirect creation of a +`Vec`, `Box`, or any standard collection, as they are not designed with +real-time programming in mind) + +- Locking a mutex - Generally, any kind of system call gives the OS the +opportunity to pause the thread, which is an unbounded operation as we don't +know how long the thread is going to be paused for + +- Waiting by looping until some condition is met (also called a spinloop or a +spinlock) + +Writing wait-free and lock-free structures is a hard task, and difficult to get +correct; however many structures already exists, and can be directly used. There +are crates for most replacements of standard collections. + +## Where in the code should real-time programming principles be applied? + +Any code that is directly or indirectly called by audio threads, needs to be +real-time safe. + +For the Bevy engine, that is: + +- In the callback of `cpal::Stream::build_input_stream` and +`cpal::Stream::build_output_stream`, and all functions called from them + +- In implementations of the [`Source`] trait, and all functions called from it + +Code that is run in Bevy systems do not need to be real-time safe, as they are +not run in the audio thread, but in the main game loop thread. + +## Communication with the audio thread + +To be able to to anything useful with audio, the thread has to be able to +communicate with the rest of the system, ie. update parameters, send/receive +audio data, etc., and all of that needs to be done within the constraints of +real-time programming, of course. + +### Audio parameters + +In most cases, audio parameters can be represented by an atomic floating point +value, where the game loop updates the parameter, and it gets picked up when +processing the next buffer. The downside to this approach is that the audio only +changes once per audio callback, and results in a noticeable "stair-step " +motion of the parameter. The latter can be mitigated by "smoothing" the change +over time, using a tween or linear/exponential smoothing. + +Precise timing for non-interactive events (ie. on the beat) need to be setup +using a clock backed by the audio driver -- that is, counting the number of +samples processed, and deriving the time elapsed by diving by the sample rate to +get the number of seconds elapsed. The precise sample at which the parameter +needs to be changed can then be computed. + +Both interactive and precise events are hard to do, and need very low latency +(ie. 64 or 128 samples for ~2 ms of latency). It is fundamentally impossible to +react to user event the very moment it is registered. + +### Audio data + +Audio data is generally transferred between threads with circular buffers, as +they are simple to implement, fast enough for 99% of use-cases, and are both +wait-free and lock-free. The only difficulty in using circular buffers is how +big they should be; however even going for 1 s of audio costs ~50 kB of memory, +which is small enough to not be noticeable even with potentially 100s of those +buffers. + +## Additional resources for audio programming + +More in-depth article about audio programming: + + +Awesome Audio DSP: \ No newline at end of file diff --git a/crates/bevy_audio/src/lib.rs b/crates/bevy_audio/src/lib.rs index cb22a2d78e8ae..19677c1552022 100644 --- a/crates/bevy_audio/src/lib.rs +++ b/crates/bevy_audio/src/lib.rs @@ -19,121 +19,6 @@ //! }); //! } //! ``` -//! -//! # Fundamentals of working with audio -//! -//! This section is of interest to anybody working with the lowest levels of the audio engine. -//! -//! If you're looking for using effects that bevy already provides, or are working within Bevy's -//! ECS, then these guidelines do not apply. This concerns code that is going to run in tandem -//! with the audio driver, that is code directly talking to audio I/O or while implementing audio -//! effects. -//! -//! This section applies to the equivalent in abstraction level to working with nodes in the render -//! graph, and not manipulating entities with meshes and materials. -//! -//! Note that these guidelines are general to any audio programming application, and not just Bevy. -//! -//! ## Under the trunk -//! -//! Some parts of the audio engine run on a high-priority thread, synchronized with the audio driver -//! as it requests audio data (or, in the case of input, notifies of new capture data). -//! -//! How often these callbacks are run by the driver depends on the audio stream settings; namely the -//! **sample rate** and the **buffer size**. These parameters are passed in as configuration when -//! creating a stream. -//! -//! Typical values for buffer size and sample rate are 512 samples at 48 kHz. This means that every 512 -//! samples of audio the driver is going to send to the sound card the output callback function is run -//! in this high-priority audio thread. Every second, as dictated by the sample rate, the sound card -//! needs 48 000 samples of audio data. This means that we can expect the callback function to be run -//! every `512/(48000 Hz)` or 10.666... ms. -//! -//! This figure is also the latency of the audio engine, that is, how much time it takes between a -//! user interaction and hearing the effects out the speakers. Therefore, there is a "tug of war" -//! between decreasing the buffer size for latency reasons, and increasing it for performance reasons. -//! The threshold for instantaneity in audio is around 15 ms, which is why 512 is a good value for -//! interactive applications. -//! -//! ## Real-time programming -//! -//! The parts of the code running in the audio thread have exactly `buffer_size/samplerate` seconds to -//! complete, beyond which the audio driver outputs silence (or worse, the previous buffer output, or -//! garbage data), which the user perceives as a glitch and severely deteriorates the quality of the -//! audio output of the engine. It is therefore critical to work with code that is guaranteed to finish -//! in that time. -//! -//! One step to achieving this is making sure that all machines across the spectrum of supported CPUs can -//! reliably perform the computations needed for the game in that amount of time, and play around with the -//! buffer size to find the best compromise between latency and performance. Another is to conditionally -//! enable certain effects for more powerful CPUs, when that is possible. -//! -//! But the main step is to write code run in the audio thread following real-time programming guidelines. -//! Real-time programming is a set of constraints on code and structures that guarantees the code completes -//! at some point, ie. it cannot be stuck in an infinite loop nor can it trigger a deadlock situation. -//! -//! Practically, the main components of real-time programming are about using wait-free and lock-free -//! structures. Examples of things that are *not* correct in real-time programming are: -//! -//! - Allocating anything on the heap (that is, no direct or indirect creation of a `Vec`, `Box`, or any -//! standard collection, as they are not designed with real-time programming in mind) -//! - Locking a mutex -//! - Generally, any kind of system call gives the OS the opportunity to pause the thread, which is an -//! unbounded operation as we don't know how long the thread is going to be paused for -//! - Waiting by looping until some condition is met (also called a spinloop or a spinlock) -//! -//! Writing wait-free and lock-free structures is a hard task, and difficult to get correct; however many -//! structures already exists, and can be directly used. There are crates for most replacements of standard -//! collections. -//! -//! ## Where in the code should real-time programming principles be applied? -//! -//! Any code that is directly or indirectly called by audio threads, needs to be real-time safe. -//! -//! For the Bevy engine, that is: -//! -//! - In the callback of `cpal::Stream::build_input_stream` and `cpal::Stream::build_output_stream`, and all -//! functions called from them -//! - In implementations of the [`Source`] trait, and all functions called from it -//! -//! Code that is run in Bevy systems do not need to be real-time safe, as they are not run in the audio thread, -//! but in the main game loop thread. -//! -//! ## Communication with the audio thread -//! -//! To be able to to anything useful with audio, the thread has to be able to communicate with the rest of -//! the system, ie. update parameters, send/receive audio data, etc., and all of that needs to be done within -//! the constraints of real-time programming, of course. -//! -//! ### Audio parameters -//! -//! In most cases, audio parameters can be represented by an atomic floating point value, where the game loop -//! updates the parameter, and it gets picked up when processing the next buffer. The downside to this approach -//! is that the audio only changes once per audio callback, and results in a noticeable "stair-step " motion -//! of the parameter. The latter can be mitigated by "smoothing" the change over time, using a tween or -//! linear/exponential smoothing. -//! -//! Precise timing for non-interactive events (ie. on the beat) need to be setup using a clock backed by the -//! audio driver -- that is, counting the number of samples processed, and deriving the time elapsed by diving -//! by the sample rate to get the number of seconds elapsed. The precise sample at which the parameter needs to -//! be changed can then be computed. -//! -//! Both interactive and precise events are hard to do, and need very low latency (ie. 64 or 128 samples for ~2 -//! ms of latency). It is fundamentally impossible to react to user event the very moment it is registered. -//! -//! ### Audio data -//! -//! Audio data is generally transferred between threads with circular buffers, as they are simple to implement, -//! fast enough for 99% of use-cases, and are both wait-free and lock-free. The only difficulty in using -//! circular buffers is how big they should be; however even going for 1 s of audio costs ~50 kB of memory, -//! which is small enough to not be noticeable even with potentially 100s of those buffers. -//! -//! ## Additional resources for audio programming -//! -//! More in-depth article about audio programming: -//! -//! Awesome Audio DSP: - #![forbid(unsafe_code)] mod audio; From 22dbdfba4f83992f349355a0393f12fb2612e753 Mon Sep 17 00:00:00 2001 From: Nathan Graule Date: Fri, 8 Mar 2024 10:54:02 +0100 Subject: [PATCH 7/8] doc(bevy_audio): review changes --- crates/bevy_audio/CONTRIBUTING.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/crates/bevy_audio/CONTRIBUTING.md b/crates/bevy_audio/CONTRIBUTING.md index 7346655d604fb..112bdc6470cc9 100644 --- a/crates/bevy_audio/CONTRIBUTING.md +++ b/crates/bevy_audio/CONTRIBUTING.md @@ -70,10 +70,10 @@ that amount of time, and play around with the buffer size to find the best compromise between latency and performance. Another is to conditionally enable certain effects for more powerful CPUs, when that is possible. -But the main step is to write code run in the audio thread following real-time -programming guidelines. Real-time programming is a set of constraints on code -and structures that guarantees the code completes at some point, ie. it cannot -be stuck in an infinite loop nor can it trigger a deadlock situation. +But the main step is to write code to run in the audio thread following +real-time programming guidelines. Real-time programming is a set of constraints +on code and structures that guarantees the code completes at some point, ie. it +cannot be stuck in an infinite loop nor can it trigger a deadlock situation. Practically, the main components of real-time programming are about using wait-free and lock-free structures. Examples of things that are *not* correct in From ec0938e912bf6b71ea00a603cb2d18b3de989034 Mon Sep 17 00:00:00 2001 From: Nathan Graule Date: Fri, 8 Mar 2024 15:11:46 +0100 Subject: [PATCH 8/8] doc(bevy_audio): fix markdownlint issues --- crates/bevy_audio/CONTRIBUTING.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/crates/bevy_audio/CONTRIBUTING.md b/crates/bevy_audio/CONTRIBUTING.md index 112bdc6470cc9..1920b1bbe0410 100644 --- a/crates/bevy_audio/CONTRIBUTING.md +++ b/crates/bevy_audio/CONTRIBUTING.md @@ -13,9 +13,9 @@ materials. Note that these guidelines are general to any audio programming application, and not just Bevy. -# Fundamentals of working with audio +## Fundamentals of working with audio -## A brief introduction to digital audio signals +### A brief introduction to digital audio signals Audio signals, when working within a computer, are digital streams of audio samples (historically with different types, but nowadays the values are 32-bit @@ -32,7 +32,7 @@ that any sound with frequencies higher than that will introduce artifacts. If you want to learn more, read about the **Nyquist sampling theorem** and **Frequency aliasing**. -## How the computer interfaces with the sound card +### How the computer interfaces with the sound card When requesting for audio input or output, the OS creates a special high-priority thread whose task it is to take in the input audio stream, and/or @@ -55,7 +55,7 @@ latency reasons, and increasing it for performance reasons. The threshold for instantaneity in audio is around 15 ms, which is why 512 is a good value for interactive applications. -## Real-time programming +### Real-time programming The parts of the code running in the audio thread have exactly `buffer_size/samplerate` seconds to complete, beyond which the audio driver @@ -94,7 +94,7 @@ Writing wait-free and lock-free structures is a hard task, and difficult to get correct; however many structures already exists, and can be directly used. There are crates for most replacements of standard collections. -## Where in the code should real-time programming principles be applied? +### Where in the code should real-time programming principles be applied? Any code that is directly or indirectly called by audio threads, needs to be real-time safe. @@ -149,4 +149,4 @@ buffers. More in-depth article about audio programming: -Awesome Audio DSP: \ No newline at end of file +Awesome Audio DSP: