4.0.10 - Nov 2024 Refresh

SFe Version 4.0.10 (Draft Specification) - November 2024 Refresh

SFe is now static (for now)

This is the tenth draft milestone of the SFe specification version 4.

We're focusing on changes that help compatibility between different SFe players, by adding features such as extended version sub-chunks, feature flags and more. We're also streamlining the format by reducing the number of variants; this has recently been a point of contention.

Making SFe variants easier to understand

We've listened to feedback, and now we're going to reduce the number of variants of SFe.

Previously, there was SFe32, SFe64L and SFe64. Now, we've postponed the divergence of features between 32-bit and 64-bit formats to version 5.0. Now, there is one SFe specification for both 32-bit and 64-bit versions of the format, and the only current difference is the chunk header format. We also plan on adding a new "dynamic" chunk header designed by SFe Team member spessasus that can scale from 32-bit to 64-bit and beyond in a future version (likely 4.5).

It is called "dynamic" because the length of the chunk size in the header is dynamic and depends on another value. This allows the format to be much more scalable, and wastes less space when compared to standard RIFF64. We will also replace the sfbk fourcc with sfen so legacy SF programs don't accidentally attempt to run an SFe bank.

We've also added a section detailing LTS versions of SFe, which will continue getting feature updates even after structurally incompatible changes (with a higher major version) have been made. For example, when the structurally incompatible SFe 5 releases, it is expected that we maintain feature additions to SFe 4 for some time after. This should strike the perfect balance between supporting legacy players and getting features included quickly!

It may be possible to also abandon the SFty sub-chunk, making the proposition of selecting a variant of SFe even simpler. Even if the SFty sub-chunk is not abandoned, it will still be much simpler for everyone.

Introducing the SFvx sub-chunk

The SFvx sub-chunk includes extended version information about the SFe bank, including:

• specification version
  • differs from ifil version
• specification type
  • drafts, milestones and final versions are supported
• draft milestone (when applicable)
• full version tag (when applicable)

By including such a sub-chunk in the ISFe-info subchunk, we can solve the problem with providing more version information without breaking compatibility with legacy SF players. The ifil value is strictly defined as two WORD values in legacy SF.

Feature flags and the flag sub-chunk

After the SFvx sub-chunk is the flag sub-chunk. This implements feature flags, and is split into branches, leaves and flags. Generally, branches correspond to types of features, leaves correspond to specific features and flags represent the features themselves.

The idea is that a bank has feature flags that are determined by the features in use by samples/instruments/presets. The SFe editor will add the feature flags automatically. The list of feature flags is included in the compatibility specification for 4.0.10, but should be moved to the same document in the final specification.

When an SFe player reads the bank, it will recognise the feature flags that it supports, but if it detects an unrecognised flag, then the user is warned. The user is also warned when flags that are not available in the specification corresponding to the SFvx sub-chunk are found, as the presence of such flags may indicate proprietary extensions.

We also include a "private-use" area (branch 240 to 255), but as the SFe license is copyleft and strongly discourages unpublished proprietary extensions, features may be moved to the main area of flags.

Other versioning changes

Previous drafts have described the first version of SFe as version 4.00. Two zeros are used due to legacy SF using this in its naming conventions (for example, the versions written in SFSPEC21.PDF and SFSPEC24.PDF are 2.01 and 2.04 respectively).

However, some programs have used the terms 2.1 and 2.4 to describe legacy SF. Therefore, we've made the decision to remove the leading zeros. This means that version 4.00 becomes 4.0 and 4.01 becomes 4.1. This is easier to understand for those who are not familiar with the legacy versioning system.

This is also a good time to mention that there will not be illogical version jumping in SFe, unlike in legacy SF. For example, E-mu updated version 1.0 to 1.5 and then to 2.00. As expected, 2.00 was replaced by 2.01, but 2.02 and 2.03 were skipped and 2.04 was the final version ever released by E-mu. Meanwhile in SFe, version 4.0 will be replaced by 4.1 or 5.0, not 4.2, 4.3 or 4.4.

In the future, we'll refer to major versions as "episodes". This naming is used to make it a bit easier for us to describe the development process of SFe versions.

Removing the info-list subchunk length limits

These limits inherited from legacy SF don't make sense for modern computers. Therefore, we're removing them, allowing bank developers to have an unlimited length for information.

Fixing a few other issues

We've also fixed some issues, including an incorrect reference to a chunk that was intended for version 4.4 being listed in section 4 as being intended for version 4.5 and some references in sections 7.4 and 7.5 to features that are to be implemented in version 4.1. We are also rewriting the definitions of more fields that convey multiple values using different bits in a similar vein to the byBankMSB and byBankLSB changes.

License changes

To make the SFe license compliant with the Open Source Definition, we've removed the requirements to share all modifications implemented in programs under the SFe license (or a compatible license) or not removing the link to the latest version of the specification (for final versions). Instead, these are simply considered "good practice".

This new version of the license supersedes the previous version. All SFe data that has been released in the past is now available without the need to share all modifications or to retain the link to the latest version.

It is likely that the SFe program specification containing all "good practice" tips will instead include these requirements in the future.

We will write the full version of the license in time, but there are other things that we need to focus on.

More features for SFe in the future!

According to feedback from team member davy7125 we're adding a few features to the plan to versions beyond 4.0!

We've got a list of baseline features that SFe should include in first implementations:

• default modulators
  • already planned for 4.1
  • using the DMOD chunk
• initial value of MIDI control changes
  • listed as "not sure it is useful though"
  • may be included in 4.1
• sample storage
  • listed as solved by Werner SF3
• round-robin
  • will be added in 4.2
  • 4.2 will consist mostly of generator enum definitions
• independent envelopes/LFOs for pitch/attenuation/filter
  • will likely be added in 4.3 but may be moved forward to 4.2
• sample modes
  • likely to be added in 4.5
• MSB/LSB banks
  • will be added in 4.0
• 64-bit mode
  • static 64-bit SFe is included in the 4.0 specification
  • dynamic SFe is 64-bit by default and will be released in 5.0
• conversion between sfz/sf2
  • this is achieved using the abstraction level system
  • AL6, AL5 and AL4 are SFZ, AL2 is SFe and AL1 is SiliconSFe
  • basic specification for AL6, AL5 and AL4 will be ready for 4.6
  • full AL3 specification will be 4.7 or 5.0
  • during episode 4, we add DLS features, and during episode 5, SFZ features

The comment also includes other features that are less-critical and will be delayed for future versions. Here is a similar list:

• multiple loops at sample level - sometime in episode 6
• different kinds of filters - very likely to be 4.3
• modulator inputs - might be as early as 4.1 but it's unknown what this means
• conditional starts/key switches - 4.7, but if significant structural changes are required, during episode 5
• exclusive class normal release - likely 4.2
• field length limitations - 4.0 for SFe64 only
• negative attenuation - 4.0, its feature flag is already there
• real attenuation - 4.0, its feature flag is already there
• sm32 chunk - 4.0

About other future plans

So, at this point, we are almost ready to release the final version of SFe version 4.0. However, we need to communicate with program developers such as FluidSynth, so we can get an agreement of features to be implemented, as well as a final specification of Werner SF3 that can be included in SFe.

Ideally, this should be done before the release of Polyphone 3 (the first planned version of Polyphone with SFe used by default). We should also communicate with other SF editing and playback programs (for example the Swami editor and the Bassmidi player) to ensure that SFe does not simply become a "Polyphone and FluidSynth Custom Features" that suffers from the same problems as SFCF. Remember that one of the fundamental goals of SFe is to tackle the inevitable problem of "format wars".

Once we've created the near-finalised text of specification version 4.0 (this will appear as a draft milestone), then we can work on creating the logos and niceties so we can release a finalised specification. The finalised specification is likely to have a somewhat different structure to the draft specifications, which closely followed the structure of legacy SF2.04. Parts that are identical ...

4.00.9 (Draft) - November 2024

SFe Version 4.00.9 (Draft Specification)

Getting closer to release

This is the November 2024 draft milestone of the SFe specification, version 4.00.

Unlike last milestone, we've included a few new features, as well as an improvement to make the specification more readable.

ASCII is now obsolete

The original legacy SF specification stated that all fields that stored a string were to use ASCII encoding.

However, since the 1990s, ASCII (or more precisely, US-ASCII) has been almost completely replaced with UTF-8 in modern computer systems. Because all ASCII text is also valid UTF-8, text that uses only ascii characters is completely compatible.

Some programs (including Polyphone) had unofficial support for UTF-8 in the ICMT subchunk, but it didn't extend to other string fields (that were still limited to ascii). Starting with 4.00.9, you can use UTF-8 in all string fields, including those found in the INFO-list and pdta-list chunks. You can now use kana, CJK characters and other non-ascii characters anywhere in SFe banks.

For the sake of completion, we've also redefined "case-sensitive" and "case-insensitive" to explicitly define the character encoding as UTF-8, rather than ascii.

wPreset changes

In the last announcement, we said that we would rework the wPreset system in 7.2 to allow the bank creator to define the use of the second bit of the wPreset word in version 4.04. Therefore, the second bit of wPreset is currently undefined for SFe versions between 4.00 and 4.03.

This is subject to change, so please review the specification every time there is an update.

Preset library management system plans

In previous drafts of SFe, we defined dwLibrary and dwGenre. However, we then realised that they were not strings, but rather DWORDs (integers), so we cannot encode strings.

Therefore, we've updated the specification to expect a DWORD that's linked to a value in an enum, which will be defined in the ISFe-list chunk. This subchunk will be defined in version 4.05.

Introducing the SFe license

A license is included for the first time to clarify what developers are allowed to do with the specification text. It's included in the specification, as well as LICENSE.MD.

It is a copyleft license that allows developers to extend the SFe specification provided that they document any changes and extra features that are implemented in their SFe compliant programs. This is to ensure that proprietary extensions to the SFe format aren't made that may misbehave when used with a player that supports a newer version of SFe.

We do not permit the removal of copyright notices, the removal of the link to the latest version of the specification, or any claim that we're affiliated with E-mu or Creative. This reduces the risk of a misunderstanding.

For draft versions, we also add the condition that the specification be clearly marked as such, because implementation of features can differ wildly between draft specifications and the corresponding final version. The license for final versions of the specification can be found in LICENSE-FINAL.MD.

Like most open-source licenses, we provide the specification "as-is" and disclaim any implied warranty.

On implementation accuracy

Section 9.7 of SFSPEC24.PDF gave information on implementation accuracy, and said that implementations are almost always going to be less accurate than the specification due to computational performance of hardware that implementations would run on.

Due to computers being much faster than in the 1990s when the specification was designed, implementation accuracy requirements are now significantly higher, with implementation of the feature flags system (coming in the next draft) becoming strongly recommended.

WernerSF3 compression

The WernerSF3 compression implementation has effectively been completed for a long time. We are just awaiting a final version of the specification of WernerSF3 from FluidSynth.

However, in this milestone, we've declared CognitoneSF4 to be a proprietary compression format due to its incompatibility with WernerSF3. We've also cited it as the reason that the .sf4 file extension is not used.

File format versioning for SFe32

We've decided to modify the file format versioning for SFe32 to make it easier to read. Please read the SFe32 specification for more information.

Defining a subset of SFe32, SFe32L

Some features found in SFe do not affect the sound output when a bank containing these features is used on a legacy SF player. This subset is called SFe32L.

Please implement the SFe32L features before implementing other features.

File extensions and ISFe-list subchunk

For this version, we've finalised the required file extension to be used. It is .sft, as suggested by spessasus.

We decided that having to use different file extensions for each variant of SFe was unnecessary. Instead, the SFty (SFe type) subchunk, found inside the ISFe-list subchunk, is used to determine the SFe variant to use, but if missing or unknown, there will be other features in the format to allow the program to determine the variant itself.

The ISFe-list subchunk, found inside the INFO-list chunk contains SFe-specific information. The first subchunk, as mentioned before, is the SFty subchunk. The next version, 4.00.10, will add the flag subchunk for feature flags. Other subchunks that are coming soon include prsw (preset switch, 4.04 and later), and plms (preset library management system, 4.05 and later).

Readability changes (4.00.9b)

Very shortly after the initial release of 4.00.9, we made a readability change that does not affect the file format itself at all, and will be a precedent for other similar readability changes. This readability change consists of removing wBank and replacing it with byBankMSB and byBankLSB.

This is completely compatible with wBank, as the actual data doesn't change at all. One WORD value (2 bytes) is replaced with two individual BYTE values, with the first byte corresponding to the bank select MSB (CC00) as used in legacy SF, and the second byte corresponding to the unused byte of the previously-used wBank value.

More experienced SF developers may continue to think of the 2-byte bank select space as a single wBank value equalling (cc00 + 1024 * cc32), but the formatting of the space as two values is easier to understand for less experienced SF developers.

Future data values that encode more than one value that correspond to different bytes may be split to make it easier to understand. It is our responsibility to make the specification as easy to understand for those who want to create SFe-compatible programs.

About future plans

The next version, 4.00.10, will include the first version of the feature flags system. SFe programs that implement the system would read the flags that the bank requests, and warn end-users if an unrecognised or unsupported flag is requested. It is planned for release in December 2024.

Since the release of 4.00.8, we've received communication from Polyphone that Polyphone 3 is planned to include support for SFe 4.0x, and it will be the default export format, with legacy SF being an option. However, Davy has said that more features should be added for SFe as soon as possible. Therefore, there is a good chance that the number of features to add in 4.02 will increase significantly. 4.01 (the DMOD and PNMM update) remains unchanged, except for the potential move of the chunks to the ISFe-list chunk (from the root of the INFO-list chunk).

Work on reverse engineering SFCF features has begun by spessasus. Ideally, we should be getting an official specification from the SynthFont author, but further reverse engineering may be required if this isn't possible. We've found out that unused generator type values are being used by SFCF, and we'll avoid using such values when adding features that require new generator type values. This is for interoperability reasons and should be okay. The plan is that we add SFCF support in version 4.03, but should there be any objection, we can simply leave out SFCF feature support and move all features forward by one version.

Versions 4.04 and 4.05 include support for manipulation of the unused byte in wPreset, and definitions of enums for dwLibrary, dwGenre and dwMorphology respectively, with more DLS features incoming after this. The last version of SFe 4 to release should have feature parity with DLS, with feature parity with SFZ and the abstraction level system fully implemented in the last version of SFe 5.

Feedback

Your feedback is appreciated! Please use the discussion page of the GitHub repository to give feedback. You can also fork and create a pull request if you have any changes that you want to make.

---

What's Changed

• Update to version 4.00.9a by @sylvia-leaf in #26

Full Changelog: 4.00.8...4.00.9

4.00.8

SFe Version 4.00.8 (Draft Specification)

Fixing Some Things

This is the eighth draft milestone of the SFe specification.

This release of the specification doesn't include new features in the format spec. In fact we'll stop mentioning it as there won't be any new features until 4.01.

Introducing the file repair specification

For 4.00.8, we've written a file repair specification for programs that fix SFe banks that are Structurally Unsound. Programs that meet this specification should help bank developers by reducing the amount of time spent fixing problems with files that don't load.

We've included both automatic and manual repair, as well as detailed explanations of each type of error that can happen.

We're looking for feedback for it, so please tell us what should and shouldn't be included in the file repair specification.

An implementation of the file repair specification will happen in the far future. It will likely not be available for version 4.00's final release.

Fixing the format outline

For a long time, the format outline found in section 4 was based on a very old draft of SFe, and was no longer representative of the current state of the specification.

There has already been confusion by some with regard to this format outline, so we've spent some time to update it to be accurate to the current version of the specification.

We've also included the ISFe sub-chunk, but there is currently no information that is defined in this sub-chunk.

Compression issues fixed

Previous versions of the SFe draft specification assumed that the data was uncompressed, which meant that Werner SF3 data was technically not compliant with the SFe spec.

We've now fixed the specification to take into account Werner SF3 compressed data. Now there should not be conflicts between SFe and the August 2021 revision of Werner SF3.

About future plans

We are planning the next few draft milestones of SFe:

• 4.00.9: SFe player reference implementation
• 4.00.10: Feature flags system
• 4.00: Final release, similar to 4.00.10 with problems fixed

Planned rework of section 7.2

We plan on removing the wPreset changes in section 7.2, and instead we would allow users to define what each bit in the wPreset value does.

If removed, then the wPreset changes would be reimplemented in 4.04 with the configuration as a part of the ISFe sub-chunk.

---

Full Changelog: 4.00.7...4.00.8

4.00.7

SFe Version 4.00.7 (Draft Specification) - October 2024

Getting Some Help

This is the 7th draft milestone of the SFe specification. Sorry that we forgot to make a pre-packaged release of the 6th milestone; we will try not to forget again.

This release of the specification does not include new features in the format spec itself, as there has been a feature freeze for SFe version 4.00, but focuses on fixing the small issues. We thank @spessasus for taking the time to fix some issues that were in 4.00.6.

In this announcement, we will also include changes that were included in 4.00.6 for the sake of completeness.

Significant structural changes

We've split the SFe specification into three different smaller specifications:

• SFe32 for full 32-bit compatibility
• SFe64L for a 64-bit format that is mostly similar to SFe32 to help users move to 64 bits more easily
• SFe64 for the more divergent 64-bit specification

Currently, there are two specifications released, SFe32 and SFe64. However, SFe64 will have structural changes that break backwards compatibility starting with version 5.00, so the first version of SFe64L, 5.00, will be based on the structure of SFe64 4.x.

Additionally, a pull request by @spessasus was merged by team member @stgiga, merging the admittedly hard-to-read separated sections of the format specification into just one document. This pull request also adds for the first time, a table of contents.

Section 7.1a was removed, because it will only be relevant when SFe64 version 5.00 is released.

Versioning system changes

Yes, we've changed the versioning system again. However, this time it is just the naming. Now, because SFe development is done on a rolling basis, and any update to the specification can be considered a draft, we've renamed the SFe drafts to draft milestones to make the purpose of these releases more clear.

Initially, we planned the file repair specification for 4.00.6. However, 4.00.6 was the version which we split SFe32 from SFe64. It was then moved to 4.00.7. However, when we decided to merge the aforementioned pull request, the structural change meant that we could no longer refer to the specification as 4.00.6a-1, so we renamed this update to 4.00.7. Now, the file repair specification is planned for version 4.00.8.

Modulators

The modulator system that we originally planned for SFe was scrapped. Instead, we will implement spessasus's DMOD sub-chunk for 4.01, and add back some of the features that were planned in this update.

Right now, discussions have been made about designing the new modulator system around the CC94 (InsertionEFX) format as used by Roland in the SC-88Pro and SC-8850.

About the ISFe sub-chunk

The ISFe sub-chunk will be used by the SFe format for SFe-specific metadata. Currently, no data is defined, and there will likely be no definitions in 4.00, however we've started to formulate plans for how we can use this sub-chunk.

As every soundfont developer knows, each soundfont player supports a different amount of the 2.04 spec, and few soundfont players support everything. Therefore, a few soundfonts (most notably GeneralUserGS by S. Christian Collins) include a built-in player test (in the case of GUGS, it is the GUTest preset).

Therefore, the plan is to use part of the ISFe sub-chunk as a sort of "feature flags" system. The SFe editor program will add the feature flags that are used by the bank to ISFe. Once the bank is played by software that doesn't support all of the features, the software can then read the ISFe sub-chunk, figure out what features are being used, and then give a warning if the bank is trying to use an SFe feature that isn't supported by the software.

Currently, we plan on developing one or two features at once, and then release a new version of the format. However, if we use ISFe to implement feature flags, then we can add many different features with each format update, and then allow the SFe program developers to implement features at their own pace. Just using the ifil version to communicate available features has the disadvantage of being unable to clearly mention which of the features are supported.

The feature flags that we plan on adding would also include features included in the base SF 2.04 format, because there are still problems with certain soundfont players not supporting soundfonts that were written to legacy SF standards.

Arguably, we could make programs reject any structures that they don't recognise. However, this would severely affect compatibility, as newer SFe files would simply not run on older players.

The ISFe sub-chunk will eventually do way more than just be a feature flags system, but this is the first idea that we are going to try to implement.

About future plans

We are planning the next few draft milestones of the SFe specification:

• 4.00.8: Redo the RIFF structure and program repair specification
• 4.00.9: SFe SDK release, start to solve the TSC implementation problem
• 4.00.10: Update the SDK with a few more things

Program specification

The program specification is mostly the same as previous versions, but has changes that correspond to the changes in 4.00.7. Namely, TSC mode support is no longer required, as it is now a separate specification to SFe itself.

Compatibility specification

In the compatibility spec, we've added information on how to handle proprietary compression formats, and we fixed a mistake in the INFO chunk information.

Where is TSC mode?

We've had some discussion about TSC mode, where the sdta chunk is placed last to extend the maximum SFe32 file size to 8 GiB. However, it's been determined that TSC mode was obviously not compatible with legacy sound card hardware, so it has been removed. Instead, TSC is now available as a separate specification: https://github.com/SFe-Team-was-taken/TSC

We will bring it back once we have the original documentation about TSC mode, we can test it on legacy sound card hardware and we can create test SFe files implementing TSC. After that, it is likely to be moved into SFe64(L) or included as a separate specification. If TSC mode is a separate specification, then we'll just implement it into SFe as a feature flag. Then, if developers don't feel like rewriting their code, then they can just detect the feature flag in TSC banks and then reject the bank.

What's coming up?

For 4.00.8, we plan on providing an SFe file repair specification for development of programs. This is to be released in late 2024 or early 2025. After that, 4.00.9 will be the first version with an implementation of SFe, based on SpessaSynth, and 4.00.10 will introduce the feature flags system implementation. If you have any suggestions for other features that we could add, or additional information that we could include in future specs, then please open an issue and describe what you would like to see.

New Contributors

• @spessasus made their first contribution in #14

Full Changelog: 4.00.5...4.00.7

4.00.5 (Draft)

SFe Version 4.00.5 (Draft Specification) - August 2024

Welcome Autumn, Welcome GitHub

This is the fifth draft of the SFe specification, and we have started to use GitHub for SFe specification development.

This release of the specification does not include new features in the format spec itself, as there has been a feature freeze for SFe version 4.00, but focuses on the release of the reference implementation of samples used for the ROM emulator that SFe implementations will need to implement.

For this reason, the changes made in this draft are mostly limited to a few formatting changes and new references to the reference implementation of the ROM samples.

SFe and GitHub

We've made the decision to host the spec on GitHub, and allow users to start issues and pull requests to improve the format.

ROM sample specification

We hope that SFe players will allow more users to enjoy soundfonts that use legacy sound card ROM samples, so SFe implementations should include an AWE ROM emulator. The problem is that the original samples are copyrighted, so we have provided a specification for samples that if met, should provide the same results as the original ROM samples. You can find this as part of the compatibility spec.

Initially, we have started with ROM samples for the 1MB wavetable that was implemented in the AWE32, but if necessary we will extend the ROM sample specification for larger wavetables.

New versioning system

In 4.00.4, we introduced a new versioning system. This system, borrowing many features from semantic versioning, makes it easy to identify:

• whether two spec versions are compatible with each other
• whether two spec versions have the same feature set
• whether or not a spec version has new features
• whether or not a spec version is a draft version

We've added an explanation of the new versioning system in section 0.1a of the format spec, and a clarification about the version of the compatibility spec.

About future plans

We've added section 1.5a, "Future Plans" to the format spec, to formalise the planned future of the format. Most importantly, the feature freeze for version 4.00 has been reached, and any more features will be considered for future versions starting from 4.01.

Program specification

Since 4.00.4, programs have been prohibited from using proprietary compression formats (such as sfArk, SFPack, SF2Pack and sfq). We made this decision based on the restrictive licensing of many (but not all) of the programs used to compress/decompress these formats, and incompatibility of these formats with major open source soundfont programs.

In SFe version 4.00, the only permitted compression format will be Werner SF3, which was created by Werner Schweer for MuseScore, and adopted widely. This format is also the reason why SFe specification versions start from 4:

• legacy SF is version 2
• Werner SF3 builds from legacy SF, and is version 3
• SFe builds from Werner SF3, and is version 4.

Compatibility specification

In the compatibility spec, we've added information on how to handle proprietary compression formats, and we fixed a mistake in the INFO chunk information.

What's coming up?

For 4.00.6, we plan on providing an SFe file repair specification for development of programs. This is to be released in late 2024 or early 2025. If you have any suggestions for other features that we could add, or additional information that we could include in future specs, then please open an issue and describe what you would like to see.