Resource system improvements

[vk2gpu]

Current setup is a bit rushed, and not as well thought out as it should be. Whilst it works, there are some improvements that could be made:

Dependencies
Correct handling of dependencies when loading resources. Rather than dependencies being loaded on demand, a dependency list should be generated on conversion. This should mean that we don't end up in the situation of kicking off more jobs than there are able to be in flight, and thus resulting in a deadlock due to the job manager running out of fibers.

Converted resources
Converted resources should probably be saved out as an intermediate format, rather than packed into binaries containing all resource data. For example, a model is stored like so:

• (file) model.fbx.converted
• • (binary data) vertex data, index data, mesh parts

It may be more beneficial to store as loose files after conversion like so:

• (folder) model.fbx.converted
• • (file) header
• • (file) vertex data
• • (file) index data
• • (file) mesh parts

This is not optimal for runtime, but gives much more flexibility later on for being able to pack resources into packages, separate all headers from the data, packing some bulk data to allow memory mapping of entire sets of resources, etc.

Packaged resources
Building on top of the changes for converted resources, some system should be in place to generate resource packages. Resources should not be addressed by the package they reside in, and the same assets should be able to exist in multiple packages. This could also include duplicating of data if appropriate, Effectively, it would just be like a directory overlay - load package -> load all resources it references -> make visible to the engine & game as their name/UUID.

Conversion cache
Either built into the idea for storing of converted resources, or separately, it would be good to have a cache for converted data. This could be used to accelerate importing by hashing the source data & other relevant data, and storing the binary output in a cache as that SHA1 hash. For operations like texture conversion, this would mean an already imported texture would just be pulled from the cache if the source data & parameters haven't changed. See "Asset File Store" in https://twvideo01.ubm-us.net/o1/vault/gdc2017/Presentations/Clyde_David_TheDataBuilding.pdf.

Basically, this would be utilized within the resource converters by hashing the appropriate source data & properties, and prior to running step of the conversion (i.e. generating a mip map level, compression to BCn, running vertex cache optimization, etc...) it would attempt to request the converted data that matches the hash. In pseudo code:

Blob sourceData;
bool sourcePropertyA;
int sourcePropertyB;
Blob convertedData;
Hash hash = Hash(sourceData, sourcePropertyA, sourcePropertyB, resourceTypeVersion);
if(!cache->Get(convertedData, hash))
{
  convertedData = convertData(sourceData, sourcePropertyA, sourcePropertyB);
  cache->Add(hash, convertedData)
}

External converter process
Right now all resource conversion is done in the engine process. Moving it out into a separate process would:

• Make a running game/editor/etc more robust since any bugs in the converters wouldn't cause a crash in the engine's process.
• Converters don't need to run on the same machine. In my case, I like to code on my laptop, but it isn't overly powerful, so conversion could just run on my desktop machine. The down side is this would add complexity at an infrastructure level (i.e. now it depends on some kind of file hosting, network).
• If an asset causes a converter to crash, the engine would simply keep trying to request the asset and not even know or care. Now it's possible to debug converters when they fail, and not need to keep relaunching the engine to get to that point.
• Converters can now perform more complex tasks, such as be an entire instance of the engine themselves for tasks such as lighting generation. No conflict with anything running inside of the engine requesting the resources. If doing this, it would be nice if the immutable resource data could be shared between processes (memory mapping would help here).

Converter metrics
Performance information, such as total time, avg. CPU & GPU load.