Engine benchmark repo

[AngelOnFira]

It was discussed at today's meetup that there was work to use a shared sample scene to compare how different Rust rendering engines are doing in terms of features.

@cwfitzgerald and @aclysma have been discussing this already as far as I can tell.

I'm thinking it would be cool to have a repo on this org with the visual results? Would love to hear the thoughts of others. Also, I'm not trying to step on any toes if there are other plans :)

[cwfitzgerald]

I definitely would be unopposed, especially if we can all use the exact same positions.

[Ralith]

The existing ECS benchmarking repo is already not well maintained. Who is volunteering labor to keep another, more complex, benchmark repo in good shape long term?

[aclysma]

I personally think this can start life as a personal repo (@cwfitzgerald and I plan to get it started and anyone else that wants to help is welcome to do so,) and once we figure out what we want it to be, we could consider at that point whether it makes sense to be a WG repo. (after we know what it actually is and who is using it.)

Even if the WG does not host the repo, it could signal-boost the existence of it (through arewegameyet.rs, newsletter, or other ways). I think maintaining actual repos is usually better done by people who are actually doing the work, although I could see there being some exceptional cases. (I think we can wait to see if this idea actually pans out and if it seems like such an exceptional case later.)

@cwfitzgerald and I discussed a little earlier today. What we are thinking about doing is producing a set of instructions and a python script imports the bistro scene https://developer.nvidia.com/orca/amazon-lumberyard-bistro and fixes some of the errors caused by importing it as FBX (everything being scaled by 0.01, materials being wrong/not using textures, duplicated mesh data, etc.)

If we have such a repo, I think it may be worth having a few utility scenes as well like a grid of spheres with metalness/roughness increasing along X/Y axis like this. (I already have a python script for generating this scene in blender.) Something lighter-weight as well like sponza or just a trimmed down version of the bistro scene.

The overall goal is to provide cleaned up assets for demoing/testing things in rust (although it may be useful elsewhere too.) This way we all know we're working from the same data and results will be reproducible.

[aclysma]

The repo we discussed is now public https://github.com/aclysma/rendering-demo-scenes and includes scripts to generate the above PBR test and to import/clean up the amazon lumberyard bistro scene

As far as moving the repo to this org, I don't have strong feelings either for/against doing so.

[AlexEne]

In general it's hard to compare even renderers since they can have different goals. For example, comparing how unreal and unity deals with a certain scene has some value, but not a lot of value I'd argue.

Various engines do different trade-offs and it's hard to capture them as part of a benchmark story.

[minecrawler]

I think there are many aspects which have to be clear before even writing the first line of a benchmark. For example: which packages qualify for comparison? Is this Ash vs Vulkano, WGPU vs Rendy or Bevy vs Amethyst? Which features should be compared? Sure, speed is important, but what about the result of the benchmarks (how "good" do certain implementations look, how accurate are they? Are they erroneous...)? What about sound, asset support, input delay, etc.? On which platforms will it be benchmarked for the official results?

There are so many things which are interesting when comparing Engines, maybe the scope should be well thought out first!