Speed up fat32 impl

[Frostman]

Hello,

We're looking to use go-diskfs to build an air-gapped installer image for our product. It includes a bunch of files (like grub tree), OCI images, binaries and etc.

The speed up is 20+ times on bigger file systems - from 540 seconds to 23 seconds for a 1G image with two partitions of ~500MB each.

The test setup is:

Intel(R) Core(TM) Ultra 7 155H, 32GiB RAM, SN740 NVMe WD 512GB
go 1.23.0
6.10.7-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC

Here are three profiling reports:

• Unmodified: v1.4.1.pdf
• Removing keys sort: v1.4.1-nosort.pdf
• Replacing map with slice: v1.4.1-nomap.pdf

[deitch]

20x! That is great. I will kick off CI here, and take a closer look tomorrow.

[deitch]

CI is green, so happy to merge this in, and thank you.

Do you mind linking to the product where you use it? I always like to learn more about how things are used in the wild.

I took a look at what you did. Instead of storing the clusters as a map, you replaced it with a slice. Whereas for a map, the indication that a cluster is not used is the fact that it does not exist in the map, for the slice it is that the value at that position in the slice is 0. A slice is not inherently faster or slowed than a map, and indexed slice lookups vs hashed map lookups should be about the same speed. So where does the speedup come from? Is it the allocateSpace(), where it needs to loop through the clusters and create a slice of keys and then sort them?

[Frostman]

@deitch, thx for merging!

Removing unused keys sort gave "only" 3x speed up, and that's b/c sort is not too slow, I think building list of keys before sort was the slower part. If you'd look at the profile, you can see that mapaccess2_fast32 called from allocateSpace is the biggest time consumer and that is just implementation of the get from the map, source https://go.dev/src/runtime/map_fast32.go. The problem is that read from the map is way more expensive then read from slice by index (on big scale). When you're reading from map you need to calculate hash and go through hash collisions/overflow. When you're reading from slice it's simply memory pointer offset.

[Frostman]

@deitch the project I've mentioned is an installer for the https://hedgehog.cloud/. We're a startup building a product for DCs.

[deitch]

project I've mentioned is an installer for the https://hedgehog.cloud/. We're a startup building a product for DCs

That is pretty cool. I will reach out offline to ask about it.