Thunderbolt 5 mesh of Mac Minis

[fkorotkov]

Hi there,

I've just got 3 M4 Pro Minis to experiment with MLX, exo, etc. One thing I noticed that when I connect all of them in a ring:

graph TD
    Mac1[Mac Mini 1] <--> Mac2[Mac Mini 2]
    Mac2 <--> Mac3[Mac Mini 3]
    Mac3 <--> Mac1

Loading

is that not all the Thunderbolt connections are active (in my example above connection between 1 and 3 is not active even though there is a cable between them). Looks like Apple is not allowing cycles in the topology. Does anyone have experience of running multiple Minis with Thunderbolt 5 between them?

I'm measuring though via iperf and it seems each hop between Minis affects throughput. I can imagine if it's not possible to have cycles in the topology then this solution might not be that scalable.

Will appreciate any insights from folks that have experience with many Minis and Thunderbolt.

[ivahno]

Also interested in a ring network topology using thunderbolt 5

[chachwick]

I'm fascinated by this project too. I'm hoping to build up a cluster in the next month.
You've probably already seen this @fkorotkov, but just in case, did you follow the Thunderbolt connection process here?

Looking forward to hearing about your progress.

[OhJayGee]

@fkorotkov Fedor - when you add the thunderbolt connects the Mac mini's span a IP network. You probably gave them IP addresses in one private network (10.0.0.0/8) or maybe (192.168.0.0/16). that means that one network is used to send packets to your 3 Mac minis - meaning the additional thunderbolt connect is not used because the 1st thunderbolt connect takes precedence via the default route. You will need to tell the two Mac minis with the additional direct thunderbolt connect that for both of those the Mac on the other side should be reached via the 2nd thunderbolt bridge - meaning you'll have to add a direct route that sends packets via your 2nd thunderbolt device.