Solver hangs indefinitely after global optimization and fails to create view transformations

[winnubstj]

On some datasets the solver initializes, performs global optimization, but then hangs indefinitely and fails to generate transformation models for the tiles.

Here is part of the log:

Starting the spark driver
Checking for /shared/scratch/spark-work/b4ccdc/.sessionId
Found /shared/scratch/spark-work/b4ccdc/.sessionId in 0 s
Use Spark IP: 10.0.1.142
[-x, /shared/scratch/nextflow/b4/fcdbe1fa668763a47ae34f263372a1/stitched-dataset.xml, -s, IP, -l, fiducials, --method, ONE_ROUND_ITERATIVE, -tm, AFFINE, --channelId, 1, --relativeThreshold, 3.5, --absoluteThreshold, 7.0, --groupChannels]
xml: /shared/scratch/nextflow/b4/fcdbe1fa668763a47ae34f263372a1/stitched-dataset.xml

...

(Thu Aug 08 20:23:38 GMT 2024): Fixing group-tile [0-1]
(Thu Aug 08 21:07:27 GMT 2024): prealigned all tiles
Shuffling took 0 ms
First apply took 763 ms
0: 4.8665754214827155 4.8665754214827155
1: 4.4960545530422475 4.8665754214827155

...


Concurrent tile optimization loop took 1273080 ms, total took 1273843 ms
(Thu Aug 08 21:28:40 GMT 2024): Global optimization of 62
(Thu Aug 08 21:28:40 GMT 2024):    Avg Error: 3.0632728220428964px
(Thu Aug 08 21:28:40 GMT 2024):    Min Error: 1.6824240273465503px
(Thu Aug 08 21:28:40 GMT 2024):    Max Error: 11.933269910172326px

The process then continues to run (single threaded) but doesn't produce additional outputs. I observe the same behavior with other methods (ONE_ROUND_SIMPLE). In this example all tiles are connected.

I tried running it locally (with ./solver) and observe the same thing.

Also a more general question, does only the optimization step run multi-threaded? I ask specifically because the prealignment step seems to take quite long (~40 mins here) and runs single threaded? Not sure if that is expected behavior.

Full log can be found here