The tests for temporal_rotate_rectangle don't pass locally on my intel mac but do in docker

[CKrawczyk]

Strangely the temporal_rotate_rectangle tests fail when run locally on my Intel MacBook Pro, but they pass without issue inside the docker container running on the same computer. Not exactly sure what is going on just yet, but it looks like the clusters resolve differently on the different OS.

Here is an example of a failing test's output:

E           -             'T0_toolIndex0_clusters_angle': [9.8, 10.0, 20.0],
E           ?                                                ^  ^^ ^
E           
E           +             'T0_toolIndex0_clusters_angle': [9.4, 9.4, 20.0],
E           ?                                                ^  ^ ^

This needs to be explored a bit more, it would not be good if the aggregation results were OS dependent...

[CKrawczyk]

Note: on my Mac I created a new environment with a fresh install of the aggregation code and got the same results.

[CKrawczyk]

After some digging, the failing tests are all using the IoU metric of the various shape reducers. The clusters are the same on both Linux and Mac, but the average values within each cluster are different.

The average values for the IoU metric rely on numerical optimization of the metric via the average_shape_IoU function. This uses

m = scipy.optimize.shgo(
    sum_distance,
    sampling_method='sobol',
    bounds=average_bounds(params_list, shape)
)

My guess is this algorithm is the cause of the difference seen in the test results. It could also be the case where the average shape is not unique (although I don't think this is likely as each clustering algorithm gives different clusters and each of them has this averaging issue).

Under the hood, the average function works by finding the shape that minimizes the sum of squares of IoU metric distances with respect to all the shapes in a cluster. It might be worth exploring if there is a more stable numerical optimizer that could be used in this case.

Display time and sigma are the only parameters that average consistently. Oddly, sigma depends on the values of the optimization function, so that is extra odd that can be the same but the other values are different...

[CKrawczyk]

Given how unstable shgo seems to be across OS I propose a switch to a different minimizer that is more deterministic. After some testing I found

bounds = average_bounds(params_list, shape)
m = scipy.optimize.direct(sum_distance, bounds=bounds, locally_biased=False, vol_tol=1e-20)

works well and is the same in both Docker and on the Mac. Also, the min it finds is lower than the min the previous method found, so a better solution is the same amount of time that is stable.

The vol_tol might need some adjusting, more testing will need to be done.

This change will also mean all IoU tests will need to be updated as the "true" values for the average will change. This should all be in its own PR so it can be easily reverted if needed.

[CKrawczyk]

Note, here is some test code that can be used on different OS that shows the issue:

import scipy
from panoptes_aggregation.reducers.shape_metric_IoU import IoU_metric, average_bounds

shape = 'temporalRotateRectangle'
params_list = [
    (510.0, 500.0, 150.0, 50.0, 10.0, 0.1),
    (490.0, 515.0, 160.0, 60.0, 12.0, 0.1),
    (510.0, 500.0, 120.0, 60.0, 15.0, 0.1)
]

def sum_distance(x):
    return sum([IoU_metric(x, p, shape)**2 for p in params_list])

bounds = average_bounds(params_list, shape)
m = scipy.optimize.shgo(sum_distance, sampling_method='sobol', bounds=bounds)
print(m)