Truncation of the eigenvalue spectrum

[dafeda]

During testing of the analysis module, we got results we were not expecting based on intuition.
To get results we would expect, we had to manually set the truncation value to 1.0 by:

analysis_module_set_var(enkf_module, ENKF_TRUNCATION_KEY_, "1.0");

The default truncation value is set to 0.98 and is documented here:

https://ert.readthedocs.io/en/latest/reference/configuration/keywords.html#enkf-truncation

After talking to Joakim and Geir, it seems as if the default truncation value is set based on the specific problem Equinor is solving.

Users can set this value by using ANALYSIS_SET_VAR ENKF_TRUNCATION 0.9, but according to both Joakim and Geir, this is probably not done.
Edit 07.04.2022: There are cases in the wild where users are setting the truncation value.

The tests where we check properties of the update algorithms are here:

https://github.com/equinor/ert/blob/main/libres/tests/analysis/test_update.cpp

To understand the tests in-depth, you should read this first:

https://ert.readthedocs.io/en/latest/theory/ensemble_based_methods.html#kalman-posterior-properties

The tests use a simple linear model, and an argument can be made that ERT was not designed to be used with such simple problems.
Still, it would be preferable if ERT somehow adaptively set the truncation value based on the problem at hand.

In the paper "Optimal reduced space for Variational Data Assimilation by Arcucci et al from 2018" a relatively simple method of setting an "optimal" truncation value is proposed, which I have implemented here:

https://github.com/dafeda/ert/blob/51a4e84068803e3b12bb02b2950cc87a7fa2814c/libres/lib/analysis/enkf_linalg.cpp#L69

    // Based on
    // Optimal reduced space for Variational Data Assimilation by Arcucci et al. 2018
    // Note: I've only skimmed the paper ...

    int num_significant = 1;

    double e = std::sqrt(sig0[0]);

    for (int i = 1; i < num_singular_values; i++) {
        if (sig0[i] > e) {
            num_significant++;
        } else {
            break;
        }
    }

Note that this code is not merged to main.

Using this method, it is not necessary to set the truncation to 1.0 in order for our current tests to pass.

I propose we devise a set of increasingly complex tests to allow us to evaluate whether this new methods is a general improvement.

Edit 07.03.2022: We've now changed the default inversion type to exact, which means that it is not necessary to set truncation value. If we want to use a full or estimated observation error covariance matrix, we will have to set a truncation value.