Missing keys in the dict returned by integrator.read_file()

[facero]

• UltraNest version: 4.3.2
• Python version: 3.9.16
• Operating System: MacOs 14.4

Description

I'm currently testing the implementation of UltraNest into gammapy the science tools for the gamma-ray Cherenkov Telescope CTAO. First results look great !
While trying to run, write, read from disk on my test example, I stumbled on this issue :
the result dict read back from the hdf5 doesn't contain the exact same keys as the dict from the run() function, in particular the parameters names.

This crashes the plotting function like cornerplot(result).

What I Did

Defining the sampler :

sampler=ultranest.ReactiveNestedSampler(parameters.names, like.fcn, transform=parameters.prior_inverse_cdf,
                                                        log_dir=self.log_dir, resume=self.resume) 
result = self._sampler.run(min_num_live_points=self.min_num_live_points, frac_remain=self.frac_remain)

Run completes just fine.

>> print(result.keys() )
dict_keys(['niter', 'logz', 'logzerr', 'logz_bs', 'logz_single', 'logzerr_tail', 'logzerr_bs', 'ess', 'H', 'Herr', 'posterior', 'weighted_samples', 'samples', 'maximum_likelihood', 'ncall', 'paramnames', 'logzerr_single', 'insertion_order_MWW_test'])

paramnames key is present.

from ultranest import integrator
seq, result_disk = integrator.read_file(log_dir='mydir', x_dim=8)
result_disk.keys()

dict_keys(['niter', 'logz', 'logzerr', 'logz_bs', 'logz_single', 'logzerr_tail', 'logzerr_bs', 'ess', 'H', 'Herr', 'posterior', 'weighted_samples', 'samples', 'maximum_likelihood', 'insertion_order_MWW_test'])

Now the following keys are missing: ['ncall', 'paramnames', 'logzerr_single']

The paramnames being the most limiting for plotting results. Is this stored in the hdf5 file ?

Looking at the source code of read_file I'm not sure where this comes into play as the there are multiple calls to other functions.

[facero]

Docs

Also it seems that this read_file function is a bit hard to find in the doc as exemplified by #149 and #128 .
I found by searching the docs with the read keyword. But it seems it's not presented in a Content or Tutorial.

A good place to showcase this important function would be in Basic Usage just before the Further Topic :
https://johannesbuchner.github.io/UltraNest/using-ultranest.html#Further-topics

x_dim argument

Also I'm not sure why the x_dim argument is mandatory in read_file(). It is used for a column counting.
But if the list of paramnames is reconstructed then this argument would become not mandatory anymore.

I tried and I could specify any value to x_dim (even 0) and it doesn't crash the read_file() but gives expectedly not the good shape for posteriors for example.

[JohannesBuchner]

ultranest can resume with changed settings, including with different parameter names or number of live points. As long as the transform and likelihood function behave the same, it can resume.

paramnames is not stored in the hdf5 file, that file is quite minimalistic and as compact as possible.
read_file is somewhat limited.

x_dim is needed, because the hdf5 table contains n=k+m columns, made from k unit cube parameters transformed into m physical parameters (k can be not equal to m). It would be good to store it though, at https://github.com/JohannesBuchner/UltraNest/blob/master/ultranest/store.py#L191

I usually resume the run and visualise the results with the sampler object.

An alternative is to work with the posterior samples (equal_weighted_post.txt) and info/results.json.

[facero]

Ok I understand for the param names. That is easy to add by hand anyway.

For the x_dim it could be indeed worth to add it. If I understood correctly, one cannot resume with a different transform or different dimension so the x_dim is fixed.

What do you think about adding a read_file example to the docs at the end of the using_ultranest.html ?