Sampler hanging above some hazy number of samples

[leerosenthalj]

I am running the sampler using MultiPool(), and keeping an eye on my activity monitor. I see eight python processes run for a few minutes, and then they stop -- but the notebook cell in which I am running the process does not complete. Is it possible that the joker.rejection_sample() code is stalling somewhere at the end of the process, after the actual sampling has been completed?

EDIT: code snippet share via email

with pm.Model():
    dv0_1 = xu.with_unit(pm.Normal('dv0_1', 0, 5.), u.m/u.s)
    s = xu.with_unit(pm.Lognormal('s', 0, 0.5), u.m/u.s)
    baseline = np.amax(time_j) - np.amin(time_k)
    params = tj.JokerPrior.default(P_min=baseline, P_max=8*baseline, s=s,
                                   v0_offsets=[dv0_1], sigma_K0=1*u.km/u.s, sigma_v = 10*u.km/u.s)

%%time
prior_samples = params.sample(2*10**8)
with schwimmbad.MultiPool() as pool:
    joker = tj.TheJoker(params, pool=pool)
    samples = joker.rejection_sample([rvdata_j, rvdata_k], prior_samples)

[leerosenthalj]

Note: sampling using a SerialPool process does not reproduce this issue.

[adrn]

If you use conda, could you dump your environment info with: conda list > conda_env.txt and send that to me? Also, it'd be handy if you could send me a small snippet of code that reproduces the behavior you are seeing. Thanks!

[adrn]

I think there are a few things going on here.

(1) I fixed some minor speed issues in prior.sample(), so please try updating with: pip install git+https://github.com/adrn/thejoker

(2) Generating that many prior samples is always going to be somewhat slow, so I would do it once and store the samples in a file on disk. I would use something like this:

import os
prior_cache_file = 'prior_samples.hdf5'
if not os.path.exists(prior_cache_file):
    prior_samples = prior.sample(2*10**8)
    prior_samples.write(prior_cache_file)

with schwimmbad.MultiPool() as pool:
    joker = tj.TheJoker(params, pool=pool)
    samples = joker.rejection_sample([rvdata_j, rvdata_k], prior_cache_file)

(3) When you pass in a JokerSamples object to rejection_sample(), the first thing it does (by default) is write it out to a temporary file. In this case, that is a ~7 GB file, so that is probably slow. This will be fixed if you use the code snippet above.

[leerosenthalj]

Hmm, now I'm seeing that theano lock issue again. Also, the bottleneck didn't seem to be in the prior sampling; that always happened in a reasonable amount of time.

…

On Tue, Nov 26, 2019 at 11:59 AM Adrian Price-Whelan < ***@***.***> wrote: I think there are a few things going on here. (1) I fixed some minor speed issues in prior.sample(), so please try updating with: pip install git+https://github.com/adrn/thejoker (2) Generating that many prior samples is always going to be somewhat slow, so I would do it once and store the samples in a file on disk. I would use something like this: import os prior_cache_file = 'prior_samples.hdf5'if not os.path.exists(prior_cache_file): prior_samples = prior.sample(2*10**8) prior_samples.write(prior_cache_file) with schwimmbad.MultiPool() as pool: joker = tj.TheJoker(params, pool=pool) samples = joker.rejection_sample([rvdata_j, rvdata_k], prior_cache_file) (3) When you pass in a JokerSamples object to rejection_sample(), the first thing it does (by default) is write it out to a temporary file. In this case, that is a ~7 GB file, so that is probably slow. This will be fixed if you use the code snippet above. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#97?email_source=notifications&email_token=ABYLHCDJLVUMCFK3YGYNKBTQVVIXVA5CNFSM4JQU6RFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFGXFVA#issuecomment-558723796>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYLHCGYDHDDQ76PC6NB3UTQVVIXVANCNFSM4JQU6RFA> .

[leerosenthalj]

I am still seeing the theano lock error message when attempting to run with MultiPool(). I am using a dataset with ~60 RVs; could that be part of the issue?

[adrn]

@leerosenthalj Coming back to this after a long time, I'm realizing that this is likely an issue with running from a Jupyter notebook. If you switch to a script, and set the compiledir via $THEANO_FLAGS for each process (see #105), it should work.

@AstroSong I made a new issue to discuss your question #106