loglike_kind='approx_differentiable' yields different estimates from 'analytical' for simple DDM

[seqasim]

I created two simple base models - the only difference between them is loglik_kind:

fully_pooled_analytic = hssm.HSSM(
    model="ddm",
    p_outlier = None,
    lapse = None, 
    data=df[['rt', 'response']] ,
    loglik_kind='analytical'
)
fully_pooled_approx = hssm.HSSM(
    model="ddm",
    p_outlier = None,
    lapse = None, 
    data=df[['rt', 'response']] ,
    loglik_kind='approx_differentiable'
)

I sampled both and compared the resulting models:

The fully_pooled_analytic model seems to perform much better. The posterior estimates differ as follows:

fully_pooled_analytic

fully_pooled_approx

Is this expected behavior for a vanilla DDM?

[AlexanderFengler]

Thank you for reporting this. Indeed this is somewhat odd.
The parameter estimates are quite different between the two likelihoods, this is usually not the case for the 'ddm'.

Quite likely the chains for the fully_pooled_approx model are sampling from a separate mode of the posterior (which is globally much less likely).

How big is the dataset that is used here?

Assuming that for the pooled model, sampling doesn't take too long, could you try this using a few bootstrapped random samples from your data (or simply a few subsets)?

I will check if I can reproduce this (or an equivalent) issue with synthetic data.

[seqasim]

Thank you for looking into this!

1. The dataset is 10312 responses (n=173 subjects). So ~ 60 responses / subject
2. I tried sub-sampling 10 subjects at a time for a couple of iterations.

The analytic model still looks pretty consistent:

However, this is the approximate model:
Boostrap 1:

Bootstrap 2:

I performed this sub-sampling a few times with distinct participants and I find the same result. The analytic model yields unimodal posteriors that are fairly stable, while the approximate model yields multimodal posteriors that vary across bootstrapped samples.

[AlexanderFengler]

Hi @seqasim,

this is indeed interesting. I ran a couple of tests on simulated data, and parameter recovery ended up looking very similar across both the analytical and approx_diff likelihoods, however I didn't capture your case appropriately because I kept the parameters consistent across datapoints.

To capture your case, I should simulate from a variety of parameters, 60 samples each and then attempt to fit the model without hierarchy.

I will run these tests as well an report back in more detail, however in the meanwhile so thoughts on what could be the culprit:

1. (not very likely but useful to rule out) This could be a problem with the priors. If you look at the priors for the approx vs. the analytical likelihoods, you will see that the approx version uses uniform priors throughout. You could instead specify equivalent uniform priors when you specify the analytical model and see if that has an impact here.

2. (more likely, afaic) This case exposes an unwanted side effect concerning a slight difference in how non-decision-times are handled across the analytical and approx case. Violating the constraint: rt - non-decision-time > 0 get's punished much less in the approx case and this creates some spurious modes in the posterior. I strongly suspect that this is what is driving problems here. Corroborating this is the much larger estimated t parameter in the approx case.

3. (not separated from 2. completely, but something that could sidestep the issue) You can try to actively set the starting point ('initval' argument when specifying the prior) of the 't' parameter to very close to 0. This can help because the posterior likely will never reach the high values of t it gets stuck at due to the problem explained under point 2.. These are very unlikely modes after all, so the chain should indeed essentially never make it there in the normal case.

One way to test this hypothesis (2): Simulate some data from the estimated parameters according to the approx model, and observe that your real data has quite a few datapoints that precede anything that can be feasible generated from the parameters according to the model fit.

Now there are three comments:

First, I will look into how we can improve alignment in how the approx and the analytical likelihoods behave when rt - non-decision-time < 0. I never ran into issues as you expose, but I now realize that this is likely because of how I tested the likelihood.

Second, your test case strongly suggests that a hierarchical mode should be the appropriate choice here, since there does seem to be a relatively strong variation in generative parameters across participants. The more these parameters would align, the less issue concerning point 2. above you should have. Notwithstanding the fact that indeed there is an issue here.