[Doubt] How to use HSSM for RT predictions on OOD samples?

[jainraj]

Describe the bug
I have a test dataset (i.e., not used for fitting) and would like to generate RT predictions (point estimates) for each sample. I thought to use model.sample_posterior_predictive(idata, test_dataset, kind='mean') (idata was output of fit() ) but it seems to generate a single value for both RT and choice - something like rt,response_mean - unsure what that is supposed to capture.

Should I use sample_posterior_predictive(idata, test_dataset, kind='pps') and then average the first column (which is RT) across draws and chains for each test sample?

HSSM version
0.2.0

To Reproduce
dataset_reg_v_hier is from the main tutorial

ddm_model = hssm.HSSM(
    data=dataset_reg_v_hier,
    model='ddm',
    p_outlier=0.05,
    include=[
        {
            "name": "v",
            "formula": "v ~ 1 + (x|subject)",
        },
    ],
)

idata = ddm_model.sample(chains=1, draws=10, tune=10)

Screenshots

The rt,response_mean is a single value per sample which doesn't seem right.

So, in the second output (call it pps_data), we would have to compute pps_data['posterior_predictive']['rt,response'][:, :, :, 0].mean(dim=['chain', 'draw'])

Is this understanding correct?

[AlexanderFengler]

Hi @jainraj ,

kind = "pps" is what you want here correct. The behavior of kind = "mean" is inherited from Bambi and embeds a bit of legacy (which is currently being worked on on the bambi side).

Essentially, kind = "mean" targets the mu parameters of a GLM model, which in our case is arbitrarily chosen to be v. So the outputs you are getting are values of v (trial wise).

Thank you for pointing towards this, I think we should essentially disable kind="mean" for our purposes. The PPC corresponding to kind="mean" in our case, would include all trial-wise model parameters (for the ddm model e.g. (v, a, z, t).

Best,
Alex

[jainraj]

Thank you @AlexanderFengler for the clear explanation.

Could you let me also know if the method to get a point prediction for RT is correct? i.e., pps_data['posterior_predictive']['rt,response'][:, :, :, 0].mean(dim=['chain', 'draw']) would essentially give me mean RT of the predictions?