Significance testing (again)

[Mcstevens]

Hi everyone... Recently, folks asked about significance testing in a post:

#421

The answers were very helpful. But, they ultimately directed people to tutorial information to learn how to get specific factor's trace information out of the sampled model (e.g., to test for the difference between two conditions, etc.). For the life of me, I can't find examples of this in any of the tutorial pages I could find online, or as part of GitHub.

Could I ask someone to provide guidance here, perhaps with specific syntax and examples? If it helps, know that what I'm ultimately hoping to do is be able to redirect some text output from python to file so that I can script up many models and summarize their results. So I'm not just looking to see a posterior plot in image form... but rather, understand an example of how to extract P or other relevant significance testing information from HSSM that will be acceptable for publication.

Also, I've seen a lot of references to tutorial information in-the-works over the past year. I've actually be delaying work I originally did in HDDM because I think I'll have a stronger study using HSSM. But I don't seem to see the sort of step-by-step examples that accompanied the HDDM work. Is this still in development? Or maybe have I been looking in the wrong places?

Many thanks,
Mike

[AlexanderFengler]

Hi @Mcstevens ,

we are going to teach a few workshops in the coming weeks (one mid July / one end of July as a start), and pushing for expanded documentation and tutorials in that context.

Really sorry about the delay, against initial intentions, we had to prioritize a whole bunch of other things first.

There is definitely going to be a tutorial along the lines of your request.

Just to mention it again though:

In principle, all you really need for (what I understand are) your purposes, is access to the traces (bayesian t-test e.g.) and/or point-wise log-likelihoods (model comparison via az.compare().

You can access the InferenceData object through which e.g. Arviz functionality will become available via [myhssmmodel].traces. You can access the traces for particular parameters as numpy arrays from there.

Best,
Alex

[frankmj]

Just as a quick addendum in advance of the fuller tuitorials: * The plots I linked to before showed how to get HDI's from posteriors but of course you can also access the HDI itself (which is often a relevant statistic for significance, ie you are interested in whether the HDI contains 0 or some other reference value). This is reported using az.summary (where you can also get the posterior mean and std etc). * If you want to just compute the proportion of samples different from a reference value (like in the HDDM tutorials), you can do something like np.mean(samples_model_angle.posterior.z_Intercept>0.5) where samples.... are what you get from the inference object, and in this case we are looking at whether the overall starting point z is biased above 0.5. Similarly you can do the same thing by subtracting two posteriors and taking the mean of their difference > 0

…

On Mon, Jun 24, 2024 at 9:52 PM Alexander Fengler ***@***.***> wrote: Hi @Mcstevens <https://github.com/Mcstevens> , we are going to teach a few workshops in the coming weeks (one mid July / one end of July as a start), and pushing for expanded documentation and tutorials in that context. Really sorry about the delay, against initial intentions, we had to prioritize a whole bunch of other things first. Best, Alex — Reply to this email directly, view it on GitHub <#473 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG7TFAKGNFEZYH53W6OUKDZJDEOHAVCNFSM6AAAAABJ2UBB66VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TQNRWGA2TA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Mcstevens]

Thanks Alex and Michael! That's exactly what I was hoping to learn... It's great I can push forward with something in the short-term, as I'm giving a talk on some data in mid-July I wanted to re-work in HSSM... That's gonna be a bit before you'd be doing those training workshops. So, this guidance is both timely and much appreciated.

Quick follow-up question for anyone -- Is there a trick of syntax to execute that command when one codes categorical IVs in the model notation? For instance, I'm playing with some data with a within-subjects "drug" or "placebo" condition. I added a + C(drug) condition into the model notation... The model itself executes fine. But when I try to access the posterior, it throws an error, e.g.:

np.mean(MODEL1.posterior.a_C(drug)>0)
Traceback (most recent call last):
File "", line 1, in
File "/home/Mstevens/onrc/data/conda2024/envs/HSSM/lib/python3.11/site-packages/xarray/core/common.py", line 285, in getattr
raise AttributeError(
AttributeError: 'Dataset' object has no attribute 'a_C'

Seems like it can't parse the ( ) parentheses around the categorical term in the output information, and falsely thinks it's 'a_C' instead of 'a_C(drug)'. I'd guess this isn't a flaw in the coding, but rather might it just be a persnickety aspect of how to make python access information? So I thought I'd ask here if anyone knows the trick, because I'm sure others looking for this info on the listserv will frequently make use of the categorical coding options. And frankly, the cool part of what I'm trying to work up for this mid-July talk includes several between- and within- conditions (some with up to 3 levels)... So it'd be a kindness to be able to rely on that automatic C(factor) coding approach rather than work up the contrast-coding by hand, lol.

Oh... and FYI -- When I use "by hand" +0.5/-0.5 coding, that posterior test works just fine, e.g.:

np.mean(MODEL2.a_drug<0)
<xarray.DataArray 'a_drug' ()> Size: 8B
array(0.98)

So that syntax thing seems unlikely to be a problem with our Linux system or our HSSM installation. Again, making me think it's just command syntax finickiness?

Thanks again. I'm looking forward to any workshop offerings, as I've been really eager to use HSSM to examine the effects of treatments on DDM parameters and fMRI-measured brain function associated with them from data we've got waiting for this.

Mike