From 4a606423c55577c5db2eb370cc3a56868a610d31 Mon Sep 17 00:00:00 2001 From: Daniel Date: Tue, 3 Sep 2024 19:09:34 +0200 Subject: [PATCH] docs --- R/equivalence_test.R | 38 ++++++++++++++++++++++++-- R/p_significance.R | 55 ++++++++++++++++++++++++++++++-------- man/equivalence_test.lm.Rd | 36 +++++++++++++++++++++++-- man/p_significance.lm.Rd | 53 ++++++++++++++++++++++++++++-------- 4 files changed, 156 insertions(+), 26 deletions(-) diff --git a/R/equivalence_test.R b/R/equivalence_test.R index bf7f22547..3f97705ce 100644 --- a/R/equivalence_test.R +++ b/R/equivalence_test.R @@ -104,8 +104,42 @@ bayestestR::equivalence_test #' Second generation p-values (SGPV) were proposed as a statistic that #' represents _the proportion of data-supported hypotheses that are also null #' hypotheses_ _(Blume et al. 2018, Lakens and Delacre 2020)_. It represents the -#' proportion of the confidence interval range (assuming a normally distributed, -#' equal-tailed interval) that is inside the ROPE. +#' proportion of the _full_ confidence interval range (assuming a normally +#' distributed, equal-tailed interval) that is inside the ROPE. +#' +#' Note that the assumed interval, which is used to calculate the SGPV, is an +#' _approximation_ of the _full interval_ based on the chosen confidence level. +#' For example, if the 95% confidence interval of a coefficient ranges from -1 +#' to 1, the underlying _full (normally distributed) interval_ approximately +#' ranges from -1.9 to 1.9, see also following code: +#' +#' ``` +#' # simulate full normal distribution +#' out <- bayestestR::distribution_normal(10000, 0, 0.5) +#' # range of "full" distribution +#' range(out) +#' # range of 95% CI +#' round(quantile(out, probs = c(0.025, 0.975)), 2) +#' ``` +#' +#' This ensures that the SGPV always refers to the general compatible parameter +#' space of coefficients, independent from the confidence interval chosen for +#' testing practical equivalence. Therefore, the SGPV of the _full interval_ is +#' similar to the ROPE coverage of Bayesian equivalence tests, see following +#' code: +#' +#' ``` +#' library(bayestestR) +#' library(brms) +#' m <- lm(mpg ~ gear + wt + cyl + hp, data = mtcars) +#' m2 <- brm(mpg ~ gear + wt + cyl + hp, data = mtcars) +#' # SGPV for frequentist models +#' equivalence_test(m) +#' # similar to ROPE coverage of Bayesian models +#' equivalence_test(m2) +#' # similar to ROPE coverage of simulated draws / bootstrap samples +#' equivalence_test(simulate_model(m)) +#' ``` #' #' ## ROPE range #' Some attention is required for finding suitable values for the ROPE limits diff --git a/R/p_significance.R b/R/p_significance.R index 263d980e3..a178b1567 100644 --- a/R/p_significance.R +++ b/R/p_significance.R @@ -6,14 +6,14 @@ bayestestR::p_significance #' @title Practical Significance (ps) #' -#' @description Compute the probability of **Practical Significance** -#' (*ps*), which can be conceptualized as a unidirectional equivalence test. -#' It returns the probability that an effect is above a given threshold -#' corresponding to a negligible effect in the median's direction, considering -#' a parameter's confidence interval. In comparison the the [`equivalence_test()`] -#' function, where the *SGPV* (second generation p-value) describes the proportion -#' of the confidence interval that is _inside_ the ROPE, the value returned by -#' `p_significance()` describes the _larger_ proportion of the confidence +#' @description Compute the probability of **Practical Significance** (*ps*), +#' which can be conceptualized as a unidirectional equivalence test. It returns +#' the probability that an effect is above a given threshold corresponding to a +#' negligible effect in the median's direction, considering a parameter's _full_ +#' confidence interval. In comparison the the [`equivalence_test()`] function, +#' where the *SGPV* (second generation p-value) describes the proportion of the +#' confidence interval that is _inside_ the ROPE, the value returned by +#' `p_significance()` describes the _larger_ proportion of the _full_ confidence #' interval that is _outside_ the ROPE. This makes `p_significance()` comparable #' to [`bayestestR::p_direction()`], however, while `p_direction()` compares to #' a point-null by default, `p_significance()` compares to a range-null. @@ -27,9 +27,9 @@ bayestestR::p_significance #' @seealso For more details, see [`bayestestR::p_significance()`]. See also #' [`equivalence_test()`]. #' -#' @details `p_significance()` returns the proportion of the confidence interval -#' range (assuming a normally distributed, equal-tailed interval) that is -#' outside a certain range (the negligible effect, or ROPE, see argument +#' @details `p_significance()` returns the proportion of the _full_ confidence +#' interval range (assuming a normally distributed, equal-tailed interval) that +#' is outside a certain range (the negligible effect, or ROPE, see argument #' `threshold`). If there are values of the distribution both below and above #' the ROPE, `p_significance()` returns the higher probability of a value being #' outside the ROPE. Typically, this value should be larger than 0.5 to indicate @@ -38,6 +38,39 @@ bayestestR::p_significance #' `p_significance()` will be less than 0.5, which indicates no clear practical #' significance. #' +#' Note that the assumed interval, which is used to calculate the practical +#' significance, is an _approximation_ of the _full interval_ based on the +#' chosen confidence level. For example, if the 95% confidence interval of a +#' coefficient ranges from -1 to 1, the underlying _full (normally distributed) +#' interval_ approximately ranges from -1.9 to 1.9, see also following code: +#' +#' ``` +#' # simulate full normal distribution +#' out <- bayestestR::distribution_normal(10000, 0, 0.5) +#' # range of "full" distribution +#' range(out) +#' # range of 95% CI +#' round(quantile(out, probs = c(0.025, 0.975)), 2) +#' ``` +#' +#' This ensures that the practical significance always refers to the general +#' compatible parameter space of coefficients. Therefore, the _full interval_ is +#' similar to a Bayesian posterior distribution of an equivalent Bayesian model, +#' see following code: +#' +#' ``` +#' library(bayestestR) +#' library(brms) +#' m <- lm(mpg ~ gear + wt + cyl + hp, data = mtcars) +#' m2 <- brm(mpg ~ gear + wt + cyl + hp, data = mtcars) +#' # probability of significance (ps) for frequentist model +#' p_significance(m, threshold = 0.6) # ROPE for mpg as outcome +#' # similar to ps of Bayesian models +#' p_significance(m2) +#' # similar to ps of simulated draws / bootstrap samples +#' p_significance(simulate_model(m)) +#' ``` +#' #' @note There is also a [`plot()`-method](https://easystats.github.io/see/articles/bayestestR.html) #' implemented in the [**see**-package](https://easystats.github.io/see/). #' diff --git a/man/equivalence_test.lm.Rd b/man/equivalence_test.lm.Rd index 86a303406..e09f96408 100644 --- a/man/equivalence_test.lm.Rd +++ b/man/equivalence_test.lm.Rd @@ -149,8 +149,40 @@ practical equivalence and accept the alternative hypothesis. Second generation p-values (SGPV) were proposed as a statistic that represents \emph{the proportion of data-supported hypotheses that are also null hypotheses} \emph{(Blume et al. 2018, Lakens and Delacre 2020)}. It represents the -proportion of the confidence interval range (assuming a normally distributed, -equal-tailed interval) that is inside the ROPE. +proportion of the \emph{full} confidence interval range (assuming a normally +distributed, equal-tailed interval) that is inside the ROPE. + +Note that the assumed interval, which is used to calculate the SGPV, is an +\emph{approximation} of the \emph{full interval} based on the chosen confidence level. +For example, if the 95\% confidence interval of a coefficient ranges from -1 +to 1, the underlying \emph{full (normally distributed) interval} approximately +ranges from -1.9 to 1.9, see also following code: + +\if{html}{\out{
}}\preformatted{# simulate full normal distribution +out <- bayestestR::distribution_normal(10000, 0, 0.5) +# range of "full" distribution +range(out) +# range of 95\% CI +round(quantile(out, probs = c(0.025, 0.975)), 2) +}\if{html}{\out{
}} + +This ensures that the SGPV always refers to the general compatible parameter +space of coefficients, independent from the confidence interval chosen for +testing practical equivalence. Therefore, the SGPV of the \emph{full interval} is +similar to the ROPE coverage of Bayesian equivalence tests, see following +code: + +\if{html}{\out{
}}\preformatted{library(bayestestR) +library(brms) +m <- lm(mpg ~ gear + wt + cyl + hp, data = mtcars) +m2 <- brm(mpg ~ gear + wt + cyl + hp, data = mtcars) +# SGPV for frequentist models +equivalence_test(m) +# similar to ROPE coverage of Bayesian models +equivalence_test(m2) +# similar to ROPE coverage of simulated draws / bootstrap samples +equivalence_test(simulate_model(m)) +}\if{html}{\out{
}} } \subsection{ROPE range}{ diff --git a/man/p_significance.lm.Rd b/man/p_significance.lm.Rd index 5d8426a6c..ca99ae9a1 100644 --- a/man/p_significance.lm.Rd +++ b/man/p_significance.lm.Rd @@ -29,22 +29,22 @@ and based on \code{\link[bayestestR:rope_range]{rope_range()}} if a Bayesian mod A data frame. } \description{ -Compute the probability of \strong{Practical Significance} -(\emph{ps}), which can be conceptualized as a unidirectional equivalence test. -It returns the probability that an effect is above a given threshold -corresponding to a negligible effect in the median's direction, considering -a parameter's confidence interval. In comparison the the \code{\link[=equivalence_test]{equivalence_test()}} -function, where the \emph{SGPV} (second generation p-value) describes the proportion -of the confidence interval that is \emph{inside} the ROPE, the value returned by -\code{p_significance()} describes the \emph{larger} proportion of the confidence +Compute the probability of \strong{Practical Significance} (\emph{ps}), +which can be conceptualized as a unidirectional equivalence test. It returns +the probability that an effect is above a given threshold corresponding to a +negligible effect in the median's direction, considering a parameter's \emph{full} +confidence interval. In comparison the the \code{\link[=equivalence_test]{equivalence_test()}} function, +where the \emph{SGPV} (second generation p-value) describes the proportion of the +confidence interval that is \emph{inside} the ROPE, the value returned by +\code{p_significance()} describes the \emph{larger} proportion of the \emph{full} confidence interval that is \emph{outside} the ROPE. This makes \code{p_significance()} comparable to \code{\link[bayestestR:p_direction]{bayestestR::p_direction()}}, however, while \code{p_direction()} compares to a point-null by default, \code{p_significance()} compares to a range-null. } \details{ -\code{p_significance()} returns the proportion of the confidence interval -range (assuming a normally distributed, equal-tailed interval) that is -outside a certain range (the negligible effect, or ROPE, see argument +\code{p_significance()} returns the proportion of the \emph{full} confidence +interval range (assuming a normally distributed, equal-tailed interval) that +is outside a certain range (the negligible effect, or ROPE, see argument \code{threshold}). If there are values of the distribution both below and above the ROPE, \code{p_significance()} returns the higher probability of a value being outside the ROPE. Typically, this value should be larger than 0.5 to indicate @@ -52,6 +52,37 @@ practical significance. However, if the range of the negligible effect is rather large compared to the range of the confidence interval, \code{p_significance()} will be less than 0.5, which indicates no clear practical significance. + +Note that the assumed interval, which is used to calculate the practical +significance, is an \emph{approximation} of the \emph{full interval} based on the +chosen confidence level. For example, if the 95\% confidence interval of a +coefficient ranges from -1 to 1, the underlying \emph{full (normally distributed) +interval} approximately ranges from -1.9 to 1.9, see also following code: + +\if{html}{\out{
}}\preformatted{# simulate full normal distribution +out <- bayestestR::distribution_normal(10000, 0, 0.5) +# range of "full" distribution +range(out) +# range of 95\% CI +round(quantile(out, probs = c(0.025, 0.975)), 2) +}\if{html}{\out{
}} + +This ensures that the practical significance always refers to the general +compatible parameter space of coefficients. Therefore, the \emph{full interval} is +similar to a Bayesian posterior distribution of an equivalent Bayesian model, +see following code: + +\if{html}{\out{
}}\preformatted{library(bayestestR) +library(brms) +m <- lm(mpg ~ gear + wt + cyl + hp, data = mtcars) +m2 <- brm(mpg ~ gear + wt + cyl + hp, data = mtcars) +# probability of significance (ps) for frequentist model +p_significance(m, threshold = 0.6) # ROPE for mpg as outcome +# similar to ps of Bayesian models +p_significance(m2) +# similar to ps of simulated draws / bootstrap samples +p_significance(simulate_model(m)) +}\if{html}{\out{
}} } \note{ There is also a \href{https://easystats.github.io/see/articles/bayestestR.html}{\code{plot()}-method}