Finding equivalence: a novel tool to investigate the effect size at which two groups are statistically equivalent

Abstract

In the learning sciences and other disciplines, researchers frequently face the challenge of deciding whether it makes sense from a statistical standpoint to combine the data set of two groups as different cohorts for subsequent analyses or not. Further, especially for studies conducted in educational settings and naturalistic environments, small effects and non-significant differences are common (Bakker et al., 2019). Therefore, it is necessary to be able to profoundly judge whether two groups are different or equivalent within a given set of threshold values and statistical assumptions. Whereas a non-significant difference is often interpreted as a reason to combine two data sets or to conclude that the groups of interest perform similarly, the absence of significant differences does not indicate equivalence of the groups (Edelsbrunner & Thurn, 2018). Meanwhile, research advocating equivalence testing has gained more attention over the last years (Lakens et al., 2018; Mehler et al., 2019). However, it remains the researcher’s decision to choose an appropriate effect size of equivalence and how to report this result. Here, I propose a new approach and ready-to-use online application to determine the precise effect size at which two groups of interest are equivalent. Based on the TOSTER package (Lakens, 2017), which aims to inform about whether two groups are statistically equivalent or not, this hereby described and novel tool accurately calculates the effect size at statistical equivalence. In other words, instead of a dichotomous judgment of equivalence based on a priori set default values including the effect size below which we would assume statistical equivalence (as by applying the TOSTER package), the presented tool does not require any effect size threshold values. Instead, it yields the exact effect size estimate at which the two groups of interest are equivalent, allowing the researcher to make a more informed decision about whether the groups should be combined for further analyses or not. Moreover, it enables the researcher to consistently and unbiasedly report their findings, also in relation to priorly reported studies as suggested by Evans and Yuan (2022), based on which they can ground their decision of combining groups or not or interpret their results. The online tool is available at https://samueltobler.shinyapps.io/findingequivalence.