Date created: | Last Updated:

Category: Project

Description: The relationship between measurement reliability and statistical power is a complex one. Where reliability is defined by classical test theory as the proportion of ‘true’ variance to total variance (the sum of true score and error variance), power is only functionally related to total variance. Therefore, to explore direct relationships between reliability and power, one must fix either true-score variance or error variance while varying the other. Here, visualisations are used to illustrate the reliability-power relationship under conditions of fixed true-score variance and fixed error variance. From these visualisations, conceptual distinctions between fixing true-score or error variance can be raised. Namely, that when true-score variance is fixed, low reliability (and low power) represents error hiding true effects. Whereas, when error variance is fixed, high reliability (and low power) represents a very small effect. I raise three observations I hope will be useful in considering the utility of measurement reliability in experimental research.