| Last Updated:
Creating DOI. Please wait...
Although identifying the idiosyncratic and shared contributions to stimulus evaluation is essential for many psychological domains, there is no established method for estimating these contributions. To examine different estimation methods (Study 1), participants rated the beauty of stimuli varying in within-person reliability and between-rater agreement: faces (high on both), objects (high on the former, low on the latter), and complex patterns (low on both). Variance component analyses (VCAs) better captured the psychometric properties of the data than other published metrics. We further examined how the VCAs estimates are affected by the number of repeated measurements and data preprocessing (Study 2). VCAs stabilized with more measures, consistent with the finding that both within-person reliability and between-rater agreement increased nonlinearly with repeated measures. Our findings suggest that researchers should collect more than two repeated measures to assess idiosyncratic versus shared contributions to stimulus evaluation. Mixed models with crossed random effects are currently the best way to render these estimates most reliable.
CC-By Attribution 4.0 International