Loading wiki pages...

Wiki Version:
<p>Studying how psychological processes unfold over time has recently become highly popular in behavioral science. Intensive time series data are often collected by coding the presence/absence of behavior across time in observational studies. Having sufficiently high interrater agreement is then obviously quintessential. Assessing interrater agreement of behaviors that are rarely shown may nevertheless lead to puzzling results, with low or negative Cohen’s Kappa values but high proportions of agreement. This influence of relative frequency is well documented, but often ignored in practice. We will show that different kinds of serial dependency as they are typically present in time series further complicate inference, because they lead to broader sampling distributions of agreement measures. We compared the sensitivity of three measures - Cohen’s Kappa, Brennan-Prediger coefficient and Gwet’s AC - to different types of serial dependency as well as to relative frequency by running a simulation study. Results revealed that Cohen’s Kappa is very sensitive to relative frequency and that all measures are affected by serial dependencies. The Brennan-Prediger coefficient remains quite effective for binary time series.</p>
OSF does not support the use of Internet Explorer. For optimal performance, please switch to another browser.
This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.

Start managing your projects on the OSF today.

Free and easy to use, the Open Science Framework supports the entire research lifecycle: planning, execution, reporting, archiving, and discovery.