Studying how psychological processes unfold over time has recently become highly popular in behavioral science. Intensive time series data are often collected by coding the presence/absence of behavior across time in observational studies. Having sufficiently high interrater agreement is then obviously quintessential. Assessing interrater agreement of behaviors that are rarely shown may nevertheless lead to puzzling results, with low or negative Cohen’s Kappa values but high proportions of agreement. This influence of relative frequency is well documented, but often ignored in practice. We will show that different kinds of serial dependency as they are typically present in time series further complicate inference, because they lead to broader sampling distributions of agreement measures. We compared the sensitivity of three measures - Cohen’s Kappa, Brennan-Prediger coefficient and Gwet’s AC - to different types of serial dependency as well as to relative frequency by running a simulation study. Results revealed that Cohen’s Kappa is very sensitive to relative frequency and that all measures are affected by serial dependencies. The Brennan-Prediger coefficient remains quite effective for binary time series.