Interpretations of task performance in cognitive studies rest on the assumption that participants are fully attentive to the tasks they agree to complete. However, with research studies being increasingly conducted online, these studies come at a cost of reduced experimental control. If participants were found to be engaging in off-task behaviours while taking part in these studies, the interpretation of study results would be called into question. To investigate this issue, we conducted a secondary data analysis across nearly 3,000 participants in various online studies to examine the prevalence of one form of off-task behaviour: media multitasking. Rates of media multitasking were found to be high, averaging 38% and ranging from 9% to 85% across studies. Our findings broadly raise questions about the interpretability of results from online studies and urge researchers to consider the likelihood that participants are simultaneously engaging in off-task behaviours while completing online research tasks.