We investigate the efficacy of a measure for detecting careless responding from participants across 12 psychological effects. We find little evidence that participants who failed or passed the attention check responded differently to the included experiments. Results suggest further research is needed to validate the use of this attention check.