We investigated whether standardized neuropsychological tests and experimental cognitive paradigms measure the same cognitive faculties. Specifically, do neuropsychological tests commonly used to assess attention measure the same construct as attention paradigms used in cognitive psychology? We built on the “general attention factor”, comprising 9 widely used experimental paradigms, identified by Huang et al. (2012). Adult participants (N = 636) completed an on-line battery (TestMyBrain.org) of six experimental paradigms [Multiple Object Tracking (MOT), Flanker Interference, Visual Working Memory (VWM), Approximate Number Sense (ANS), L/T Visual Search Task, and Gradual Onset Continuous Performance Task (Grad CPT)] and six neuropsychological tests [Trail Making Test versions A & B (TMT), Digit Symbol Coding (DSC), Forward and Backward Digit Span, Letter Cancellation, Spatial Span, and Arithmetic]. Assessments were counterbalanced across participants and took roughly 90 minutes to compete. Five factors were identified: 1) attentional capacity (MOT, VWM, DSC, Spatial Span), 2) search (Visual Search, TMT, Letter Cancellation); 3) Digit Span; 4) Arithmetic; and 5) sustained attention (GradCPT). ANS and Flanker Interference did not load on to any factor. We conclude that Digit Span and Arithmetic tests should not be classified as attention tests. Digit Symbol Coding and Spatial Span tap attentional capacity, while TMT-A, TMT-B, and Letter Cancellation tap search (or attention-shifting) ability. These five tests can be classified as attention tests. We hope that these results provide inspiration for the development of new clinical assessments based on experimental attention paradigms.