*
Background: Along with Open Science practices, the replication crisis
has been argued to be a symptom of an underlying measurement crisis or
theory crisis. Validity is a central concept in these reflections.
Within validity theory, construct validity has become a dominant
approach since its proposition in the 1950s. It has also been
persuasively argued to be fundamentally flawed, and its tools of the
trade (e.g., factor analysis, internal consistency estimates, and
convergent and divergent validity) have been shown to lack diagnostic
value as to the validity of a measurement instrument for a given construct.
Methods:Narrative response models are tools to reason about and study
the validity of psychological measurement instruments. Unlike the
construct validity approach, they allow studying howmeasurement
instruments work. They do so by providing a procedure for devising
probes for cognitive interviews or meta questions for response process
evaluation.
Findings:The procedure will be illustrated at the hand of an
investigation of the Questionnaire of Unpredictability in Childhood. As
results, the narrative response models, used probes, and results will be
presented. These results illustrate the item-level validity of each item.
Discussion:Describing narrative response models for an existing
measurement instrument is a challenging task. However, once developed,
they provide a concrete foothold for interrogating a measurement
instrument’s validity. By examining heterogeneity in participants’
response processes at the hand of the steps in a narrative response
model, cognitive interviews or response process evaluation can directly
provide evidence as to a measurement instrument’s (lack of) validity.*
