Most measures of social judgment biases are [1] low in reliability because they use a single response, [2] not indicative of individual differences in bias because they use between subjects manipulations, [3] inflexible because they are designed for a particular domain, and [4] ambiguous about magnitude of bias because there is no objectively correct answer. We developed a measure of social judgment bias, the Judgment Bias Task, that is [1] reliable by using multiple responses, [2] indicative of individual differences by using within-subjects manipulations, [3] adaptable for assessing a variety of social judgments, [4] able to identify magnitude of bias compared to objectively correct answers, and [5] can be completed within six minutes. In nine pre-registered studies (Total *N* > 9,000) we establish the method and accumulate evidence for its effective use. A reliable, flexible, objective, efficient assessment of individual differences in social judgment biases will accelerate theoretical and empirical progress.