To perceive reflectances of surfaces, the visual system transforms the physical variable luminance into the perceptual variables of brightness (perceived luminance) or lightness (perceived reflectance). While lightness perception as a whole is barely understood various image-computable models aim to predict brightness differences seen by human observers in a range of classic phenomena. These take stimulus images as input and return the result of model computations as output image. To evaluate model performance this output then needs to be compared to psychophysical measurements of brightness perception. However, there is no principled way of mapping 2D model output to psychophysical brightness judgments (often scalars). Lacking such a linking function, models are said to “predict” brightness effects already when pixel differences in the output are of the same sign as the average brightness judgments across observers. Even for single trials, sign comparisons ca n only s how whether model output is monotonically related to psychophysical measurements; to characterize the shape of the linking function magnitudes of brightness effects should be considered. We present specific cases where quantitative predictions by models to parametric variations of stimuli, are not simply linked to brightness matches for those stimuli. For example, brightness phenomena often “work best” when both targets are at a luminance approximately halfway between the lowest and highest luminances of the stimulus. Psychophysical magnitude can change substantially when deviating from these intermediate luminance values, which models fail to capture. This suggests that the complex nature of human brightness perception requires explicitly including linking functions in computational models. [This work has been supported by research grants of the German Research Foundation DFG MA5127/3-1 and MA5127/4-1]