Viewers use contextual information to visually explore complex scenes. Object recognition is facilitated by exploiting object-scene relations (which objects are expected in a given scene) and object-object relations (which objects are expected based on the occurrence of other objects). Semantically inconsistent objects deviate from these expectations, and thus they tend to capture viewers’ attention (i.e., semantic inconsistency effect). Some objects fit the identity of a scene more or less than others, yet semantic inconsistencies have hitherto been operationalised as binary (consistent vs. inconsistent). In an eye-tracking experiment (N = 21 adults), we study the semantic inconsistency effect in a continuous manner by using the linguistic similarity of an object to the scene category and to other objects in the scene. We find that both highly consistent and highly inconsistent objects are viewed more than other objects (U-shaped relationship), revealing that the (in)consistency effect is more than a simple binary classification.