Embodied cognition theories propose that abstract concepts are grounded in a variety of exogenous and endogenous experiences which may be flexibly activated across contexts and tasks. In three experiments, we explored how semantic size (i.e., the magnitude, dimension or extent of an object or a concept) of abstract (vs. concrete) concepts is mentally represented. We show that abstract size is metaphorically associated with the physical size of concrete objects (Experiment 1) and can produce a semantic-font size congruency effect comparable to that demonstrated in concrete words during online lexical processing (Experiment 2). Critically, this size congruency effect is large when a word is judged by its physical (font) size but significantly smaller when it is judged by its emotionality (Experiment 3), regardless of concreteness. Our results suggest that semantic size of abstract concepts can be grounded in visual size, which is activated adaptively under different task demands. The present findings advocate flexible embodiment of semantic representations, with an emphasis on the role of task effects on conceptual processing.