Facial symmetry is purportedly attractive, though methods for measuring preference for facial symmetry vary between studies. Some studies have used a two-alternative forced choice (2AFC) task, while others have used a ratings task. How researchers manipulate facial symmetry also varies; some studies have used faces manipulated to be more (or perfectly) symmetrical, while others have used faces manipulated to be more asymmetrical. Here, across three studies, we evaluate and compare these different methods. In Studies 1 and 2 (N = 340 and 256 respectively), we compare facial symmetry preferences as measured by the 2AFC and ratings tasks. Across both studies, we consistently found a significant preference for facial symmetry when using the 2AFC task, but not with the ratings task. Also, correlations between facial symmetry preferences as measured by the two tasks were weak or showed no association. In Study 3, 159 participants rated the attractiveness of faces manipulated to be either symmetrical or more asymmetrical. The asymmetrical faces were rated as significantly less attractive compared to the original faces, while the difference in attractiveness ratings between the original and symmetrical versions was comparatively much smaller. These studies suggest that preference for facial symmetry depends greatly on the study design.