The [paper can be viewed here][1]. The figures in the article appear in low resolution. You can find high-resolution images in the ["Files" tab][2]. # Summary of the findings We did not build a privacy-invading tool. We studied existing facial recognition technologies, already widely used by companies and governments, to see whether they can detect political orientation from expressionless faces. Governments and companies have been developing such capabilities for quite some time (e.g., see [patent1][3], [patent2][4], [patent3][5], or this [startup][6]). We took carefully standardized facial images of 591 participants while controlling for self-presentation, facial expression, head orientation, and image properties. Participants removed all jewelry, make-up, and shaved facial hair. Their hair was pulled back using hair ties, hair pins, and a headband while taking care to avoid flyaway hairs. The images were cropped in a tight rectangle from the top of the forehead (the start of the hairline) to the bottom of the chin. The resulting facial image looked like this: ![enter image description here][7] Next, we asked the participants to fill-in a political orientation questionnaire to estimate their political orientation. Next, we showed their facial images to human raters and a facial recognition algorithm, making sure their predictions are not based on age, gender, and ethnicity. The accuracy achieved by humans (r = .21) and the algorithm (r = .22) was comparable. It was relatively high. In fact, it was on par with how well job interviews predict job success, or alcohol drives aggressiveness. The algorithm’s predictive accuracy was even higher (r = .31) when it leveraged information on participants’ age, gender, and ethnicity. Moreover, the associations between facial appearance and political orientation seem to generalize beyond our sample: The predictive model derived from standardized images (while controlling for age, gender, and ethnicity) could predict political orientation (r ≈ .13) from naturalistic images of 3,401 politicians from the United States, the United Kingdom, and Canada. The analysis of facial features associated with political orientation revealed that conservatives tended to have larger lower faces. # Privacy implications Widespread use of facial recognition technology poses dramatic risks to privacy and civil liberties. Ubiquitous CCTV cameras and giant databases of facial images, ranging from public social network profiles to national ID card registers, make it alarmingly easy to identify individuals, as well as track their location and social interactions. Pervasive surveillance is not the only risk brought about by facial recognition. Apart from identifying individuals, facial recognition algorithms can identify their traits. Like humans, facial recognition algorithms can accurately infer gender, age, ethnicity, or emotions. Unfortunately, the list of personal attributes that can be inferred from the face extends well beyond those few obvious examples. Past work has shown that widely used facial recognition technologies can expose people’s [sexual orientation][8] and [political orientation][9] from social media images. This work shows that the latter can be also predicted from carefully standardized passport-like facial images. While [many other digital footprints are revealing of political orientation and other intimate traits][10], facial recognition can be used without subjects’ consent or knowledge. Facial images can be easily (and covertly) taken by law enforcement or obtained from digital or traditional archives, including social networks, dating platforms, photo-sharing websites, and government databases. They are often easily accessible; Facebook and LinkedIn profile pictures, for instance, can be accessed by anyone without a person’s consent or knowledge. Thus, the privacy threats posed by facial recognition technology are, in many ways, unprecedented. Some may doubt whether the accuracies reported here are high enough to cause concern. Yet, our estimates are unlikely to constitute an upper limit of what is possible. Progress in computer vision and AI is unlikely to slow down anytime soon. Moreover, [even modestly accurate predictions can have tremendous impact when applied to large populations in high-stakes contexts, such as elections][11]. For example, even a crude estimate of an audience’s psychological traits can drastically boost the efficiency of propaganda and mass persuasion. We hope that scholars, policymakers, engineers, and citizens will take notice. # How can we prevent the risks of facial recognition technology? Unfortunately, even the best privacy-protecting laws and technologies are unlikely to guarantee privacy at all times and for everybody. One can regulate Facebook, Google, and U.S. intelligence agencies, but one cannot expect that hackers, startups with little to lose, or foreign intelligence services will also follow the rules. The digital environment is very difficult to police; data can be easily moved across borders, stolen, or recorded without users’ consent. Also, most people want some of their social media posts, blogs, or profiles to be public. Few would be willing to cover their faces while interacting with others. As this and other studies show, this is enough to invade their privacy. Consequently, our safety hinges not on our right to privacy (which can be maliciously invaded), but on the protection of our human rights and tolerance of societies and governments. In order for the post-privacy world to be safer and hospitable, it must be inhabited by well-educated people who are radically intolerant of intolerance and obey laws assertively protecting the rights of people to do and think as they please (as long as they are not hurting others). # Why should there be links between political orientation and facial appearance? Scientific literature documents numerous mechanisms that could account for such links. Those mechanisms can be grouped into three causal pathways: ![enter image description here][12] Mind→face pathway (aka the Dorian Gray effect) includes mechanisms through which psychological traits may influence facial features. While we tend to think of facial features as relatively fixed, there are many factors that influence them in both the short and long term: emotional states, facial care, diet, substance use, physical health, injuries, exposure to sunlight and harsh environmental conditions, etc. Such face-altering factors are likely moderated by political orientation. Liberals, for example, tend to smile more intensely and genuinely, which leads to the emergence of different expressional wrinkle patterns. Conservatives tend to be healthier, consume less alcohol and tobacco, and have a different diet—which, over time, translates into differences in skin health and the distribution and amount of facial fat. Face→mind pathway includes mechanisms through which facial features may influence political orientation. Facial appearance predicts life outcomes such as the length of the prison sentence, occupational success, educational attainments, chances of winning an election, and income. Those outcomes, in turn, likely influence political orientation. For instance, negative first impressions could—over a person’s lifetime—reduce their earning potential and status, and thus increase their support for wealth redistribution and sensitivity to social injustice, shifting them toward the liberal end of the political spectrum. Face←factor→mind pathway embraces factors that influence both: facial appearance and political orientation. Those factors include—among others—genes, hormones, or prenatal exposure to substances. For example, twin studies show that genes are responsible for over 50% of variation in both facial features and political orientation. Similarly, prenatal and postnatal testosterone levels affect facial shape and correlate with political orientation. Furthermore, prenatal exposure to nicotine and alcohol affects facial morphology and cognitive development (which has been linked to political orientation). While some of these (and similar) mechanisms may be misconceived or have a negligible effect, given their number and diversity, it would be remarkable—or even extraordinary—if none of them have any merit. # FAQ #### To what extent was the classification enabled by demographics? Naturally, one can predict political orientation from demographics (e.g., women tend to be more liberal). Thus, we controlled for age, gender, and ethnicity. (Allowing the algorithm to benefit from these variables, further increased the accuracy, to r =.31 #### What facial features enabled the classification? Facial recognition algorithms are not directly interpretable. In other words, it is not immediately clear what facial features have enabled the classification. Thus, we analysed the correlation between a number of facial features and political orientation. The analysis revealed that conservatives tended to have larger lower faces. #### There are surely some facial features, like mustache or long hair in males, which make it very easy to guess political orientation. Those traits are (weakly) correlated with political orientation, as our [previous research][13] shows. Yet, in this study, we used carefully standardized facial images taken in the laboratory while controlling for self-presentation, facial expression, head orientation, and image properties. Participants removed all jewelry and—if necessary—shaved facial hair. Face wipes were used to remove cosmetics until no residues were detected on a fresh wipe. Their hair was pulled back using hair ties, hair pins, and a headband while taking care to avoid flyaway hairs. We then cropped the images in a tight rectangle from the top of the forehead (the start of the hairline) to the bottom of the chin, with the ears included. #### Facial images used in this study were taken in the lab. Could similar predictions be made using social media profile pictures? Yes. As shown in our [previous research][14] political orientation can be also predicted from profile images. #### What was the classification accuracy? The political orientation variable was not binary (e.g., it was not liberal vs conservative) but linear: we computed a 'conservatism' score for each particpant. Since there are no "classes" there is no classification accuracy. Instead we report correlation between the actual political orientation and the predicted political orientation. It equals r=.2, which is comparable with the predictive power of job interviews when predicting job success, or the predictive power of alcohol intoxication when predicting aggressive behaviors. #### Correlation of r=.2 seems very small, doesn't it? It's not small at all. In fact, it is about an average effect size in psychological science. Richard and colleagues’ (2003) review of more than 25,000 psychological studies encompassing 8 million participants found an average effect size of r = .21, a finding mirrored by Fraley and Marks (2007). More recently, Gignac and Szodorai (2016) observed an even lower average effect at r = .19. Many consequential effects are much lower than r=.2. For example, the scarcity principle (the cognitive bias boosting the perceived value of scarce resources) manifests at r = .12. The self-serving bias (tendency to blame external factors rather than oneself for failures) comes in at r = .10. The pain-relieving effect of Ibuprofen is approximately r = .14. The use of antihistamines to prevent the effects of allergies, such as runny nose or sneezing, is around r = .11. The association between smoking and the risk of lung cancer is r = .08 and it remains one of the leading preventable causes of deaths globally, killing over 8 million people per year. Taking Aspirin reduces the chances of heart attack with an effect size of r = .03. In other words, r=.2 is quite an effect. #### Can you apply the software you developed to my face? We did not develop any software - we just showed that widely available facial recognition algorithms inherently differentiate between these two groups. Moreover, applying such predictions is precisely what we are warning against and discouraging in our work. There are better ways to learn more about oneself (and a few better ways to invade someone's privacy!). Sadly, there are startups out there that have been doing precisely that for quite some time. #### Are you physiognomists? Don’t dismiss those results (and thus the associated risks) simply because they remind you of the claims made by physiognomists. This makes no sense. Without a doubt, physiognomy was based on unscientific studies, superstition, anecdotal evidence, and racist pseudo-theories. The fact that its claims were unsupported, however, does not automatically mean that they are all wrong: Sadly and unfortunately, some of such claims may have been correct, perhaps by a mere accident. Scholars, policymakers, and citizens should take notice. #### Could the accuracy be even higher? Our work is intended as a warning that predictions of this kind can be made with worrying accuracy, rather than an attempt to estimate the maximum accuracy of such predictions. We used basic tools and a relatively small sample of images. Those deploying such methods in practice are using much more sensitive algorithms and may have access to millions rather than hundreds of images. # This must be wrong! #### This must be wrong—this is pseudoscience! We get some feedback along these lines. And quite frankly, we would be delighted if our results were wrong. Humanity would have one less problem, and we could join others writing self-help bestsellers about how power-posing makes you bolder, smiling makes you happier, and seeing pictures of eyes makes you more honest. (Unfortunately, all these findings do not seem to replicate.) #### This must be wrong; it is widely known that there are no links between faces and character traits! Unfortunately, this belief is not supported by evidence. Quite the contrary: Many studies have shown that people can determine others’ political views, personality, sexual orientation, honesty, and many other traits from their faces. Moreover, humans’ relatively low accuracy when judging such traits does not necessarily mean that those traits are not prominently displayed on the face. People may simply have a limited ability to detect or interpret the cues—a limitation that does not necessarily apply to algorithms. #### This must be wrong; I read that physiognomists believed that criminals were part ape! Well, it seems that physiognomists were at least partially correct, as [we are all 100% ape][15]. Without a doubt, physiognomy was based on unscientific studies, superstition, anecdotal evidence, and racist pseudo-theories. The fact that its claims were unsupported, however, does not automatically mean that they are all wrong. Some of physiognomists’ claims may have been correct, perhaps by a mere accident. Physiognomists were clearly wrong when they claimed that they could accurately judge character based on facial appearance. Modern scientific studies have shown that we are not very accurate at this task. However, the same studies consistently show that we are better than chance, revealing that faces contain at least some information about one’s character. Thus, physiognomists’ main claim—that character is to some extent displayed on one’s face—is supported by modern science. # Code and data The "Files" section includes: - Code used to produce the results (code_shared.R) - Participants' data (participants.RData) - Participants' facial descriptors (vgg.RData) - Participants' facial properties from Face++ (faceplusplus.RData) - Human judgments of participants' faces (human_judges.RData) - Politicians' data (politicians_to_share.RData) # Selected media Coverage: - [PsyPost][16] - [Fox News][17] - [Gizmodo][18] [1]: https://psycnet.apa.org/fulltext/2024-65164-001.html [2]: https://osf.io/nuz2m/files/ [3]: https://patents.google.com/patent/WO2014068567A1/en [4]: https://patents.google.com/patent/US20160019411A1/en [5]: https://patents.google.com/patent/US20210089759A1 [6]: https://www.washingtonpost.com/news/innovations/wp/2016/05/24/terrorist-or-pedophile-this-start-up-says-it-can-out-secrets-by-analyzing-faces/ [7]: https://awspntest.apa.org/ftasset/journals/amp/ofp/amp0001295/images/amp_amp0001295_fig2a.gif [8]: https://docs.google.com/document/d/11oGZ1Ke3wK9E3BtOFfGfUQuuaSMR8AO2WfWH3aVke6U/edit# [9]: www.nature.com/articles/s41598-020-79310-1 [10]: https://www.pnas.org/doi/10.1073/pnas.1218772110 [11]: https://www.pnas.org/content/114/48/12714 [12]: https://psycnet.apa.org/ftasset/journals/amp/ofp/amp0001295/images/amp_amp0001295_fig1a.gif [13]: https://www.nature.com/articles/s41598-020-79310-1 [14]: https://www.nature.com/articles/s41598-020-79310-1 [15]: https://www.amazon.com/Descent-Man-Illustrated-Charles-Darwin-ebook/dp/B00O6C4QJY/ [16]: https://www.psypost.org/artificial-intelligence-can-predict-political-beliefs-from-expressionless-faces/ [17]: https://www.foxnews.com/politics/ai-can-predict-political-orientations-blank-faces-researchers-fear-serious-privacy-challenges [18]: https://gizmodo.com/ai-can-tell-your-political-affiliation-just-by-looking-1851430714