Face perception is a fundamental aspect of human social interaction, yet most research on this topic has focused on single modalities and specific aspects of face perception. Here, we present a comprehensive multimodal dataset for emotion perception and judgment. Specifically, this dataset includes EEG data from 97 unique neurotypical participants through 8 experiments, fMRI data from 19 neurotypical participants, single-neuron data from 16 neurosurgical patients (22 sessions), eye tracking data from 24 neurotypical participants, and behavioral and eye tracking data from 18 participants with ASD and 15 matched controls. Overall, this multimodal dataset provides a comprehensive exploration of facial emotion perception, emphasizing the importance of integrating multiple modalities to gain a holistic understanding of this complex cognitive process. This dataset provides a key missing link between human neuroimaging and neurophysiology literature, and facilitates the study of neuropsychiatric populations. Wang S, Yu R, Tyszka JM, Zhen S, Kovach C, Sun S, et al. The human amygdala parametrically encodes the intensity of specific facial emotions and their categorical ambiguity. Nature communications. 2017;8(1):14821. Sun S, Zhen S, Fu Z, Wu D-A, Shimojo S, Adolphs R, et al. Decision ambiguity is mediated by a late positive potential originating from cingulate cortex. NeuroImage. 2017;157:400-14. doi: https://doi.org/10.1016/j.neuroimage.2017.06.003. Sun S, Yu R, Wang S. A neural signature encoding decisions under perceptual ambiguity. eneuro. 2017;4(6). Wang S. Face size biases emotion judgment through eye movement. Scientific reports. 2018;8(1):317. Sun S, Yu H, Yu R, Wang S. Functional connectivity between the amygdala and prefrontal cortex underlies processing of emotion ambiguity. bioRxiv. 2023:2023.01. 24.525116. Sun S, Webster PJ, Wang Y, Yu H, Yu R, Wang S. Reduced Pupil Oscillation During Facial Emotion Judgment in People with Autism Spectrum Disorder. Journal of autism and developmental disorders. 2023;53(5):1963-73.