Research on pitch processing in autism spectrum disorder (ASD) has mainly focused on Western speakers, who generally showed increased low-level pitch processing and intact/enhanced musical pitch processing, but impaired linguistic pitch processing. To test whether this is also the case for speakers of tone languages, the present study investigated pitch processing in music and language in Mandarin speakers with and without ASD. Thirty individuals with ASD and thirty-three typically developing (TD) controls participated in a set of lexical tone and intonation perception tasks and two pitch threshold tasks. The main results showed that Mandarin speakers with ASD showed elevated pitch thresholds and worse lexical tone and intonation perception compared to their typically developing counterparts. However, after taking background measures (e.g., musical training, verbal and nonverbal IQ, and nonverbal short-term memory) into consideration, no group difference was observed across the different tasks. Rather, verbal IQ, nonverbal IQ, and/or musical training were significant predictors for task performance. These findings suggest that pitch processing ability of Mandarin speakers with ASD may be moderated by intelligence and musical training background. Overall, the present results provide a foundation for future work to explore how different language backgrounds may affect pitch processing in individuals with ASD.