It is often thought that the diffusion model explains all effects related to the speed-accuracy tradeoff (SAT) but this has previously been examined with only a few SAT conditions or only a few subjects. Here we collected data from 20 subjects who performed a perceptual discrimination task with five different difficulty levels and five different SAT conditions (5000 trials/subject). We found that the five SAT conditions produced robustly U-shaped curves for (i) the difference between error and correct response times (RTs), (ii) the ratio of the standard deviation and mean of the RT distributions, and (iii) the skewness of the RT distributions. Critically, the diffusion model where only drift rate varies with contrast and only boundary varies with SAT could not account for any of the three U-shaped curves. Further, allowing all parameters to vary across conditions revealed that both the SAT and difficulty manipulations resulted in substantial modulations in every model parameter, while still providing imperfect fits to the data. These findings demonstrate that the diffusion model cannot fully explain the effects of SAT and establishes three robust but challenging effects that models of SAT should account for.