**Abstract** The role of slow oscillations and spindles during sleep on memory retention has become an area of great interest in the recent decade. Accordingly, there are multiple studies that examine the efficacy of acoustic stimulation during sleep to facilitate slow oscillations and associated memory retention. Here, we run meta-analyses on a current set of 14 studies that use audible noise-burst sound stimulation to modulate overnight retention of word pairs (kS = 12 studies, kES = 14 effect sizes, n = 206 subjects). Our meta-analyses demonstrate a steady, yearly decline in effect size that accounts for 91.8% of the heterogeneity between studies. We find that the predicted effect on memory retention in 2013 favored the acoustic stimulation condition at dδ = 0.99 (95% CI [0.49, 1.49]), while the predicted effect in 2021 declined to a moderate and significant effect favoring no acoustic stimulation at dδ = −0.39 (95% CI [−0.73, −0.05]). Our meta-regression model finds no coded study-level characteristics could account for the decline in effect sizes over time other than the publication date alone. Using available data, we estimate that 34% of subjects are not actually blind to the acoustic stimulation condition due to hearing acoustic stimulation during sleep. In addition, we find that that the test-retest reliability of memory retention scores is nearly zero (ρd = 0.01, 95% CI [−0.18, 0.21]), and through simulation demonstrate the impact this has on statistical power and observed effect sizes. Based on our analyses, we discuss the need for larger sample sizes, true placebo controls, age range restrictions, open-data sharing, and improvements in the reliability of memory retention tasks. **Data** Data used in the present meta-analysis is contained within the "Sleep Meta" CSV file under the "Data and Code." All code to process said data is contained within the same folder.