Date created: | Last Updated:

Category: Project

Description: Decision Trees (DT), one method falling under the umbrella of Exploratory Data Mining (EDM; McArdle & Ritschard, 2013), has seen an increasing amount of use in psychological research. This can be attributed to its easily interpretable tree structure, along with its propensity to automatically capture interactions among predictors. However, the creation of tree structures comes at the expense of generalizability – namely that slight perturbations in the sample can result in dramatically different results. The purpose of this paper was to investigate this instability (reliability) and to provide metrics to quantify the propensity for the results to generalize. This was examined across multiple DT algorithms and datasets to determine whether there were differences in stability, and what factors accounted for these differences. There were large differences in both stability and predictive performance across algorithms, with some favoring simplistic, stable trees, and others large, unstable, but highly predictive models. Finally, this comparison was applied to three empirical examples – predicting dementia with various cognitive scales and demographic factors, using depression scales to predict suicidal ideation, and predicting science achievement in Kindergarten. In sum, we demonstrate that measuring the stability of DT algorithms has utility in comparing multiple algorithms and methods, providing additional information to researchers as to both the viability and usefulness of using DT algorithms in their research.