Loading wiki pages...

Wiki Version:
<h1>SMART Mental Health Prediction Tournament</h1> <p><strong>Stratified Medicine Approaches foR Treatment Selection (SMART)</strong></p> <p><strong>Abstract:</strong> In the SMART Mental Health Prediction Tournament, 13 teams from around the world will compete to see who can build the best predictive model for anxiety and depression treatment response. Each team will be provided with the same large, anonymized mental health treatment outcome dataset from the UK’s national health system. A separate, held-out test sample will be used to determine the winner. This will provide a level playing field for evaluating each model’s efficacy, but it will also allow participants and, ultimately, the field, to understand the reasons for the advantages and disadvantages of each strategy, under different conditions. Head-to-head comparisons of the best approaches for selecting mental health treatments will, we believe, yield knowledge that can be used to maximize the efficiency of mental health care delivery in the future.</p> <p>The first aim of the tournament will be to learn more about different methodological approaches to building predictive models for treatment selection in mental health. We also hope to contribute to the conversation about barriers to implementation of precision medicine approaches in real-world clinical settings, informed in part by work we will undertake with key stakeholders (service-users and clinicians) as part of this project.</p> <p>The second aim is to produce an algorithm that could be used to inform treatment decisions in the UK’s IAPT (Improving Access to Psychological Therapies) services, with the goal of enhancing efficiency, efficacy, and service. </p> <h2>Contents</h2> <p><strong>History</strong> </p> <p>Growing interest in personalized medicine has resulted in a proliferation of research aimed at understanding individual differences in response to treatment. In 2016, researchers from around the world came together for the Treatment Selection Idea Lab (TSIL) to discuss precision medicine approaches in mental health. Many exciting methods for modeling individual differences in treatment response were presented, all of which had been applied to different datasets. As no head-to-head comparisons had been performed, those who had attended were left without a full understanding of the relative strengths and weakness of the approaches in different contexts. </p> <p>We were inspired by a talk given at TSIL2016 by Barb Mellers about her and Phil Tetlock’s <a href="https://www.gjopen.com" rel="nofollow">Good Judgment</a> project, a tournament of superforecasters in which Tetlock and Mellers compared different individuals’ and groups’ approaches to prediction to better understand the process of forecasting. </p> <p><strong>The UK Mental Healthcare System</strong></p> <p>The UK mental healthcare system is organized based on the <a href="https://www.nice.org.uk/guidance/cg123/chapter/1-guidance" rel="nofollow">stepped-care</a> model, which presents treatment options hierarchically. Step-1 involves contact with a general medical practitioner and may involve pharmacotherapy or “watchful waiting”. IAPT (Improving Access to Psychological Therapies) comprises low intensity (Step-2 – e.g., guided self-help, computerized CBT) and high intensity (Step-3 – e.g., 1-on-1 treatment with a clinician who specializes in a specific intervention) psychological therapies. The majority (in this sample ~80%) of patients treated in IAPT begin in low intensity (LI) treatment. Patients can be stepped up or down at any point of their contact with IAPT services, typically those who don’t respond to LI are “stepped up” to high intensity (HI) treatment (~15% of this sample), while an additional ~20% (in this sample) are treated at HI from the outset. LI treatments are briefer in terms of the number of sessions and the length of treatment sessions compared to HI treatments. Further, LI treatments are delivered by staff with post-graduate diplomas in LI therapies whereas HI treatments are delivered by a mixture of professionals. HI treatments are therefore considerably more expensive for IAPT services to deliver than LI therapies, in terms of staff costs, overheads and other resources. </p> <p><strong>Goals of the Tournament</strong></p> <p>The aim of this tournament is to test statistical approaches that could be used to improve the process by which patients are allocated to LI or HI treatment in IAPT, such that a given treatment selection model might improve patient outcomes and the efficiency of resource use within IAPT services. </p> <p><strong>Project Timeline</strong></p> <ul> <li>December, 2017 – Teams receive training data</li> <li>March 16, 2018 – Teams turn in Phase I predictive models </li> <li>June 26, 2018 – Tournament results presented and winner announced at TSIL2018</li> </ul> <p><strong>Sample Details</strong></p> <p>A dataset of a large (~N=6,000) cohort of patients with depression and/or anxiety problems treated in an IAPT service in Leeds, England. This dataset will have two sets of potential predictors: the set of variables that are routinely collected at all IAPT services across the UK, and an enriched set of variables that have been assessed specifically in the Leeds IAPT services. A second separate sample from Cumbria IAPT (N=~1000) will be used as a second held-out test sample for both standard and enriched models. A separate validate sample (~N=30,000) will be used to test the generalizability of the winning models (standard variables only).</p> <p><em>Variables Routinely Collected at IAPT Sites</em>: Diagnosis, Age, Gender, Ethnicity, Disability Status, Employment Status, Comorbid long-term physical condition, GAD-7 (Spitzer et al. 2006), PHQ-9 (Kroenke et al. 2001), Work and Social Adjustment Scale (Mundt et al. 2002), IAPT Phobias Scales</p> <p><em>Enriched Variables</em>: Chronicity, Number of prior treatment episodes, Family history of mental health problems, Outcome Expectancy (Lutz et al. 2007), Index of Multiple Deprivation (socioeconomic status)(McLennan et al. 2011)</p> <p>The primary dataset includes N=6000 cases of <strong>non-randomized</strong> data. 1000 of these cases constituted the sample analyzed by Delgadillo, Moreea and Lutz (2016). These cases, plus 3000 randomly selected cases of the remaining 5000, will serve as the common training dataset with which the teams will develop their algorithms and allocation strategies. The remaining 2000 cases will be held out as a test sample.</p> <p><strong>Outcome Variable</strong></p> <p>After considering many different outcome metrics as well as current IAPT core metrics (please see the “Outcome metric explanation” document for more details), we developed an adapted binary outcome metric based on the three metrics currently used in IAPT (recovery, reliable change, reliable recovery). The SMART outcome variable is defined for three possible cases in the tables below. Important definitions include: 1) Recovery is defined by being below caseness (PHQ-9 &lt;= 9; GAD-7 &lt;= 7) post-treatment. 2) “reliable change” is defined as improving at least 6 points for the PHQ-9 and at least 5 points for the GAD-7. 3) If a patient is classified as a “case” (PHQ-9 &gt;= 10; GAD-7 &gt;= 8), then a change score that equals or exceeds 50% would meet criteria for positive change on that measure. 4) “reliable and clinically significant deterioration” is defined as moving from being below caseness pre-treatment to above caseness post-treatment with an increase of 6 or more on the PHQ-9 and as an increase of 5 or more on the GAD-7.</p> <p>![Outcome rule for patients who start above caseness on both PHQ9 and GAD7](<a href="http://osf.io/dx6hj/download" rel="nofollow">http://osf.io/dx6hj/download</a> =500x500) Table 1a.</p> <p>![Outcome rule for patients who start above caseness only on PHQ9](<a href="http://osf.io/m9td8/download" rel="nofollow">http://osf.io/m9td8/download</a> =500x500) Table 1b.</p> <p>![Outcome rule for patients who start above caseness only on GAD7](<a href="http://osf.io/y43w6/download" rel="nofollow">http://osf.io/y43w6/download</a> =500x500) Table 1c.</p> <p><strong>Data Analysis Plan</strong></p> <p><em>Model Construction</em></p> <p>The primary models on which teams will be evaluated will only use the standard set of variables; however, teams will be allowed to submit a secondary set of models that can use the "enriched variables". If those models indicate meaningful added predictive value from those variables, a cost-benefit analysis will be performed. </p> <p>Teams will be provided with four datasets – two that only include the standard variables (one without imputed data and one with imputed data) and two that include the enhanced variables (again, with and without imputed data).</p> <p>Teams must submit one set of 3 algorithms (described below) that rely only on the standard variables, and a second set of 3 algorithms that can be informed by the enriched variables. This is because only the standard variables are currently collected across all IAPT services in the UK, and thus a model that required the enriched variables would not be useful for IAPT unless and until they changed assessment practices across the entire system.</p> <p>Teams will be allowed to use any available methodological approach to build their predictive model, although <strong>the first model must produce a prognostic prediction of outcome in LI. The second model must generate a prognostic prediction of outcome for those assigned directly to HI</strong>. Prognostic models do not consider interactions between predictive variables and treatment condition, and thus do not explicitly capture the expected “differential” response between LI and HI. </p> <p><strong>The third model that teams will submit will produce a prediction of the differential benefit of HI over LI</strong>. This prediction could come from two separate prognostic models (one in HI and one in LI), the predictions of which are subtracted, or from a single model that directly generates a differential prediction. This differential prediction must be amenable to the evaluation scheme described below (which requires a difference between <em>probability</em> of good outcome in HI vs LI, and not, for example, predicted differential benefit in terms of a continuous outcome like PHQ-9).</p> <p>If possible, all models should be built and submitted in the R software environment; other programs can be used.</p> <p>Once each team has submitted their candidate predictive model using ‘routinely collected IAPT variables’ (see Table 1), the predictive accuracy and performance of each team’s model will be tested in the test and validation datasets: (1) the held out test subset of cases in the Leeds IAPT dataset; (2) the second test sample from Cumbria IAPT; (3) the wider validation multi-centre dataset including data from 4 other IAPT services from the Northern IAPT Practice Research Network.</p> <p><em>Model Evaluation</em></p> <p>Two different types of approaches will be used to evaluate the SMART tournament models. The first type of approach will focus on the accuracy of the predictions generated by each of the prognostic models. The second evaluation will focus on the accuracy of the differential predictions. These evaluations will determine the model(s) that win the SMART tournament.</p> <p>One goal of the tournament is to come to a consensus as a group about which approach (e.g., propensity score analysis) we should use to account for the non-randomized nature of the data. One approach that has been proposed is a double-robust weighting scheme (available as a SAS routine), which could help account for the effects of any observed predictors on treatment allocation, and would aim to allow us to evaluate the models as if the test and validation samples had been randomized to LI and HI. The primary evaluations will not rely on these approaches, but secondary analyses accounting for the non-randomized nature of the data will be presented.</p> <p><em>Accuracy Evaluation:</em> The LI and HI prognosis models will be evaluated using brier scores, the deviance statistic, and ROC curves (with AUC). The differential models will be evaluated in the following way: within the test and validation samples, patients will be arrayed for each model based on the predicted differential benefit (from smallest predicted benefit of HI over LI to largest predicted benefit). Then, a sliding window will be used to calculate the observed differential benefit of HI over LI at each point along that spectrum. The accuracy of these models will be calculated by comparing the predicted differential benefit at each point to the observed differential benefit.</p> <p><strong>Teams</strong></p> <p>Team 1) <a href="http://web.sas.upenn.edu/derubeis/outcome-research/" rel="nofollow">Rob DeRubeis</a>, Jack Keefe, Colin Xu, and Thomas Kim (University of Pennsylvania)</p> <p>Team 2) <a href="http://kapelner.com/publications" rel="nofollow">Adam Kapelner</a>, Alina Levine (Queens College in New York City) and <a href="http://joshuawiley.com" rel="nofollow">Joshua Wiley</a> (Monash University)</p> <p>Team 3) <a href="https://www.researchgate.net/profile/Adam_Chekroud" rel="nofollow">Adam Chekroud</a>, Chief Scientist, <a href="http://yygssingapore.yale.edu/people/abhishek-chandra" rel="nofollow">Abhishek Chandra</a>, Chief Technology Officer, <a href="https://publichealth.yale.edu/biostat/people/ralitza_gueorguieva-1.profile" rel="nofollow">Ralitza Gueorguieva</a>, Chair of Biostatistics, <a href="http://medicine.yale.edu/psychiatry/people/john_krystal.profile" rel="nofollow">John Krystal</a>, Chair of Psychiatry, <a href="https://psychology.yale.edu/people/kevin-anderson" rel="nofollow">Kevin Anderson</a>, PhD Candidate, <a href="https://psychology.yale.edu/people/thomas-oconnell" rel="nofollow">Thomas O'Connell</a>, PhD Candidate, <a href="https://psychology.yale.edu/people/stefan-uddenberg" rel="nofollow">Stefan Uddenberg</a>, PhD Candidate, <a href="https://psychology.yale.edu/people/yoonho-chung" rel="nofollow">Yoonho Chung</a>, Postdoctoral Associate, <a href="http://childstudycenter.yale.edu/faculty_people/gianfilippo_coppola.profile" rel="nofollow">Gianfilippo Coppola</a>, Asisstant Professor, <a href="https://psychology.yale.edu/people/michael-lopez-brau" rel="nofollow">Michael Lopez-Brau</a>, PhD Candidate (Yale University, Spring Health)</p> <p>Team 4) <a href="https://www.uni-trier.de/index.php?id=9524&L=2" rel="nofollow">Wolfgang Lutz</a>, Julian Rubel, Anne-Katharina Deisenhofer, Brian Schwartz, Björn Bennemann, (Universität Trier), <a href="http://www.dynamicpsychlab.com?id=9524&L=2" rel="nofollow">Aaron Fisher</a> (University of California, Berkeley)</p> <p>Team 5) <a href="https://www.ucl.ac.uk/brain-sciences/case-studies/2017/sep/meet-researcher-steve-pilling" rel="nofollow">Steve Pilling</a>, <a href="http://www.ucl.ac.uk/pals/people/profiles/research-staff/rob-saunders" rel="nofollow">Rob Saunders</a> and <a href="http://www.ucl.ac.uk/pals/people/profiles/research-staff/joshua-buckman" rel="nofollow">Joshua Buckman</a> (University College London)</p> <p>Team 6) <a href="https://www.sheffield.ac.uk/psychology/staff/academic/jaime_delgadillo.uk/pals/people/profiles/research-staff/joshua-buckman" rel="nofollow">Jaime Delgadillo</a> and <a href="https://www.sheffield.ac.uk/psychology/staff/academic/michael-barkham" rel="nofollow">Michael Barkham</a> (University of Sheffield)</p> <p>Team 7) <a href="https://www.hcp.med.harvard.edu/faculty/core/ronald-c-kessler-phd" rel="nofollow">Ronald Kessler</a>, <a href="https://www.researchgate.net/profile/Ekaterina_Sadikova" rel="nofollow">Ekaterina Sadikova</a> (Harvard Medical School), and <a href="http://www.alexluedtke.com" rel="nofollow">Alex Luedtke</a> (Fred Hutchinson Cancer Research Center). </p> <p>Team 8) <a href="https://liberalarts.utexas.edu/psychology/faculty/smitsja1" rel="nofollow">Jasper Smits</a>, <a href="https://liberalarts.utexas.edu/psychology/faculty/jshumake" rel="nofollow">Jason Shumake</a>, <a href="https://labs.la.utexas.edu/beevers/" rel="nofollow">Christopher Beevers</a>, <a href="https://liberalarts.utexas.edu/imhr/graduate-students/profile.php?id=dap3463" rel="nofollow">Derek Pisner</a>, <a href="https://liberalarts.utexas.edu/imhr/graduate-students/profile.php?id=papinis" rel="nofollow">Santiago Papini</a> (University of Texas at Austin)</p> <p>Team 9) <a href="https://www.researchgate.net/profile/Andrea_Niles" rel="nofollow">Andrea Niles</a> (University of California San Francisco)</p> <p>Team 10) <a href="https://www.uel.ac.uk/staff/f/cynthia-fu" rel="nofollow">Cynthia Fu</a> (University of East London), <a href="https://www.med.upenn.edu/apps/faculty/index.php/g275/p32990" rel="nofollow">Christos Davatzikos</a> and <a href="https://www.med.upenn.edu/apps/faculty/index.php/g275/p4257219" rel="nofollow">Yong Fan</a> (University of Pennsylvania)</p> <p>Team 11) <a href="https://www.researchgate.net/profile/Clarissa_Bauer-Staeb2" rel="nofollow">Clarissa Bauer-Staeb</a>, <a href="http://www.bath.ac.uk/psychology/staff/kate-button/index.html" rel="nofollow">Katherine Button</a>, <a href="http://www.bath.ac.uk/imi/people/commercial-research-associates.html#Catherine" rel="nofollow">Catherine Barnaby</a>, <a href="http://www.maths.bath.ac.uk/~jjf23/" rel="nofollow">Julian Faraway</a> (University of Bath)</p> <p>Team 12) <a href="https://www.universiteitleiden.nl/en/staffmembers/marjolein-fokkema#tab-1" rel="nofollow">Marjolein Fokkema</a> (Leiden University), <a href="http://www.annafreud.org/about-us/meet-the-leadership-team/professor-miranda-wolpert-director-of-innovation-evaluation-and-dissemination/" rel="nofollow">Miranda Wolpert</a>, Elisa Napoleone and Julian Edbrooke-Childs (Anna Freud Center).</p> <p>Team 13) <a href="https://aifredhealth.com/team.html" rel="nofollow">David Benrimoh</a>, <a href="https://aifredhealth.com/team.html" rel="nofollow">Robert Fratila</a>, <a href="https://www.researchgate.net/profile/Matthew_Krause2" rel="nofollow">Matthew Krause</a> (<a href="http://aifredhealth.com/team.html" rel="nofollow">aifred health</a>, McGill University)</p> <p><strong>Advisors</strong></p> <p>We are grateful that David Clark, Steve Pilling, and Michael Barkham have agreed to act as advisors to the SMART tournament. Their insights into IAPT will help maximize the potential of the project by ensuring that the structure and goals of the tournament are aligned with the real-world context and needs of IAPT services, clinicians, and service-users. </p>

Start managing your projects on the OSF today.

Free and easy to use, the Open Science Framework supports the entire research lifecycle: planning, execution, reporting, archiving, and discovery.

Create an Account Learn More Hide this message