Home

Menu

Loading wiki pages...

View
Wiki Version:
***A word document of the preregistration for this study can be found in the preregistration folder.*** ---------- **Preregistration Title:** From 2006 to 2018: Understanding thirteen years of temperament and ability data **Lead/corresponding author:** Elizabeth M. Dworak, Department of Psychology, Northwestern University **Contact:** elizabeth.knowlton@northwestern.edu **Date of pre-registration:** July 26, 2019 ---------- **Introduction** **Research Question/Hypothesis** This study will investigate cohort shifts of personality traits and cognitive ability. Although data were collected cross-sectionally, we will examine yearly trends of age, gender, education, and geographic origin from 2006 to 2018. While we posit that results are natural temporal changes, there is a possibility that shifts are due to sampling methods. Over the 13 year lifespan of the Synthetic Aperture Personality Assessment Project (SAPA-Project), our lab has successfully collected data from over 1,000,000 participants utilizing a web based survey. Understanding the differences in our data is crucial and will inform ongoing and future research how to best proceed when collecting large volumes of cross-sectional data. **Methods** **Sampling Plan** The archival data for this study were collected cross-sectionally from 2006 to 2018 through the SAPA-Project, a free web-based personality survey. Participants have found the website through varying mechanisms including google, Google advertisements, posts on Reddit (n.d.), twitter (charlesmurray, 2017), and other various online articles; for example, the Washington Post (Guarino, 2018) article on personality types. **Sample Characteristics** The data used in this sample were collected from April 2006 to November 2018 from over 1,000,000 participants. Based on previous papers (Condon, Roney & Revelle, 2017a), we know that data were collected from a minimum of 220 countries/territories, with the United States composing approximately 69% of the sample. From results presented at the 2019 International Society for Intelligence Research (Dworak, Revelle, & Condon, 2019b), we know from April 2006 to November 2018 there were 381,887 participants from the United States between the ages of 18 to 90 (M=33.6; Median=28.0). Of this subsample, approximately 65% were female and participants most frequently reported their education level as currently attending college. However, because the current project plans to use participants under the age of 18 and participants from outside of the United States, we expect the demographics found by Condon, Roney, and Revelle (2017a) to be more representative of the current subsample. In order to remain as blind to the data as possible, we do not have the exact percentages for this study’s participant demographics. From previous studies, we expect the sample to be a majority female and range from 11 and 90 years old. Because our temperament items are not intended to work with younger participants, we will remove individuals below the age of 16 before examining the data. Participants above the age of 90 will also be removed from the sample as they could be an outlier for our temperament or ability data (i.e. it is unclear to us what motivation an individual above the age 90 would have to seek out a personality survey). From April 2006 to August 2010 Education was measured using six categories (excluding associate degree and currently in graduate or professional school) and from August 2010 to February 2017 education was measured with seven categories (excluding associate degree). Starting in 2017 and currently, education is measured using eight categories. These categories allow participants report less than 12 years, high school graduate, currently in college/university, some college/university, but did not graduate, associate degree (2yr), college/university degree (4yr), currently in graduate or professional school, or graduate or professional school degree. **Previous Exposure to Data (Variables or Sample)** As the lead/corresponding author will be conducting the data analyses, we have included a detailed description of their exposure to the data prior to this project. Due the co-investigators’ greater exposure to the data, they were not consulted in generating the research questions. **Overall data.** Most recently, the 2006 to 2018 data were used by the lead/corresponding author to examine the means and standard errors of cognitive ability domains for adults (ages 18 to 90) from the United States (Dworak, Revelle, & Condon, 2019b). Additionally, the mean scores of 35 International Cognitive Ability Resource (ICAR) items, 60 ICAR items, matrix reasoning, letter and number series, verbal reasoning, and three dimensional rotation were examined across birth year and age to see whether scores were increasing or decreasing. Likewise, the author also examined demographic information and how rates of educational attainment changed for these participants. From this presentation, the authors are aware that there is a general decrease in cognitive ability scores within in the United States adult subsample regardless of increases in education attainment. While the lead /corresponding author used items that overlap with the present study, there is a plethora of untouched data. This includes the author not examining any information pertaining to participants outside the United States, participants under the age of 18, or any information regarding temperament-based personality traits. A copy of the author’s oral presentation can be accessed here: https://sapa-project.org/emd/presentations/dworak_isir19.pptx **2006-2010.** The 2006-2010 data have been used by the lead/corresponding author to conduct preliminary analysis on select data for an oral presentation abstract submission to the 2019 International Society for the Study of Individual Differences. For this analysis, average scores of matrix reasoning (M=0.51), verbal reasoning (M=0.65), and letter and number series (M=0.61) were found from 35 ICAR items for the first 28,497 participants. These 28,497 participants acted as a placeholder for the estimated number of participants collected in 2006 (approximately one fourth of the participants in this dataset) until dates could pulled from the raw/uncleaned data. In addition to this, the 2006 average scores of consciousness (M=-0.43), agreeableness (M=-0.10), emotional stability (M=0.08), intellect (M=0.11), and extraversion (M=0.01) were examined for this submission. For this preliminary analysis, descriptive statistics of demographic information were also examined. This included finding the average and range of participants’ geographic origin, education, gender, and age to help understand the preliminary analysis; for example the decrease on ability items in the 2010-2017 and the 2017 dataset. The 2006-2010 dataset has also been used by the lead/corresponding author for an oral presentation at the 2019 International Society for Intelligence Research (see section Overall Data). **2010-2013 Open Source.** The 2010 to 2013 ICAR data is currently posted on Dataverse (Condon, & Revelle, 2015a). Although the lead/corresponding author has not worked directly with the relevant measures in this dataset, she has read a study presented at the 2018 International Society for Intelligence Research. In this study, Young and Keith (in review) examined group differences on ICAR items. Specifically, this study found that the 16 item ICAR and 60 item ICAR norming sample were slightly biased towards men across the four domains. Likewise, age had a non-monotonic effect on all but verbal reasoning. **2010-2017.** The 2010-2017 data were been used by the lead/corresponding author for an oral presentation at the 2019 International Society for Intelligence Research (see section Overall Data). **February - November 2017.** The 2017 datasets have been primarily used by the lead/corresponding author for class work over the last two years. Class projects described in this section are those that used variables relevant to the current study. This includes cognitive ability scores from the 60 item ICAR (Condon & Revelle, 2014) and personality trait scores or items from the 135 SAPA Personality Inventory (SPI-135; Condon, 2018) five factor and 27 factor models. Class projects using these data included running regressions of stress on openness, conscientiousness, extraversion, agreeableness, and neuroticism and running an exploratory factor and cluster analysis on stress, and the SPI-135 traits of anxiety, well-being, emotional stability, and irritability. The lead/corresponding author has also used this dataset to run exploratory and confirmatory structural equation models to test the relationship between ability and the Big Five personality traits on latent factors of health. Beyond coursework, these data have also been used for three conference submissions. Presentations at the 2018 European Conference of Personality (Knowlton, Revelle, & Condon, 2018) and 2019 Association for Research in Personality (Dworak, Revelle, & Condon, 2019a) used the coping item “How do you most often cope with stress?” and the SPI-135. Specifically, Knowlton, Revelle, and Condon (2018) used cross-validation and bootstrapping exploration of the 27 factor personality traits, 5 factor personality traits, and the 135 personality items to see what best predicted coping behaviors. Likewise, Dworak, Revelle, and Condon (2019a) correlated zero order correlations of dummy coded coping behaviors to coping profiles developed across the 135 personality items (SPI-135) for each coping behavior. This resulted in correlations of coping behaviors with coping behavior profiles. This project also used exploratory cluster analysis of coping profiles using iclust and Schmid-Leiman exploratory factor analysis of coping profiles to examine patterns of coping behaviors. In addition to this, the 2017 dataset was used to conduct preliminary analysis for an oral presentation abstract submission to the 2019 International Society for the Study of Individual Differences. This preliminary analysis found average scores of matrix reasoning (M=0.45), verbal reasoning (M=0.61), and letter and number series (M=0.48) from 35 ICAR items. In addition to this, the 2017 average scores of consciousness (M=-0.05), agreeableness (M=-0.10), neuroticism (M=-0.15), openness (M=-0.30), and extraversion (M=-0.03) were examined for this submission. **February 2017 - November 2018.** The 2017-2018 dataset has been used by the lead/corresponding author for an oral presentation at the 2019 International Society for Intelligence Research (see section Overall Data). **Data Access** Data is currently available to the authors in the research laboratories Box Folder. Because the lead/corresponding author does not have ownership over the SAPA-project, the type of data posted with this study will be determined by the co-investigators. Currently, the temperament (or non-ability personality) sections of the SAPA-Project collected from August 2010 to February 2017 are currently published on the Harvard Dataverse (Condon & Revelle, 2015b; Condon & Revelle, 2018; Condon, Roney, & Revelle, 2017b; Condon, Roney, & Revelle, 2017c) and available for public use. Pending approval of the co-investigators, the April 2006 to August 2010 and February 2017 to November 2018 datasets will become available through the same open source data repository and use the same format established by previous datasets. In addition to these temperament datasets, cognitive ability data collected between August 2010 to May 2013 by the SAPA-Project has been posted on Dataverse (Condon & Revelle, 2015a). Pending approval of the co-investigators, the April 2006 to August 2010, June 2013 to February 2017, and February 2017 to November 2018 datasets will become available through the same open source data repository using the same format established by previous datasets. Foreseeably, 2017-2018 data will not be posted immediately as questions that pertain to these data are still being investigated by members of the research lab. **List of Measures** The demographics this study will examine include age, gender, education level, and geographic origin. We are interested in including these variables because changes in participant demographics could be influencing the structure of our temperament and ability data. In addition to this, demographics will allow us to test for invariance across groups. Beyond demographics, we are also interested in investigating changes in non-ability personality (temperament). During the 13 years of data collection, temperament has been measured using the 100-item International Personality Item Pool (Goldberg, 1999) and more recently the SPI-135 (Condon, 2018). The personality traits included in these scales are conscientiousness, agreeableness, emotional stability, neuroticism, intellect, openness, and extraversion. In addition to temperament, this study will look for changes in ability using the International Cognitive Ability Resource (ICAR; Condon & Revelle, 2014). Specifically, we are interested in seeing if different domains of ability or full ability scores have changed over our 13 years of data. The four domains of interest include matrix reasoning, verbal reasoning, letter and number series, and three dimensional rotation. **List of Variables** **Ability.** Dependent variables to be used in this study include domain specific ability scores (matrix reasoning, verbal reasoning, letter and number series, and three dimensional rotation) from the International Cognitive Ability Resource (ICAR; Condon & Revelle, 2014) and overall ability scores (35 ICAR items; 60 ICAR items). Ability items are administered through the SAPA-Project using a multiple-choice format with eight response options. Of the options, only one answer is correct. During data cleaning items are scored the participant giving the correct answer (1) or an incorrect answer (0). Skipped questions are coded as NA and not included in the analysis. Domain scores and total scores will be calculated by finding each participant’s average score across answered times. ***Matrix reasoning.*** Matrix reasoning is composed of 11 items that contain 3x3 arrays of geometric shapes. Within this grid, one of the nine shapes is intentionally excluded. Participants are then prompted to respond by choosing which of six geometric shapes best fits within the stimuli. This task is most often compared to stimuli seen in the Raven’s Progressive Matrices. Matrix reasoning items are meant to measure nonverbal reasoning, visual processing, and fluid reasoning. Recent work by Young (in prep) showed that when validated against the WAIS-IV matrix reasoning scores were primarily related to fluid reasoning and visual processing. ***Verbal reasoning.*** Verbal reasoning is composed of 16 items that use various general knowledge, logic, and vocabulary questions. Verbal reasoning items are generally thought to measure comprehension-knowledge, but when validated against the WAIS-IV verbal reasoning scores were primarily related to visual processing (Young, in prep). ***Letter and number series.*** From April 2006 to August 2010, letter and number series was composed of 8 items. To improve this measure, collection of a ninth item began August 18, 2010 along with the continues collection of the original 8 items. Letter and number series items present participants with a sequence of digits or letters and asks the participant to choose the digit or letter that occurs next. Letter and number series items are generally thought to measure computational/mathematical reasoning, but recent work by Young (in prep) showed that when validated against the WAIS-IV letter and number series scores were related to fluid reasoning. ***Three dimensional rotation.*** Three dimensional rotation is composed of 24 items that present participants with a marked cubic shape. Participants are then asked to choose a possible rotation of the shape. Three-dimensional rotation items measure visuospatial and mental rotation, however when validated against the WAIS-V, scores only had a moderate positive correlation with visual processing (Young, in prep). Collection of these items began in May 2011 and are ongoing. ***Overall ability from 35 ICAR items.*** A participant’s overall ability will be scored using participants means from the 35 ICAR items. The 35 ICAR items are composed of 8 letter and number series items, 11 matrix reasoning items, and 16 verbal reasoning items. We justify examining the 35 variables in addition to the 60 ICAR items is due to the history of SAPA-Project. From April 2006 to August 2010 data were only collected for the 35 ICAR items. By including this shorter list of items, this paper will be able to examine how ability has changed between cohorts from 2006 to 2018. ***Overall ability from 60 ICAR items.*** A participant’s overall ability will be scored using participants means from the 60 ICAR items. The 60 ICAR items are composed of the 9 letter and number series items, 11 matrix reasoning items, 24 three dimensional rotation items, and 16 verbal reasoning items. We justify examining the 60 variables in addition to the 35 ability items is due to the history of SAPA-Project. In August 2010 one additional letter and number series item was added and in May 2011 24 three dimensional rotation items were added to data collection. Using this longer list of items, this paper will also be able to examine how scores on the 60 ICAR items have changed between cohorts from 2011 to 2018. **Temperament.** This study will also examine scores of the Big Five personality traits (Digman, 1990; Goldberg, 1990). Depending on the scale used, these traits can either be referred to as conscientiousness, agreeableness, neuroticism, openness to experience, and extraversion or conscientiousness, agreeableness, emotional stability, intellect, and extraversion. Responses to these items were recorded through the SAPA-Project using a Likert-scale that asked participants to report how much a question described them from 1 (Very Inaccurate) to 6 (Very Accurate). To generate trait scores for each participant, Item Response Theory (IRT) will be used for each of the personality trait scales; resulting in five trait scores distributed between -4 and 4. Higher scores, or scores closer to 4 would be indicative of higher levels of a trait (i.e. extremely conscientiousness, extremely agreeable, extremely neurotic, extremely open to experiences, and extremely extraverted) whereas a -4 would indicate the individual has low levels of a trait (i.e. extreme lack of conscientiousness, extremely disagreeable, extremely emotionally stable, extremely closed to experiences, and extremely introverted). ***Conscientiousness.*** From 2006 to 2013, conscientiousness was primarily collected using a 20-item scale from the 100-item International Personality Item Pool (Goldberg, 1999). Although collection of these items is continuing, starting in 2013 the SAPA-Project began to collect additional items that would go into the 14-item conscientiousness scale. These items would later become part of the SPI-135 (Condon, 2018). Between these scales of conscientiousness, only six items overlap. We justify examining conscientiousness using the 20-item IPIP scale because items have been collected over the lifespan of the SAPA-Project. However, we also believe it is important to examine the 14-item SPI scale because of the rigorous effort that went in to selecting these items and its prioritized use within our lab’s research. Using this these scales, this paper will examine how scores on the IPIP have changed between cohorts from 2006 to 2018 and how scores on the SPI have changed between cohorts from 2013 to 2018. ***Agreeableness.*** From 2006 to 2013, agreeableness was primarily collected using a 20-item scale from the 100-item International Personality Item Pool (Goldberg, 1999). Although collection of these items is continuing, starting in 2013 the SAPA-Project began to collect additional items that would go into the 14-item agreeableness scale. These items would later become part of the SPI-135 (Condon, 2018). Between these scales of agreeableness, only two items overlap. We justify examining agreeableness using the 20-item IPIP scale because items have been collected over the lifespan of the SAPA-Project. However, we also believe it is important to examine the 14-item SPI scale because of the rigorous effort that went in to selecting these items and its prioritized use within our lab’s research. Using this these scales, this paper will examine how scores on the IPIP have changed between cohorts from 2006 to 2018 and how scores on the SPI have changed between cohorts from 2013 to 2018. ***Emotional stability/Neuroticism.*** From 2006 to 2013, emotional stability was primarily collected using a 20-item scale from the 100-item International Personality Item Pool (Goldberg, 1999). Although collection of these items is continuing, starting in 2013 the SAPA-Project began to collect additional items that would go into the 14-item neuroticism scale. These items would later become part of the SPI-135 (Condon, 2018). Between these scales of emotional stability and neuroticism, five items overlap. We justify examining emotional stability using the 20-item IPIP scale because items have been collected over the lifespan of the SAPA-Project. However, we also believe it is important to examine neuroticism using the 14-item SPI scale because of the rigorous effort that went in to selecting these items and its prioritized use within our lab’s research. Using this these scales, this paper will examine how scores on the IPIP have changed between cohorts from 2006 to 2018 and how scores on the SPI have changed between cohorts from 2013 to 2018. ***Intellect/Openness to experience.*** From 2006 to 2013, intellect was primarily collected using a 20-item scale from the 100-item International Personality Item Pool (Goldberg, 1999). Although collection of these items is continuing, starting in 2013 the SAPA-Project began to collect additional items that would go into the 14-item openness to experience scale. These items would later become part of the SPI-135 (Condon, 2018). Between these scales of intellect and openness, six items overlap. We justify examining intellect using the 20-item IPIP scale because items have been collected over the lifespan of the SAPA-Project. However, we also believe it is important to examine openness to experience using the 14-item SPI scale because of the rigorous effort that went in to selecting these items and its prioritized use within our lab’s research. Using this these scales, this paper will examine how scores on the IPIP have changed between cohorts from 2006 to 2018 and how scores on the SPI have changed between cohorts from 2013 to 2018. ***Extraversion***. From 2006 to 2010, extraversion was primarily collected using a 19 item scale from the 100-item International Personality Item Pool (Goldberg, 1999). While extraversion was originally composed of a 20-item scale, one variable was dropped for analysis due to an error with data storage (an ability variable with a similar naming convention overwrote the temperament item during data collection and storage). The item “Am a very private person” would then begin to be properly stored after 2010. Although collection of these items is continuing in addition to the misrecorded extraversion item, starting in 2013 the SAPA-Project began to collect additional items that would go into the 14-item extraversion scale. These items would later become part of the SPI-135 (Condon, 2018). Between these scales of extraversion, only two items overlap. We justify examining extraversion using the 20-item IPIP scale because items have been collected over the lifespan of the SAPA-Project. However, we also believe it is important to examine the 14-item SPI scale because of the rigorous effort that went in to selecting these items and its prioritized use within our lab’s research. Using this these scales, this paper will examine how scores on the IPIP have changed between cohorts from 2006 to 2018 and how scores on the SPI have changed between cohorts from 2013 to 2018. **Demographics.** ***Geographic origin.*** Participant’s geographic origin was collected by asking a participant their country/territory of origin. In response to “Where did you grow up?” participants are able to choose from a drop-down menu of 235 countries and territories. Rather than comparing every country and territory, geographic origin will be coded into one of two categories: United States or Rest of World. ***Age.*** Age was collected by asking their participants to indicate their age. During data cleaning, any participants under the age of 16 and above the age of 90 will be removed. Although we have participants down to 11 years old, because the Big Five personality traits do not fully develop until adolescence and our temperament items are not meant to be administered to participants this young, we believe examining temperament and ability in these participants is outside the scope of this project. When testing for invariance, age will be broken up into two year age groups and life stages. ***Gender.*** From April 2006 to February 2017 participants reported gender/biological sex by choosing from a drop-down menu between male, female, and prefer not to answer. It was not until February 2017 that the option other was included. Because we have only been collecting this additional category for two years, gender will be binary in our analysis. ***Education.*** From April 2006 to August 2010 education was measured using six categories (excluding associate degree and currently in graduate or professional school) and from August 2010 to February 2017 education was measured with seven categories (excluding associate degree). Starting in February 2017 and continuing, education is measured using eight categories. These categories allow participants report less than 12 years, high school graduate, currently in college/university, some college/university, but did not graduate, associate degree (2yr), college/university degree (4yr), currently in graduate or professional school, or graduate or professional school degree. **Plans for Data Analysis** **Psychometric information.** ***Reliability index.*** For the analysis, we will report the overall Cronbach's alpha, omega hierarchical, and omega total (Revelle & Condon, in press; Revelle & Zinbarg, 2009; Zinbarg, Revelle, Yovel, & Li, 2005) for each ability subscale and Big Five personality trait. In addition to this, alpha and omega reliabilities will be reported for each year of collection to assist with understanding trends within the data. Given our interest in seeing how the data changed over the last 13 years, we not include cutoffs for these variables. ***Test-retest reliability.*** As the data in this study was collected cross-sectionally, we do not plan to examine or report test-retest reliability. ***Structural validity and measurement invariance.*** We plan to conduct a multi-group confirmatory factor analysis of ability based on the ability domains proposed by Condon and Revelle (2014). This analysis will be used to establish that the 60 ICAR items is a bifactor model composed of a general factor of ability four lower level factors. Similarly, the 35 ICAR items will be examined using a multi-group confirmatory factor analysis to confirm there is a bifactor model with a general factor of ability and three lower level factors; this additional analysis is conducted to allow us to look at the 35 ability items that have been over the 13 years of data collection. For the Big Five personality traits, a confirmatory factor analysis will be performed to examine the fit of the five factor items for both the 100-item IPIP (Goldberg, 1999) collected from 2006 to 2010 and the SPI-135 (Condon, 2018) collected from 2013 to 2018. Fit indices of CFI, TLI, and RMSEA will be used to evaluate our models. Evaluations of these models will be based on Brown and Cudeck’s (1992) thresholds. A good fit will be considered a RMSEA ≤ .05, CFI ≥ .95 and TLI ≥ .95. An adequate fit will be considered a RMSEA range from .06 to .10, a CFI from .90 to .94, and a TLI from .90 to .94. A poor fit will be considered a RMSEA >.10, CFI < .90, and TLI < .90. Measurement invariance will also be examined for both models of ability and temperament to establish configural, metric, and scaler invariance exists across gender, age, education, or geographic origin. We know from previous research, that the 16 item ICAR and 60 item ICAR norming sample were slightly biased towards men across all domains. Furthermore, age has a non-monotonic effect for all domains but verbal reasoning (Young & Keith, in review). In addition to looking at measurement invariance and structural validity using comparisons of first-order factor structure in multi-group confirmatory factor analysis, we will also fit a multiple indicator multiple cause (MIMIC) model to the data to examine the influence of age, gender, education, and geographic origin on the latent factors of ability and temperament for each year. By doing so, we aim to examine how the structure of participant responses changes based the year the survey was completed. Again, fit indices of CFI, TLI, and RMSEA will be used to evaluate our models based on Brown and Cudeck’s (1992) thresholds. ***Convergent/discriminant validity.*** In addition to reporting the confirmatory factor structure of the latent factors of ability and temperament, this study will calculate and report correlations between subscale (latent factor) items. Rather than reporting yearly correlation tables for each subscale, this study will report aggregate item correlations. As such, there will be one correlation table for matrix reasoning, one for verbal reasoning, two for letter and number series, one for three dimensional rotation, two for conscientiousness, two for agreeableness, one for neuroticism, one for emotional stability, one for openness to experience, one for intellect, and two for extraversion. Justification for some subscales reporting two correlations is due to a change in the measure being used for data collection on that latent factor. ***Predictive and criterion validity.*** As this study aims to provide a descriptive overview of changes and structure, we do not plan to report predictive validity. ***Construct validity.*** From previous research and norming studies, we expect that latent factors from the ICAR measures are measuring forms of cognitive ability. This includes moderate to strong correlations with cognitive ability and achievement tests such as combined SAT scores, composite ACT scores, and the Shipley-2 (Condon & Revelle, 2014). More recently, the latent model of ICAR measures have been shown to have a strong correlation (r=.955) with the WAIS-IV (Young, in prep). Additionally, after correcting for restriction of range and reliability, ICAR scores had strong correlations with general IQ (r= .815) and Full Scale IQ (r= .82). We also expect the latent factors from the International Personality Item Pool and the SAPA Personality Inventory to be measuring the Big Five personality traits. Specifically, Goldberg (1999) found the International Personality Item Pool strongly correlated with the Revised NEO Personality Inventory (r=.73; corrected r=.94). Because the 5 factor scales for the SPI was developed by pulling the top 14 items from the 696 non-ability personality items, we also expect that these scales are measuring the Big Five personality traits. As such, detailed construct validity can be found in the scales’ development (Condon, 2018) or by referencing the original scales used in the development: 100 item IPIP (Goldberg, 1999), 300 items from the IPIP-NEO (Goldberg, 1999), 240 items from the IPIP-HEXACO inventory (Ashton, Lee, & Goldberg, 2007), 100 items from the Big Five Aspect Scales (DeYoung, Quilty, & Peterson, 2007), 48 items from the Questionnaire Big Six scales (Thalmayer, Saucier, & Eigenhuis, 2011), and 79 items from the revised Eysenck Personality Questionnaire (exluding the lie scale; Eysenck, Eysenck, & Barrett, 1985). ***Type of Analysis.*** The study will examine how average ability scores and personality trait scores have changed across the 13 years of data collection. As such, this study will examine and compare year to year mean scores and standard errors for each dependent variable. In addition to this, we will examine year to year scores based on the participants’ geographic origin, age, gender, and education at the year they completed the ability or temperament item. For ability, this will result in mean test scores (by domain and overall) in relation to the year of testing. For temperament, this will result in mean test scores by Big Five personality trait. Rather than aggregating data by cohort, this analysis will allow us to compare participants at the same age across years to see how sampling has changed. For example, we could compare 18 and 19 year olds that took temperament or ability items in 2006 to 18 and 19 year olds that completed temperament or ability items in 2014. Beyond looking at these means, we will also report correlations between the item domains/traits (see section Convergent/discriminant validity) along with alpha and omega reliabilities (see section Reliability index). As we are interested in seeing if our items have performed similarly across the 13 years of data collection, this study will use multi-group confirmatory factor analysis to test for invariance in the ability domains and the Big Five personality traits (see section Structural validity and measurement invariance). In addition to this, we will use MIMIC models to fit the data to examine the influence of age, gender, education, and geographic origin on the latent factors of ability and temperament for each year. By doing so, we aim to examine how the structure of participant responses changes based the year the survey was completed. **Evaluation Criteria** **Cut-offs.** We will use the categorical descriptions and quantitative cutoffs specified above in the section Psychometric Information. All correlations will be interpreted according to their effect sizes (p values will not be used, as their results are negligible with this large of a sample). We will use the following categorical descriptions from Funder and Ozer (2019) to describe our correlations: negligible (r’s [.00, .049]), very small (r’s [.05, .09]), small (r’s [.10, .19]), moderate (r’s [.20, .29]), large (r’s [.30, .39]), and very large (r’s [.40, 1.0]). **Adjustments.** As we will not use p values as evaluation criteria in this study, correction for multiple comparisons does not apply. **Expectations.** Given the large sample size, we expect this study to be overpowered. **Syntax/code.** Because the lead/corresponding author does not have ownership over the SAPA-project, posting cleaning syntax will be dependent on the agreement and approval of the co-investigators of this project. If the cleaning syntax is posted, parts of the scoring syntax will be redacted. Specifically, we are unable to provide the scoring syntax for the ability items used in this analysis because it would reveal the correct answers for these items. Final code for the main analyses will be included in the supplement of the project. **Data exclusions.** Mentioned previously, data from the sample will exclude participants under the age of 16 and above the age of 90. Because our non-ability personality items are not intended to work with younger participants, we will remove individuals below the age of 16 before examining the data. Participants above the age of 90 will also be removed as they could be an outlier for our temperament or ability data. In addition to this, because we only collected gender using a drop-down menu using the options male, female, and prefer not to answer from April 2006 to February 2017, we will only use binary gender in our analysis. As such, all participants that indicated their gender as other starting in 2017 will have their response cleaned as NA. Any participants that responded prefer not to answer for any demographic questions will also have their response cleaned as NA. As this study also aims to examine changes in at both temperament and ability items, participants that did not complete any ability items will also be removed during the cleaning process. **Missing Data** The data collected by the SAPA-Project are massively missing completely at random, causing most of the data to be missing. Although missing data are frequently cited as a limitation, Revelle et al. (2016) has documented out lab methods in dealing with MMCAR data. Beyond these missing data, there is a possibility that SPI-135 analysis will need to begin in 2014 rather than 2013. Justification for this is that if collection on all of the SPI-135 items began too late in the year, that they may not be representative of the full year. If we choose to not examine the 2013 SPI-135 data, this information will be disclosed in the final manuscript. **Analyses Run With and Without Covariates** As many of the planned analyses do not require covariates, relevant variables will be contained in the models. **Follow-up Analyses** At this time, the authors are not expecting additional exploratory analyses for the 13 years of data. However, if MIMIC models are not informative regarding structure changes in ability or temperament scores, the authors would be interested in further exploring if college major causes participants to vary. In addition to this, given that the SAPA-project is presented in English, the authors would be interested in exploring how invariance or the structure of the results changes when examining geographic origin with the categories of the United States, primarily English-speaking countries, and the rest of the world. Other possible exploratory analyses include exploratory structural equation modeling approaches. All exploratory analyses will be presented in the paper as exploratory and will report the previously mentioned statistics. Some of the data used in this paper was used for an analysis looking for a Flynn Effect or reverse Flynn Effect in the 13 years of SAPA-Project ability data. Some findings of this study can be found in the section Overall data. Unlike this project, the project looking for cognitive change will use regression and only included adult participants (ages 18-90) from the United States. Because the results of this research are separate from what is reported in this paper, a separate pre-registration for this 13 year analysis paper was created to account for these analyses (see Looking for a Flynn Effect: Examining shifts in cognitive ability within the SAPA-Project at https://osf.io/kmgx8/). **Funding Sources** This research was supported in part by grant SMA-1419324 from the National Science Foundation to William Revelle and David Condon. **References** Ashton, M. C., Lee, K., & Goldberg, L. R. (2007). The IPIP-HEXACO scales: An alternative, public-domain measure of the personality constructs in the HEXACO model. *Personality and Individual Differences, 42*(8), 1515–1526. https://doi.org/10.1016/j.paid.2006.10.027 Browne, M. W., & Cudeck, R. (1992). Alternative Ways of Assessing Model Fit. *Sociological Methods & Research, 21*(2), 230–258. https://doi.org/10.1177/0049124192021002005 charlesmurray. (2017, August 28). Want to kill part of the day and also contribute to science? A full-scale personality inventory is waiting for you: Https://sapa-project.org/ [Twitter post]. Retrieved July 22, 2019, from https://twitter.com/charlesmurray/status/902230016859262977 Condon, D. M. (2018). The SAPA Personality Inventory: An empirically-derived, hierarchically-organized self-report personality assessment model. https://doi.org/10.31234/osf.io/sc4p9 Condon, D. M., & Revelle, W. (2014). The International Cognitive Ability Resource: Development and initial validation of a public-domain measure. *Intelligence, 43*, 52–64. Condon, D. M., & Revelle, W. (2015a). *Selected ICAR data from the SAPA-Project: Development and initial validation of a public-domain measure.* https://doi.org/10.7910/DVN/AD9RVY Condon, D. M., & Revelle, W. (2015b). *Selected personality data from the SAPA-Project: 08Dec2013 to 26Jul2014.* https://doi.org/10.7910/DVN/SD7SVE Condon, D. M., & Revelle, W. (2018). *Selected personality data from the SAPA-Project: 18aug2010 to 08dec2013.* https://doi.org/10.7910/DVN/C40K1U Condon, D. M., Roney, E., & Revelle, W. (2017a). A SAPA Project update: On the structure of phrased self-report personality items. *Journal of Open Psychology Data, 5*(1), 1–8. https://doi.org/10.5334/jopd.32 Condon, D. M., Roney, E., & Revelle, W. (2017b). *Selected personality data from the SAPA-Project: 22Dec2015 to 07Feb2017.* https://doi.org/10.7910/DVN/TZJGAT Condon, D. M., Roney, E., & Revelle, W. (2017c). *Selected personality data from the SAPA-Project: 26Jul2014 to 22Dec2015.* https://doi.org/10.7910/DVN/GU70EV DeYoung, C. G., Quilty, L. C., & Peterson, J. B. (2007). Between facets and domains: 10 aspects of the Big Five. *Journal of Personality and Social Psychology, 93*(5), 880–896. https://doi.org/10.1037/0022-3514.93.5.880 Digman, J. M. (1990). Personality Structure: Emergence of the Five-Factor Model. *Annual Review of Psychology, 41*(1), 417–440. https://doi.org/10.1146/annurev.ps.41.020190.002221 Dworak, E. M., Revelle, W., & Condon, D. M. (2019a, June). *Coping behavior profiles across personality items.* Presented at the Association for Research of Personality, Grand Rapids, MI. Retrieved from https://sapa-project.org/emd/presentations/dworak_arp19.pdf Dworak, E. M., Revelle, W., & Condon, D. M. (2019b, July). *Looking for a Flynn Effect: Examining shifts in cognitive ability within the SAPA project.* Presented at the International Society for Intelligence Research, Minneapolis, MN. Retrieved from https://sapa-project.org/emd/presentations/dworak_isir19.pdf Eysenck, S. B. G., Eysenck, H. J., & Barrett, P. (1985). A revised version of the psychoticism scale. *Personality and Individual Differences, 6*(1), 21–29. https://doi.org/10.1016/0191-8869(85)90026-1 Funder, D. C., & Ozer, D. J. (2019). Evaluating Effect Size in Psychological Research: Sense and Nonsense. *Advances in Methods and Practices in Psychological Science, 2*(2), 156–168. https://doi.org/10.1177/2515245919847202 Goldberg, L. R. (1990). An alternative “description of personality”: The Big-Five factor structure. *Journal of Personality and Social Psychology, 59*(6), 1216–1229. https://doi.org/10.1037/0022-3514.59.6.1216 Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), *Personality Psychology in Europe* (Vol. 7, pp. 7–28). Tilburg, The Netherlands: Tilburg University Press. Guarino, B. (2018, September 17). Scientists identify four personality types. Retrieved July 22, 2019, from The Washington Post website: https://www.washingtonpost.com/science/2018/09/17/scientists-identify-four-personality-types/?noredirect=on&utm_term=.2d12d0ef3c14 Knowlton, E. M., Revelle, W., & Condon, D. M. (2018, July). *Personality and coping: The best personality trait predictors of coping strategies.* Presented at the European Conferences on Personality, Zadar, Croatia. Retrieved from https://sapa-project.org/emd/presentations/knowlton_ecp18.pdf Reddit. (n.d.). Sapa-project.org on reddit.com. Retrieved July 22, 2019, from https://www.reddit.com/domain/sapa-project.org/ Revelle, W., & Condon, D. M. (in press). Reliability from α to ω: A Tutorial. In *Psychological Assessment*. Retrieved from http://personality-project.orgwww.personality-project.org/revelle/publications/rc.pa.19.pdf Revelle, W., Condon, D. M., Wilt, J., French, J. A., Brown, A., & Elleman, L. G. (2017). Web and phone based data collection using planned missing designs. In N. G. Fielding, R. M. Lee, & G. Blank (Eds.), *Handbook of Online Research Methods*. (pp. 578–594). https://doi.org/10.4135/9781473957992.n33 Revelle, W., & Zinbarg, R. E. (2009). Coefficients Alpha, Beta, Omega, and the glb: Comments on Sijtsma. *Psychometrika, 74*(1), 145–154. https://doi.org/10.1007/s11336-008-9102-z Thalmayer, A. G., Saucier, G., & Eigenhuis, A. (2011). Comparative validity of brief to medium-length Big Five and Big Six Personality Questionnaires. *Psychological Assessment, 23*(4), 995–1009. https://doi.org/10.1037/a0024165 Young, S. R. (in prep). Valid cognitive ability measures in the public domain: A convergent validity study of the ICAR16 using the WAIS-IV. Young, S. R., & Keith, T. Z. (in review). Age and sex invariance of the International Cognitive Ability Resource. Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach’s α, Revelle’s β, and Mcdonald’s ωH: Their relations with each other and two alternative conceptualizations of reliability. *Psychometrika, 70*(1), 123–133. https://doi.org/10.1007/s11336-003-0974-7
OSF does not support the use of Internet Explorer. For optimal performance, please switch to another browser.
Accept
This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.
Accept
×

Start managing your projects on the OSF today.

Free and easy to use, the Open Science Framework supports the entire research lifecycle: planning, execution, reporting, archiving, and discovery.