Data Preprocessing & Analysis

This page describes the steps for preprocessing and analyzing the data for this project.

Preprocessing

Follow these simple steps to preprocess data:

1. If you do not have Python v3 installed locally, download Anaconda and follow these instructions to setup a Python v3 environment.

2. Download the entire directory to your local hard disk

3. Unzip the contents of the directory.

4. The final .csv file will be outputted into the cleanedData directory, which is empty at first. You can also save intermediate files at each preprocessing stage by modifying the script as shown below.

   saveIntermed <- TRUE
#saveIntermed <- FALSE

    setwd("~/Downloads/mec_osfPipeline/Rscripts") #or wherever you saved your directory
    source(1_mec1_clean_step1.R)
    
(This script should take  ~15 minutes to run)

5. Run Spectral_encoding_trim.py from the Dictionaries/moral_emo folder. For example, open Gitbash for Windows and run the following.

source activate py35
cd ~/mec_osfPipeline/dictionaries/moral_emo
python Spectral_encoding_trim.py

(This script should take ~40 minutes to run)

Tip: you can run steps 5-7 in separate command line windows to run in parallel

6. Run Spectral_encoding_trim.py from the Dictionaries/moral_emo_pos folder. For example, open Gitbash for Windows and run the following.

source activate py35
cd ~/mec_osfPipeline/dictionaries/moral_emo_pos
python Spectral_encoding_trim.py

(This script should take ~40 minutes to run)

7. Run Spectral_encoding_trim.py from the Dictionaries/moral_emo_neg folder. For example, open Gitbash for Windows and run the following.

source activate py35
cd ~/mec_osfPipeline/dictionaries/moral_emo_neg
python Spectral_encoding_trim.py

(This script should take ~40 minutes to run)

8. Source the 2_mec1_clean_step2.R file located in Rscripts. Or type these commands into R or RStudio.

   setwd("~/Downloads/mec_osfPipeline/Rscripts") #or wherever you saved your directory
   source(2_mec1_clean_step2.R)
   

   (This script should take ~30 seconds to run)

Analysis

You can use SAS or R to explore and analyze the preprocessed files (e.g., MEC_SASpreproc_Marriage.csv) located in the Cleaned Data folder.

However, if you want to follow along with our analysis scripts, be sure to check out the SAS Scripts folder, and use the scripts in there. Be sure to use the data files in the SAS scripts folder to prevent any import or incorrect variable name errors (e.g., use All.sas together with MEC_SASpreproc_all_updated)

The data files in the Cleaned Data folder are identical with the exception of some variable naming and removal of "NA"s for missing data that can cause errors in the SAS data import procedure.