# Repository Contents - **ratings_data** : Logfiles for behavioural ratings experiment. - **recordings** : Clips for the 3 recording subjects used as inputs to the MCGM and FACS-based model pipelines. Due to the extremely large file sizes of the original recording files, we only include the clips after they have already been cut to time and cropped to square ready for analysis (essentially skipping pre-processing steps - see more detailed notes on workflow) - **scripts** : Scripts for analysis of video clips and of the ratings experiment (see more detailed notes on workflow). ---------- # Detailed structure ## Recordings Descending from top-level directory: - **S[1-3]** : Subject directories - ***cropped_[prefix/suffix]_clips_MCGM*** : Directory contains clips used as inputs to the MCGM processing pipeline. Clips have been cut to the prefix/suffix timestamps, and cropped to a square bounding box centred on the face, then downsampled to 128x128 resolution. *bbox_info.hdf5* file contains details of the bounding boxes used to crop the frames. - ***cropped_[prefix/suffix]_clips_FACS*** : Directory contains clips used as inputs to the FACS-based processing pipeline. Clips have been cut to the prefix/suffix timestamps, and cropped to square, but left at full resolution (1080p). - ***MCGM_reference*** : Directory contains the starting reference frame used for the MCGM pipeline (both the original frame, and the frame after cropping to square and down-sampling to 128x128). *bbox_info.hdf5* contains details on the bounding box used to crop the frame. - ***clip_attrs.csv*** : Contains details on the class membership and phrase details of each clip. ## Scripts Details are listed in (approximate) order of the workflow. ---------- - **preproc** : Pre-processing scripts mostly handle clipping the prefix/suffix phrases from the original recordings. Due to extremely large file sizes, the original recordings are not provided, so some of these scripts will not run as they are missing their inputs. - ***gen_attr_files.py*** : Creates clip_attrs.csv files. - ***extract_prefix_clips.py*** *(won't run, inputs not provided)* : Uses a semi-automated procedure to extract prefix portions of each phrase based on analysis of audio spectrogram. - ***extract_suffix_clips.py*** *(won't run, inputs not provided)* : Uses a semi-automated procedure to extract suffix portions of each phrase by using Google Cloud Speech-To-Text to generate a transcript and timestamps for each word. - ***correct_extracted_clips.py*** *(won't run, inputs not provided)* : Applies manual corrections to cutting of clips where necessary. ---------- - **MCGM** : Scripts for running full MCGM analysis pipeline. - ***crop+rescale_clips.py*** *(won't run, inputs not provided)* : Takes clips extracted in pre-processing and crops and downscales them for input to MCGM pipeline. Uses a face detection algorithm to place a square bounding box centred on the face in each clip, then downsamples to 128x128 resolution. Outputs are in the *cropped_[prefix/suffix]_clips_MCGM* directories in the *recordings* directory. - ***extract_reference_frames.py*** *(won't run, inputs not provided)* : Extracts reference frames from original recordings, used as initial reference images for MCGM calculation. Uses same face detection process as above to crop frame to bounding box centred on the face, and downsamples frame. Outputs are in the *MCGM_reference* directories in the *recordings* directory. - ***clips2mat.py*** : Extracts frames from video files and stores them in a MATLAB .mat file. This is due to limitations of the MATLAB install on our cluster which leaves it unable to read the video files directly. To save on storage the .mat files are not included here, but can be generated using this script. - ***[xsubj_]do_MCGM_warps.m*** : Uses MCGM model to calculate warp fields and generate warped textures. Note - script is intended to leverage the parallel processing capabilities of a cluster computer, but can be run locally if the number of requested workers is reduced. do_MCGM_warps.m and xsubj_do_MCGM_warps.m are used for the within- and cross-subject analysis pipelines respectively. - ***[xsubj_]do_MCGM_PCA.m*** : Performs principal components decomposition of MCGM feature space. Runs PCA on both full dataset and within cross-validation scheme. do_MCGM_PCA.m and xsubj_do_MCGM_PCA.m are used for the within- and cross-subject analysis pipelines respectively. - ***[xsubj_]do_LDA_xval.m*** : Determines cross-validated classification accuracy using LDA classifier. do_LDA_xval.m and xsubj_do_LDA_xval.m are used for the within- and cross-subject analysis pipelines respectively. - ***do_LDA_projection.m*** : Fits LDA to full dataset and samples images along discriminant dimension. Only applies to within-subject pipeline. Outputs are used as inputs to the FACS based analysis of the LDA projection, and also provide the stimuli for the ratings experiment. - ***extract_LDA_proj_FACS_features.ps1*** : Extracts FACS features (pose and AU intensities) for the images projected along the LDA discriminant dimension. Note that this requires the [OpenFace toolbox][1]. - ***analyse_LDA_proj_FACS_features.R*** : Takes extracted FACS features for LDA projection and plots them against position along the discriminant dimension. ---------- - **FACS** : Scripts for running full FACS analysis pipeline. - ***crop_clips.py*** *(won't run, inputs not provided)* : Takes clips extracted in pre-processing and crops them to the central square of the frame. Frames are left at their original resolution. Outputs are in the *cropped_[prefix/suffix]_clips_FACS* directories in the *recordings* directory. - ***do_feature_extraction.ps1*** : Extracts FACS-based features (pose and AU intensities) for each frame across clips. Note that this requires the [OpenFace toolbox][1]. - ***[xsubj_]do_LDA_xval.m*** : Determines cross-validated classification accuracy using LDA classifier. do_LDA_xval.m and xsubj_do_LDA_xval.m are used for the within- and cross-subject analysis pipelines respectively. ---------- - **do_decodingAccStats.R** : Runs statistics on cross-validated LDA classification accuracies from MCGM and FACS pipelines. Tests accuracies against chance and performs contrasts between models. ---------- - **rating_experiment** : Experiment materials and analysis code for behavioural ratings experiment. - ***runExperiment.psyexp, conditions.csv, PNG files*** : PsychoPy experiment file and associated stimulus materials for running experiment. Data files are in the *ratings_data* directory in top-level of repository. - ***doStats.R*** : Script runs analysis of data. Correlates ratings against position along discriminant dimension and checks inter-rater reliability. ---------- - **matlab_utils** : Includes the MCGM toolbox and assorted other MATLAB utility scripts used by various other processing stages. ---------- [1]: https://github.com/TadasBaltrusaitis/OpenFace