# Repository contents ## Subject directories Contain the video recordings and dynamic caricatures for each of the three subjects. For convenience, directories are provided as zip files. Each subject directory contains the following: * `cropped_reference.png` - The reference image used by the McGM pipeline. For convenience, this is placed in the top-level of the subject directory, but in practice the scripts would create it nested within further sub-directories. * `recordings` - Sub-directory contains the video clips used as inputs to the McGM/PCA pipeline. Due to the extremely large file sizes of the original recordings, we only include the clips after they have already been cut to the prefix portion of each phrase and cropped to square (essentially skipping the video pre-processing steps - see more detailed notes on workflow below). The videos provided should be sufficient to run the main McGM and PCA processing steps. Each directory contains the following: * *cropped_clips* - Directory containing the video files, stored as MP4 files with lossless compression. * *clip_attrs.csv* - Gives details of each clip, including the phrase type ("Good news" or "I'm sorry to say"), and the phrase number and original content. * `caricatures` - Sub-directory contains the dynamic caricatures derived from the second-order PCA. To keep the file sizes reasonable, these videos have been put through a slight (but lossy) MP4 compression, whereas the ones actually produced by the scripts are uncompressed AVI files. The original uncompressed caricatures can be generated by running the analysis pipeline on the provided recordings. Also, for convenience, this directory is placed in the top-level of the subject directory, but in practice the scripts would create it nested within further sub-directories. Each directory contains the following: * *[gn/ists]-videos* - Sub-directories for the "Good news" and "I'm sorry to say" videos respectively. * Each of those then contain further sub-directories for each clip (named according to the clip number). * Finally, each clip sub-directory contains the caricatured videos for that clip, named according to the caricature level (e.g. the 0.5 anti-caricature is shown in the `0.50.mp4` file, the -0.25 anti-face anti-caricature is shown in the `-0.25.mp4` file, etc.). For visualisation purposes, video montages of all the caricature levels are also provided, both at full- and half-speed playback. ## Scripts Contains all of the code needed to reproduce the full analysis pipeline. The scripts below are listed in (appproximate) order of the workflow. ### preproc ***Video pre-processing*** \ Performed first. Outputs variously passed on to audio pre-processing and McGM/PCA pipelines. > Some of the video pre-processing scripts operate on the original recording > files, which are not provided here due to the large file sizes. These scripts > will not work, but are still provided for transparency. * `extract_ELAN_timestamps.py` *(won't run - inputs not provided)* - An initial rough cut of each clip from the original recordings was created by manually extracting the timestamps using ELAN. This script extracts those timestamps from the xml files and saves them to CSV, so they can be used by subsequent scripts to create more precise cuttings. * `gen_attrs_files.py` *(won't run - inputs not provided)* - Generates the *clip_attrs.csv* files giving details of each clip. * `extract_clip_transcripts.py` *(won't run - inputs not provided)* - Uses *Google's Cloud API speech recognition* algorithm to extract transcripts and timestamps for each word in each phrase. Outputs are used as basis for cutting prefix portions of each phrase. * `cut_clips.py` *(won't run - inputs not provided)* - Uses the transcript timestamps to cut the clips to the prefix portion of each phrase. This process is somewhat error prone, and manual corrections are often necessary... * `correct_cut_clips.py` *(won't run - inputs not provided)* - Applies manual corrections to the cutting of the prefix clips. Requires timestamp CSV files to be manually edited to correct the timestamps, then this script will re-cut any clips marked for correction accordingly. * `crop+rescale_clips.py` *(won't run - inputs not provided)* - Crops each clip to a square bounding box centred on the face, then rescales to a resolution of 128x128 pixels. The outputs are used by the McGM pipeline. The outputs are provided in this repository, in the *cropped_clips* sub-directories within each subject's *recordings* directory. * `extract_reference_frames.py` *(won't run - inputs not provided)* - Extracts reference image from original recordings, then crops and rescales it using the same process as above. The outputs are used by the McGM pipeline. The outputs are provided in this repository, within each subject's directory (albeit not with the right directory structure). * `clips2mat.py` - Takes the cropped and resized clips and exports the frames to a MAT file. This is a workaround for an issue with the MATLAB install on our cluster which prevents it from reading the uncompressed videos directly. ***Audio pre-processing*** \ Performed after video-preprocessing. Outputs form the inputs to the dynamic time warping. * `extract_audio.py` - Extracts the audio data (raw waveforms and amplitude envelopes) from the clips and saves them to HDF5 files. Technically this won't run as it's configured to use the video files before being cropped and rescaled (produced by *\[correct_\]cut_clips.py*) which are not provided here, but it should give the same results if it were run on the cropped and rescaled clips (produced by *crop+rescale_clips.py*) which are provided. * `cut_audio_to_onsets.py` - The cut clips still include a brief period before the onset of each vocalisation, so as to include any facial movements commencing before the vocalisation. However, as there is little audio during this period, the time warping is unlikely to perform well. This script re-cuts the the audio streams to start closer to the onset of the vocalisations. Outputs are appended to the HDF5 files. This process is somewhat error prone, and manual corrections are necessary... * `correct_audio_onsets.py` - Provides a simple GUI for applying manual corrections to the re-cutting of the audio onsets. Plots the amplitude envelope, and the user may click to indicate the corrected onset. Script will then re-cut the audio streams accordingly, and update the HDF5 files. ### MCGM Code for running the McGM, registering each input frame to the reference image. * `do_MCGM.m` - Runs the McGM registration. * `mcgm2` - Contains source and compiled code for the McGM itself. For convenience, this is provided as a zip file. ### DTW Code for running dynamic time warping on audio streams, computing temporal alignments between clips. * `do_DTW.py` - Runs the dynamic time warping. Saves the warping curves for the audio streams, but also downsamples these to be used for the video streams. ### PCA Code for running first- and second-order PCAs, and creating dyanmic caricatures. * `do_PCA1.m` - Runs first-order PCA on McGM outputs. * `backproj_PCA1.m` - Samples points along first few components of the PCA₁ space and back-projects to image space for visualisation. * `dtw_interpolate_PCA1.m` - Interpolates PCA₁ scores between frames, based on the dynamic time warping curves computed on the audio streams (see above). After this, PCA₁ timeseries will be the same length for each clip, and should be temporally aligned over clips. * `average_dtw_interpolated_PCA1.m` - Averages temporally aligned PCA₁ timeseries over clips. Saves those out, and also back-projects them to the image space for visualisation. * `do_PCA2.m` - Runs second-order PCA on temporally aligned PCA₁ scores. * `backproj_PCA2.m` - Samples points along first few components of PCA₂ space and back-projects to image space for visualisation. * `caricature_PCA2.m` - Generates dynamic caricatures from PCA₂ space and back-projects them to the image space. Lightly compressed versions of the outputs are provided in the *caricatures* sub-directory within each subject directory (albeit not with the right directory structure). ## Ratings Experiment Contains materials, data, and scripts for the perceptual ratings experiment: * `data` - Directory contains the raw data files from each rater. * `stats` - Directory contains statistical outputs produced by analysis script (see *do_analysis.R* script). * `set1` - Directory contains stimuli, condtions list, and PsychoPy experiment file for running the experiment. There would be one directory for each of the 10 image sets, but for simplicity we only include the first one. Directories for the other sets would be identical except for the stimulus lists (the *copy_videos.py* script can generate the remaining stimulus lists). * `compress_caricatures_PCA2.ps1` - Script is used to generate compressed versions of the caricature videos, suitable for use in an online experiment. See the *caricature_PCA2.m* script in the main scripts directory for the code used to generate the original uncompressed video files. * `copy_videos.py` - Script copies the compressed videos into the directory structure used for the experiment and generates the corresponding stimulus list files. * `do_analysis.R` - Script runs statistical analysis of data. Tests effect of caricaturing on ratings via an ordinal logistic regression.