**Introduction** ------------ This multimodal dataset was acquired in a controlled experiment on a driving simulator (see Ref. [1]). The set includes data for $n=68$ volunteers that drove the same highway under four different conditions: No distraction, cognitive distraction, emotional distraction, and sensorimotor distraction. The experiment closed with a special driving session, where all subjects experienced a startle stimulus in the form of unintended acceleration - half of them under a mixed distraction, and the other half in the absence of a distraction. During the experimental drives key response variables and several explanatory variables were continuously recorded. The response variables included speed, acceleration, brake force, steering, and lane position signals, while the explanatory variables included perinasal electrodermal activity (EDA), palm EDA, heart rate, breathing rate, and facial expression signals; biographical and psychometric covariates as well as eye tracking data were also obtained. This dataset enables research into driving behaviors under neatly abstracted distracting stressors, which account for many car crashes. The set can also be used in physiological channel benchmarking and multispectral face recognition **Data Records** ------------ The repository's data is organized per subject under three major directories: (1) Raw Thermal Data - 1.54 TB in size. (2) Structured Study Data - 57.5 GB in size. (3) R-Friendly Study Data - 40.1 MB in size. In these directories, the subject folders are named Txxx, where xxx stands for the subject number. At the repository's root, a spreadsheet named \`Dataset-Table-Index.xlsx\' gives an exhaustive enumeration of the dataset's files in the Raw Thermal Data and Structured Study Data directories. Files expected to be present and are present indeed in these two directories, are denoted by **1**; there are 4,960 such files. Files expected to be present, which are not present for technical and other reasons, are denoted by **0**; there are 236 such cases. Files that are not supposed to be present due to the experimental design are denoted by **NA**; there are 544 such cases. Files that have been redacted due to IRB restrictions, are denoted by **IRB**; there are 40 such cases. Files associated with derivative thermal variables, such as perinasal EDA, which could not be extracted due to the presence of facial hair, are denoted by **N**; there are 144 such cases. Files that are present, but found during the technical validation process to be marred by noise, are denoted by **-1**; there are 60 such files and we do not recommend using their data. For convenience of access, the \`Dataset-Table-Index.xlsx\' spreadsheet also contains designators about the gender and age group of each subject, although this demographic information is nested within the Structured Study Data directory. The R-Friendly Study Data directory is a reformatting of the quantitative variables contained in the Structured Study Data directory. It contains, however, one variable that is not present in the Structured Study Data directory due to its late curation. This variable is the eye tracking variable, and the `Dataset-Table-Index.xlsx' spreadsheet lists its present and missing data in a specially highlighted column. Raw Thermal Data -------------------------- In the Raw Thermal Data directory, each subject's folder contains the facial thermal sequences for all experimental sessions. These binary files are named `Txxx-SESSION\_CODE.dat', where xxx stands for the subject number and SESSION CODE holds the acronym of the experimental session, that is, B for the Baseline, PD for the Practice Drive, RD for the Relaxing Drive, ND for the Normal Drive, CD for the Cognitive Drive, ED for the Emotional Drive, MD for the Sensorimotor Drive, and FDL or FDN for the Failure Drive. Each .dat file is accompanied by an .inf file in text format. The header of each .inf file has three numbers: The first number denotes the number of thermal frames contained in the corresponding .dat file; the second number denotes the width of each thermal frame; and, the third number denotes the height of each thermal frame. The body of each .inf file contains the timestamps of all thermal frames contained in the corresponding .dat file. The .dat files can be accessed via the S-Interface (Data Citation [D1]), which uses the information in the corresponding .inf files to properly open .dat files and process them. Structured Study Data ------------------------------- In the Structured Study Data directory, each subject's folder has three files containing biographic and psychometric data; they are colocated with subfolders containing data corresponding to experimental sessions,thus forming a hierarchical data sturcture reflective of the experimental design. Specifically: **Txxx.b: Biographic Data - Session Independent.** This Excel file contains Gender {Male,Female} information; Age {Integer} information; and, Age Group {Old,Young} information. **Txxx.bar: Psychometric Data - All Sessions.** This Excel file contains the NASA TLX scores the subject provided after each experimental session, following the Relaxing Drive (RD). NASA TLX is a multi-dimensional psychometric, measuring perceived Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration. **Txxx.tp: Trend Psychometric Data - Session Independent.** This Excel file contains the (State-) Trait Anxiety Inventory (STAI) score and the Type A/B Personality score. The subfolders containing data pertaining to experimental sessions (treatments), are named `Y SESSION\_CODE', where Y stands for the order of the session in the experimental timeline and SESSION\_CODE holds the acronym of the session. The Baseline [BL], Practice Drive [PD], Relaxing Drive [RD], and Failure Drive - either Loaded [FDL] or Nonloaded [FDN], have constant order across subjects: BL$\rightarrow$1, PD$\rightarrow$2, RD$\rightarrow$3, FDL or FDN$\rightarrow$8. The loaded drives, that is, Normal Drive [ND], Cognitive Drive [CD], Emotional Drive [ED], and Sensorimotor Drive [MD] have randomized order, blocked by gender and age group; thus, their order, drawing from the set $[4\dots7]$, differs from subject to subject. Each session folder contains up to nine files holding measurements from different modalities, and a stimulus file, where needed, holding information about the stressors/events that applied/took place during the session. Specifically, the stimulus file is named `Txxx-zzz.stm', where xxx is the subject number and zzz is the session's order number. Each stressor/event is specified in a separate row that lists its: Start Time [in s], End Time [in s], and Type [Other $\equiv$ 0, Analytical Questions $\equiv$ 1, Mathematical Questions $\equiv$ 2, Emotional Questions $\equiv$ 3, Texting $\equiv$ 4, Texting and Talking $\equiv$ 5, Failure Event $\equiv$ 6]. The multimodal measurements for each experimental session are held in files named Txxx-zzz.MEASUREMENT\_CODE, where xxx is the subject number, zzz is the session's order number, and MEASUREMENT\_CODE refers to the measurement channel. Specifically: **Txxx-zzz.BR: Breathing Rate Signal.** This Excel file contains three synced columns: Frame #; Time; and, Breathing Rate signal. Hence, scanning each row from left to right we find the chronological rank order of the instantaneous measurement, the time [in s] the measurement was taken with respect to the beginning of the session, and the value of the breathing rate signal at that time [in bpm]. No breathing rate measurements were recorded during the baseline (B) sessions. **Txxx-zzz.HR: Heart Rate Signal.** This Excel file contains three synced columns: Frame #; Time; and Heart Rate signal. Hence, scanning each row from left to right we find the chronological rank order of the instantaneous measurement, the time [in s] the measurement was taken with respect to the beginning of the session, and the value of the heart rate signal at that time [in bpm]. No heart rate measurements were recorded during the Baseline (B) sessions. **Txxx-zzz.peda: Palm EDA Signal.** This Excel file contains three synced columns: Frame #; Time; and, Palm EDA signal. Hence, scanning each row from left to right we find the chronological rank order of the instantaneous measurement, the time [in s] the measurement was taken with respect to the beginning of the session, and the value of the palm EDA signal at that time [in ${\text k}{\Omega}$]. No palm EDA measurements were recorded during the Baseline (B) sessions. **Txxx-zzz.pp: Perinasal EDA Signal.** This Excel file contains four synced columns: Frame #; Time; Perinasal EDA signal; and, Noise Reduction (NR) Perinasal EDA signal. Hence, scanning each row from left to right we find the chronological rank order of the instantaneous measurement, the time [in s] the measurement was extracted with respect to the beginning of the session, the value of the perinasal EDA signal at that time [in $^{\circ}\text{C}^2$], and its smoothed over value [also in $^{\circ}\text{C}^2$]. Perinasal EDA measurements were the only physiological measurements recorded during the Baseline (B) sessions. **Txxx-zzz.avi2: Perinasal Region of Interest Thermal Video.** This avi file is the thermal video recording of the perinasal region of interest (ROI) during session zzz. The S-Interface extracts this ROI video out of the raw thermal facial imagery, thanks to the tracker reported by Zhou et al. (see Ref. [2]). It is upon this tracked ROI that the physiological signal extractor reported by Shastri et al. (see Ref. [3]) operates, yielding the perinasal EDA signal. **Txxx-zzz.res: Performance Response Variables.** This Excel file contains 7 synced columns: Frame #; Time; Speed signal; Acceleration signal; Brake Force signal; Steering signal; and, Lane Position signal. Hence, scanning each row from left to right we find the chronological rank order of the instantaneous measurement, the time [in s] the composite measurement was taken with respect to the beginning of the session; and, the values at that time of speed [in km/h], acceleration [in $^\circ$], brake force [in N], steering [in rad], and lane position [in m]. **Txxx-zzz.avi3: Operational Theater Video.** This avi file is the first-person video recording of what the driver sees during the session zzz. The green dot identifies the direction of the driver's gaze, superimposed on the video by the eye-tracker. **Txxx-zzz.FACS: FACS Signals.** This Excel file contains 10 synced columns: Frame #; Time; Anger signal; Contempt signal; Disgust signal; Fear signal; Joy signal; Sad signal; Surprise signal; and, Neutral signal. Hence, scanning each row from left to right we find the chronological rank order of the instantaneous measurement, the time [in s] the composite measurement was taken with respect to the beginning of the session; and, the normalized values of the composite FACS vector at that time, extracted via the CERT software. These normalized values indicate levels of anger, contempt, disgust, fear, joy, sad, surprise, and neutral feelings, respectively. **Txxx-zzz.avi1: Facial Video.** This avi file is the facial video recording during session zzz. It is upon this video that the CERT software operates to extract the FACS signals. R-Firendly Study Data --------------------- The R-Friendly Study Data directory has a copy of the physiological and performance signals in a flat format, and features some additional data, too. In the flat format, all signals for a subject are arranged column-wise in a single file (no session subdirectories). Each such subject file is enhanced by the addition of columns bearing traveled distance [in m] and explicit eye tracking data (gaze x position, gaze y position, left pupil diameter, right pupil diameter). Distance and explicit eye tracking data do not exist in the original hierarchical data structure saved in the Structured Study Data directory; their curation was an afterthought. The R-Friendly Study Data directory allows easy addition of such quantitative variables to the dataset during post-study work. **References** ---------- [1] Pavlidis, I., M. Dcosta, S. Taamneh, M. Manser, T. Ferris, R. Wunderlich, E. Akleman, and P. Tsiamyrtzis. "Dissecting driver behaviors under cognitive, emotional, sensorimotor, and mixed stressors." *Scientific Reports* 6 (2016). [2] Zhou, Yan, Panagiotis Tsiamyrtzis, Peggy Lindner, Ilya Timofeyev, and Ioannis Pavlidis. "Spatiotemporal smoothing as a basis for facial tissue tracking in thermal imaging." *IEEE Transactions on Biomedical Engineering* 60, no. 5 (2013): 1280-1289. [3] Shastri, Dvijesh, Manos Papadakis, Panagiotis Tsiamyrtzis, Barbara Bass, and Ioannis Pavlidis. "Perinasal imaging of physiological stress and its affective potential." *IEEE Transactions on Affective Computing* 3, no. 3 (2012): 366-378. Data Citations -------------- [D1] Buddharaju, P., Khatri, A., and Pavlidis, I. **S-Interface (formerly OTACS)** Dataset accession number or DOI (2016): [https://figshare.com/articles/OTACS_Software/4244273/4].