# **Toadstool: A Dataset for Training Emotional Intelligent Machines Playing Super Mario Bros** # ### **[[preprint]][1]** **[ \[github\]][2]** ### ![enter image description here][3] In this dataset, we present a dataset called **Toadstool** that aims to contribute to the field of reinforcement learning, multimodal data fusion, and the possibility of exploring emotionally aware machine learning algorithms. Furthermore, the dataset can also be useful to researchers interested in facial expressions, biometric sensors,sentiment analysis, and game studies. The dataset consists of video, sensor, and demographic data collected from ten participants playing Super Mario Bros. The sensor data is collected through an Empatica E4 wristband, which provides high-quality measurements and is graded as a medical device. We believe that the presented dataset can be interesting for a manifold of researchers to explore different exciting questions. ### Folder structure and data formats ### The structure of the main folder: - [Main directory] - **participants [directory]**: contains the information of each participant. This includes the video of them playing, the controller input of each game frame, and the Empatica E4 wristband sensor data. - **scripts [directory]**: holds a set of Python scripts meant to aid the user in getting an easy start to using the dataset. The files include a script for replaying gameplay using the provided controller inputs, a script for matching the gameplay session to the facial expression video, and a script for matching the raw signal outputs to the gameplay session. - **protocol.pdf**: is the protocol used to collect the video game session data. - **questionnaire.pdf**: is the questionnaire that was filled out by each participant before starting the game session. - **questionnaire_answers.csv**: is a summary of all the an-swers to the questionnaire. - **consent.pdf**: is the consent form that was signed by each participant. - **LICENSE**: is the file that signifies which license in which the dataset is distributed under. Each participant directory holds the sensor data collected from the Empatica E4 wristband, a JSON file containing information about the participant's game session, the video recording of the participant playing the game stored in ".avi" format, and another JSON file which holds some information about the facial expression video. The sensor data is stored in another directory within each participant; whichs contents are listed below. - **ACC.csv**: contains the data collected from the 3-axis accelerometer sensor in the range [-2g, 2g] sampled at 32 Hz. The accelerometer measures the movement of the wearer. - **EDA.csv**: holds the data collected by the EDA sensor sampled at 4 Hz. EDA measures the electrical conductivity of the skin and measurements have been proven to be correlated with emotions since the late 1800s. EDA is also sometimes called psychogalvanic reflex or skin conductance. - **BVP.csv**: contains the data collected from the photoplethysmograph sensor, which measures the BVP, and is sampled at 64 Hz. - **IBI.csv**: stores the interbeat intervals (IBI). The IBI measures the time interval between individual heartbeats and can be used to estimate the instantaneous heart rate as well as heart rate variability. The wristband calculates the values contained within this file based on the BVP signals. - **HR.csv**: contains the average heart rate values, computed in spans of 10 seconds. The heart rate measures the number of times a person’s heartbeats per minute. Similar to the IBI, these values are calculated based on the BVP signal. - **TEMP.csv**: holds the information collected by the thermome-ter, which is the temperature of the person playing the game expressed in degrees Celsius (°C) sampled at 4Hz. - **info.txt**: gives a brief description of all variables collected by the wristband. ### Term of use ### The license for the *Toadstool* dataset is Attribution-NonCommercial 4.0 International. More information can be found here: https://creativecommons.org/licenses/by-nc/4.0/legalcode [1]: https://osf.io/4v9mp [2]: https://github.com/simula/toadstool [3]: https://files.osf.io/v1/resources/qrkcf/providers/osfstorage/5e578160ef5d89007406a2b6?mode=render