Analysis Approach
We split the analyses into two different sections: confirmatory (in which we examined whether we replicated the original effect) and exploratory (where we explored whether other models provided a better fit for the effect, how other body parts respond in terms of peripheral temperature to social exclusion [vs. inclusion], and whether some of the effects are moderated by various social variables or climate).
Data preparation
After our data import, we excluded finger temperatures that are certainly considered extreme (i.e., below 0 degrees and above 50 degrees Celsius). In addition, we only kept temperature data in if they had a valid timestamp. We left missing data as missing. For our participants, we excluded them if they had answered less than 80% of the questions. Participants who failed attention check item (“To indicate that you read this item carefully, please mark the neutral rating”) included in the Social Thermoregulation protocol were excluded from further analysis.
Confirmatory analyses - focal replication effect
To explore the main effect, we followed the analytical strategy as in the original study. In the original study, the authors conducted a growth model obtained as a general mixed-model by predicting participants’ peripheral temperature on the index finger of the non-dominant hand as a dependent variable with the time of measurement as a continuous independent variable, the experimental manipulation (exclusion vs. inclusion) as a dichotomous independent variable, and their interaction. In our analyses, we followed the same strategy, but included random effects of lab as well, while we also analyzed the effect separately for each lab.
To check for the robustness of the main interaction effect to outliers, we will perform a jackknife resampling sensitivity analysis by removing one participant at a time from the sample and rerunning the confirmatory model on each of the subsamples (Purić & Opačić, 2013).
*Important files for confirmatory analysis:*
[Complete script for confirmatory analysis][1].
[Rmarkdown file for confirmatory analysis][2].
Replication criteria
To examine whether we replicated the effect or not, we had three different criteria for replication: weak, medium, and strong (Wittmann et al., 2022). For the weak criterion, if the effect was significant and in the same direction, we considered the effect replicated. For the medium criterion, if the effect was significant and if the confidence interval of the effect size overlapped, we considered the effect to have replicated. For the strong criterion, we tested the difference between the original study and the replication study’s estimate through Z-tests. Only a significant difference resulting from the superiority of the confirmatory effect size estimate over the exploratory effect size estimate was considered a strong replication.
Exploratory analyses
We then proceeded to a number of exploratory analyses, which can be split up into the three aforementioned classes: different body parts, social and individual variables, and climatic variables. To conduct the analyses, we split the data into a training (⅔) and testing (⅓) set (the exact division will depend on how many participants we will have in the final dataset). We took a conditional approach to exploratory data analyses: a priori, we planned to rely on neural nets to efficiently explore our data. If power was insufficient for neural nets, we opted to proceed to multi-random coefficient modeling analysis (Nezlek, 2021). If we proceeded to multi-random coefficient modeling analyses, we only target a subset of variables for moderation, but we would then make ⅔ of the dataset available to other researchers to allow for independent exploratory research and pre-registration prior to making the remainder of the ⅓ testing data available.
Multi-Random Coefficient Modeling
Hierarchically nested data, such as in this case (with skin temperature as a dependent variable and social exclusion versus inclusion, various individual difference variables, and site as predictor variables) is the perfect candidate for running multi-random coefficient modeling analysis (Nezlek, 2001). The two-level model that we can specify in this case is the following:
Level 1: Peripheral skin temperature.
Level 2: Respondent level (group condition, variables at the individual level, including, but not limited to attachment, STRAQ-1, social network variables).
In other words, units of the analysis at level 1 are nested within aggregate units at level 2, (i.e., respondents). The variance of each of the variables at level 1 is decomposed into within-individual and between-individual parts. The main advantage of multi-random coefficient modeling is the ability to model random effects, thus enabling more accurate parameter estimates and tests of significance than ordinary least squares regression (Raudenbush & Bryk, 2002). Multi-random coefficient modeling is ideally suited here because it enables a straightforward and accurate estimation of the amount of intra-individual variations in peripheral skin temperature across the testing situation, and across labs. The unique advantage of MRCM is the possibility to investigate whether the peripheral skin temperature varies across the participants assigned to a different group. It also enables decomposing the amount of variance stemming from the stable (inter-individual differences) and the variance originating from unstable factors (intra-individual differences).
To keep the analysis as focused as possible in this first phase, we decided to only focus on a subset of our variables:
Body parts. First, we replicated the confirmatory analyses for the pinky finger of the non-dominant hand, the wrist of the non-dominant hand, and the supraclavicular area on the non-dominant side. If we detected an effect in the training set, we went to replicate the effect in the testing set.
Social and individual variables. For each of the body parts for which peripheral temperature was collected, we examined potential moderating effects on the relationship between experimental condition and skin temperature changes. Variables from the social thermoregulation protocol for which we expected to have a moderating effect were included in the exploratory analysis (attachment, STRAQ-1 subscales, social network diversity, height, weight, and sex).
Climatic and ambient temperature variables. We also explored the moderating role of climatic variables (minimum and maximum temperature and minimum and maximum humidity of the day, the day before, and the two days before for that location, as well as average and standard deviation of the temperature of the year before) and ambient temperature measured in the lab. In order to examine climatic variables in our multi-random coefficient model, we clustered them based on distance from the equator.
We may proceed to other analyses if the power is sufficient (e.g., neural nets).
[1]: https://osf.io/9qyt4
[2]: https://osf.io/z95bc