**May 17th:** Data collection and procession is finished. We will put the results of the so far planned analyses (this thread) into the [file folder][1]. The same for potential other analyses. The open data set will also be made available there soon. **April 27th, 2022:** Further exploratory analysis to be carried out: 1 Conduct preregistered confirmatory in an explorative manner to see if the hypotheses can be raised again or modified for a confimatory trial with new data. Omit the second item of the manipulation control for case exclusion; otherwise use syntax preregistered below: *drop if manipulationcheck ==2* 2 Why did so many participant disagree with "Drinking plain water over years can only be investigated in an observational study."? (variable *manipulationcheck2*) Rank correlations with all *viewpoint_** items to see if this rather reflects certain stances or indicates an invalid item/example *for num 1/21: spearman manipulationcheck2 viewpoints_X , print(.05) pw* 3 Relation of *manipulationcheck2* with age group and scientific field? *regress manipulationcheck2 ib1.field regress manipulationcheck2 ib1.subfield* **April 27th, 2022:** We hereby record that the confirmatory analyses preregistered (here below) are not possible. This plan has **failed**, **no confirmatory analyses will be conducted**. The reason is that out of 176 (107 with outcome information) participants, only 11 (8) fulfilled the below preregistered inclusion criteria. This is mainly due to the second item on the manipulation check ("Drinking plain water over years can only be investigated in an observational study"), which only 15 (11) answered as expected (agree "strongly" or "very strongly"). Instead, we will explore around the preregistered hypotheses in order to eventually modify them and/or generate new hypotheses. We will note plans for this here as soon as they are available. Besides, data will be provided as Open Data as soon as the follow-up assessment is completed. [LimeSurvey][2] will be used to conduct the study. Study course (all items in consecutive order will be published before starting the data collection): 1. Informed consent 2. Randomization into 3 groups 2. Sociodemographical/academical information 3. Research example 4. Partially different items across groups 5. For all groups: Assessment of 18 items on attitudes towards scientific reasoning + 2 outcome items (on motivation to deal with explicit causal methods) 6. Final items **Inclusion criteria to test the hypotheses** 1. Informed consent provided 2. Correct answers to 2 control questions on understanding the research example *"The example describes a (fictional) dialogue between two scientists." No! "'Drinking plain water over years can only be investigated in an observational study.' Please rate how much you agree with that statement." Answers on 5-point Likert scale must be "strongly" or "very strongly" 3. Outcome information is there (see definition below)* **Outcome assessment across three groups** All participants are presented the following example: > A recent study investigated whether drinking one litre of clear water > per day would reduce the risk of dementia in late life. The study > managed to look at 100,000 individuals longitudinally over the > lifespan and came up with a statistically significant and stable > estimate (neglectably small standard error) of a 50% risk reduction > for dementia in persons who drank at least one litre of clear water > (for at least 10 years). The result was adjusted for age and sex. Then, they are asked to rate 2 items on 5-point Likert scale (results not of primary interest): > A causal conclusion must not be made with a non-experimental study > like this. > > A causal conclusion is unavoidable, because one must decide whether to > start drinking plain water to prevent dementia. Participants are **randomized** into three groups (with equal probability through LimeSurvey: *{if(is_empty(RandomGroup),rand(1,3),RandomGroup)}* **Controls (coded 1):** not asked to reflect on whether the last two items are in conflict + not asked to reflect on whether science would benefit from addressing causality in examples like this **Group that reflects on conflict only (coded 2):** This group is asked: > "A causal conclusion must not be made with a non-experimental study like this." "A causal conclusion is compelling to decide whether to start drinking plain water to prevent dementia." > Do you believe that these two statements are in coflict? (Free text item for qualitative. explorative analysis) **Group that ALSO reflects on benefits (coded 3):** > Do you believe that research could move forward if it adressed > causality in non-experimental instances like this (using sound methods > like adjusting for common causes of factor and outcome)? Take at least 60 seconds for your answer. Write down your thoughts. > (Free text item for qualitative. explorative analysis) **Outcome definition** Arithmetic mean of these two items (both mandatory and assessed on 5-point Likert scale): > I feel motivated to address causality outside experiments in my field. > > I feel motivated to dig into methods if necessary for this purpose. **Hypothesis 1:** Group 2 (conflict) shows lower motivation than the control group (no reflection). **Hypothesis 2:** Group 3 (conflict + benefits) shows higher motivation than the control group. One-sided tests with α =.05 See the syntax for operationalisation and sample size calculation. **Planned exploratory analysis** 1. Compare the 3 groups across the other 18 items (item scores) to propose some new hypotheses on a) what reflection on conflict and benefits might trigger b) barriers that might be resist such reflections (items that do not differ across groups) 2. Analyse the example assessment items "A causal conclusion must not be made with a non-experimental study like this" and "A causal conclusion is unavoidable, because one must decide whether to start drinking plain water to prevent dementia" in relation with the subsequent 18 +2 items. 3. Qualitative investigation of stances and barriers mentioned in the free text items 4. Other unanticipated exploratory analyses [1]: https://osf.io/msn9r/files/ [2]: https://www.limesurvey.org/de/