Background
Null hypothesis significance testing (NHST) causes researchers to draw erroneous conclusions from their data. NHST is still
ubiquitously used as a default procedure to make inferences about population effects. Despite it’s popularity, NHST has
received much criticism in the past decades. The core of this criticism is that NHST is a form of indirect inference; a conclusion
about a population effect (the null hypothesis) is drawn from the p-value that provides the probability of the data or more
extreme data, given the null hypothesis. However, the probability of a null population effect depends greatly on the prior probability of a null-hypothesis being ‘true’. This is an important underlying reason that many conclusions from research are
false and it makes NHST unsuitable as the default procedure to draw conclusions from empirical data. A number of alternatives
have been developed that overcome this pitfall, such as Bayesian inference methods, informative hypothesis testing and
a-priori inferential statistics. However the uptake of these methods has been limited. Therefore, the questions are why many
scientists keep on using NHST and how this information can be used to create a paradigm shift in the scientific community.
Aims
1. To explore views on NHST among all relevant
stakeholders in the scientific system (first 14 months).
2. To develop strategies to implement use of alternative methods for drawing conclusions from empirical data (last 10 months).
Methods
Theoretical framework
The theoretical framework for this study is the diffusion of innovation theory of Rogers. In this proposal the innovation consists
of the idea that our alternatives provide more valid and more responsible methods for drawing conclusions from empirical data
than NHST. An important feature of Rogers’ theory is that it distinguishes between phases in the adoption process, which
coincide with different types of people. In this project we will specifically focus on early adopters. They form the second group to
adopt an innovation, after the innovators themselves. This group involves a great deal of opinion leadership and serves as role
models for the rest of the system.
Strategy
The project consists of two phases, largely coinciding with the two research aims.
Phase I (months 1-14): At the start of the study we will form an interest group that will provide input for a list of alternatives for
drawing conclusions from empirical data. Next, we will conduct interviews with all relevant stakeholders from the science
system. We aim to enroll 5-10 editors, 8-10 senior researchers, 8-10 junior researchers, 8-10 lecturers in statistics and 5-7
program leaders from funding agencies. The topic list for the semi-structured interviews will contain key characteristics from
Rogers’ theory as well as topics derived from the theory of planned behavior. This will result in information on the perceived
barriers and facilitators and will also allow us to identify early adopters. Subsequently we will organize four focus groups (one
per stakeholder group) to validate and also to enrich and further explore the data gathered during the interviews.
Phase II (months 15-24): We will organize three search conferences with the ‘early adopters’ as identified in the first phase of
the project complemented by other early adopters if needed. A search conference is a participative planning method that
enables people to create a strategy for the most desirable future, in our case, with respect to the way conclusions are drawn
from empirical data. The participants take responsibility for carrying out the strategy themselves. The potential barriers and
facilitators identified from the first phase of the study serve as input for these search conferences. A total of 20-30
representatives of all four groups of stakeholders will be included.
In the first search conference we will discuss the outline of the new paradigm for drawing conclusions in empirical sciences.
Information from the interviews and focus groups will be used for each of the
alternatives from the perspective of the different stakeholder groups. The second search conference will address a strategy to
achieve the desirable paradigm shift in research practice, the result of which will become visible in future published material.
The third search conference will address the way in which educational programs need to be adapted in order to teach students
the desirable alternative way(s) of drawing conclusions from empirical data. The deliverable of the search conferences is a
proof of concept for the strategy, ready to be incorporated in a specific implementation plan.
The final months of the project will be spent on processing the input form the search conferences into specific implementation
plans and for other implementation activities.