Main content

Home

Menu

Loading wiki pages...

View
Wiki Version:
**Update Feb 22th, 2023:** Revised version of paper II has been submitted (abstract and paper structure below were updated). **Update May 10th, 2022:** Paper II is ready for submission: "Means to valuable exploration: II. How to explore data to modify existing claims and create new ones" Link to [submitted manuscript][1] ***Abstract:*** Transparent exploration in science invites novel discoveries by stimulating new or modified claims about hypotheses, models, and theories. In this second article of two consecutive parts, we outline how to explore data patterns that inform such claims. Transparent exploration should be guided by two contrasting goals: comprehensiveness and efficiency. *Comprehensiveness* calls for a thorough search across all variables and possible analyses as to not to miss anything that might be hidden in the data. *Efficiency* adds that new and modified claims should withstand severe testing with new data and give rise to relevant new knowledge. Efficiency aims to reduce false positive claims, which is better achieved if a bunch of results is reduced into a few claims. Means for increasing efficiency are methods for filtering local data patterns (e.g. only interpreting associations that pass statistical tests or using cross-validation) and for smoothing global data patterns (e.g. reducing associations to relations between a few latent variables). We suggest that researchers should condense their results with filtering and smoothing before publication. Coming up with just a few most promising claims saves resources for confirmation trials and keeps scientific communication lean. This should foster the acceptance of transparent exploration. We end with recommendations derived from the considerations in both parts: an exploratory research agenda and suggestions for stakeholders such as journal editors on how to implement more valuable exploration. These include special journal sections or entire journals dedicated to explorative research and a mandatory separate listing of the confirmed and new claims in a paper’s abstract. ***Structure:*** **Introduction** **Goals of exploration** - Exploration as a quantitative quest for novelty - Comprehensive exploration and the explorative search-space - Efficient exploration **Exploring around existing claims** - Exploring along an existing hypothesis - Exploring with specification curves - Exploring within a theory’s or model’s degrees of freedom **Creating new claims** - Local versus global data patterns - Filtering local data patterns - Individual versus community-driven filtering - Smoothing global data patterns **Planning exploration and transparency on how one has explored* **Further research agenda for exploration: where to explore, what and how to explore** **Recommendations to stakeholders** **Discussion** **Update May 10th:** Paper I has been accepted on April 6th. **Update February 28th, 2022:** Paper I had been [revised][2]. The [revision][3] has been submitted. *Revised abstract:* > Exploration research data has enormous potential to modify and create > hypotheses, models, and theories. Harnessing the potential of > transparent exploration would replace the common, flawed purpose of > intransparent exploration to produce results that appear to be > confirmative through hidden analytical steps. For transparent > exploration to succeed, however, methodological guidance, elaboration > and implementation in the publication system is required. We present > some basic conceptions to stimulate further development. In this first > of two part, we describe the current blending of confirmatory and > exploratory research and propose how to separate the two via severe > testing. A claim is confirmed if it passes a test that it probably > would have failed if the claim were false. Such a severe test makes a > risky prediction. It adheres to an evidential norm with a threshold, > usually p < α = .05, but other norms are possible, for example, with > Bayesian approaches. Adherence requires control against questionable > research practices like p-hacking, and at present, pre-registration > seems to be the most feasible method for this. Analyses that do not > adhere to a norm or where this cannot be controlled should be > considered as exploratory. We propose that exploration serves to > modify or create new claims that are likely to pass severe testing > with new data. Confirmation and exploration, if sound and transparent, > benefit from one another. The second part will provide suggestions on > how to achieve this and how to implement more transparent exploratory > research. The paper now has the following structure: **Introduction** **The blending of confirmation and exploration** - Manifestations of blending - The pressure to produce seemingly confirming results - Transparent exploration helps to reduce the pressure* **Differentiating confirmation and exploration** - Only confirmation uses an evidential norm - A norm must be used, but which norm is disputable - Evidential norms and Mayo’s theory of severe testing - Bayesian severity - At the cost of falling short of a norm, exploration opens the door for novelty - Global versus local claims **Transparent exploration serves confirmation and scientific communication** - With transparency, explorative research practices are no longer questionable - Concatenated exploration The project is part of the Open Science Initiative of the faculty of Psychology at the Dresden technical university (OSIP), https://osf.io/whrzg/ Former versions: [https://psyarxiv.com/mtzqj][4] [https://psyarxiv.com/psfqw/][5] [1]: https://psyarxiv.com/r5gmy [2]: https://drive.google.com/drive/folders/1IKee9MVxzsx6J1Zn10SxcGOlKjUADkMV [3]: https://psyarxiv.com/yb4km [4]: https://psyarxiv.com/mtzqj [5]: https://psyarxiv.com/psfqw/
OSF does not support the use of Internet Explorer. For optimal performance, please switch to another browser.
Accept
This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.
Accept
×

Start managing your projects on the OSF today.

Free and easy to use, the Open Science Framework supports the entire research lifecycle: planning, execution, reporting, archiving, and discovery.