This page contains useful resources concerning the Cyclic Evaluation Process that we developed for the evaluation and optimization of web-based interventions.
The Cyclic Evaluation Process consists of four phases:
1. A theoretical analysis of intended use;
2. A quantitative analysis of actual use;
3. A further exploration of user perceptions with a think-aloud study combined with semi-structured interviews;
4. An optimization phase
![A visual overview of the Cyclic Evaluation Process][1]
**Phase 1: Theoretical analysis of intended use**
The first phase has the aim of envisioning how the intervention is supposed to be used by end-users and how it would possibly lead to behavior change. Acyclic behavior change diagrams (ABCDs) can help visualize the active ingredients of the intervention and all underlying assumptions. ABCDs should not be seen as an intervention planning framework on its own (e.g. Intervention Mapping) but rather as a complementary tool to such frameworks, visualizing the active components of behavior change interventions. Our paper on ABCDs can be found here: https://doi.org/10.1080/21642850.2022.2149930 [OPEN ACCESS], and a free tool to create an ABCD is available here: https://matherion.shinyapps.io/ABCD---Shiny-App/
![A simple overview of a causal-structural chain of an ABCD][2]
**Phase 2: Quantitative analysis of actual use**
After studying intended use in the first phase, the second phase entails studying actual use of the intervention. In our case, we used Matomo. Our paper on the first two phases in our evaluation of the Dutch sexual health intervention Sense.info (in collaboration with Soa Aids Nederland) can be found here: https://academic.oup.com/heapro/advance-article/doi/10.1093/heapro/daab190/6433101?login=true [OPEN ACCESS]
**Phase 3: Think-aloud study and semi-structured interviews**
Based on the previous two phases, assumptions can be made on use and impact of the intervention. These assumptions and perceptions of users can then be further explored using a think-aloud study and semi-structured interviews. Participants are asked to navigate the intervention while thinking out loud. Afterwards they are interviewed, with a specific focus on the things mentioned during the think-aloud phase as well as on relevant behavioral determinants. Our paper (Preprint) on the results of the think-aloud study that we executed during our evaluation of Sense.info can be found here: https://doi.org/10.31234/osf.io/edypj
**Phase 4: Optimizing elements**
Based on the three preceding steps, possibilities for optimization of the intervention will have emerged. At this point, the cycle starts again: new elements can be developed using the approach used in the first phase of the Cyclic Evaluation Process. Once the optimized elements are online, the cycle starts again, with another round of web-analytics and a second think-aloud study. Our paper on the optimization of Sense.info is currently in preparation.
We have written two reports on the evaluation and optimization of Sense.info (in Dutch), that can be found in this OSF environment, under <files>.
Also, a presentation on the Cyclic Evaluation Process was given during the 36th Annual Conference of the European Health Psychology Society in 2022, for which the PowerPoint presentation is available under <files>. Another presentation was given at the 37th EHPS conference, about the development and evaluation of role-model stories for sexual health prevention using the CEP (also available under <files>).
This work was supported by ZonMw—the Netherlands Organization for Health Research and Development (project number: 555003022).
[1]: https://files.osf.io/v1/resources/s437k/providers/osfstorage/63c9671ad1a7b7014f6419cc?mode=render
[2]: https://files.osf.io/v1/resources/s437k/providers/osfstorage/63d3caa886b982005dbac41c?mode=render
