#### [← Back to Nomenclature](https://osf.io/2tqj7/wiki/Nomenclature/)
## Experiment Design
This wiki article lays out the design and the execution plan of the experiment to test whether the Framework developed for introductory programming courses is effective in enhancing the beginning students' ability to program concepts without overwhelming them with the syntax of a new programming language.
### Nomenclature
Understanding of this project's experiment design relies heavily on the reader's knowledge of a few key terms used throughout this document. Please refer to [Nomenclature](https://osf.io/2tqj7/wiki/Nomenclature/) article of this wiki to review the definitions of those key terms applicable to this project.
### The Environment
The experiment for the project will be conducted throughout the **Winter 2018** quarter at **California Polytechnic State University**, during which the first-year computing major students (Software Engineering, Computer Science, Computer Engineering) take the course Fundamentals of Computer Science I (CPE-101).
The primary investigator will be the instructor of a single offering (section) of CPE-101, and there will be around 30 students enrolled in the offering of the course. The offering will be composed of three hours of common lecture time and two separate three-hour lab times per week.
All 30 students will be enrolled in the same lecture hours in which they will receive instruction on introductory programming concepts. During the lectures, concrete examples for the concepts discussed and important techniques necessary to complete the class assignments will be presented, but no step-by-step guide on how to construct a full solution for the assignments will be given.
One half (approximately 15) of the students will be treated as a control group and enrolled in the lab hours separately from an experimental group (also approximately 15).
The control group they will be taught the methods of programming in a manner that is consistent with the traditional approach, which encourages jumping right into the implementation with the standard programming language chosen for the course.
The experimental group will be presented with the Framework and taught the methods of programming with minimal reliance on the standard programming language chosen for the course.
### The Framework
The framework to be utilized in the experimental groups' lab hours are mainly composed of the following teaching methods, as described in the **Overview** section of Home article of this wiki:
- **Design Recipe** from [*How to Design Programs*](http://www.htdp.org/2001-01-18/Book/node14.htm), with augmentations to distinguish functional arguments from console and file system I/Os
- **Code Outlining** to externalize the problem-solving steps in easily readable and sharable format
- **Peer Review Process** for students to learn from each other and to practice effectively communicate different ideas for the given problem
- **Automatic Code Template Generation** to minimize the cognitive load spent on familiarizing oneself with the syntactic structure of the implementation language
- **Automatic Unit Test Generation** to reduce the introductory-level learning curve to test-driven development while still encouraging thinking through the given problem sufficiently prior to implementation
As the secondary goal in the **Overview** of this project stated, the development of electronic tools and the workflow around such tools to allow the efficient integration of the above framework is also being considered.
However, important to note is that the above framework does not necessarily require any extra tools to be implemented in a classroom setting. The main goal of this project, to validate this framework, can and will be achieved in a manner that does not introduce coupling of the framework to any tools developed for efficient integration.
### The Execution
Throughout the ten-week duration of the quarter, two groups of students (control and experimental, each enrolled in different lab hours) will receive the same lectures, but will be presented with different tools and procedures to complete lab and project assignments.
The control group will be presented with the traditional methods of test-driven development and restricted peer evaluation or collaboration, whereas the experimental group will receive instructions with heavy emphasis on code outline construction, peer review of the outlines and feedback exchange, and test value (example) compositions.
There will be planned quizzes and "midterm" exams throughout the quarter, which will present all students with the same questions, including a small number of questions designed to test the *ability to program* as defined in the **Nomenclature** section.
During the three-hour final exam that is scheduled at the very end of the academic term, which will be the same exam given to all students enrolled in the course university-wide, a question similar to the ones presented in the midterms will be included.
By collecting the scores earned on the exam problems by different groups (control and experimental during the midterm, control and experimental during the final exam, and potentially all other students during the final exam) and analyzing the differences, we will be able to validate if the Framework was effective in making a statistically significant difference in the experimental groups' *ability to program*.
In addition to validating the Framework based on the exam results, in-person interviews, surveys, and inspection of the student assignment submissions will be performed throughout the academic term to determine the students' reaction to the integration of the Framework into the teaching of the course.
Collection and analysis of student reaction data is expected to reveal the magnitude of marginal *friction* introduced by integrating the Framework into the course. In the case that this friction is determined to be overshadowing the benefits the Framework is expected to produce in students' *ability to program*, a set of suggestions for potentially reducing such friction shall be provided with the publication of the benefits of the Framework alone.
For more detailed information on data collection and analysis, please refer to [Validation Plan](https://osf.io/2tqj7/wiki/Validation%20Plan/) article of this wiki.
<br>
#### [→ Continue to Validation Plan](https://osf.io/2tqj7/wiki/Validation%20Plan/)