We are a group of researchers (Economics, Psychology, other) running a field experiment with a large US-based international charity. This opportunity was provided through our affiliation with [Donor Voice](http://agitator.thedonorvoice.com/behavioral-science/).The charity is sending out a massive email campaign (approximately 330,000 emails) in the broad 'Thanksgiving season' of 2018. The email will randomly (without blocking/stratification) vary whether or not these emails present a statement about the charity was 'able to provide with just $10'. *Note*: It is not clear at present how this measure was calculated by the charity/ We may run other treatments in future campaigns as part of this same project. We are running this subject to the final say of the charity. We have proposed that the Treatment emails (but not the control emails) will include a sentence/fragment such as the following in both a captioned photo in the email, and the email text: > Last year, we were able to provide [general provision of an outcome here relevant to the charity] to a [recipient unit] with just $10. We are not clear yet on exactly the data we will obtain. However the individual data we expect to gain from the emails are: - Whether they made a donation (yes/no) - Donation amount - Email open rates (these shouldn't vary by treatment as we didn’t change the subject line) - Click through (whether they clicked on a link: yes/no) - Number of click throughs (how many links they clicked on) We plan to perform standard nonparametric statistical tests of the affect of this treatment on 1. Average donation amount (including zeroes) 2. Incidence and number of people making a donation between control and test. 3. The likelihood of donating exactly the $10 mentioned in the email. 4. As a secondary concern, the 'clickthrough'. rates 5. Rates of unsubscription from the mailing list (if available) If the data is available, we will also aim to measure the impact on the *long-run* participation and donations of these email recipients over the course of subsequent promotions. In particular, we will focus on Fisher's exact test (for incidence) and the standard rank sum and t-tests for the donation amounts. If the aforementioned results are not statistically significant at the p=0.05 level or better, we do not plan to include statistical controls nor to do any interactions/differentiation of our results. We will report confidence intervals on our estimates, and make inferences on reasonable bounds on our effect, even if it is a 'null effect'. Response rates in previous such emails were extremely low: approximately 1 per 3,000 emails. Our power calculations suggest that we have .29 power to detect a 50% effect, and 0.90 power to detect approximately a 100% (doubling) on incidence: ```statmod::power.fisher.test(0.0003,0.00045,150000,150000,alpha=0.01,nsim=10000)``` $\rightarrow$ .2896 power This is probably considered an 'underpowered test'. We need roughly a 100% effect (a doubling of the donation incidence) to have 80% power here. ```statmod::power.fisher.test(0.0003,0.0006,150000,150000,alpha=0.01,nsim=10000) ``` $\rightarrow$ 0.8963 Because of this limited power, we may ask the charity to run this trial a second time with an equivalent-sized sample. With a doubling of our sample we have roughly 0.8 power to detect a 50% impact on incidence: ``` stats::power.prop.test(,0.000327,0.000327*1.5,0.01,.8) ``` --> n = 357007 (this refers to the number per treatment) *** *Update 27 Nov 2018*: After having seen the emails the charity sent out (but not the outcomes), the treatment emails seem to present an outcome for each $10 that the recipients may not interpret literally, as it seems somewhat overstated. (We may ask the charity to present a more realistic, evidence-based measure for a future trial.) The email recipients may interpret the treatment will interpret this as something like "even donations as small as $10 help towards our mission"... We are considering how to interpret the results of this trial in light of this. *** In response to unanticpated deviations from our plan or 'surprises', we will follow [Columbia Green Lab standard operating protocol](https://github.com/acoppock/Green-Lab-SOP) as is possible and reasonable. **Exclusions**: we will exclude any emails that bounced or were not opened from our analysis. The latter is under the assumption that nothing about the email was visibly different before opening it between control and treatment, thus no differential selection. (27 Nov 2018: This seems reasonable given the emails the charity has shown us.) If we see this does not hold after implementation, we will not make this exclusion. Finally, our 'stopping rule': We will not ask the charity to stop this treatment in the middle of this campaign. We aim to continue this treatment in future charity appeals until we can statistically bound (with 95% confidence) the impact of the treatment on both incidence and average donation within a margin of 1/3 of the incidence and average donation in the control condition. *Update 27 Nov 2018*: After having seen the emails the charity sent out, we are considering asking for a change in any future trials, implying that the data may not be easily pooled.