Contemporarily, experimental investigations of revealed preference choice consistency utilize different tasks interchangeably. However, the reliability of choice consistency measurements among (inter-method) and within tasks (test-retest) has not been determined so far. Hence, it is unclear whether estimations of choice consistency fulfill a basic requirement of valid psychometric measures. Further, it is unclear how far results from different studies using different methodologies are comparable. In the study described here, we investigated the reliability of two established and one novel choice consistency tasks in an online-experiment under non-incentivized conditions in the choice domain of social decisions. Our results confidently indicate generally poor inter-method reliability and at best moderate test-retest reliability for the two indices, the Critical Cost Efficiency Index (CCEI) and the Houtman-Maks-Index (HMI), with the CCEI being the tentatively more reliable measure. This is especially concerning, since the full experiment (including test and retest measurement) lasted on average less than 45 minutes. Hence, it appears that estimations of choice consistency do not fulfill a basic requirement of valid psychometric measures. Further, results from different studies using different methodologies should not be compared without caution. Future work should investigate the impact of incentivization as well as the choice domain generality of our results.