Date created: | Last Updated:

Category: Project

Description: This is the accompanying repository for the paper "A Rigorous Behavior Assessment of CNNs Using a Data-Domain Sampling Regime" Abstract: We present a data-domain sampling regime for quantifying CNNs’ graphic perception behaviors. This regime lets us evaluate CNNs’ ratio estimation ability in bar charts from three perspectives: sensitivity to training-test distribution discrepancies, stability to limited samples, and relative expertise to human observers. After analyzing 16 million trials from 800 CNN models and 6,825 trials from 113 human participants, we arrived at a simple and actionable conclusion: CNNs can outperform humans and their biases simply depend on the training-test distance. We show evidence of this simple, elegant behavior of the machines when they interpret visualization images. Contents: The repository contains source code (model training, statistical analysis, figure plotting), human & CNN outputs, and all figures used in the paper. Statistical analysis and figure plotting code can be executed in Google Drive: https://drive.google.com/drive/folders/19Ybm-wpsaY-dL5OM-Ee_mf7ntw3WpHCs?usp=sharing. Our studies are pre-registered at: Study I (robustness): https://osf.io/t65su Study II (stability): https://osf.io/mzcky Study III (human-AI comparison): https://osf.io/myt7g