Date created: | Last Updated:

Category: Other

Description: The main challenge during this reproduction is the combination of a closed source model (Chat GPT) and available models (like CLIP). Using the models provided by the authors, the results were reproducible (Recreated Table 2). Using Ubuntu the results were partly not recreatable on Ubuntu. On Windows the training was successful. Thus, there is a system dependence of the reproducibility when retraining the models. Another challenge was the volume of models and corresponding inference. Thus I only chose a subset of models to retrain and perform inference on. A part of the inference in the paper was done manually in the paper, which I was not able to recreate. I was able to fully recreate Table1, Table 3, Table 5, Table 6 and Table 7 from the original paper. I would count the overall reproduction as a success.