# Learning-based Prioritization of Test Cases in Continuous Integration of Highly-Configurable Software This repository contains the data used in the article **Learning-based Prioritization of Test Cases in Continuous Integration of Highly-Configurable Software**. Our article introduces and evaluate two strategies for the application of a Test Case Prioritization learning-based approaches in Continuous Integration of Highly-Configurable Software. ## Table of Contents - [Repository](#requirements) - [Datasets](#datasets) - [Results](#results) - [Analysis](#analysis) - [License](#license) ## Repository The repository is organized as follows: ``` A Multi-Armed Bandit Approach for Test Case Prioritization in Continuous Integration environments │ Wiki └───Datasets │ │ Wiki | | libssh-mirror | | └───total │ │ | | Failures by CI Cycle │ │ | | Test Case Volatility | | └───variants | | | └───CentOS7-openssl 1.0.x-x86-64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───centos7-openssl_1.0.x-x86-64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───Debian-openssl 1.0.x-aarch64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───Debian.cross.mips-linux-gnu │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───fedora-libgcrypt-x86_64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───fedora-mbedtls-x86-64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───fedora-mbedtls-x86_64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───Fedora-openssl 1.1.x-x86-64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───fedora-openssl_1.1.x-x86-64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───fedora-openssl_1.1.x-x86-64-release │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───fedora-openssl_1.1.x-x86_64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───fedora-openssl_1.1.x-x86_64-fips │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───fedora-openssl_1.1.x-x86_64-minimal │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───fedora-undefined-sanitizer │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───freebsd-x86_64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───mingw32 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───mingw64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───pages │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───tumbleweed-openssl_1.1.x-x86-64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───tumbleweed-openssl_1.1.x-x86-64-release │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───tumbleweed-openssl_1.1.x-x86_64-clang │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───tumbleweed-openssl_1.1.x-x86_64-gcc │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───tumbleweed-openssl_1.1.x-x86_64-gcc7 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───tumbleweed-undefined-sanitizer │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───ubuntu-openssl_1.1.x-x86_64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───undefined-sanitizer │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───visualstudio-x86 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility | | | └───visualstudio-x86_64 │ │ │ │ | Failures by CI Cycle │ │ | │ | Test Case Volatility └───Results | | Wiki │ └───Time Budget 10% | └───Time Budget 50% | └───Time Budget 80% └───Analysis | | Technical Report (technicalreport.pdf) | | └───address-sanitizer | | └───CentOS7-openssl | | └───CentOS7-openssl 1.0.x-x86-64 | | └───centos7-openssl_1.0.x-x86-64 | | └───Debian-openssl 1.0.x-aarch64 | | └───Debian.cross.mips-linux-gnu | | └───fedora-libgcrypt-x86_64 | | └───fedora-mbedtls-x86-64 | | └───fedora-mbedtls-x86_64 | | └───Fedora-openssl | | └───Fedora-openssl 1.1.x-x86-64 | | └───fedora-openssl_1.1.x-x86-64 | | └───fedora-openssl_1.1.x-x86-64-release | | └───fedora-openssl_1.1.x-x86_64 | | └───fedora-openssl_1.1.x-x86_64-fips | | └───fedora-openssl_1.1.x-x86_64-minimal | | └───fedora-undefined-sanitizer | | └───freebsd-x86_64 | | └───mingw32 | | └───mingw64 | | └───pages | | └───tumbleweed-openssl_1.1.x-x86-64 | | └───tumbleweed-openssl_1.1.x-x86-64-release | | └───tumbleweed-openssl_1.1.x-x86_64-clang | | └───tumbleweed-openssl_1.1.x-x86_64-gcc | | └───tumbleweed-openssl_1.1.x-x86_64-gcc7 | | └───tumbleweed-undefined-sanitizer | | └───ubuntu-openssl_1.1.x-x86_64 | | └───undefined-sanitizer | | └───visualstudio-x86 | | └───visualstudio-x86_64 ``` ## Datasets In this study we used a real-world software system and its 31 variants. The datasets regarding the selected system are available in the [Datasets][1] component. To support the data extraction and modeling we developed a tool attached in this repository and available at GitHub. ## Results The Results are available in the [Results][2] component. We separated the results from the Analysis component to provide a better visualization and organization. The results were obtained from 30 independent runs using the Time-Ranked Reward (TimeRank) function, as well as considering three time budgets (10%, 50%, and 80%). ## Analysis The analysis of the results is available in the [Analysis][3] component. In such a component, there are results regarding the comparison between COLEMAN4HCS and Random approach. Both techniques are evaluated for two strategies proposed for HCS context. Each folder in the Analysis component contains the following plots: - Accumulative: - NAPFD values along CI Cycles using TimeRank (file **ACC_NAPFD_timerank.pdf**) function. - Reward values along CI Cycles from TimeRank (file **ACC_Reward_timerank.pdf**) function. - Overview: - NTR distribution values obtained using TimeRank (file **NTR_timerank.pdf**) function. - Kruskal-Wallis test: - NAPFD values found in TimeRank (file **NAPFD_Kruskal_timerank.pdf**) function. - RFTC values found in TimeRank (file **RFTC_Kruskal_timerank.pdf**) function. - Prioritization Time values found in TimeRank (file **PrioritizationTime_Kruskal_timerank.pdf**) function. - Variation along CI Cycles: - NAPFD variation values along CI Cycles found in TimeRank (file **NAPFD_Variation_timerank.pdf**) function. ## License This project is licensed under the terms of the Creative Commons Attribution 4.0 International Public License. [1]: https://osf.io/98zx5/ [2]: https://osf.io/ah8gb/ [3]: https://osf.io/8pr3w/