1. Project Title and Overview

Namu Go (New York University Abu Dhabi) and Nelson Mbigili (New York University Abu Dhabi)

• Paper Title: Replication of Flakify: A Black-Box, Language Model-Based Predictor for Flaky Tests

• Authors: Sakina Fatima, Taher A. Ghaleb, Lionel Briand

• Replication Team: Namu Go and Nelson Mbigili

• Course: CS-UH 3260 Software Analytics, NYUAD

• Brief Description:

  • Original Paper Summary:

  The original paper introduces Flakify, a black-box, language model-based predictor for flaky tests that relies exclusively on the source code of test cases. By leveraging CodeBERT and deep learning classification heads, the authors demonstrate that Flakify can effectively identify non-deterministic test behavior without requiring project-specific features or access to production code, outperforming traditional machine learning classifiers that rely on manually engineered features.

  • Replication Scope Summary:

  This repository replicates and evaluates key claims from the original paper. In particular, this replication:

  1. Manual Data Validation: Performs a qualitative analysis by randomly selecting two flaky and two non-flaky tests from the FlakeFlagger dataset to validate labels and identify specific test smells associated with flakiness.
  2. RQ1 Reproduction: Reproduces the core performance experiments by retraining the Flakify model from scratch using both the FlakeFlagger and IDoFT datasets. The results, including precision, recall, and F1-score, are independently computed and compared against the benchmarks reported by the authors.
  3. Cross-Dataset Generalization: Assesses the model's robustness by performing inference on a set of 20 unseen tests from the FlakyCat dataset. This evaluation explores the "Capture Rate" of the trained models.

  All scripts, datasets, and evaluation artifacts needed to reproduce the replication results are provided in this repository.

---

2. Repository Structure

This repository has the following structure

- README.md:           Main project documentation containing the paper summary, replication 
                       scope, and high-level results of the study.
- INSTRUCTS.md:        Technical step-by-step guide for setting up the environment and 
                       executing the training and inference scripts.
- requirements.txt:    List of Python dependencies and specific versions 
- dataset/:            Input directory containing the raw CSVs and Java source files for 
                       FlakeFlagger, IDoFT, and FlakyCat datasets.
- src/:                The workspace with Python training/inference logic, 
                       automation shell scripts, and the Java smell detection tool.
- replication_results/ Output directory storing all artifacts generated by this study, 
                       including new model weights (.pt) and prediction CSVs.
- results/:            Reference directory containing the original weights and performance 
                       benchmarks provided by the paper's authors (Fatima et al.).
- lib/:                Dependency folder with external Java .jar files (Eclipse JDT) 
                       needed for the static analysis of Java test code.

3. Setup Instructions and Replication Guide

Prerequisites

• Operating System: Linux or macOS
• Programming Labguage: Python
• Tools: git, pip, virtualenv or venv
• codebert-base-local To be run locally, download here the listed files below, save them in a directory codebert-base-local and save it the src folder

    Inside src/codebert-base-local
      ├── config.json
      ├── merges.txt
      ├── pytorch_model.bin
      ├── tokenizer_config.json
      └── vocab.json

• Pre computed Weights Because of storage limitations, the binary files with weights from our training are saved in this Hugging face repository. To use our weights, download and save all files inside replication_results folder

All required Python packages and exact versions are specified in requirements.txt.

---

Replication Guide

1. Clone the Repository

   git clone <repository-url>
   cd Replication_FlakyTestsPredictor

2. Set Up Environment Create a virtual environment and install the required dependencies:

   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt

3. Replication Steps

   1. Replication Task 1: Manual Classification Analysis:

   Two flaky and two non-flaky tests were randomly selected and saved in replication_results/Classification Analysis/Flaky and Non Flaky tests - Selected.csv. The analysis can be found in report.md file within the same directory.

   2. Replication Task 2: Retraining the Model

   Note: Training the model is computationally intensive and was performed on an HPC cluster. To start the process use the following commands

   1. Navigate to the source directory:

   cd src/

   2. Run Cross-Validation Training: Execute the following to retrain on both datasets:

   bash Flakify_predictor_cross_validation.sh FlakeFlagger
   bash Flakify_predictor_cross_validation.sh IDoFT

   3. Run Per-Project Validation:

   bash Flakify_predictor_per_project.sh FlakeFlagger
   bash Flakify_predictor_per_project.sh IDoFT

   Results and weights (.pt files) are saved in replication_results/.

   3. Replication Task 3: Runing Flakify on FlakyCat Dataset

   Using the script selectRandom20.py in the src directory, we randomly selected 20 tests from the Flakycat dataset. The Output saved to dataset/FlakyCat/flakycat_input.csv in the required format.

   To run flakify on it, we modified the original script which included training the model into Flakify_Predict_FlakyCat.py which uses the weights computed in Task 2 to predict flakiness of tests in the given csv file.

   In ordet to run it use the following command:

   bash Flakify_Predict_FlakyCat.sh

   The results will be available in replication_results//Flakify_predictions.csv and the summary printed out in the terminal

---

4. GenAI Usage

• Tool Used: Google Gemini

• How It Was Used:

  • Environment & Dependency Management: Assisted in resolving dependencies version conflicts and setting up a virtual environment for the project.