Skip to content

INSTRUCTIONS.md

Latest commit

 

History

History
177 lines (125 loc) · 16.1 KB

INSTRUCTIONS.md

File metadata and controls

177 lines (125 loc) · 16.1 KB
Raw

Overview

The paper makes the following claims on pg. 2 (Comments to AEC reviewers after :):

  1. A collection of benchmark programs: these are POSIX shell programs that perform a variety of tasks.

  2. Permanent and scalable storage of varying inputs: "permanent" means Zenodo; "scalable" means Brown university servers, which allows for faster download compared to Zenodo. There are three input sizes---min take about a few minutes, small take a few hours, and full take several days---but note AEC reviewers are encouraged to only try the first two sizes.

  3. Tooling and automation for reuse: additional scripts automating dependency setup, input validation, and correctness checking.

  4. Characterization and implications: additional infrastructure performing static and dynamic program analysis of the benchmark programs.

This artifact targets the following badges:

  • Artifact available: Reviewers are expected to confirm that the benchmark programs, their inputs, and automation scripts are all publicly available (about 10 minutes).
  • Artifact functional: Reviewers are expected to confirm sufficient documentation, key components as described in the paper, and execution with min inputs (about 20 minutes); execution with small inputs (3 hours)is encouraged only after completion of the full artifact evaluation.
  • Results reproducible: Reviewers are expected to reproduce key results of sections 3, 5, 6, and 7 of the paper (2 hours).

Artifact Available (10 minutes)

Confirm that the benchmark programs, their inputs, and automation scripts are all publicly available:

  1. The benchmark code is permanently hosted at: https://github.com/binpash/benchmarks

  2. Data are available on permanent (i.e., archival-level durable, but slow) and scalable (i.e., fast, but not archival-level durable) storage:

    • Permanent archival storage on Zenodo, split across multiple DOIs due to Zenodo's max 50GB limit (AEC Reviewers: this is slow, do not try to download—just confirm their existence): code & software dependencies, small inputs; full in 5 parts: 1, 2 3, 4, 5.
  1. Additional scripts are available in the infrastructure/ directory of the repository.

AEC Reviewers: From this point on, scripts use the Brown links, as Zenodo is significantly slower.

Artifact Functional (20 minutes, optionally 3 hours)

Confirm sufficient documentation, key components as described in the paper, and execution with min inputs (about 20 minutes):

At this point, run git clone https://github.com/binpash/benchmarks and cd benchmarks.

Quickstart: Running a single benchmark (e.g., nlp): To quickly execute a specific benchmark such as nlp, invoke the top-level main.sh script—which will set up a Debian-based container image, install dependencies, download benchmark-specific min inputs, and execute the benchmark (2 minutes):

./main.sh nlp

The reason this is easier to evaluate in a container is that some scripts assume that some dependencies on the host system (e.g. git). The scripts also assume a docker executable in the PATH, which can be configured with the KOALA_CONTAINER_CMD.

Complete run: Running all benchmarks: To execute the full set of benchmarks, invoke the top-level all.sh script—which essentially runs the previous command in a loop (20 minutes):

./all.sh

Not recommended on the first run:

Both scripts above take additional optional parameters: Using --small will execute the benchmark(s) with the small inputs (about 3 hours) and --bare will execute the benchmarks on the host system (which might not satisfy dependencies). To replace the shell interpreter, set the KOALA_SHELL variable — e.g., KOALA_SHELL="bash --posix".

Results Reproducible (about 3 hours)

The key results of the analysis are the following:

  1. Principal Component Analysis (§3, Fig. 2);
  2. Syntactic Characterization & Analysis (§5, Fig. 4 & 5);
  3. Dynamic Characterization & Analysis (§6, Fig. 6).

Most of these results are easily reproducible by running specific scripts in the repo. A part of PCA depends on embeddings, whose calculation incurs some financial costs due to OpenAI—thuse we have cashed these embeddings in the repo. Although summarized earlier in the paper (§3), PCA depends on the results of the static and dynamic analyses, so we start with no. 2 and 3 before returning to no. 1.

Preparation: The dynamic analysis requires running Koala in a pre-set container due to elevated privileges:

mkdir -p /tmp/plots
sudo docker pull ghcr.io/binpash/benchmarks:latest
sudo docker run -it --rm -v "/tmp/plots":/tmp/plots ghcr.io/binpash/benchmarks:latest bash
./setup.sh # Inside the container

Static characterization: To generate the static characterization, run the following commands:

./infrastructure/static-analysis.sh /tmp/plots

This will place the heatmap plot showing the results of the static analysis in the /tmp/plots directory on the host system.

Dynamic characterization: To generate the dynamic characterization, run the following commands: Note: This step will run the entire suite again, now having tracing enabled (~3 hours).

./infrastructure/dynamic-analysis.sh --small

Principal component analysis: Part of the PCA analysis involves sending the source code of the benchmarks to a remote embedding model using OpenAI's API; for convenience and cost concerns, we provide the results the embedding model in infrastructure/data/embeddings.csv (generatable by running python3 infrastructure/do_embedding.py).

./infrastructure/pc-analysis.sh /tmp/plots/pca.pdf

Optional: Applying benchmarks to various systems (1–3 days)

This section is about setting up and running other systems on the Koala benchmarks. Crucially:

  • The difficulty of evaluating prototype systems using Koala depends on these systems (i.e., not our contribution) as well as access to specific hardware (e.g., large multiprocessors).
  • The authors of these systems are free to make any modifications to the benchmarks, the inputs, or the scripts according to the contributions claimed by their systems, not Koala.

Therefore, this evaluation outside the scope of the Koala artifact evaluation.

Contact

For questions or bug reports, please contact koala@brown.edu or open an issue on GitHub.

Appendix I: All Inputs

The table below contains all links to the inputs. AEC reviewers: Some of these inputs are extremely large and hosted on low-bandwidth permanent-storage services such as Zenodo. To confirm existence, simply click on the URL to to start a download and then stop downloading a few seconds later.

Benchmark Min Small Full
aurpkg Github Repo Brown Zenodo Brown Zenodo
bio Github Repo Brown Zenodo Brown Zenodo
covid-mts Github Repo Brown Zenodo Brown Zenodo
file-enc Github Repo Brown Zenodo Brown Zenodo
log-analysis Github Repo Brown Zenodo Brown Zenodo
makeself No inputs No inputs No inputs
max-temp Github Repo Brown Zenodo Brown Zenodo
media-conv Github Repo Brown Zenodo Brown Zenodo
nlp Brown Brown Zenodo Brown Zenodo
oneliners Brown Brown Zenodo Brown Zenodo
riker Github Repo Brown Zenodo Brown Zenodo
sklearn Github Repo Brown Zenodo Brown Zenodo
unix50 Github Repo Brown Zenodo Brown Zenodo
vps-audit No inputs No inputs No inputs
web-index Brown Brown Zenodo Brown Zenodo