Automation_jager

Automated PR Evaluation and Multi-Architecture Docker Image Builder

This project provides a comprehensive automation system for:

Building universal Docker images supporting linux/amd64 and linux/arm64 architectures
Automated PR (Pull Request) evaluation with test execution and validation
Docker-based test harness for reproducible testing environments

Features

🚀 Multi-Architecture Docker Builds

Build universal Docker images that work on both Intel/AMD and ARM architectures
Single OCI-compliant tarball containing both architecture variants
Automatic architecture selection when loading images

🔬 PR Evaluation System

Automated PR cloning and evaluation
Language and test framework detection
Docker-based isolated test execution
Comprehensive test result categorization and metadata generation
Support for Python, JavaScript, TypeScript, Go, Rust, Java, C#, and Ruby

🛠️ Robust Automation

Self-healing Docker builds
Configurable test execution with retries
Detailed logging and error reporting
Workspace management and cleanup

Documentation

This project is organized into distinct modules, each with its own detailed documentation:

Module	README	Description
PR Evaluation System	automation_script/README.md	Full guide to the two-phase PR evaluation pipeline: Docker image building, test execution, patch evaluation, metadata generation, and VeloraHarness integration
Multi-Arch Build Utilities	automation_script_build_multi/README.md	Utility functions for multi-architecture Docker image builds, including command execution helpers
Installation Guide	INSTALL.txt	Step-by-step manual installation instructions for all supported platforms

For a quick overview, continue reading this README. For in-depth usage of the PR evaluation system (command options, output format, supported languages, test identifier formats, VeloraHarness integration), see automation_script/README.md.

Quick Start

Automatic Installation (Recommended)

# Clone the repository
git clone <repository-url>
cd Automation_jager

# Run the installation script
chmod +x install.sh
./install.sh

The installation script will automatically:

Detect your operating system
Install Python 3.8+, Docker, Docker Buildx, and GitHub CLI
Configure Docker permissions
Install Python dependencies

Manual Installation

See INSTALL.txt for detailed manual installation instructions.

Installation

This project provides multiple ways to install dependencies:

Option 1: Automated Installation (Recommended)

./install.sh

Interactive script that installs all required dependencies automatically.

Option 2: Manual Installation

Follow the comprehensive guide in INSTALL.txt for platform-specific manual installation steps.

Option 3: Python Dependencies Only

python3 -m pip install -r requirements.txt

Note: This project uses Python standard library modules, so no external packages are required.

Prerequisites

System Requirements

Python 3.8 or higher
Docker 20.10 or higher
Docker Buildx plugin
GitHub CLI (optional but recommended for PR operations)
4GB RAM (8GB recommended)
10GB free disk space

Supported Operating Systems

macOS 10.15 or higher
Ubuntu 20.04 or higher
Debian 10 or higher
Fedora 35 or higher
RHEL/CentOS 8 or higher

Verify Installation

# Check Python
python3 --version  # Should be 3.8+

# Check Docker
docker --version
docker info

# Check Buildx
docker buildx version

# Check GitHub CLI (optional)
gh --version

Project Structure

Automation_jager/
├── build_universal_image.py      # Multi-arch Docker image builder
├── install.sh                     # Automated installation script
├── requirements.txt               # Python dependencies
├── INSTALL.txt                    # Detailed installation guide
├── README.md                      # This file (start here)
│
├── automation_script/             # Main automation package
│   ├── README.md                 # Detailed module documentation (see Documentation section)
│   ├── __init__.py
│   ├── main_orchestrator.py      # Main entry point for PR evaluation
│   ├── part1_build_and_base.py   # Docker build and base testing
│   ├── part2_patch_and_evaluate.py  # Patch application and evaluation
│   │
│   ├── config.py                  # Configuration and constants
│   ├── docker_builder_new.py     # Docker image generation
│   ├── docker_runner.py           # Container test execution
│   ├── docker_healing.py          # Self-healing Docker builds
│   ├── container_runner.py        # Advanced container operations
│   │
│   ├── git_operations.py          # Git utilities
│   ├── git_wrappers.py            # High-level Git operations
│   ├── github_api.py              # GitHub API interactions
│   │
│   ├── language_detection.py     # Language/framework detection
│   ├── environment.py             # Environment detection and setup
│   ├── test_results.py            # Test result processing
│   ├── test_targeting.py          # Test identification
│   │
│   ├── metadata.py                # Metadata structure definitions
│   ├── metadata_generator.py     # Metadata creation
│   ├── collect_29_fields.py      # Extended metadata collection
│   ├── artifacts.py               # Artifact management
│   ├── organize_outputs.py       # Output organization
│   ├── repo_configs.py            # Repository-specific configurations
│   │
│   ├── cleanup.py                 # Workspace cleanup
│   ├── cleanup_workspaces.py      # Bulk workspace cleanup
│   ├── utils.py                   # Utility functions
│   ├── validate_fix.py            # Fix validation
│   └── _archived/                 # Legacy/deprecated code
│
├── automation_script_build_multi/ # Multi-build utilities
│   ├── README.md                  # Module documentation (see Documentation section)
│   ├── __init__.py
│   └── utils.py                   # Command execution helpers
│
├── F2P_finder/                    # FAIL_TO_PASS test discovery CLI/tooling
│   ├── cli.py
│   ├── infer.py
│   └── ...                        # Additional finder modules
└── F2P_finder_data/               # Finder runtime/output data directory

Usage

1. PR Evaluation System

The main orchestrator provides automated PR evaluation with test execution:

Full Workflow (Build + Test + Evaluate)

python3 -m automation_script.main_orchestrator \
    https://github.com/owner/repo/pull/123 \
    /path/to/workspace

With Language/Test Command Overrides

python3 -m automation_script.main_orchestrator \
    --language rust \
    --test-cmd "cargo test --manifest-path engine/Cargo.toml" \
    https://github.com/owner/repo/pull/123 \
    /path/to/workspace

The orchestrator currently runs the full evaluation workflow in one command (BASE run, test-only patch run, full patch run, and metadata generation).

Performance Options

# Fast mode with shallow clone
python3 -m automation_script.main_orchestrator \
    --shallow-clone \
    https://github.com/owner/repo/pull/123 \
    /path/to/workspace

# Reuse existing Docker image
python3 -m automation_script.main_orchestrator \
    --reuse-image pr-eval:base-abc123 \
    https://github.com/owner/repo/pull/123 \
    /path/to/workspace

Get Help

python3 -m automation_script.main_orchestrator --help

2. Universal Docker Image Builder

Build multi-architecture Docker images:

Basic Usage

python3 build_universal_image.py \
    --dockerfile <path-to-dockerfile> \
    --output <output.tar> \
    --repo_url <git-repo-url> \
    --commit <commit-sha>

Example

python3 build_universal_image.py \
    --dockerfile ./Dockerfile \
    --output ./my-app.tar \
    --repo_url https://github.com/example/my-app.git \
    --commit abc123def456

Parameters

Parameter	Required	Description
`--dockerfile`	Yes	Path to the Dockerfile to build
`--output`	Yes	Path for the output .tar file (OCI format)
`--repo_url`	Yes	Repository URL (passed as REPO_URL build arg)
`--commit`	Yes	Commit SHA (passed as BASE_COMMIT build arg)
`--context`	No	Build context directory (default: current directory)
`--builder-name`	No	Name of the buildx builder (default: velora-builder)
`--verbose` or `-v`	No	Enable verbose logging

Advanced Examples

With custom build context:

python3 build_universal_image.py \
    --dockerfile ./docker/Dockerfile.prod \
    --output ./dist/image.tar \
    --repo_url https://github.com/myorg/myrepo.git \
    --commit $(git rev-parse HEAD) \
    --context ./src

With verbose logging:

python3 build_universal_image.py \
    --dockerfile ./Dockerfile \
    --output ./image.tar \
    --repo_url https://github.com/example/repo.git \
    --commit abc123 \
    --verbose

Get Help:

python3 build_universal_image.py --help

How It Works

PR Evaluation Workflow

Part 1: Build + Base Testing
- Clone repository and fetch PR references
- Detect language and test framework
- Build Docker image with all dependencies
- Run base tests (without PR changes)
- Save state for Part 2
Part 2: Patch + Evaluation
- Generate patch files (full, test-only, code-only)
- Verify patches apply cleanly
- Run test-patch-only tests (should FAIL)
- Run full-patch tests (should PASS)
- Categorize tests (F2P, P2P, etc.)
- Generate comprehensive metadata
- Organize outputs and artifacts

Multi-Architecture Docker Build

Pre-flight Checks: Verifies Docker and Buildx are available
Builder Setup: Creates or uses a Docker Buildx builder with docker-container driver
Multi-Arch Build: Builds for both linux/amd64 and linux/arm64 simultaneously
Caching: Uses local cache (.buildx-cache/) to speed up subsequent builds
OCI Output: Creates a single universal .tar file containing both architectures

Using the Output Image

Loading the Image

The output .tar file is OCI-compliant and works on any architecture:

# Load the image
docker load < my-app.tar

# Docker automatically selects the correct architecture
# - On Intel/AMD machines: uses linux/amd64 variant
# - On ARM machines (Graviton, Apple Silicon): uses linux/arm64 variant

Verifying the Image

# List loaded images
docker images

# Inspect the image to see supported platforms
docker image inspect <image-name> | grep Architecture

Distributing the Image

The .tar file is portable and can be:

Copied to other machines
Uploaded to cloud storage
Distributed to colleagues
Imported on any Docker-compatible system

Build Cache

The script uses a local cache directory (.buildx-cache/) to speed up builds:

Location: ./.buildx-cache in the working directory
Purpose: Caches layers to speed up subsequent builds
Cleanup: Delete the directory to clear cache if needed

# Clear build cache
rm -rf .buildx-cache

Dockerfile Requirements

Your Dockerfile should:

Accept REPO_URL and BASE_COMMIT as build arguments:
```
ARG REPO_URL
ARG BASE_COMMIT
```
Use multi-architecture compatible base images (e.g., ubuntu:24.04)
Avoid architecture-specific binaries unless conditionally installed

Example Dockerfile Structure

FROM ubuntu:24.04

ARG REPO_URL
ARG BASE_COMMIT

# Your build steps here
RUN apt-get update && apt-get install -y git

# Clone repository
RUN git clone "${REPO_URL}" /app
WORKDIR /app
RUN git checkout "${BASE_COMMIT}"

# Rest of your build...

Troubleshooting

Common Issues

Docker Permission Denied

# Add your user to docker group
sudo usermod -aG docker $USER

# Then log out and log back in, or run:
newgrp docker

Buildx Not Available

# Install Docker Buildx plugin (Ubuntu/Debian)
sudo apt-get update
sudo apt-get install docker-buildx-plugin

# Or on macOS with Homebrew:
brew install docker-buildx

Language/Test Detection Fails

Use manual overrides:

python3 -m automation_script.main_orchestrator \
    --language python \
    --test-cmd "pytest" \
    https://github.com/owner/repo/pull/123 \
    /path/to/workspace

Docker Build Fails

Check logs and consider using self-healing features (automatic) or manual intervention:

# View unified workflow logs
cat /path/to/workspace/<pr_folder>/logs/workflow.log

Builder Creation Fails

# Remove existing builder and let script recreate it
docker buildx rm multi-arch-builder

# Then run the script again

Build Fails for One Architecture

This can happen if:

Base image doesn't support that architecture
Architecture-specific binaries are being installed
Solution: Check Dockerfile for architecture-specific commands

Slow Cloning for Large Repositories

Use shallow clone:

python3 -m automation_script.main_orchestrator \
    --shallow-clone \
    https://github.com/owner/repo/pull/123 \
    /path/to/workspace

Getting More Help

See automation_script/README.md for detailed PR evaluation usage, supported languages, output formats, and VeloraHarness integration
Check INSTALL.txt for detailed installation guidance
Run scripts with --help flag for usage information
Review logs in workspace directory under logs/
Verify all prerequisites are properly installed

Performance Tips

Use Build Cache: The script caches layers automatically
Parallel Builds: Both architectures build in parallel
Network Speed: First build downloads images; subsequent builds are faster
Use .dockerignore: Exclude unnecessary files from build context

Architecture Support

The script builds for:

linux/amd64: Intel and AMD x86_64 processors
linux/arm64: ARM 64-bit processors (AWS Graviton, Apple Silicon, etc.)

To modify architectures, edit the PLATFORMS constant in the script:

PLATFORMS = "linux/amd64,linux/arm64"

FAQ

Q: Can I build for just one architecture?
A: Yes, modify PLATFORMS = "linux/amd64" in the script.

Q: How large will the output file be?
A: Approximately 2x the size of a single-architecture image (contains both variants).

Q: Can I push to Docker Hub instead of creating a tar file?
A: Yes, replace --output=type=oci,dest=<file> with --push and tag the image.

Q: Does this work with private registries?
A: Yes, ensure you're logged in with docker login before running.

Q: Can I add more architectures?
A: Yes, add to PLATFORMS (e.g., "linux/amd64,linux/arm64,linux/arm/v7"), but ensure base images support them.

Supported Languages

The PR evaluation system automatically detects and supports:

Python - pytest, unittest
JavaScript/TypeScript - npm test, jest, mocha
Go - go test
Rust - cargo test
Java - maven, gradle
C# - dotnet test
Ruby - rake test, rspec

If auto-detection fails, use --language and --test-cmd flags to override.

Output Artifacts

After successful PR evaluation, the workspace contains:

workspace/<pr_folder>/
├── artifacts/                  # base/, test_patch_only/, pr/ test outputs
├── docker_images/              # Saved Docker image tar + generated Dockerfile
├── logs/                       # Unified workflow log (workflow.log)
├── metadata/                   # instance.json + swe-bench-instance.json
├── patches/                    # pr.patch, test.patch, code.patch
└── repo/                       # Cloned repository

Advanced Configuration

Environment Variables

DOCKER_BUILDKIT=1 - Enable BuildKit (recommended)
DOCKER_BUILD_JOBS - Number of parallel build jobs

Custom Configurations

Edit automation_script/config.py for:

Docker build settings
Timeout configurations
Test execution parameters
Logging preferences

Contributing

When contributing:

Follow existing code style and structure
Test changes thoroughly with different PR types
Update documentation for new features
Ensure all linters pass

Support

For issues or questions:

See automation_script/README.md for detailed usage and troubleshooting of the PR evaluation system
Check INSTALL.txt for installation issues
Review Troubleshooting section above
Run scripts with --help or --verbose flags
Check logs in workspace directory
Verify all prerequisites are installed correctly

Additional Resources

PR Evaluation Guide: See automation_script/README.md for the full pipeline docs
Multi-Build Utilities: See automation_script_build_multi/README.md for build helper utilities
Installation Guide: See INSTALL.txt
Python Dependencies: See requirements.txt
Automated Install: Run install.sh

License

This tool is provided as-is for automated PR evaluation and multi-architecture Docker image building.

Happy Automating! 🚀

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
F2P_finder		F2P_finder
automation_script		automation_script
automation_script_build_multi		automation_script_build_multi
.DS_Store		.DS_Store
.gitignore		.gitignore
INSTALL.txt		INSTALL.txt
README.md		README.md
build_universal_image.py		build_universal_image.py
install.sh		install.sh
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation

Automation_jager

Table of Contents

Features

🚀 Multi-Architecture Docker Builds

🔬 PR Evaluation System

🛠️ Robust Automation

Documentation

Quick Start

Automatic Installation (Recommended)

Manual Installation

Installation

Option 1: Automated Installation (Recommended)

Option 2: Manual Installation

Option 3: Python Dependencies Only

Prerequisites

System Requirements

Supported Operating Systems

Verify Installation

Project Structure

Usage

1. PR Evaluation System

Full Workflow (Build + Test + Evaluate)

With Language/Test Command Overrides

Performance Options

Get Help

2. Universal Docker Image Builder

Basic Usage

Example

Parameters

Advanced Examples

How It Works

PR Evaluation Workflow

Multi-Architecture Docker Build

Using the Output Image

Loading the Image

Verifying the Image

Distributing the Image

Build Cache

Dockerfile Requirements

Example Dockerfile Structure

Troubleshooting

Common Issues

Docker Permission Denied

Buildx Not Available

Language/Test Detection Fails

Docker Build Fails

Builder Creation Fails

Build Fails for One Architecture

Slow Cloning for Large Repositories

Getting More Help

Performance Tips

Architecture Support

FAQ

Supported Languages

Output Artifacts

Advanced Configuration

Environment Variables

Custom Configurations

Contributing

Support

Additional Resources

License

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages