🧠 Cortex

Next-generation AI proxy and routing infrastructure

Getting Started • Documentation • Contributing • Community

📑 Table of Contents

• About
• Features
• Getting Started
  • Prerequisites
  • Installation
  • Quick Start
• Usage
• Configuration
• Architecture
• Tools
• Development
• Documentation
• Roadmap
• Contributing
• License
• Community
• Acknowledgments

📖 About

Cortex is a high-performance AI infrastructure project written in Rust. Its primary component, Rook, acts as an intelligent AI proxy and router designed to simplify and harden your AI application stack.

In an ecosystem with multiple LLM providers (OpenAI, Anthropic, Google, etc.), Cortex provides a unified, OpenAI-compatible interface with enterprise-grade features out of the box. It solves common challenges such as provider outages, high latency, and lack of observability by providing automatic failover, efficient caching, and comprehensive audit logging.

✨ Features

• 🔌 OpenAI-compatible API: Drop-in replacement for existing OpenAI clients.
• ✨ Multi-provider routing: Support for OpenAI, Anthropic, Ollama, Gemini, and Groq.
• 🔄 Automatic fallback: Resilience against provider failures with configurable fallback chains.
• 🚀 Built-in caching: Reduce costs and latency with TTL-based in-memory caching.
• 📊 Audit logging: Full traceability of requests and responses backed by SQLite.
• 🏗️ Clean Architecture: Built with Domain-Driven Design (DDD) for high maintainability.
• 🦀 Rust-powered: Exceptional performance, memory safety, and concurrency.
• 🔒 Production-ready: Includes health checks, observability (tracing, metrics), and encryption.

🚀 Getting Started

Prerequisites

• Rust 1.81 or later (the version is pinned in rust-toolchain.toml)
• Cargo (included with Rust)
• (Optional) Just task runner: cargo install just
• (Optional for dashboard) Node.js 18+ and pnpm

Installation

From source

git clone https://github.com/dallay/cortex.git
cd cortex
cargo build --release -p rook

Binary releases

Download pre-built binaries from Releases for:

• Linux (x86_64, aarch64)
• macOS (Intel, Apple Silicon)
• Windows (x86_64, aarch64)

Quick Start

1. Configure Rook: Create a config file at ./rook.toml (or use the ROOK_CONFIG env var to point elsewhere):

   [server]
   host = "127.0.0.1"
   port = 3773
   
   [providers.openai]
   api_key = "sk-..."
   enabled = true

2. Run the server:

   # Using cargo
   ROOK_CONFIG=./rook.toml cargo run --release -p rook
   
   # Using binary
   ./target/release/rook

3. Test the proxy:

   curl http://localhost:3773/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "gpt-4",
       "messages": [{"role": "user", "content": "Hello!"}]
     }'

🛠 Usage

Multi-provider Fallback

Configure a fallback chain to ensure your application stays online even if a provider fails:

# In your rook.toml
[[fallback_chains]]
name = "robust-gpt4"
chain = ["openai/gpt-4", "anthropic/claude-3-opus"]

Dashboard Access

Cortex comes with a built-in dashboard for management. On a fresh database, Rook enters bootstrap mode. Check the logs for a setup token and open the dashboard (if running locally, typically http://localhost:4747 if you run it via just run-dashboard).

⚙️ Configuration

See the Configuration Guide for detailed options.

Quick reference:

• Server: host, port, timeouts, encryption settings.
• Providers: API keys, endpoints, health check intervals.
• Cache: TTL, size limits.
• Audit: log location, rotation.

🏗 Architecture

Cortex follows Clean Architecture principles with clear separation of concerns:

cortex/
├── apps/rook/              # Binary entry point, DI bootstrap
├── crates/
│   ├── domain/             # Business logic
│   │   ├── shared-kernel/  # Zero-deps common types
│   │   └── rook-core/      # Domain models and ports
│   ├── application/        # Use cases
│   │   └── rook-usecases/  # Orchestration logic
│   └── infrastructure/     # External adapters
│       ├── providers-*/    # LLM provider clients
│       ├── cache-memory/   # Caching implementation
│       ├── audit-sqlite/   # Audit logging
│       └── transport-axum/ # HTTP server
└── docs/                   # Documentation

See Architecture Documentation for details.

🧰 Tools

Tool	Status	Description
Rook	✅ Active	AI proxy/router with multi-provider support
Future tools	🚧 Planned	TBD

🛠 Development

Setup

# Clone and setup
git clone https://github.com/dallay/cortex.git
cd cortex
just setup

Development commands

# Format code
just fmt

# Run linter
just clippy

# Run tests
just test

# Watch mode (auto-check, test, clippy)
just dev

# Full CI locally
just ci-local

Testing

# Run all tests
cargo test --workspace

# Run specific crate tests
cargo test -p rook-core

# Run E2E tests (requires Docker)
just test-e2e

📚 Documentation

• Architecture
• Configuration
• Providers
• API Reference

🗺 Roadmap

Current Focus

• 🔨 Dynamic provider registry
• 🔐 Enhanced authentication and authorization
• 📈 Advanced metrics and observability

Planned Features

• Rate limiting and quota management
• Request/response transformation
• Plugin system
• WebUI dashboard enhancements
• Streaming support optimization
• Additional provider integrations

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Follow the Rust Code of Conduct.

📄 License

This project is licensed under the MIT License.

👥 Community

• 💬 GitHub Discussions - Questions and ideas
• 🐛 Issue Tracker - Bug reports and feature requests

🙏 Acknowledgments

Built with:

• Rust - Systems programming language
• Tokio - Async runtime
• Axum - Web framework
• Rusqlite - SQLite bindings

Special thanks to all contributors and the Rust community.