RustyQ: An Asynq Clone (Rust & Redis Task Queue)

Project Intention

This project aims to create a simplified, practical clone of Asynq, built entirely with Rust and Redis. The primary goal is educational: by building a task queue from scratch, you'll learn the fundamentals and intricacies of Rust programming and Redis database operations.

Why Rust?

Rust is a modern systems programming language offering powerful features:

• Performance: Fast and efficient code execution.
• Safety: Memory safety without a garbage collector.
• Concurrency: Built-in support for asynchronous programming patterns.

Why Redis?

Redis is a highly performant in-memory data store:

• Simplicity: Easy-to-use data structures like lists, streams, and hashes.
• Speed: Ultra-fast data access and manipulation, ideal for task queue operations.
• Learning: Offers practical exposure to key-value storage, pub/sub, and stream processing.

What You'll Gain

By developing this Asynq-inspired task queue, you'll:

• Gain practical Rust programming experience.
• Understand Redis concepts deeply through hands-on implementation.
• Learn asynchronous programming and concurrent task execution.
• Explore real-world systems design, error handling, and scalability concerns.

Features

• Guaranteed at least one execution of a task
• Scheduling of tasks
• Retries of failed tasks
• Automatic recovery of tasks in the event of a worker crash
• Weighted priority queues
• Strict priority queues
• Low latency to add a task since writes are fast in Redis
• De-duplication of tasks using unique option
• Allow timeout and deadline per task
• Allow aggregating group of tasks to batch multiple successive operations
• Flexible handler interface with support for middlewares
• Ability to pause queue to stop processing tasks from the queue
• Periodic Tasks
• Support Redis Sentinels for high availability
• Integration with Prometheus to collect and visualize queue metrics
• Web UI to inspect and remote-control queues and tasks
• CLI to inspect and remote-control queues and tasks

Intended Audience

• Rust beginners and intermediate developers.
• Developers new to Redis.
• Individuals interested in building asynchronous and distributed systems.

Getting Started

To start:

• Familiarize yourself with Rust basics using the Rust Book.
• Install Redis locally or via Docker.
• Clone the project and experiment with task enqueueing and processing.

Happy learning and coding!