2XKO Dashboard

A production-ready data pipeline and analytics platform for managing creators and clips data across multiple platforms (Twitch, YouTube, Reddit, Twitter). This project migrates from AITable to a robust PostgreSQL + Metabase stack for better performance, scalability, and analytics capabilities.

🎯 Project Overview

This system provides:

• Automated Data Pipeline: Nightly sync jobs from AITable to PostgreSQL
• Multiple UI Layers: Metabase for analytics, Supabase Studio for data management, custom React dashboard
• Normalized Database Schema: Optimized for performance and scalability
• Real-time Analytics: Fast queries and visualizations for content creators and their highlights

🏗️ Architecture

• Database: PostgreSQL hosted on Supabase
• Analytics: Metabase for business intelligence and saved views
• Data Management: Supabase Studio for spreadsheet-style editing
• Custom Dashboard: React application for specialized views
• Data Pipeline: Python sync service with automated scheduling

📊 Data Models

• creators: Creator information with platform-specific data
• clips: Clip data with foreign key relationships to creators
• clip_tags: Normalized many-to-many tag relationships
• SQL Views: Pre-built analytics queries for common use cases

🚀 Quick Start

Prerequisites

• Python 3.12+
• Supabase account and project
• Environment variables configured

Installation

1. Clone the repository:

git clone <repository-url>
cd 2xko-dashboard

2. Install dependencies:

uv sync

3. Set up environment variables:

cp .env.example .env
# Edit .env with your Supabase credentials

4. Test database connectivity:

python tests/test_db_connectivity.py

🔧 Configuration

Environment Variables

Create a .env file with the following variables:

DATABASE_URL=postgresql://user:password@host:port/database
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_KEY=your-anon-key
AITABLE_API_KEY=your-aitable-key

Database Setup

Run the database setup script to create tables and views:

python setup_database.py

📁 Project Structure

2xko-dashboard/
├── main.py                 # Main application entry point
├── setup_database.py       # Database schema setup
├── setup_env_guide.py      # Environment setup guide
├── docs/                   # Documentation
│   └── groq-work.md        # Technical implementation details
├── tests/                  # Test files and connectivity reports
│   ├── test_db_connectivity.py
│   ├── database_demo.py
│   └── *.md               # Test reports and summaries
└── .taskmaster/           # Task management configuration
    ├── config.json
    └── docs/prd.txt       # Product Requirements Document

🧪 Testing

The project includes comprehensive database connectivity tests:

• Basic Connectivity: test_db_connectivity.py
• Comprehensive Testing: test_comprehensive_connectivity.py
• Database Operations: test_database_operations.py
• Final Validation: test_final_connectivity.py

Run all tests:

python -m pytest tests/

📈 Current Status

✅ Completed Tasks:

• Project repository setup and configuration
• Supabase database setup and connectivity testing
• Environment variable configuration
• Database connection validation
• Comprehensive test suite

🔄 In Progress:

• Database schema creation (creators, clips, clip_tags tables)
• Data import script development
• Metabase configuration

📋 Upcoming:

• Automated sync job implementation
• Custom React dashboard development
• API layer creation

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests to ensure connectivity
5. Submit a pull request

📄 License

[Add your license information here]

🔗 Related Documentation

• Product Requirements Document
• Technical Implementation Details
• Database Connectivity Reports

📞 Support

For questions or issues, please refer to the documentation or create an issue in the repository.