A modern web application for automatic video captioning using OpenAI's Whisper speech recognition model, integrated with Remotion for real-time caption rendering and styling.
Whisper Remotion is a full-stack video captioning platform that enables users to:
- Upload videos to AWS S3 with secure presigned URLs
- Extract audio from video files using FFmpeg
- Generate captions using Whisper.cpp (OpenAI's Whisper model)
- Render captions with Remotion player in real-time
- Customize caption styles with multiple presets (bottom-centered, top-bar, karaoke)
- Preview videos with synchronized captions before export
The application provides a seamless workflow from video upload to caption generation and preview, all within a modern React-based interface.
- Next.js 16 - React framework with App Router
- React 19 - UI library
- TypeScript - Type safety
- Tailwind CSS 4 - Styling
- ShadCN UI - Accessible component primitives
- Remotion - Video rendering and caption display
@remotion/player- Video player component@remotion/captions- Caption utilities@remotion/rounded-text-box- Styled caption boxes@remotion/google-fonts- Font loading
- Lucide React - Icons
- Next.js API Routes - Serverless API endpoints
- AWS SDK v3 - S3 integration for file storage
- Whisper.cpp - Speech-to-text transcription
- FFmpeg - Video/audio processing
- Docker - Containerization
- Alpine Linux - Lightweight base images
- AWS S3 - Object storage
- ESLint - Code linting
- Prettier - Code formatting
- TypeScript - Static type checking
- Node.js 22+ and npm
- Docker and Docker Compose (for containerized setup)
- AWS account with S3 bucket configured
- FFmpeg installed (if running locally without Docker)
-
Clone the repository
git clone <repository-url> cd whisper-remotion
-
Install dependencies
npm install
-
Set up environment variables
Create a
.env.localfile in the root directory:# AWS Configuration AWS_REGION=ap-south-1 AWS_ACCESS_KEY_ID=your_access_key_id AWS_SECRET_ACCESS_KEY=your_secret_access_key AWS_BUCKET_NAME=whisper-remotion # API Configuration NEXT_PUBLIC_API_BASE_URL=http://localhost:3000/api # Model Configuration MODEL_PATH=/app/models/ggml-base.bin
-
Run the development server
npm run dev
Open http://localhost:3000 in your browser.
Development Environment:
docker-compose -f docker-compose.dev.yaml up --buildProduction Environment:
docker-compose up --buildThe Docker setup includes:
- Whisper.cpp built from source
- FFmpeg for audio extraction
- Pre-configured Whisper model (ggml-base.bin)
- Optimized multi-stage builds for minimal image size
AWS_ACCESS_KEY_ID- AWS access keyAWS_SECRET_ACCESS_KEY- AWS secret keyAWS_BUCKET_NAME- S3 bucket nameAWS_REGION- AWS region (default:ap-south-1)
MODEL_PATH- Path to Whisper model (default:/app/models/ggml-base.bin)NEXT_PUBLIC_API_BASE_URL- API base URL for client (default:http://localhost:3000/api)
The system follows a modular, layered architecture optimized for scalable, cloud-native caption generation. Below is an overview of the main components and their interactions:
- Built with Next.js App Router and React, it provides a modern UI and serves API endpoints.
- Handles all user interactions, video uploads, caption style customization, and playback.
- Maintains application state (uploaded videos, selected caption style, status) via React Context.
- Uses AWS S3 as secure cloud storage for raw videos, audio files, and generated caption files (JSON).
- The frontend acquires secure S3 presigned URLs (upload/download) via API routes.
- FFmpeg: Extracts audio from uploaded videos for processing.
- Whisper.cpp: Processes audio on the backend, using OpenAI’s speech-to-text model, supporting multiple languages.
- Caption Converter: Custom module transforms Whisper transcriptions to Remotion-compatible caption format.
- Remotion Player: Advanced video player with live, styled captions. Supports custom styles (standard, karaoke, news ticker) and live preview.
- All layers communicate via serverless Next.js API routes.
- The architecture supports horizontal scaling (stateless processing; AWS S3 as persistent storage).
Typical Flow:
- User uploads a video via frontend; file is sent to S3.
- API triggers FFmpeg to extract audio.
- Whisper.cpp transcribes audio to a caption JSON.
- Caption converter formats captions for Remotion.
- Remotion Player in frontend displays captions in real time.
whisper-remotion/
├── app/ # Next.js App Router
│ ├── api/ # API routes
│ │ ├── health/ # Health check endpoint
│ │ ├── upload-url/ # Generate S3 presigned URLs
│ │ └── video/ # Video processing endpoints
│ │ └── [file_key]/
│ │ ├── audio/ # Extract audio from video
│ │ ├── transcription/# Generate captions
│ │ └── signed-url/ # Get video playback URL
│ ├── layout.tsx # Root layout
│ ├── page.tsx # Home page
│ └── globals.css # Global styles
│
├── components/ # React components
│ ├── ui/ # Reusable UI components
│ │ ├── alert.tsx
│ │ ├── button.tsx
│ │ ├── card.tsx
│ │ └── ...
│ └── video/ # Video-related components
│ ├── video-component.tsx # Remotion video component
│ ├── caption-renderer.tsx # Caption rendering logic
│ ├── unified-video-player.tsx
│ ├── remotion-player.tsx
│ └── ...
│
├── config/ # Configuration files
│ └── settings.ts # Environment settings
│
├── context/ # React context providers
│ └── video-context.tsx # Video state management
│
├── lib/ # Utility libraries
│ ├── awsS3.ts # AWS S3 operations
│ ├── axios.ts # HTTP client configuration
│ └── utils.ts # General utilities
│
├── types/ # TypeScript type definitions
│ ├── app.d.ts # Application types
│ └── caption-styles.ts # Caption style types
│
├── utils/ # Utility functions
│ ├── caption-converter.ts # Whisper to Remotion format
│ ├── ffmpeg.ts # FFmpeg operations
│ └── whisper.ts # Whisper CLI wrapper
│
├── Dockerfile # Production Docker image
├── Dockerfile.dev # Development Docker image
├── docker-compose.yaml # Production compose config
├── docker-compose.dev.yaml # Development compose config
├── next.config.ts # Next.js configuration
├── tsconfig.json # TypeScript configuration
└── package.json # Dependencies and scripts
- Secure video upload via AWS S3 presigned URLs
- Automatic audio extraction from video files
- Support for multiple video formats (MP4, etc.)
- Automatic speech-to-text using Whisper.cpp
- Timestamp synchronization
- Support for multiple languages
- Configurable model size (base, small, medium, large)
- Real-time caption preview with Remotion player
- Multiple caption style presets:
- Bottom Centered - Standard subtitle style
- Top Bar - News-style top banner
- Karaoke - Word-by-word highlighting
- Customizable styling (colors, fonts, positioning)
- Responsive text fitting and multi-line support
- Drag-and-drop video upload
- Real-time processing status
- Error handling and notifications
- Dark mode support (via next-themes)
Generate a presigned URL for video upload.
Response:
{
"uploadUrl": "https://...",
"fileKey": "uploads/videos/uuid.mp4"
}Extract audio from uploaded video.
Response:
{
"audioPath": "/tmp/audio-uuid.wav"
}Generate captions from audio file.
Request:
{
"audioPath": "/tmp/audio-uuid.wav"
}Response:
{
"transcript": { ... },
"transcriptKey": "transcripts/uuid.json"
}Get presigned URL for video playback.
Response:
{
"url": "https://..."
}Health check endpoint.
npm run dev # Start development server
npm run build # Build for production
npm run start # Start production server
npm run type-check # Type check without emitting
npm run lint:check # Check linting errors
npm run lint:fix # Fix linting errors
npm run format:check # Check code formatting
npm run format:fix # Format codeThe project uses:
- ESLint for linting (Next.js config)
- Prettier for code formatting
- TypeScript strict mode
Run npm run lint:fix and npm run format:fix before committing.
-
Build the production image:
docker build -t whisper-remotion . -
Run the container:
docker run -p 3000:3000 \ -e AWS_ACCESS_KEY_ID=your_key \ -e AWS_SECRET_ACCESS_KEY=your_secret \ -e AWS_BUCKET_NAME=your_bucket \ whisper-remotion
Ensure all required environment variables are set in your deployment environment. For production, use secure secret management (AWS Secrets Manager, Vercel Environment Variables, etc.).
The Dockerfile automatically downloads the Whisper model during build. For custom models, mount a volume or modify the MODEL_PATH environment variable.
We welcome contributions! Please follow these guidelines:
-
Fork the repository and create a feature branch
git checkout -b feature/amazing-feature
-
Make your changes following the code style guidelines
-
Test your changes thoroughly
npm run type-check npm run lint:check npm run build
-
Commit your changes with clear, descriptive messages
git commit -m "feat: add amazing feature" -
Push to your fork and open a Pull Request
- Follow the existing code style and conventions
- Write clear commit messages (conventional commits)
- Add tests for new features when applicable
- Update documentation for API changes
- Ensure all checks pass before submitting PR
- Be respectful and inclusive
- Provide constructive feedback
- Help others learn and grow
This project is licensed under the MIT License - see the LICENSE file for details.
- OpenAI Whisper - Speech recognition model
- Whisper.cpp - C++ implementation
- Remotion - Video rendering framework
- Next.js - React framework
- Radix UI - UI component primitives
For issues, questions, or suggestions:
- Open an issue on GitHub
- Check existing issues and discussions
- Review the documentation
Built with ❤️ using Next.js, Remotion, and Whisper