Go Telemetry Storage 🚀

A high-performance, asynchronous Go backend service that manages file metadata and orchestrates direct-to-cloud file transfers using MinIO.

This project demonstrates scalable system design by separating heavy file storage from lightweight metadata management, utilizing Go's concurrency model for background processing.

🏗️ Architecture & Core Features

• Direct-to-Storage Uploads (Bypassing the Server): Uses AWS SDK to generate Presigned URLs, allowing clients to upload/download heavy files directly to/from the S3 bucket without bottlenecking the Go backend.
• Asynchronous Worker Pool: Utilizes Go channels and Goroutines to handle post-upload processing (checksum calculation, status updates) without blocking the client's HTTP request.
• Data Integrity: Implements SHA-256 checksum validation to detect and prevent duplicate file uploads.
• Soft Deletion & Background Cleanup: Queues deleted files for asynchronous permanent removal from the storage bucket to optimize space.

🛠️ Tech Stack

• Language: Go (Golang)
• Database: MongoDB (Metadata source of truth)
• Storage: MinIO (Binary object storage)
• Concurrency: Native Go Channels and WaitGroups

📡 API Reference

Base URL: /api/v1

1. File Upload Flow

Method	Endpoint	Purpose
POST	/files/upload-request	Request Upload: Client sends file metadata (name, size). Server validates, creates a "Pending" DB record, and returns an S3 Presigned PUT URL.
POST	/files/{file_id}/complete	Confirm Upload: Client calls this after successfully uploading to S3. Triggers the Go background worker to calculate checksums and mark the file as "Completed".

2. Metadata & Retrieval

Method	Endpoint	Purpose
GET	/files	List Files: Returns a paginated list of the user's uploaded files and their current processing status.
GET	/files/{file_id}	Get Details: Returns specific metadata for a single file (e.g., checking if processing is finished).
GET	/files/{file_id}/download	Request Download: Server verifies file status and returns an S3 Presigned GET URL for secure, direct download from the storage bucket.

3. File Management

Method	Endpoint	Purpose
DELETE	/files/{file_id}	Delete File: Soft deletes the file record in PostgreSQL and sends an async task to the worker pool to permanently delete the binary object from S3.

🚀 Getting Started

Prerequisites

• Go 1.20+
• Docker & Docker Compose (for MongoDB and MinIO)

Installation

1. Clone the repository:

   git clone [https://github.com/krishna102001/go-telemetry-storage](https://github.com/krishna102001/go-telemetry-storage)
   cd go-telemetry-service

2. Start the storage and database infrastructure:

   docker-compose up -d

3. Run the application:

   go run main.go