Skip to content

README.md

Latest commit

 

History

History
98 lines (72 loc) · 2.08 KB

README.md

File metadata and controls

98 lines (72 loc) · 2.08 KB

TinyDFS - Distributed File System (Java)

A lightweight, educational Distributed File System implemented in pure Java.
It simulates file chunking, block distribution, metadata management, and integrity handling similar to Google GFS and Hadoop HDFS.

Features

  • File Chunking: Splits files into 64KB blocks
  • Distributed Storage: Stores blocks across multiple simulated DataNodes
  • SHA-256 Integrity: Each block is hashed for verification
  • Metadata Management: NameNode tracks filenames and block IDs
  • Console Interface: Commands to store, list, retrieve, and corrupt files
  • Local Simulation: Entire DFS runs locally without networking

Getting Started

Prerequisites

  • Java Development Kit (JDK) 8 or higher

Installation

  1. Clone this repository
  2. Navigate to the project directory

Running the Application

Compile:

javac TinyDFS.java

Run:

java TinyDFS

Use the console commands shown below.

Available Commands

Store a file:

put "your content here" filename.txt

List files:

list

Retrieve a file:

get filename.txt

Simulate corruption:

corrupt filename.txt

Project Structure

  • TinyDFS.java — Core logic (NameNode, DataNode, DFS Client)
  • dfs_storage/ — Auto-created storage directory
  • node_1/, node_2/, etc. — DataNode block storage folders
  • README.md — Documentation

How It Works

Write Path

Input → Split 64KB blocks → Hash with SHA-256 → Round-robin node assignment → Save to disk

Read Path

Metadata lookup → Fetch blocks → Reassemble file → Output

Metadata

Uses:

ConcurrentHashMap<String, List<UUID>>

Technologies Used

  • Java SE (JDK 8+)
  • SHA-256 hashing
  • Local file system for DataNode simulation
  • ConcurrentHashMap for metadata

Future Improvements

  • Block replication
  • DataNode heartbeat system
  • Java Socket-based networking
  • Automatic integrity verification

License

Open source for educational and personal use.

Acknowledgements

Inspired by concepts from Google File System (GFS) and Hadoop HDFS, simplified for learning.