[FEATURE] The question about RocksInodeStore

[zuston]

After reviewing the RocksInodeStore, I realized it’s not strictly necessary. All metadata operations on fs_dir are already persisted to RocksDB, and the current usage in the codebase is mainly to reload and reconstruct the entire directory tree.

Given that, I don’t think RocksDB is the only viable option for checkpointing. Would it be possible to serialize the in-memory view to a file and use that instead? Removing RocksDB could potentially simplify the design and improve overall metadata performance.

I might have misunderstood the design, and I’d really appreciate it if you could share more about the rationale behind this approach.

[jlon]

Core Role of RocksDB in Curvine

Your observation is very insightful! Indeed, on the surface, RocksInodeStore appears to be mainly used for rebuilding the in-memory directory tree, but it actually serves much more complex and critical responsibilities:

1. Multi-dimensional Data Indexing & Querying

RocksDB doesn't just store the directory tree - it maintains 5 independent Column Families:

// curvine-server/src/master/meta/store/rocks_inode_store.rs:29-33
pub const CF_INODES: &'static str = "inodes";      // inode metadata
pub const CF_EDGES: &'static str = "edges";        // parent-child relationships
pub const CF_BLOCK: &'static str = "block";        // block metadata
pub const CF_LOCATION: &'static str = "location";  // block location info
pub const CF_MOUNTPOINT: &'static str = "mountpoints"; // mount points

These data structures support:

• Efficient prefix queries: prefix_scan for finding subdirectories
• Block location queries: Fast lookup of all replica locations by block_id
• Mount point management: Supporting multi-tenant filesystem

2. High-Performance Block Location Queries

This is a crucial point! Look at this function:

// curvine-server/src/master/meta/store/rocks_inode_store.rs:63-75
pub fn get_locations(&self, id: i64) -> CommonResult<Vec<BlockLocation>> {
    let prefix = RocksUtils::i64_to_bytes(id);
    let iter = self.db.prefix_scan(Self::CF_LOCATION, prefix)?;
    // Fast lookup of all replica locations for a specific block
}

In large-scale distributed filesystems, these queries are extremely frequent. With pure in-memory storage, memory consumption would be enormous.

3. Transactional Batch Operations

// curvine-server/src/master/meta/store/rocks_inode_store.rs:183-237
impl<'a> InodeWriteBatch<'a> {
    // Support atomic batch writes
    pub fn add_location(&mut self, id: i64, loc: &BlockLocation)
    pub fn write_inode(&mut self, inode: &InodeView) 
    pub fn write_block(&mut self, meta: &BlockMeta)
}

This ensures ACID properties for metadata operations.

4. Efficient Raft Snapshot Implementation

RocksDB's checkpoint mechanism is designed for Raft snapshots:

// curvine-server/src/master/journal/journal_loader.rs:277-286
fn create_snapshot(&self, node_id: u64, last_applied: u64) -> RaftResult<SnapshotData> {
    let fs_dir = self.fs_dir.read();
    let dir = fs_dir.create_checkpoint(last_applied)?; // RocksDB checkpoint
    let data = RaftUtils::create_file_snapshot(&dir, node_id, last_applied)?;
}

RocksDB checkpoints are incremental and atomic, much more efficient than serializing the entire in-memory tree.

Why Can't We Simply Replace It with Serialization?

Performance Issues:

1. Memory consumption: Block location information in large-scale filesystems would consume GB-level memory
2. Query efficiency: RocksDB's LSM-Tree structure is optimized for range and prefix queries
3. Snapshot efficiency: Serializing entire in-memory trees is much slower than RocksDB checkpoints

Functional Limitations:

1. Transaction support: Pure in-memory solutions struggle to provide ACID guarantees
2. Crash recovery: RocksDB provides WAL and fault recovery mechanisms
3. Data consistency: Distributed environments require reliable persistence

Wisdom of Design Trade-offs

Curvine adopts a hybrid architecture:

• Hot data (directory tree structure): Kept in memory for millisecond-level access
• Cold data (block locations, metadata): Stored in RocksDB, queried on-demand
• Dual-write mechanism: Memory + RocksDB synchronized updates

This design achieves an excellent balance between performance and reliability.

Summary

RocksDB here isn't optional - it handles:

1. Large-scale block location indexing
2. Efficient Raft snapshots
3. Transactional metadata operations
4. Crash recovery guarantees

While a pure serialization approach might seem simpler, in production environments at scale, it would encounter bottlenecks in performance, memory consumption, and reliability.

Your thinking is very valuable! This design does add complexity, but it's a necessary engineering trade-off that distributed storage systems must make to support production-grade workloads.

The key insight is that Curvine isn't just storing a directory tree - it's managing a complex distributed metadata system where:

• Block locations need frequent, efficient queries
• Transactional consistency is critical
• Raft snapshots must be fast and reliable
• The system must handle petabyte-scale data with millions of files

In this context, RocksDB isn't just a storage choice - it's an essential component that enables the system to scale to production requirements.

[zuston]

Thank you for your patient reply, I will learn it carefully.

[szbr9486]

hi @zuston I converted this issue to discussion