Skip to content

Consider default 128 token overlap #110

@jzombie

Description

@jzombie

Currently, these are set at 64, and probably should be 128 or higher.

As a default these can be overridden anyway.

impl Default for ChunkingStrategy {
    fn default() -> Self {
        Self {
            max_window_tokens: 1024,
            overlap_tokens: vec![64],    // 👈 default overlap = 64 tokens
            summary_fallback_weight: 0.35,
            summary_fallback_tokens: 512,
            chunk_weight_floor: 0.1,
            preprocessors: Vec::new(),
        }
    }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions