Currently, these are set at 64, and probably should be 128 or higher.
As a default these can be overridden anyway.
impl Default for ChunkingStrategy {
fn default() -> Self {
Self {
max_window_tokens: 1024,
overlap_tokens: vec![64], // 👈 default overlap = 64 tokens
summary_fallback_weight: 0.35,
summary_fallback_tokens: 512,
chunk_weight_floor: 0.1,
preprocessors: Vec::new(),
}
}
}
Currently, these are set at 64, and probably should be 128 or higher.
As a default these can be overridden anyway.