CharacterBench — local regression testing for character/memory changes

[RayzerCat76]

I’m validating a small local tool called CharacterBench that may fit the same privacy-first workflow as LettuceAI: repeatable regression testing for character behaviour after model, prompt, memory, or card changes.

The problem I’m trying to solve is the manual retest loop: a change feels better overall, but one relationship, knowledge boundary, memory behaviour, or personality trait quietly gets worse.

CharacterBench 0.2.1 currently works like this:

1. Import a standard Character Card JSON/PNG.
2. Review the starter checks it derives from fields already present in the card.
3. Run those checks locally against an Ollama model.
4. Save the result as a local baseline.
5. Change one thing and rerun.
6. Inspect matching unchanged checks that meaningfully improved or regressed.

It is not a model leaderboard, and the alpha deliberately treats scores as diagnostic signals rather than scientific measurements. If the test set changes, it avoids pretending the overall scores are directly comparable.

The local UI binds to localhost, CharacterBench adds no telemetry, and saved baselines contain score metadata/test fingerprints rather than prompts, responses, transcripts, lore, or card contents. The project is Apache-2.0 licensed.

I’m not claiming a LettuceAI integration yet. I’m looking for the product failure cases first: a real card that imports badly, a starter check that misunderstands the character, a false positive/negative, or a regression it actually catches after a model/memory change.

Website: https://characterbench-alpha.netlify.app
Source/release: https://github.com/RayzerCat76/CharacterBenchmark
Validation thread: RayzerCat76/CharacterBenchmark#1

If anyone here already compares providers/models or changes memory settings and then manually retests a character, one sanitized real run would be much more useful to me than general feedback.

[RayzerCat76]

Focused follow-up for v0.2.1-alpha: I’m trying to get one real privacy-safe tester run, not broader promotion. If CharacterBench fits a character/memory workflow you already use, please report one concrete outcome: a regression it correctly catches, an obvious failure it misses, an in-character response it falsely flags, or setup friction that stops you reaching a useful result.

Tester guide: https://github.com/RayzerCat76/CharacterBenchmark/blob/main/TESTER_GUIDE.md
Live site: https://characterbench-alpha.netlify.app
Release: https://github.com/RayzerCat76/CharacterBenchmark/releases/tag/v0.2.1-alpha
Validation issue: RayzerCat76/CharacterBenchmark#1

Sanitized evidence is enough; please omit private prompts, character data and API keys.