Experimental version - temporary AI generated docs
Loreval measures how well language models do logic and reasoning on grid-based spatial puzzles. It's a web app for playing and designing puzzles, plus the harness that hands the same puzzles to an LLM and scores what comes back — no rubric, no judge model. A puzzle is either solved or it isn't.
Two tasks:
- Solving — given a puzzle as text, can the model produce a valid move sequence? Doors need switches hit first, agents block each other, one-way tiles and color-changing paint force ordering, not just pathfinding.
- Designing — given a difficulty, grid size, and a set of mechanics, can the model produce a well-formed, solvable puzzle that actually uses them?
Why grids: outcomes are verifiable by execution, fresh test items can be minted forever, and a failed attempt names the exact move where the plan broke.
Levels are plain text — a grid of characters, a legend, and agent declarations. Compact enough to fit in a prompt, which is exactly how a model receives it.
level "Paint Run" 9x7
grid = [
W W W W W W W W W,
W R R R W W W W W,
W R W S W W W W W,
W R W W W N W W W,
W R D R R R E P W,
W W W W W W W G W,
W W W W W W W W W,
]
tile S = tiles.switch(blue)
tile D = tiles.door(blue)
tile N = tiles.paint(green)
tile E = tiles.door(green)
tile P = tiles.paint(orange)
tile G = tiles.goal(orange)
agent(orange) start(1,1) and reach(7,5)
Tiles: wall, floor, empty, goal, door, switch, paint, one-way, lock. The interesting ones carry preconditions — a door opens only for a matching color or a matching switch, paint changes both what an agent can walk through and which exit it may claim. That turns a route into an ordering problem: the right detours, in the right order, before the last step.
Levels can be hand-written, drawn in the visual designer, or generated by a model. All three read and write the same text.
One engine decides whether a move is legal. The designer, interactive playback, the scorer and the solver all go through it, so there is no separate "eval version" of the game that can drift from the one you play.
Scoring replays a model's move list one step at a time. Each move comes back as one of five outcomes:
| Outcome | Meaning |
|---|---|
moved |
applied — the agent changed tiles |
illegal |
rejected: wall, grid edge, or another agent |
blocked |
accepted but nothing happened: shut door, lock, wrong side of a one-way |
no-agent |
no agent stands on the square the move names |
bad-direction |
not one of up/down/left/right |
A rejected move leaves the agent where it was while the plan assumes it moved, so every later coordinate is stale and reports no-agent. One mistake cascades. The statistics therefore track the first failure separately — that's the root cause; the rest is echo.
A deterministic search runs over the same engine before any model is called, and answers two things:
- Is this solvable? An unsolvable level is a broken test item, not a hard one.
- What's the optimum? "Solved in 57 moves" means nothing on its own. "57 against an optimum of 48" means the plan was 19% long.
The search is exponential in the number of agents. It's bounded by a node cap and a two-second budget, and yields while it works so the page stays responsive. Boards too large to settle report "not determined" rather than a guess.
Pick a level, tick some models, choose how many runs each, hit Run. Attempts go out a couple at a time and land in the table as they finish.
Keys are yours: Anthropic, OpenAI, Google, Groq, OpenRouter, or any OpenAI-compatible endpoint. They stay in your browser and are passed through to the provider — never persisted server-side. Custom model IDs are accepted, so a new model doesn't need a release here.
Three kinds of failure stay apart on purpose, because collapsing them lets one hide another: a request that never returned is a transport error and is excluded from the solve rate; a reply with no parsable move list is a format failure; a parsable plan the engine rejects is a reasoning failure.
Per model:
- Solve rate, with a 95% confidence interval and pass@k. Three solves out of three is not a 100% model — the interval says roughly 44–100%, and the table says so.
- vs optimal — typical winning length against the solver's optimum. Measured on the mean, not the best run, which would reward whichever attempt got luckiest.
- Illegal-move rate, and the root cause of each failed attempt.
- Dies at — how far into a plan the first rejected move appears. Separates "wrong from the first step" from "nearly had it".
- Time and tokens, with spread.
- Cost, in total and per solve. No rates ship with the app: published prices change and vary by account, and a stale number would skew the whole table. Enter your own; unpriced models read as a dash rather than a guess.
Charts render the same numbers where a picture is faster — overlapping confidence intervals, distance from optimal, how attempts first fail. Sessions are kept in the browser and export to CSV, one row per attempt.
Working: the DSL and engine, a visual designer with a two-way code editor, model solve attempts with step-by-step playback and a retry loop, model-generated levels, the deterministic solver, and multi-provider eval sessions with statistics, charts and CSV export.
Not yet: fixed benchmark suites, an interactive solve mode where the model sees the board after every move, logic gates and compound win conditions, and a CLI. Accounts and level sharing are built but unrouted until authentication ships.
No model comparison numbers are published. Run it with your own keys and get your own.
The roadmap/ folder holds the design notes, including the intended split into a shared core consumed by the web app, a generator, the eval harness and a CLI alike — so the app becomes one client among several rather than holding all the logic.
npm install
npm run dev # Vite dev server
npm run build # typecheck + production build
npm test
npm run lintNo environment variables are needed to run the app — model calls use the key you enter in the UI, and the proxy holds no credentials of its own. Supabase powers accounts and level sharing; without it the app boots and says so, and the designer, campaign and eval pages work regardless. To enable it, put these in a local .env (gitignored):
VITE_SUPABASE_URL=...
VITE_SUPABASE_ANON_KEY=...
Deploys to Vercel; the API routes are serverless functions.
Stack: React 19 + TypeScript, Vite, Tailwind CSS v4, React Router, Vitest, Supabase, Motion.