Feature: Replay timeline jumps via keyframe states

[tetrisface]

I have embarked on this experimentative pipedream. Please do tell if there is a better issue. I searched for replay and keyframe.

First goal is to rewind a replay and do timeline jumps without desyncs or speedups.

I'm trying to keep desync resolutions documented so its possible to refactor the POC into regression tests and some separate PR's that can be useful even if the main "replay bundle" store might not.

I do believe I'm following approach number 2 in this comment ("perfect sync-safe saves") #1388 (comment)

Discussion of the feature: #2704

Other related issues:
#1179
#350

So outside of this goal (but very interesting) is the later optimization stage to try to shrink the storage and transfer of the keyframes.

WIP branches:
https://github.com/tetrisface/RecoilEngine/tree/keyframes-rewind-catchup
https://github.com/tetrisface/Beyond-All-Reason/tree/replay

2026-06-09 update
even though the agent might succeed it might not even be possible to work towards merging anything meaningful due to the scale of the work. but i will let it stew for a bit more out of curiosity.

2026-06-13
large body of work. but I dont think the POC has to grow much more now.

DISCLAMER: Completely done with AI (GPT-5.5 on max effort)

[burnhamrobertp]

Saw this while digging into the replay-seek / save-load side of things, and it might be useful for the regression-test angle you mentioned in the description.

It's a small tool that runs a scenario twice - one continuous run that saves partway through, then a second run that loads that save and continues - and reports the first frame where the two runs' sync checksums stop matching. On current master they split right after the reload (about frame 1020 in my run), which fits with the loaded game not reproducing the original yet.

It pulls the checksums straight out of the two replays with demotool, so there's no gadget or special build involved. demotool didn't actually build on master though, so that's a separate little fix it leans on:

• the comparator: #3031
• demotool build fix: #3030

Looks like your frame-level diagnostics are already comparing restored vs uninterrupted runs from inside the engine - this does a similar thing from the outside, off the recorded checksums, so it could give an objective number for how far the keyframe work gets (does that first-split frame move out or disappear as more checkpoint state gets restored). Happy to point it at your branch and see where it lands. Is that close to how you're measuring it, or are you doing it a different way?

[burnhamrobertp]

Tried building the branch to run some sync checks against it and hit a wall on the entt submodule. Your tip commit (609ba98) bumps it to 8981c2171624138db5255e36a009c3553b6ae360, but that commit isn't fetchable from skypjack/entt (the URL the submodule points at) - GitHub returns "no commit found," and it's not on any branch/tag/PR I can pull. Looks like the bump just didn't get pushed along with the branch.

Could you push it, or point the submodule at a fork that has it? As-is the branch can't be cloned and built from scratch, which also blocks running it headless or through any batch setup.

If you remember, I'm also curious what's in it. The engine's own entt pin is still the old v3.10.3 and the ECS save/load still uses the v3.10-era API (e.g. registry.each), so it reads more like a small local patch than a version jump - was it a determinism tweak to entt itself (entity-id recycling / storage iteration order, that kind of thing)? I tried subbing in stock entt to get unblocked and it wasn't a clean drop-in, so if your change matters for sync I'd rather build against the real thing than guess.

[tetrisface]

Full disclosure: I don't understand a lot of this. All of it was generated by AI. Under guidance but heavily just document driven.

But yes, I think we are employing the same kind of verification method just that mine does the work inside via the sync logging. Most of the state representations is very much black box to me but its fun to learn some new and advanced concepts while putting the agent to work.

the current harness does the comparison from inside the replay/checkpoint flow via Lua/engine log markers. It compares the emitted SYNCCHECK hashes after restore/resume against the original replay sequence, so a mismatch gives immediate feedback about the restore frame/range that failed.

Tried building the branch to run some sync checks against it and hit a wall on the entt submodule. Your tip commit (609ba98) bumps it to 8981c2171624138db5255e36a009c3553b6ae360, but that commit isn't fetchable from skypjack/entt (the URL the submodule points at) - GitHub returns "no commit found," and it's not on any branch/tag/PR I can pull. Looks like the bump just didn't get pushed along with the branch.

Pushed it here now tetrisface/entt@8981c21
And yes, as I understand, this is just bypassing a guard to use deterministic entity ids for our use case.

It does actually seem to work though. the ssf files are huge. each ssf is about 10x larger/68MB during the synccheck runs.

Lots of graphical glitches and a long jump freeze.

Here are two scripts to record and demo. i have other scripts that trigger checkpoint jumps that are very useful for the ai to jump and verify. like -StartFrame 30 -TargetFrame 120 to jump from frame 30 to 120.

replay-record-jumpable.ps1

param(
	[string]$WorkspaceRoot = (Resolve-Path (Join-Path $PSScriptRoot "..")).Path,
	[string]$BarDataRoot = (Join-Path $env:LOCALAPPDATA "Programs\Beyond-All-Reason\data"),
	[string]$EngineTag = "2025.06.24",
	[string]$SpringExe = "",
	[string]$GameSource = "",
	[string]$MapName = "Comet Catcher Remake 1.8",
	[string]$MapArchive = "",
	[int]$DurationFrames = 1500,
	[int]$CheckpointInterval = 23,
	[int]$SynctestSeed = 0,
	[int]$TimeoutSeconds = 420
)

$ErrorActionPreference = "Stop"

function Quote-ProcessArgument([string]$Argument) {
	if ($Argument -notmatch '[\s"]') {
		return $Argument
	}
	return '"' + ($Argument -replace '"', '\"') + '"'
}

function Get-FreeUdpPort() {
	$udp = [System.Net.Sockets.UdpClient]::new(0)
	try {
		return ([System.Net.IPEndPoint]$udp.Client.LocalEndPoint).Port
	} finally {
		$udp.Dispose()
	}
}

function Resolve-RunRelativePath([string]$PathText, [string]$BaseDir) {
	$cleanPath = $PathText.Trim()
	if ([System.IO.Path]::IsPathRooted($cleanPath)) {
		return $cleanPath
	}
	return Join-Path $BaseDir $cleanPath
}

function Find-ReplayCheckpointBundle([string]$DemoPath, [string]$InfoLog, [string]$WriteDir) {
	$sessionMatches = [regex]::Matches($InfoLog, "\[ReplayCheckpoint\] wrote session manifest (?<path>[^\r\n]+?session\.json)")
	for ($i = $sessionMatches.Count - 1; $i -ge 0; --$i) {
		$sessionPath = Resolve-RunRelativePath $sessionMatches[$i].Groups["path"].Value $WriteDir
		$bundlePath = Split-Path -Parent $sessionPath
		if (Test-Path -LiteralPath $bundlePath -PathType Container) {
			return $bundlePath
		}
	}

	$legacyBundle = "$DemoPath.replay-checkpoints"
	if (Test-Path -LiteralPath $legacyBundle -PathType Container) {
		return $legacyBundle
	}

	return $legacyBundle
}

if (-not $SpringExe) {
	$SpringExe = Join-Path $BarDataRoot "engine\recoil_$EngineTag\spring.exe"
}
if (-not $GameSource) {
	$GameSource = Join-Path $WorkspaceRoot "Beyond-All-Reason"
}
if (-not $MapArchive) {
	$MapArchive = Join-Path $BarDataRoot "maps\comet_catcher_remake_1.8.sd7"
}
if ($SynctestSeed -le 0) {
	$SynctestSeed = Get-Random -Minimum 1 -Maximum ([int]::MaxValue)
}

$springExePath = (Resolve-Path $SpringExe).Path
$gameSourcePath = (Resolve-Path $GameSource).Path
$mapArchivePath = (Resolve-Path $MapArchive).Path

if (-not (Test-Path -LiteralPath $springExePath -PathType Leaf)) {
	throw "spring.exe not found: $springExePath"
}
if (-not (Test-Path -LiteralPath $gameSourcePath -PathType Container)) {
	throw "BAR game source not found: $gameSourcePath"
}
if (-not (Test-Path -LiteralPath $mapArchivePath -PathType Leaf)) {
	throw "Map archive not found: $mapArchivePath"
}
if ($DurationFrames -lt 1) {
	throw "DurationFrames must be positive"
}
if ($CheckpointInterval -lt 1) {
	throw "CheckpointInterval must be positive"
}

$stamp = Get-Date -Format "yyyyMMdd_HHmmss"
$runDir = Join-Path $WorkspaceRoot ".cache\replay-jumpable\run_$stamp"
$dataDir = Join-Path $runDir "data"
$gamesDir = Join-Path $dataDir "games"
$mapsDir = Join-Path $dataDir "maps"
$writeDir = Join-Path $runDir "write"
$configPath = Join-Path $runDir "springsettings.cfg"
$scriptPath = Join-Path $runDir "startscript_jumpable.txt"
$summaryPath = Join-Path $runDir "summary.json"
$replayScriptPath = Join-Path $runDir "replay_windows.ps1"
$hostPort = Get-FreeUdpPort
$gameId = [guid]::NewGuid().ToString("N")

New-Item -ItemType Directory -Force -Path $gamesDir, $mapsDir, $writeDir | Out-Null
New-Item -ItemType Junction -Path (Join-Path $gamesDir "Beyond-All-Reason.sdd") -Target $gameSourcePath | Out-Null

$mapDest = Join-Path $mapsDir (Split-Path $mapArchivePath -Leaf)
try {
	New-Item -ItemType HardLink -Path $mapDest -Target $mapArchivePath | Out-Null
} catch {
	Copy-Item -LiteralPath $mapArchivePath -Destination $mapDest
}

@"
[Game]
{
	[allyTeam0]
	{
		startrectright = 0.2;
		startrectbottom = 1;
		startrectleft = 0;
		numallies = 0;
		startrecttop = 0;
	}

	[allyTeam1]
	{
		startrectright = 1;
		startrectbottom = 1;
		startrectleft = 0.8;
		numallies = 0;
		startrecttop = 0;
	}

	[team0]
	{
		Side = Armada;
		Handicap = 0;
		RgbColor = 0.99609375 0.546875 0;
		AllyTeam = 0;
		TeamLeader = 0;
	}

	[team1]
	{
		Side = Cortex;
		Handicap = 0;
		RgbColor = 0.546875 0.546875 0.99609375;
		AllyTeam = 1;
		TeamLeader = 0;
	}

	[modoptions]
	{
		synctest_autostart = 1;
		synctest_start_frame = 1;
		synctest_total_frames = $DurationFrames;
		synctest_area_fraction = 0.45;
		synctest_area_offset_x = 0.2;
		synctest_area_offset_z = 0.2;
		synctest_unit_multiplier = 0.25;
		synctest_seed = $SynctestSeed;
		synctest_gameover = 1;
		replaycheckpoint_disable_dev_reloaders = 1;
		startmetal = 10000;
		startmetalstorage = 20000;
		startenergy = 10000;
		startenergystorage = 20000;
	}

	[player0]
	{
		IsFromDemo = 0;
		Name = ReplayJumpableHost;
		Team = 0;
		Spectator = 1;
		rank = 0;
	}

	[ai0]
	{
		Host = 0;
		IsFromDemo = 0;
		Name = InactiveAI(1);
		ShortName = NullAI;
		Team = 0;
		Version = 0.1;
	}

	[ai1]
	{
		Host = 0;
		IsFromDemo = 0;
		Name = InactiveAI(2);
		ShortName = NullAI;
		Team = 1;
		Version = 0.1;
	}

	hostip = 127.0.0.1;
	hostport = 0;
	numplayers = 1;
	startpostype = 2;
	mapname = $MapName;
	ishost = 1;
	numusers = 3;
	gametype = Beyond All Reason `$VERSION;
	GameID = $gameId;
	GameStartDelay = 1;
	myplayername = ReplayJumpableHost;
	nohelperais = 0;
}
"@ | Set-Content -Encoding ASCII -Path $scriptPath

@"
DemoFromDemo = 0
Fullscreen = 0
Windowed = 1
XResolution = 1280
YResolution = 720
Sound = 0
snd_volmaster = 0
snd_volmusic = 0
music = 0
music_loadscreen = 0
WorkerThreadCount = 0
HostPortDefault = $hostPort
ReplayCheckpointDisableDevReloaders = 1
RecordDemo = 1
ReplayCheckpointAutoRecord = 1
ReplayCheckpointRecordInterval = $CheckpointInterval
"@ | Set-Content -Encoding ASCII -Path $configPath

$previousSpringDataDir = $env:SPRING_DATADIR
$env:SPRING_DATADIR = $dataDir

Write-Host "Recording jumpable replay"
Write-Host "  run: $runDir"
Write-Host "  engine: $springExePath"
Write-Host "  game: $gameSourcePath"
Write-Host "  map: $MapName"
Write-Host "  duration_frames: $DurationFrames"
Write-Host "  checkpoint_interval: $CheckpointInterval"
Write-Host "  synctest_seed: $SynctestSeed"

$processStartInfo = [System.Diagnostics.ProcessStartInfo]::new()
$processStartInfo.FileName = $springExePath
$processStartInfo.WorkingDirectory = Split-Path $springExePath -Parent
$processStartInfo.WindowStyle = [System.Diagnostics.ProcessWindowStyle]::Hidden
$processStartInfo.UseShellExecute = $false
$processStartInfo.Arguments = (@("--config", $configPath, "--write-dir", $writeDir, $scriptPath) |
	ForEach-Object { Quote-ProcessArgument $_ }) -join " "

$process = [System.Diagnostics.Process]::Start($processStartInfo)
try {
	$deadline = (Get-Date).AddSeconds($TimeoutSeconds)
	while (-not $process.HasExited -and (Get-Date) -lt $deadline) {
		Start-Sleep -Milliseconds 500
		$process.Refresh()
	}
	if (-not $process.HasExited) {
		Stop-Process -Id $process.Id -Force -ErrorAction SilentlyContinue
		throw "Jumpable replay recording timed out after $TimeoutSeconds seconds"
	}
} finally {
	$env:SPRING_DATADIR = $previousSpringDataDir
}

$infoLogPath = Join-Path $writeDir "infolog.txt"
if (-not (Test-Path -LiteralPath $infoLogPath -PathType Leaf)) {
	throw "infolog.txt not found: $infoLogPath"
}

$infoLog = Get-Content -LiteralPath $infoLogPath -Raw
if ($infoLog -match "(?i)\b(desync|sync error)\b|differs from the host's copy") {
	throw "Potential sync/archive problem found in $infoLogPath"
}

$demo = Get-ChildItem -LiteralPath (Join-Path $writeDir "demos") -Filter "*.sdfz" -ErrorAction SilentlyContinue |
	Sort-Object LastWriteTime -Descending |
	Select-Object -First 1
if (-not $demo) {
	throw "No recorded demo found under $writeDir\demos"
}

$checkpointBundle = Find-ReplayCheckpointBundle $demo.FullName $infoLog $writeDir
$checkpointCount = 0
if (Test-Path -LiteralPath $checkpointBundle -PathType Container) {
	$checkpointCount = @(Get-ChildItem -LiteralPath $checkpointBundle -Filter "replaycheckpoint_*.ssf" -ErrorAction SilentlyContinue).Count
}
if ($checkpointCount -le 0) {
	throw "Replay checkpoint bundle is missing or empty: $checkpointBundle"
}

$summary = [ordered]@{
	engine_tag = $EngineTag
	scenario = "jumpable"
	gametype = "Beyond All Reason `$VERSION"
	map = $MapName
	duration_frames = $DurationFrames
	synctest_seed = $SynctestSeed
	sync_hash = [ordered]@{
		available = $false
		path = $null
		digest = $null
		frame_count = 0
		first_frame = $null
		last_frame = $null
	}
	checkpoint_bundle = $checkpointBundle
	checkpoint_count = $checkpointCount
	best_episode = 0
	episodes = @(
		[ordered]@{
			episode = 0
			game_id = $gameId
			duration_frames = $DurationFrames
			demo_path = $demo.FullName
		}
	)
}
$summary | ConvertTo-Json -Depth 10 | Set-Content -Encoding ASCII -Path $summaryPath

$replayScriptLines = @(
	"`$RunDir = `"$runDir`"",
	"`$RunData = `"$dataDir`"",
	"`$SpringExe = `"$springExePath`"",
	"`$Demo = `"$($demo.FullName)`"",
	"`$ReplayWrite = Join-Path `$RunDir `"replay_write`"",
	"`$Config = Join-Path `$RunDir `"replay_springsettings.cfg`"",
	"`$HostPort = Get-Random -Minimum 20000 -Maximum 60000",
	"`$env:SPRING_DATADIR = `$RunData",
	"New-Item -ItemType Directory -Force -Path `$ReplayWrite | Out-Null",
	"`"DemoFromDemo = 0`nFullscreen = 0`nWindowed = 1`nSound = 0`nsnd_volmaster = 0`nsnd_volmusic = 0`nmusic = 0`nmusic_loadscreen = 0`nWorkerThreadCount = 0`nHostPortDefault = `$HostPort`nReplayCheckpointDisableDevReloaders = 1`nReplayCheckpointUseBundle = 1`nReplayTimelineCheckpointJumps = 1`" | Set-Content -Encoding ASCII -Path `$Config",
	"Set-Location (Split-Path `$SpringExe -Parent)",
	"& `$SpringExe --config `$Config --write-dir `$ReplayWrite --window `$Demo"
)
$replayScriptLines | Set-Content -Encoding ASCII -Path $replayScriptPath

Write-Host "Jumpable replay written to $runDir"
Write-Host "  summary: $summaryPath"
Write-Host "  replay_script: $replayScriptPath"
Write-Host "  demo: $($demo.FullName)"
Write-Host "  checkpoint_bundle: $checkpointBundle ($checkpointCount saves)"

replay-demo-jumpable.ps1

param(
	[Parameter(Mandatory = $true)]
	[string]$RunDir,
	[string]$SpringExe = "",
	[string]$OutputSuffix = "",
	[int]$WindowWidth = 0,
	[int]$WindowHeight = 0
)

$ErrorActionPreference = "Stop"

function Quote-ProcessArgument([string]$Argument) {
	if ($Argument -notmatch '[\s"]') {
		return $Argument
	}
	return '"' + ($Argument -replace '"', '\"') + '"'
}

function Convert-ExtendedPath([string]$Path) {
	if ($Path -and $Path.StartsWith("\\?\")) {
		return $Path.Substring(4)
	}
	return $Path
}

function Get-FreeUdpPort() {
	$udp = [System.Net.Sockets.UdpClient]::new(0)
	try {
		return ([System.Net.IPEndPoint]$udp.Client.LocalEndPoint).Port
	} finally {
		$udp.Dispose()
	}
}

$resolvedRunDir = (Resolve-Path $RunDir).Path
$summaryPath = Join-Path $resolvedRunDir "summary.json"
$runData = Join-Path $resolvedRunDir "data"

if (-not (Test-Path -LiteralPath $summaryPath -PathType Leaf)) {
	throw "summary.json not found: $summaryPath"
}
if (-not (Test-Path -LiteralPath $runData -PathType Container)) {
	throw "Run data directory not found: $runData"
}
if (($WindowWidth -lt 0) -or ($WindowHeight -lt 0)) {
	throw "WindowWidth and WindowHeight must be non-negative"
}
if (($WindowWidth -eq 0) -ne ($WindowHeight -eq 0)) {
	throw "Set both WindowWidth and WindowHeight, or leave both as 0"
}

$summary = Get-Content -LiteralPath $summaryPath -Raw | ConvertFrom-Json
$demoPath = Convert-ExtendedPath ([string]$summary.episodes[$summary.best_episode].demo_path)
if (-not (Test-Path -LiteralPath $demoPath -PathType Leaf)) {
	throw "Demo not found: $demoPath"
}

if (-not $SpringExe) {
	$generatedReplayScript = Join-Path $resolvedRunDir "replay_windows.ps1"
	if (-not (Test-Path -LiteralPath $generatedReplayScript -PathType Leaf)) {
		throw "SpringExe is required when replay_windows.ps1 is missing"
	}

	& powershell -ExecutionPolicy Bypass -File $generatedReplayScript
	if ($LASTEXITCODE -ne 0) {
		throw "Replay failed"
	}
	return
}

$springExePath = (Resolve-Path $SpringExe).Path
if (-not (Test-Path -LiteralPath $springExePath -PathType Leaf)) {
	throw "spring.exe not found: $springExePath"
}

if ($OutputSuffix -eq "") {
	$OutputSuffix = "demo_" + (Get-Date -Format "yyyyMMdd_HHmmss")
}
if ($OutputSuffix -notmatch "^[A-Za-z0-9_.-]+$") {
	throw "OutputSuffix may contain only letters, numbers, underscore, dash, and dot"
}

$writeDir = Join-Path $resolvedRunDir "replay_demo_write_$OutputSuffix"
$configPath = Join-Path $resolvedRunDir "replay_demo_springsettings_$OutputSuffix.cfg"
$hostPort = Get-FreeUdpPort

$configLines = @(
	"DemoFromDemo = 0",
	"Fullscreen = 0",
	"Windowed = 1",
	"Sound = 0",
	"snd_volmaster = 0",
	"snd_volmusic = 0",
	"music = 0",
	"music_loadscreen = 0",
	"WorkerThreadCount = 0",
	"HostPortDefault = $hostPort",
	"ReplayCheckpointDisableDevReloaders = 1",
	"ReplayCheckpointUseBundle = 1",
	"ReplayTimelineCheckpointJumps = 1"
)
if ($WindowWidth -gt 0) {
	$configLines += "XResolution = $WindowWidth"
	$configLines += "YResolution = $WindowHeight"
}
($configLines -join "`n") + "`n" | Set-Content -Encoding ASCII -Path $configPath

New-Item -ItemType Directory -Force -Path $writeDir | Out-Null

Write-Host "Launching jumpable replay demo"
Write-Host "  run: $resolvedRunDir"
Write-Host "  engine: $springExePath"
Write-Host "  demo: $demoPath"
Write-Host "  write_dir: $writeDir"
Write-Host "  config: $configPath"
Write-Host ""
Write-Host "Use the bottom replay timeline to jump. Checkpoint jumps are enabled."

$previousSpringDataDir = $env:SPRING_DATADIR
$env:SPRING_DATADIR = $runData
try {
	$processStartInfo = [System.Diagnostics.ProcessStartInfo]::new()
	$processStartInfo.FileName = $springExePath
	$processStartInfo.WorkingDirectory = Split-Path $springExePath -Parent
	$processStartInfo.UseShellExecute = $false
	$processStartInfo.Arguments = (@("--config", $configPath, "--write-dir", $writeDir, "--window", $demoPath) |
		ForEach-Object { Quote-ProcessArgument $_ }) -join " "

	$process = [System.Diagnostics.Process]::Start($processStartInfo)
	$process.WaitForExit()
	if ($process.ExitCode -ne 0) {
		throw "Replay failed with exit code $($process.ExitCode)"
	}
} finally {
	$env:SPRING_DATADIR = $previousSpringDataDir
}

BEYOND.1.MP4

[bruno-dasilva]

ok but that video is really funny lol