Persist terminal sessions across quit with zmx

.gitignore

@@ -10,6 +10,9 @@ Macterm/Resources/*
 
 build/
 
+# Swift/zig build caches (e.g. a local .build from tooling) — never committed.
+.build/
+
 # Generated by xcodegen from project.yml
 Macterm.xcodeproj/
 

AGENTS.md

@@ -6,7 +6,7 @@ A macOS terminal emulator built with SwiftUI and libghostty. Single-window app w
 
 ```bash
 mise install          # Install tools (gh, swiftformat, swiftlint, xcodegen, xcbeautify)
-mise run setup        # Download pre-built GhosttyKit.xcframework
+mise run setup        # Download pre-built GhosttyKit.xcframework + zmx binary
 mise run run          # Build and launch (debug)
 mise run logs         # Stream live logs from the debug app (--release for release app)
 mise run format       # Auto-fix formatting with swiftformat
@@ -19,7 +19,7 @@ mise run build        # Release build + DMG
 
 Requires macOS 14+, Swift 6.0+. Liquid glass and a few chrome refinements are macOS 26 (Tahoe) features that degrade gracefully on older systems (gated behind `WindowAppearance.glassSupported` / `#available`). GhosttyKit is a pre-built xcframework from `thdxg/ghostty` (a fork that adds CI builds); no zig toolchain needed.
 
-`GhosttyKit.xcframework` and the `Macterm/Resources/{ghostty,terminfo,…}` contents are gitignored artifacts downloaded by `mise run setup` — **every fresh checkout, including a git worktree, must run `mise run setup`** before it can build. Don't symlink them from another checkout: `setup.sh` only re-downloads when the artifact is _absent_ (presence check, not version check), so a symlinked copy silently goes stale. To refresh a stale artifact, delete it and re-run setup.
+`GhosttyKit.xcframework`, the `Macterm/Resources/{ghostty,terminfo,…}` contents, and the bundled `Macterm/Resources/zmx/zmx` binary are gitignored artifacts downloaded by `mise run setup` — **every fresh checkout, including a git worktree, must run `mise run setup`** before it can build. Don't symlink them from another checkout: `setup.sh` only re-downloads when the artifact is _absent_ (presence check, not version check), so a symlinked copy silently goes stale. To refresh a stale artifact, delete it and re-run setup. (`GhosttyKit` comes from the `thdxg/ghostty` release; `zmx` from the `thdxg/zmx` release — both prebuilt by CI, never compiled locally, since zig 0.15.2 can't link against the macOS 26+/Xcode 26.4 SDK locally.)
 
 ## Releasing & Updates
 
@@ -95,6 +95,17 @@ Two non-obvious terminfo facts (the regression behind #39/#40 — broken `TERM=x
 - **Never set TERMINFO ourselves.** At shell spawn libghostty _unconditionally overwrites_ `TERMINFO` with `dirname(GHOSTTY_RESOURCES_DIR)/terminfo`, so the bundle layout must make that derivation land on the dir we ship — terminfo MUST be a sibling of `ghostty/`, never inside it. `BundledResourcesTests` asserts this invariant.
 - **The terminfo tree uses the macOS hashed layout** — `terminfo/78/xterm-ghostty` (`x` = 0x78), not `terminfo/x/...`. It's a `tic -x` compiled tree shipped verbatim.
 
+### Session Persistence (zmx)
+
+Terminal shells survive app quit and reattach on relaunch, backed by [zmx](https://github.com/neurosnap/zmx) — a Zig session multiplexer that passes raw PTY bytes through (no tmux-style screen re-parse). Every surface's shell runs under `zmx attach macterm-<sessionID>`, injected via ghostty's **`command-wrapper`** config (a patch carried by the `thdxg/ghostty` fork — see below): the wrapper argv is prepended to the *fully resolved* shell command (after `login(1)` + shell integration), so OSC 7 cwd / OSC 133 framing — and thus live tab titles and cwd-on-split — stay intact.
+
+- **Bundled binary.** Built by `thdxg/zmx` CI (single squashed patch commit on upstream, like the ghostty fork) and downloaded by `setup.sh` to `Macterm/Resources/zmx/zmx` (gitignored), embedded into the bundle at `Contents/Resources/zmx/zmx` by `embed-zmx.sh` (a `project.yml` post-compile phase). Never compiled locally — zig 0.15.2 can't link against the macOS 26+/Xcode 26.4 SDK.
+- **`ZmxClient`** (`Macterm/System/ZmxClient.swift`) — cache-free wrapper around the binary. `ZmxSessionID` (`macterm-<uuid>`), `ZmxSocketBudget` (bypass wrapping when `sun_path` would overflow — never gate kills on it), `ZmxSessionListParser` (the `zmx ls` tab-delimited parser), `ZmxAttach.resolveLaunch` (interactive → argv wrapper; declared `run:` → `/bin/sh -c` string), and `ZmxReaper.orphans` (pure orphan-selection). A 5s subprocess timeout + zombie-reap bounds every call so a stuck daemon can't hang the close/quit path.
+- **Stable session id.** `Pane.sessionID` (a UUID **distinct from** `Pane.id`, which is regenerated on restore) is persisted in `PaneSnapshot`. On restore the rebuilt pane reuses it, so its surface reattaches the live daemon instead of spawning fresh. `PaneSnapshot` also persists `workingDirectory`.
+- **Detach vs. kill.** `pane.destroySurface()` only *detaches* (the daemon lives on). `pane.killPersistentSession()` is the explicit kill, called **only on permanent close** — `TerminalTab.removePane`, `AppState.removeProject`/`closeTab`, and `executeLayoutPlan`'s `panesToDestroy`. Transient teardown (`unloadProject`, window hide, tab-switch churn) never kills. On quit with `terminateSessionsOnQuit` on, `applicationWillTerminate` kills all sessions **synchronously** (`ZmxClient.killSessionsBlocking`) — a detached `Task` would never run before the process exits.
+- **Launch reaper.** `AppState.restoreSelection` calls `ZmxClient.reapOrphans(knownSurfaceIDs:)` — kills `macterm-*` sessions with `clients == 0` that no restored pane claims (crash/force-quit orphans). Prefix-scoped (`macterm-` only, never a co-resident `supa-*` Supacode session); a `nil` probe or `clients > 0` spares everything.
+- **Quit.** Quitting detaches by default (shells reattach next launch). `Preferences.terminateSessionsOnQuit` (default off, Settings → Session Persistence) kills all sessions on quit instead; while off, the "processes will be killed" quit prompt is suppressed since nothing is lost.
+
 ### Tests (`MactermTests/`)
 
 One `XxxTests.swift` per production type, mirroring the source path. `@testable import Macterm` + `@MainActor` on test classes.
@@ -136,7 +147,7 @@ One `XxxTests.swift` per production type, mirroring the source path. `@testable
 - Workspaces → `~/Library/Application Support/Macterm/workspaces_v3.json`; projects → `projects.json`; wrapper configs (`macterm-defaults.conf`, `macterm-overrides.conf`) in the same directory. The directory name comes from `appDisplayName` (`CFBundleDisplayName`), so debug builds use `Macterm Debug/` — fully separate data per build, mirroring the bundle-ID split.
 - `Pane` IDs are not preserved across restarts — restore creates fresh UUIDs.
 - Declarative layouts are an _authorable_ file at `.macterm/layout.yaml` in the project root, applied/saved on demand via `applyLayout`/`saveLayout`. An unparseable file surfaces `LayoutFileError` and is never applied. JSON schema at `assets/layout.schema.json` — keep it in sync when layout types change.
-  - A committed layout file is the source of truth: on relaunch, `restoreSelection` skips the workspace snapshot for any project that has one; a project's first open auto-applies it (with no live panes the reconcile is pure-spawn, never destructive).
+  - On relaunch, `restoreSelection` restores **every** project's workspace snapshot (including projects with a committed layout file), so a layout project's live tabs/splits and each pane's persisted `sessionID` come back and its zmx sessions reattach. **Reopen never prompts and never re-applies the layout** — a restored snapshot always wins. The committed `.macterm/layout.yaml` only seeds a workspace when there's **no** snapshot to restore (a genuine first open, or first launch after the layout was committed): `autoApplyLayoutOnFirstOpen` applies it silently, guarded on `workspaces[id] == nil` so a restored snapshot makes it a no-op. (Applying a layout to an already-open project remains an explicit user action via `applyLayout`, which still stages `pendingLayoutApply` for confirmation when destructive.)
   - `save` records a pane's `run:` as its **live** foreground command (`ghostty_surface_foreground_pid` → `ProcessInspector` argv via `KERN_PROCARGS2`) — an idle prompt saves no `run`. `shell:` is recorded only when the pane sits in a _non-default_ shell. `run` and `shell` are mutually exclusive on a leaf.
   - `apply` (`LayoutReconciler`) matches live panes to declared ones by that same live `(run, cwd)`; a pane that quit its declared command is respawned. A plain-shell leaf matches an idle pane positionally, but a declared `shell:` only reuses an idle pane running that shell (basename compare). The live-command/live-shell lookups are injected closures (default `ProcessInspector`), so the logic is unit-testable.
   - The file's top-level `name:` is the project it was saved for — a mismatch on apply stages a confirmation warning. Tab `name:` is the tab's title, matched to live tabs during reconcile.
@@ -166,7 +177,6 @@ Macterm-side settings flow through `Preferences` (UserDefaults); ghostty-shaped
 
 ## Known Limitations
 
-- **No process persistence** — closing the app kills all shells (recommend tmux/zellij).
 - **Single window only** — multi-window would require a tmux-like daemon; out of scope.
 - **Not code-signed with a Developer ID** — first launch needs `xattr -cr /Applications/Macterm.app` (or Homebrew install); Sparkle updates verify EdDSA after that.
 - **Pane IDs not stable across restarts** — fresh views are created on restore.

Macterm/App/AppState.swift

@@ -88,6 +88,16 @@ final class AppState {
     @ObservationIgnored
     private var processNameTimer: Timer?
 
+    /// Guards against overlapping `zmx ls` refreshes of the foreground-resolver
+    /// cache: a slow probe shouldn't pile up behind the 250ms poll.
+    @ObservationIgnored
+    private var zmxForegroundRefreshInFlight = false
+
+    /// zmx session-persistence client. Defaults to the live binding; injectable
+    /// so tests can drive the launch-time orphan reaper without a real daemon.
+    @ObservationIgnored
+    var zmx: ZmxClient = .live
+
     init(workspaceStore: WorkspaceStore = WorkspaceStore()) {
         self.workspaceStore = workspaceStore
         autoTileObserver = NotificationCenter.default.addObserver(
@@ -112,6 +122,14 @@ final class AppState {
     /// workspaces. Each pane only republishes (and triggers a tab re-render)
     /// when its name actually changes, so this is cheap when nothing's moving.
     func refreshAllForegroundProcesses() {
+        // Refresh the zmx session→daemon-leader-pid cache that ProcessInspector
+        // uses to see past the `zmx attach` wrapper to the real foreground
+        // process. One `zmx ls` per tick (not per pane), off-main; the cache is
+        // read synchronously and is at most one tick stale (daemon pids are
+        // stable per session). A guard drops the refresh when one is already in
+        // flight so a slow `ls` can't pile up. Only when panes exist.
+        refreshZmxForegroundCacheIfNeeded()
+
         // Shell/raw-mode detection (KERN_PROCARGS2 + open/tcgetattr per pane)
         // and the quiet-settle only matter when the status indicator is shown;
         // skip them in icon mode so the default poll stays as cheap as before
@@ -136,6 +154,23 @@ final class AppState {
         if didAcknowledgeCompletion { saveWorkspaces() }
     }
 
+    /// Kick an off-main `zmx ls` to refresh `ZmxForegroundResolver`'s
+    /// session→daemon-leader-pid cache, unless one is already running or no
+    /// panes exist (nothing to resolve, and no reason to spawn a subprocess).
+    private func refreshZmxForegroundCacheIfNeeded() {
+        guard !zmxForegroundRefreshInFlight else { return }
+        let hasPanes = workspaces.values.contains { ws in
+            ws.tabs.contains { !$0.splitRoot.allPanes().isEmpty }
+        }
+        guard hasPanes else { return }
+        zmxForegroundRefreshInFlight = true
+        Task { [zmx] in
+            let map = await zmx.sessionLeaderPIDs()
+            ZmxForegroundResolver.updateCache(map)
+            await MainActor.run { self.zmxForegroundRefreshInFlight = false }
+        }
+    }
+
     private func recordProjectVisit(_ projectID: UUID) {
         projectRecency.push(projectID)
         UserDefaults.standard.set(projectRecency.items.map(\.uuidString), forKey: recencyKey)
@@ -165,29 +200,43 @@ final class AppState {
         hasRestoredSelection = true
         let snapshots = workspaceStore.load()
         let valid = Set(projects.map(\.id))
-        // A committed layout file is the source of truth: skip restoring the
-        // session snapshot for any project that has one, leaving its workspace
-        // nil so it rebuilds from `.macterm/layout.yaml` on open (below / on
-        // first select). Projects with no layout file restore their snapshot.
-        let pathByID = Dictionary(projects.map { ($0.id, $0.path) }, uniquingKeysWith: { a, _ in a })
-        for ws in WorkspaceSerializer.restore(from: snapshots, validIDs: valid)
-            where !LayoutFile.exists(atProjectRoot: pathByID[ws.projectID] ?? "")
-        {
+        // Restore every project's session snapshot — including layout projects.
+        // The snapshot carries the live tabs/splits and each pane's `sessionID`,
+        // so a layout project's panes reattach their zmx sessions and its live
+        // layout is remembered, instead of being silently overwritten by the
+        // declared `.macterm/layout.yaml` on every launch. The committed layout
+        // is offered (a confirmable apply), not force-applied over a restored
+        // session — see `autoApplyLayoutOnFirstOpen` (first-open seed only) and
+        // `offerLayoutIfChanged`.
+        for ws in WorkspaceSerializer.restore(from: snapshots, validIDs: valid) {
             workspaces[ws.projectID] = ws
         }
         if let id = Preferences.shared.activeProjectID,
            let project = projects.first(where: { $0.id == id })
         {
             activeProjectID = id
             recordProjectVisit(id)
-            // Build the active project from its layout file if it has one (its
-            // snapshot was skipped above); otherwise the restored snapshot stands
-            // and `ensureWorkspace` only creates a default when neither exists.
+            // Reopen is always silent — never prompt. A restored snapshot wins
+            // (reattach + remembered live layout); only when there's no snapshot
+            // does the committed layout seed the workspace, applied silently.
+            // `autoApplyLayoutOnFirstOpen` guards on `workspaces[id] == nil`, so
+            // a restored snapshot makes it a no-op automatically.
             autoApplyLayoutOnFirstOpen(project)
             ensureWorkspace(projectID: id, path: project.path)
             acknowledgeActiveTab(projectID: id)
             warmFocusedProject()
         }
+
+        // Reap zmx sessions left behind by a crash / force-quit: any macterm-*
+        // session with no attached client that no restored pane claims. Runs
+        // after workspaces are built so the known set is complete. Detached but
+        // claimed sessions (a pane that will reattach lazily on select) are
+        // spared because their surfaceID is in the known set.
+        let known = Set(workspaces.values
+            .flatMap(\.tabs)
+            .flatMap { $0.splitRoot.allPanes() }
+            .map(\.sessionID))
+        Task { [zmx] in await zmx.reapOrphans(knownSurfaceIDs: known) }
     }
 
     func saveWorkspaces() {
@@ -314,6 +363,8 @@ final class AppState {
         logger.debug("removeProject: \(projectID, privacy: .public)")
         if let ws = workspaces[projectID] {
             for pane in ws.tabs.flatMap({ $0.splitRoot.allPanes() }) {
+                // Project removed for good → kill each pane's persisted session.
+                pane.killPersistentSession()
                 pane.destroySurface()
             }
         }
@@ -345,6 +396,8 @@ final class AppState {
         else { return }
         logger.debug("closeTab: \(tabID, privacy: .public) project=\(projectID, privacy: .public)")
         for pane in tab.splitRoot.allPanes() {
+            // Tab closed for good → kill each pane's persisted session.
+            pane.killPersistentSession()
             pane.destroySurface()
         }
         ws.closeTab(tabID)
@@ -633,7 +686,11 @@ final class AppState {
         }
 
         // Destroy surfaces only AFTER the new trees no longer reference them.
+        // A pane the reconcile dropped is gone for good (not reused by any
+        // declared node), so kill its persisted session — otherwise it would
+        // orphan a clients==0 zmx daemon until the next launch's reaper.
         for pane in plan.panesToDestroy {
+            pane.killPersistentSession()
             pane.destroySurface()
         }
 

Macterm/App/MactermApp.swift

@@ -254,6 +254,19 @@ final class AppDelegate: NSObject, NSApplicationDelegate {
 
     func applicationWillTerminate(_: Notification) {
         onTerminate?()
+        // zmx sessions outlive the app by design (reattach on relaunch). Only
+        // when the user opted into terminate-on-quit do we kill them all here —
+        // and it must be a BLOCKING kill: a detached Task would never be
+        // scheduled before the process exits, so the per-pane fire-and-forget
+        // path (killPersistentSession) silently no-ops at quit. Collect the
+        // session ids and tear them down synchronously, bounded so a wedged
+        // daemon can't hang quit.
+        guard Preferences.shared.terminateSessionsOnQuit else { return }
+        let sessionIDs = (appState?.workspaces.values ?? [:].values)
+            .flatMap(\.tabs)
+            .flatMap { $0.splitRoot.allPanes() }
+            .map { ZmxSessionID.make(surfaceID: $0.sessionID) }
+        ZmxClient.live.killSessionsBlocking(sessionIDs)
     }
 
     func applicationShouldTerminate(_: NSApplication) -> NSApplication.TerminateReply {
@@ -273,6 +286,14 @@ final class AppDelegate: NSObject, NSApplicationDelegate {
     /// Walk every workspace + the quick terminal and emit one row per pane
     /// whose ghostty surface still has a foreground process running.
     private func collectRunningProcessRows() -> [RunningProcessRow] {
+        // With session persistence active (zmx bundled + under budget) and
+        // terminate-on-quit off, every shell survives quit and reattaches on
+        // relaunch — so nothing is actually lost and the "processes will be
+        // killed" prompt is wrong. Skip it. (When the user opts into
+        // terminate-on-quit, sessions DO die, so fall through and prompt.)
+        if ZmxClient.live.executableURL() != nil, !Preferences.shared.terminateSessionsOnQuit {
+            return []
+        }
         var rows: [RunningProcessRow] = []
         let projectsByID = Dictionary(
             uniqueKeysWithValues: (projectStore?.projects ?? []).map { ($0.id, $0) }