Skip to content

Follow changes reads only the folders that changed - #44

Merged
JE-Chen merged 4 commits into
mainfrom
dev
Nov 20, 2026
Merged

JE-Chen merged 4 commits into
mainfrom
dev

Conversation

@JE-Chen

@JE-Chen JE-Chen commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Releases what dev has added since 0.2.15: Follow changes reads only the folders that changed, instead of rescanning the whole scan.

Only the changed folder is read (U-20261120-01)

  • A change lists only the folder whose own entries changed. Its subfolders are kept, matched by name and file identity (ScanOptions.keep, core/refresh.py), so a file saved directly in the scanned folder no longer rescans everything. A subfolder replaced meanwhile is read whole.
  • All changed folders are read in one worker and put back into the same nodes, so the tree keeps its position, expanded folders and selection.
  • The five-minute full check is gone. Past 256 changed folders, their parent folders are read whole instead of the root.
  • The native feed keeps running while folders are read. The scope is captured again only when subfolders were added, removed or replaced, and an NTFS journal feed resumes from the position held when the refresh began.

No gaps and no needless whole rescans (U-20261120-03)

  • Moves, renames, compression and other source operations now stop only the capture that reads the tree. The feed keeps reporting, so a cancelled review no longer rescans the root. A feed lost during a Recycle Bin move, permanent delete or Undo is still withdrawn by it.
  • A hard link made anywhere on an NTFS volume no longer rescans the scan. The file's names are looked up by file number, and only folders holding one inside the scan are read.
  • The journal is read on at once while records wait. Directory notifications use a 256 KiB buffer (64 KiB over a network). An inotify feed re-reads the folders it newly watches once armed.
  • Changes are coalesced for 0.5 s and polled every 0.25 s (was 1 s and 1 s).
  • macOS is supported through one FSEvents stream (core/mac_watch.py). It reports folders, and it reports dropped events (whole root) and events it could not keep up with (that folder, read whole).
  • Japanese and Korean "unsupported" strings and the Japanese inotify label were wrong machine translations; they are fixed.

Verified natively, with two fixes (U-20261120-02)

  • On this machine (administrator, readable C: journal), both the NTFS journal and directory notifications, in a hidden window, on made-up folders:

    Case Journal Directory notifications
    One file added in the scanned folder 1.05 s; only that folder read 0.75 s
    New folder, plus a file added during the handover 2.01 s; both files shown 1.78 s
    Subfolder deleted 0.74 s 0.73 s
    60 s idle nothing read nothing read

    No case rescanned the whole root.

  • Worst window stall on a 120,208-entry tree: 29–85 ms.

  • The result navigation redrew every page icon on each refresh (about 100 ms); icons are now drawn once per colour.

Tests (U-20261120-04)

  • Two timing-sensitive tests that failed once each on loaded CI runners (test_elevation_worker, test_recurring_review) wait longer.

Not verified

  • FSEvents ran only on the macOS CI runner: no real Mac, Finder operations or external disks.
  • Linux inotify handover (old and new feeds briefly together, near the watch limit) is covered only by CI.
  • Long-term use on real folders. Tracked in progress.md #144.

Test plan

  • ruff check .
  • Related tests locally: 61 files, 880 passed (the known long-path test excluded)
  • New: test_folder_refresh.py; refresh, handover, operation and inotify cases in test_change_watch_gui.py; journal handover, name lookup, read-on, buffer and FSEvents cases in test_change_watch.py
  • Native: NTFS journal hard-link test (local and Windows CI); FSEvents core and window tests (macOS CI)
  • Tests workflow on dev green for 4044196

A change now lists only the folder whose own entries changed and keeps
its subfolders (ScanOptions.keep, core/refresh.py), so a file saved
directly in the scanned folder no longer rescans the whole scan. All
changed folders are read in one worker and refilled in place, so the
view keeps its place, expansion and selection.

The native feed no longer stops while folders are read. The scope is
captured again only when subfolders changed, and the new feed takes over
once armed; an NTFS journal feed reads on from where the refresh began.
Directory notifications report the real folder, too many changes read
their parent folder whole instead of the root, and the five-minute full
reconciliation is gone.
Verified Follow changes natively on the USN journal and on directory
notifications: a change in the scanned folder reads only that folder,
a folder added during the hand-over is caught from the held journal
position, and nothing rescans the whole root.

Two costs found on the way: the result navigation redrew every page
icon whenever its texts were updated (about 100 ms after each refresh),
and a folder removed before it was read asked for its folder again even
when that folder had just been read in the same pass.
Source operations now stop only the capture, which reads the tree; the
feed keeps reporting, so an operation leaves no gap and a cancelled
review no longer reads the whole root. A feed lost during a Recycle Bin
move, permanent delete or Undo is still withdrawn by it.

A hard link made anywhere on an NTFS volume no longer reads the whole
scan: the file's names are looked up by file number and only folders
holding one inside the scan are read. The journal is read on at once
while records wait, directory notifications get a 256 KiB buffer, and
an inotify feed reads the folders it newly watches once armed.

macOS follows through one FSEvents stream, which reports folders and
says when it dropped events or could not keep up.
The elevation tests released their held launch after 2 s as a fallback;
on a loaded runner that came first, so the launch had ended before the
close request looked. The recurring review waited 10 s for a scan,
analysis, review and move in a row. Both wait longer now.

The name lookup test simulated a gone file with OSError's winerror
argument, which only Windows sets; it sets the attribute itself now.
@sonarqubecloud

Copy link
Copy Markdown

@JE-Chen
JE-Chen merged commit 1ad5afc into main Nov 20, 2026
35 checks passed
@JE-Chen
JE-Chen deployed to windows-signing November 20, 2026 08:34 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
windows-signing — 40441964 Deployed Nov 20, 2026 by JE-Chen via build-exe #18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant