Conversation
…inks Two daemon/plugin robustness fixes (replaces the earlier "sync symlinks on daemon fork" attempt, which had no effect — the wipe happened after it). 1. Stop detached daemons that lost their PID file (utils/daemon/Daemon.py). A daemon is launched via `venv_exec` with `Daemon(daemon_id='<id>')` embedded in the executed code, so the daemon_id stays visible in the process command line. When the PID file is lost (the launcher exited without cleanup, leaving an orphaned daemon), `Daemon.pid` now recovers the PID by scanning for that marker, so status reflects reality instead of a false "stopped". `kill()`/`quit()` reap EVERY process matching the daemon_id (SIGTERM then SIGKILL) — no more `pkill meerschaum` or hunting the id in htop. New helpers: `_find_detached_pids`, `_kill_detached_processes`. 2. Don't remove valid plugin symlinks during sync (plugins/__init__.py). `sync_plugins_symlinks()` removed any symlink whose target wasn't in the CURRENT `PLUGINS_DIR_PATHS`. That path list is process-global and transiently falls back to the host default when `MRSM_PLUGINS_DIR` is momentarily absent (observed across daemon threads / replace_env contexts), so a sync in that window deleted another root's project plugin symlinks from the shared per-root `.internal/plugins` — making a `plugin:`-backed job fail to import. Now only TRULY stale symlinks (target gone) are reaped; valid cross-dir symlinks are left in place. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`replace_env`'s finally cleared `os.environ` entirely (`clear()` + `update(old_environ)`) before restoring it. In a multi-threaded daemon (job target + check-jobs RepeatTimer) a concurrent `sync_plugins_symlinks` could read `MRSM_PLUGINS_DIR` as absent during that blank window, fall back to the host plugins dir, and never create a project's plugin symlinks — so a `plugin:<name>` background job failed to import its plugin. - Restore `os.environ` by diffing instead of clearing it: keys present in `old_environ` are never removed, so `MRSM_PLUGINS_DIR` never momentarily vanishes. - Serialize the process-global env/paths swap under a new `_replace_env_lock` (held only around the mutation, not across `yield`). Complements the dev1 only-reap-stale symlink fix (which stopped the wipe class); this stops the never-created class. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`unload_plugins(remove_symlinks=True)` deleted a plugin's symlink from the per-root `.internal/plugins` directory unconditionally. That directory is SHARED by every process operating on the root, including a `plugin:<name>` background job started moments earlier. `mrsm compose start jobs` loads the project plugins, starts the job daemon (which needs the symlink), then unloads the plugins it loaded — and the unload removed the symlink out from under the running daemon, so its plugin import failed with "Plugin '<name>' cannot be found". This was the never-created/never-persisted symlink class behind the `compose start jobs` failures (`compose up` happened to dodge it). Apply the same only-reap-stale invariant already used in `sync_plugins_symlinks`: only remove a symlink whose target no longer exists. In-memory unloading (popping `sys.modules`) is process-local and is all that unloading needs; symlink lifecycle is owned by `sync_plugins_symlinks`. Verified against a throwaway compose project with a `plugin:`-backed job: `compose start jobs` (in-process AND isolation: subprocess), stop+start persistence, and `compose up` all keep the symlink and import the plugin; genuinely stale symlinks (target gone) are still cleaned. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When `replace_env` changes the active root or plugins directory — and on exit when it restores the previous one — the `plugins` package lingered in `sys.modules` with a `__path__` still pointing at the prior scope's `.internal/plugins`. A subsequent plugin import (e.g. an in-process scope switcher's "load back" step) then re-discovered project plugins under the wrong scope: `Plugin(...).__file__` resolved against the current `PLUGINS_RESOURCES_PATH` while submodule discovery walked the stale directory, so a plugin doing a module-level `from_plugin_import` of a sibling failed with "Unable to import plugin 'X'" and connector-providing plugins could fail to register their connectors. - Add `meerschaum.plugins.invalidate_plugins_cache()`: pop the cached `plugins` package and its submodules from `sys.modules` and reset `_loaded_plugins` so the next import rebuilds against the current `PLUGINS_RESOURCES_PATH`. - Call it from `replace_env` on both enter and exit whenever the root or plugins-dir scope actually changed. - Add `tests/test_plugins_scope.py` (fails without the fix). This is the core counterpart to the compose-side workaround (compose v2.3.4), which popped the stale package after its scope-restore; the core fix closes the class for every in-process scope switcher. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a plugin-loading regression exposed by 3.4.4: when
replace_envchanges the active root / plugins-dir scope (and on exit when it restores the previous one), thepluginspackage lingered insys.moduleswith a__path__pointing at the prior scope's.internal/plugins. A subsequent plugin import then re-discovered plugins under the wrong scope:Plugin(...).__file__resolved against the currentPLUGINS_RESOURCES_PATHwhile submodule discovery walked the stale directory →from_plugin_importof a sibling failed withUnable to import plugin 'X', andThis bit
mrsm composein-process scope switching (load project plugins → run subaction → restore env → "load back" host plugins). It surfaced after 3.4.4 stoppedunload_pluginsfrom deleting still-valid plugin symlinks (a correct fix for shared per-root symlinks used by background-job daemons) — the project symlinks now survive unload and remain re-discoverable via the stale__path__.Changes
meerschaum.plugins.invalidate_plugins_cache()— pops the cachedpluginspackage + submodules fromsys.modulesand resets_loaded_plugins, so the next import rebuilds against the currentPLUGINS_RESOURCES_PATH.replace_envon enter and exit whenever the root or plugins-dir scope actually changed.tests/test_plugins_scope.py— regression test (verified to fail without the fix).Notes
v2.3.4), which popped the stale package after its own scope-restore. This core fix closes the class for every in-process scope switcher; compose's pop becomes belt-and-suspenders.🤖 Generated with Claude Code