test: isolate integration harness runtime

[wenchanghan]

What's broken / annoying

A developer machine can have the installed claude-smart plugin running while the same developer is also debugging this checkout.

Before this PR, the source integration test used the same default ports as the installed plugin: backend 8071, embedding 8072, and dashboard 3001. The test also runs the normal start/stop scripts. That meant a local e2e run could accidentally stop, replace, or confuse the claude-smart services the developer depends on day to day.

There was also a quieter risk when reusing a test home: an old .claude-smart/.env could keep managed Reflexio connection keys and point the source test at hosted Reflexio instead of the local test backend.

What's changing

• Source e2e now runs on test-only ports by default: backend 18071, embedding 18072, dashboard 13001.
• The harness refuses production ports unless the developer explicitly sets CLAUDE_SMART_INTEGRATION_ALLOW_PROD_PORTS=1.
• The harness uses a temporary HOME plus explicit local Reflexio storage/config paths, so test data stays separate from the installed plugin's data.
• Reused test homes have managed Reflexio connection keys removed before services start: REFLEXIO_URL, REFLEXIO_API_KEY, and REFLEXIO_USER_ID.
• The sanitized test .env is written as private (0600) because it may contain local settings or secrets.
• Embedding startup wait is configurable with EMBEDDING_HEALTH_TIMEOUT_SECONDS; the default is now long enough for slower local model warmup.
• DEVELOPER.md now separates safe source e2e from production-like private tarball validation.

Local machine flows

flowchart TD
  A["One developer machine"] --> P["Daily installed claude-smart"]
  P --> P1["Default ports: 8071 / 8072 / 3001"]
  P --> P2["Real local data: ~/.reflexio and ~/.claude-smart"]
  P --> P3["Used by Claude Code, Codex, and OpenCode"]

  A --> E["Source e2e from this checkout"]
  E --> E1["Test ports: 18071 / 18072 / 13001"]
  E --> E2["Temporary HOME and isolated Reflexio data"]
  E --> E3["Safe for normal source debugging"]

  A --> T["Production-like validation"]
  T --> T1["Build a private tarball"]
  T --> T2["Install through normal host paths"]
  T --> T3["Uses default ports and real local data, like a user install"]

Loading

How the product behaves afterwards

Installed claude-smart behaves the same as before. The backend, embedding daemon, and dashboard still use 8071, 8072, and 3001; Claude Code, Codex, and OpenCode still share the same local services and data.

Only the developer integration harness changes its defaults. Developers can now debug source changes without disturbing their daily installed plugin. When they need to test exactly like production, they should build and install a private tarball instead of running source e2e on production ports.

How it was verified

• bash -n tests/integration/integration.sh
• git diff --check
• uv run --project plugin --with pytest pytest tests/test_install_scripts.py::test_integration_harness_prepares_vendored_reflexio tests/test_install_scripts.py::test_integration_harness_defaults_to_isolated_runtime tests/test_install_scripts.py::test_integration_harness_rejects_production_ports_without_opt_in tests/test_install_scripts.py::test_integration_harness_rejects_non_numeric_ports tests/test_install_scripts.py::test_integration_harness_sanitizes_reused_managed_env tests/test_host_integration_e2e.py -q
• uv run --project plugin --with pytest pytest tests/test_install_scripts.py -q
• bash tests/integration/integration.sh
• Checked production listeners before and after the full local e2e run: backend stayed on PID 56668 at 8071, embedding stayed on PID 56666 at 8072, and dashboard stayed on PID 82939 at 3001.
• Checked test ports after cleanup: nothing remained listening on 18071, 18072, or 13001.

Reviewer notes

• The test changes stay in the existing install-script test module, next to the prior integration harness checks.
• The production-port rejection cases are now one parametrized test instead of three copy-pasted tests.
• The .env sanitizer test now proves comment preservation and the intentional 0600 privacy behavior.

Summary by CodeRabbit

• Documentation
  • Updated local validation guidance with two workflows: isolated integration testing and production-style tarball validation, including documented port behavior and embedding startup timeout.
• Bug Fixes
  • Integration now defaults to isolated non-production ports and blocks production-like ports unless explicitly opted in.
  • Improved sandbox safety by canonicalizing the sandbox HOME, setting isolated runtime locations, and sanitizing reused Reflexio environment values.
  • Embedding health checks now use EMBEDDING_HEALTH_TIMEOUT_SECONDS.
• Tests
  • Added smoke/integration-harness coverage for isolated defaults, production-port rejection, numeric port validation, and environment sanitization (including file permission checks on non-Windows).

[coderabbitai]

📝 Walkthrough

Walkthrough

The integration harness now defaults to isolated ports, rejects unsafe production-like port usage without explicit opt-in, validates sandbox home handling, sanitizes Reflexio environment state, and uses a configurable embedding health timeout. Tests and documentation were updated to cover the new local validation paths.

Changes

Isolated integration harness

Layer / File(s)	Summary
Port defaults and opt-in guard tests/integration/integration.sh	The harness now defaults backend, embedding, and dashboard ports to isolated values, documents the opt-in example, and exits with code 2 when production-like ports are selected without CLAUDE_SMART_INTEGRATION_ALLOW_PROD_PORTS=1.
Sandbox home and Reflexio sanitization tests/integration/integration.sh	The script canonicalizes and validates INTEG_HOME and HOME, rejects unsafe CLAUDE_SMART_INTEG_HOME values, exports isolated Reflexio paths and URL state, unsets Reflexio auth/user variables, rewrites the integration .env file without managed Reflexio keys, and uses EMBEDDING_HEALTH_TIMEOUT_SECONDS for embedding polling.
Smoke tests for harness behavior tests/test_install_scripts.py	Adds pytest coverage for isolated defaults, production-port rejection, non-numeric port rejection, and sanitization of reused managed .env files.
Local validation documentation DEVELOPER.md	Documents the safe source e2e workflow and the production-like tarball validation path.

Estimated code review effort: 3 (Moderate) | ~25 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 11.11% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: isolating the integration harness runtime.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tests/test_install_scripts.py (1)

51-57: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

_isolated_env doesn't scrub CLAUDE_SMART_INTEGRATION_ALLOW_PROD_PORTS, breaking test isolation.

The production port rejection tests depend on this env var being unset, but _isolated_env only pops BASH_ENV and ENV. If a developer has CLAUDE_SMART_INTEGRATION_ALLOW_PROD_PORTS=1 in their shell environment, os.environ.copy() will inherit it and the script won't refuse production ports — causing test_integration_harness_rejects_production_ports_without_opt_in to fail or silently pass incorrectly.

🔧 Proposed fix

 def _isolated_env(tmp_path: Path) -> dict[str, str]:
     env = os.environ.copy()
     env["HOME"] = str(tmp_path)
     env["SHELL"] = "/bin/sh"
     env.pop("BASH_ENV", None)
     env.pop("ENV", None)
+    env.pop("CLAUDE_SMART_INTEGRATION_ALLOW_PROD_PORTS", None)
     return env

Also applies to: 1716-1747

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_install_scripts.py` around lines 51 - 57, The test helper
_isolated_env is not fully isolating the shell environment because it leaves
CLAUDE_SMART_INTEGRATION_ALLOW_PROD_PORTS inherited from os.environ. Update
_isolated_env in test_install_scripts.py to scrub that variable alongside
BASH_ENV and ENV so the production-port rejection tests stay deterministic; use
the existing _isolated_env helper and the tests that call it, including
test_integration_harness_rejects_production_ports_without_opt_in.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@tests/test_install_scripts.py`:
- Around line 51-57: The test helper _isolated_env is not fully isolating the
shell environment because it leaves CLAUDE_SMART_INTEGRATION_ALLOW_PROD_PORTS
inherited from os.environ. Update _isolated_env in test_install_scripts.py to
scrub that variable alongside BASH_ENV and ENV so the production-port rejection
tests stay deterministic; use the existing _isolated_env helper and the tests
that call it, including
test_integration_harness_rejects_production_ports_without_opt_in.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 66379ec8-db0c-4979-a6e7-e87a77b38a5e

📥 Commits

Reviewing files that changed from the base of the PR and between 0c129df and 87c8385.

📒 Files selected for processing (3)

• DEVELOPER.md
• tests/integration/integration.sh
• tests/test_install_scripts.py

✅ Files skipped from review due to trivial changes (1)

• DEVELOPER.md

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/integration/integration.sh