Skip to content

feat: add Clojure language extractor - #3767

Open
dansreis wants to merge 5 commits into
Graphify-Labs:v8from
dansreis:feat/clojure-extractor
Open

dansreis wants to merge 5 commits into
Graphify-Labs:v8from
dansreis:feat/clojure-extractor

Conversation

@dansreis

Copy link
Copy Markdown

What does this PR do?

Adds a Clojure / ClojureScript extractor for .clj, .cljs, .cljc and .edn, backed by the clojure grammar already bundled in tree-sitter-language-pack. No new dependency: the new clojure extra reuses the pin shared with the r and erlang extras, and uv.lock only gains the extra tables.

Per file it extracts the namespace, every def* form (functions, macros, multimethods and defmethods, vars, protocols with their method signatures, records/types with their method implementations, and custom definers such as deftest or helix/defnc), Java :import classes, contains / implements edges, and intra-file calls edges.

A registered LanguageResolver then links :require targets to the namespace node(s) that declare them (or one external node per undefined namespace) and resolves cross-namespace calls through aliases, :refer, :refer :all and :use to the unique matching definition, with the same god-node guard the other resolvers use. Aliased record constructors (alias/->Record) become references edges.

Precision safeguards: names bound by parameters, let-style forms, destructuring, letfn and catch shadow same-named top-level defs; reify / proxy / extend-* inside a body emit implements edges rather than treating method heads as calls; Java interop, quoted data, comment forms and #_ discards are skipped. Operator characters in symbols (<!!, foo?) are mapped to readable id tokens so distinct definitions never collide.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Tests or CI
  • Refactor
  • Security fix

How was this tested?

uv sync --frozen --extra clojure
uv run --frozen pytest tests/test_clojure_extractor.py -q        # 16 passed
uv run --frozen pytest tests/test_extractors_registry.py tests/test_detect.py \
  tests/test_extract_cli.py tests/test_extract.py tests/test_languages.py \
  tests/test_analyze.py tests/test_build.py -q                     # 1090 passed
uv run --frozen python -m tools.skillgen --check                   # check OK
ruff check graphify/extractors/clojure.py tests/test_clojure_extractor.py

Field-tested on ring-clojure/ring (81 files) and clojure/core.async (46 files)
plus two private codebases (~270 files): zero parse errors, zero ambiguous
resolutions, and >99% of cross-namespace calls into the corpus resolved.
Ring's 404 top-level definitions were compared against a regex ground truth;
all are extracted.

Local runs were on Python 3.14 (macOS). The code uses from __future__ import annotations and no 3.11+ syntax.

Graphify-specific checklist

  • I added or updated tests for behavior changes.
  • I updated documentation or confirmed that no documentation is needed. (README extras table and file-types table.)
  • I updated generated skill artifacts when changing their source fragments. (None changed; skillgen --check passes.)
  • I considered compatibility across supported Python versions.
  • I confirmed that no API keys, generated graph data, or local-only files are included.

Handle .clj/.cljs/.cljc/.edn via the clojure grammar already bundled in
tree-sitter-language-pack. Extracts namespaces, def* forms, protocols,
records/types, :import classes, and resolves :require targets and
cross-namespace calls in a post-extraction pass.
Field-tested on ring, core.async and two internal projects (400 files):

- mangle operator characters in node ids so <!! / >!! and foo? / foo!
  no longer collapse onto one node
- exclude record method implementations from cross-namespace call
  resolution; calls resolve to the protocol signature instead
- register namespaced custom definers (helix/defnc, t/deftest) and
  definitions nested in top-level when/if/do/let forms
- resolve alias/->Record and alias/map->Record constructors
Audited against ring (81 files, 404 top-level defs):

- names bound by parameters, let-style forms, destructuring, letfn and
  catch shadow same-named top-level defs, so (handler req) on a handler
  parameter is no longer a call to the handler function
- reify/proxy/extend-* forms inside a body emit implements edges and walk
  method bodies without treating impl heads as calls
No new packages: the extra reuses the already-locked
tree-sitter-language-pack, so only the extra tables change.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.


Graphify review — findings

Adds a Clojure/ClojureScript AST extractor covering .clj/.cljs/.cljc/.edn, wiring the new clojure extra and extract_clojure into the dispatch, extension, and language-family tables so these files classify as code and route to the grammar-backed extractor. Emits nodes for the ns form, def* forms (functions, macros, multimethods, vars, protocols, records, types), protocol/record methods, and Java :import targets, plus contains/implements and intra-file calls edges — driven by head-symbol inspection since the bundled grammar has no defn/ns concept, with a def-prefix heuristic (and a denylist) to catch custom definers like deftest. Records cross-namespace calls and :require targets as raw entries and resolves them across files via resolve_clojure_namespaces after all files are extracted.

Worth a look

  • Clojure extraction now raises for files without an ns form — graphify/extractors/clojure.py · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Arrow-prefixed Clojure functions never resolve as calls — graphify/extractors/clojure.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Qualified Clojure calls can be suppressed by unrelated refer-all namespaces — graphify/extractors/clojure.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Function-wide bound set suppresses valid calls outside local binding scope — graphify/extractors/clojure.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Namespaced fdef custom definer is never recognized — graphify/extractors/clojure.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 3094 functions depend on the 493 functions this change touches.

Health — this change adds coupling hotspots:

  • new: extract() — 703 callers, 45 callees
  • new: _rebuild_code() — 144 callers, 55 callees
  • new: detect() — 112 callers, 15 callees
  • new: to_obsidian() — 38 callers, 14 callees
  • new: _extract_generic() — 18 callers, 29 callees
  • new: save_manifest() — 40 callers, 11 callees
  • new: to_json() — 58 callers, 7 callees
  • new: extract_js() — 87 callers, 4 callees
  • …and 80 more — each is listed as a finding

Verification — 3094 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 2775 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

299 of 299 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • tests/test_affected_cli.py — full-run-safety
  • tests/test_affected_member_seed.py — full-run-safety
  • tests/test_agents_platform.py — full-run-safety
  • tests/test_analyze.py — impact, full-run-safety
  • tests/test_anthropic_custom_endpoint.py — full-run-safety
  • tests/test_antigravity_install.py — full-run-safety
  • tests/test_apm_fallback_version.py — full-run-safety
  • tests/test_architecture_doc.py — full-run-safety
  • tests/test_astro_extraction.py — impact, full-run-safety
  • tests/test_astro_import_ids.py — impact, full-run-safety
  • tests/test_atomic_canvas_export.py — impact, full-run-safety
  • tests/test_atomic_version_stamp.py — full-run-safety
  • tests/test_atomic_writes.py — impact, full-run-safety
  • tests/test_backend_env_isolation.py — full-run-safety
  • tests/test_backend_extras.py — full-run-safety
  • tests/test_benchmark.py — full-run-safety
  • tests/test_benchmark_raw_graph.py — full-run-safety
  • tests/test_build.py — impact, full-run-safety
  • tests/test_build_merge_dedup_scope.py — full-run-safety
  • tests/test_build_merge_hyperedges_and_prune.py — full-run-safety
  • tests/test_build_merge_shrink_guard.py — full-run-safety
  • tests/test_builtin_global_type_refs.py — impact, full-run-safety
  • tests/test_cache.py — full-run-safety
  • tests/test_callflow_html.py — full-run-safety
  • tests/test_cargo_introspect.py — impact, full-run-safety
  • tests/test_cargo_missing_manifest.py — full-run-safety
  • tests/test_carried_hyperedge_remap.py — impact, full-run-safety
  • tests/test_case_sensitive_resolution.py — impact, full-run-safety
  • tests/test_charmap_encoding.py — impact, full-run-safety
  • tests/test_chunking.py — impact, full-run-safety
  • tests/test_cjs_module_extension.py — impact, full-run-safety
  • tests/test_claude_cli_backend.py — impact, full-run-safety
  • tests/test_claude_md.py — full-run-safety
  • tests/test_cli_broken_pipe.py — full-run-safety
  • tests/test_cli_export.py — impact, full-run-safety
  • tests/test_cli_help.py — full-run-safety
  • tests/test_clojure_extractor.py — impact, changed-test, full-run-safety
  • tests/test_cluster.py — full-run-safety
  • tests/test_cobol_extractor.py — impact, full-run-safety
  • tests/test_codebuddy.py — full-run-safety
  • tests/test_community_hub_labels.py — full-run-safety
  • tests/test_community_labels_skill.py — impact, full-run-safety
  • tests/test_confidence.py — impact, full-run-safety
  • tests/test_corrupt_graph_json.py — full-run-safety
  • tests/test_cpp_nested_and_cli.py — impact, full-run-safety
  • tests/test_cpp_objc_cross_file_calls.py — impact, full-run-safety
  • tests/test_cpp_preprocess.py — full-run-safety
  • tests/test_cross_extension_reexport_self_cycle.py — impact, full-run-safety
  • tests/test_cross_language_call_resolution.py — impact, full-run-safety
  • tests/test_cross_repo_external_call_guards.py — impact, full-run-safety
  • … and 249 more

non-code file(s) changed (README.md, pyproject.toml, tests/fixtures/new_languages/sample.clj, uv.lock) → running the full suite for safety (a code graph can't see config/fixture/data deps)

changed code file(s) with no mapped test (README.md, graphify/extractors/__init__.py, graphify/extractors/clojure.py, pyproject.toml) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

· 2 grounded finding(s) anchored inline below; 86 more finding(s) on lines outside this diff (see the check run).

add_edge(caller, candidates[0], "calls", "remote_call", raw)


def extract_clojure(path: Path) -> dict:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — extract_clojure()

fans out to 18 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expected for a new language extractor: extract_clojure is the per-file entry point and fans out to the same helper set (_make_id, _read_text, the libspec/import parsers, node/edge builders) that extract_erlang and extract_commonlisp do. The callee count reflects the number of Clojure forms handled, not shared coupling with the rest of the package; nothing outside extractors/clojure.py gained callers.

add_edge(owner_id, method_id, "contains", item)
bodies.append((impl_values[1:], method_id))

def process_definition(form: Node, head: tuple[str | None, str]) -> bool:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — process_definition()

fans out to 9 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

process_definition is the dispatch over the def* families (fn, var, type, protocol, defmethod, extend-*, custom definers); its nine callees are the per-family handlers within the same file. Splitting it further would only move the fan-out into a table. Happy to restructure if maintainers prefer a dispatch dict, but it is local to the extractor.

…ames

- shadowing is now lexically scoped: a let/fn/catch binding hides a
  same-named def only inside that form, not the whole definition
- arrow-prefixed function names (->seconds) resolve as calls; only names
  matching a record/type are treated as constructors, locally and in the
  cross-namespace resolver
- a qualified call resolves only in its own namespace, never widened by
  :refer :all namespaces
- top-level forms in files without an ns form attach to the file node
- fix a misleading comment that listed s/fdef as a custom definer
@dansreis

dansreis commented Sep 22, 2026 •

Copy link
Copy Markdown
Author

Addressed the review findings in cf0cb87:

  • Raises without an ns form: can't reproduce. Files with no ns, empty files, scripts and EDN all extract fine; the file node is used as the namespace. Top-level calls in such files now attach to it instead of being dropped.
  • Arrow-prefixed functions: fixed. ->foo is only a constructor when a matching record exists, otherwise it's a normal call. Ring's ->seconds resolves now.
  • Qualified calls and refer-all: hardened. A qualified call is looked up only in its own namespace.
  • Function-wide shadowing: fixed. Bindings now shadow only inside their own form.
  • s/fdef: not a definer, it annotates an existing fn. Only the comment was wrong; corrected.

Tests added for each. Suite and the ring/core.async field run are green.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.


Graphify review — findings

Adds first-class Clojure/ClojureScript/EDN support: a new extract_clojure extractor parses .clj/.cljs/.cljc/.edn files into namespace, def*, protocol, record/type, and :import nodes plus contains/implements and intra-file calls edges by inspecting each top-level form's head symbol. Registers the extension family across analyze, detect's CODE_EXTENSIONS, and the extract dispatch/resolver tables, with resolve_clojure_namespaces wiring cross-namespace calls and :require targets after all files are extracted. Documents the new clojure extra and supported-language rows in the README.

Worth a look

  • process_definition uses undefined variable custom_id — graphify/extractors/clojure.py · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • definterface is denylisted as a non-definition — graphify/extractors/clojure.py:53 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Clojure operator escaping still collides with valid symbol names — graphify/extractors/clojure.py:205 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • pending_impls resolves protocol_ns but never uses it for target lookup — graphify/extractors/clojure.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • extend-protocol method implementations lose their protocol metadata — graphify/extractors/clojure.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 3100 functions depend on the 499 functions this change touches.

Health — this change adds coupling hotspots:

  • new: extract() — 707 callers, 45 callees
  • new: _rebuild_code() — 144 callers, 55 callees
  • new: detect() — 112 callers, 15 callees
  • new: to_obsidian() — 38 callers, 14 callees
  • new: _extract_generic() — 18 callers, 29 callees
  • new: save_manifest() — 40 callers, 11 callees
  • new: to_json() — 58 callers, 7 callees
  • new: extract_js() — 87 callers, 4 callees
  • …and 81 more — each is listed as a finding

Verification — 3100 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 2781 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

299 of 299 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • tests/test_affected_cli.py — full-run-safety
  • tests/test_affected_member_seed.py — full-run-safety
  • tests/test_agents_platform.py — full-run-safety
  • tests/test_analyze.py — impact, full-run-safety
  • tests/test_anthropic_custom_endpoint.py — full-run-safety
  • tests/test_antigravity_install.py — full-run-safety
  • tests/test_apm_fallback_version.py — full-run-safety
  • tests/test_architecture_doc.py — full-run-safety
  • tests/test_astro_extraction.py — impact, full-run-safety
  • tests/test_astro_import_ids.py — impact, full-run-safety
  • tests/test_atomic_canvas_export.py — impact, full-run-safety
  • tests/test_atomic_version_stamp.py — full-run-safety
  • tests/test_atomic_writes.py — impact, full-run-safety
  • tests/test_backend_env_isolation.py — full-run-safety
  • tests/test_backend_extras.py — full-run-safety
  • tests/test_benchmark.py — full-run-safety
  • tests/test_benchmark_raw_graph.py — full-run-safety
  • tests/test_build.py — impact, full-run-safety
  • tests/test_build_merge_dedup_scope.py — full-run-safety
  • tests/test_build_merge_hyperedges_and_prune.py — full-run-safety
  • tests/test_build_merge_shrink_guard.py — full-run-safety
  • tests/test_builtin_global_type_refs.py — impact, full-run-safety
  • tests/test_cache.py — full-run-safety
  • tests/test_callflow_html.py — full-run-safety
  • tests/test_cargo_introspect.py — impact, full-run-safety
  • tests/test_cargo_missing_manifest.py — full-run-safety
  • tests/test_carried_hyperedge_remap.py — impact, full-run-safety
  • tests/test_case_sensitive_resolution.py — impact, full-run-safety
  • tests/test_charmap_encoding.py — impact, full-run-safety
  • tests/test_chunking.py — impact, full-run-safety
  • tests/test_cjs_module_extension.py — impact, full-run-safety
  • tests/test_claude_cli_backend.py — impact, full-run-safety
  • tests/test_claude_md.py — full-run-safety
  • tests/test_cli_broken_pipe.py — full-run-safety
  • tests/test_cli_export.py — impact, full-run-safety
  • tests/test_cli_help.py — full-run-safety
  • tests/test_clojure_extractor.py — impact, changed-test, full-run-safety
  • tests/test_cluster.py — full-run-safety
  • tests/test_cobol_extractor.py — impact, full-run-safety
  • tests/test_codebuddy.py — full-run-safety
  • tests/test_community_hub_labels.py — full-run-safety
  • tests/test_community_labels_skill.py — impact, full-run-safety
  • tests/test_confidence.py — impact, full-run-safety
  • tests/test_corrupt_graph_json.py — full-run-safety
  • tests/test_cpp_nested_and_cli.py — impact, full-run-safety
  • tests/test_cpp_objc_cross_file_calls.py — impact, full-run-safety
  • tests/test_cpp_preprocess.py — full-run-safety
  • tests/test_cross_extension_reexport_self_cycle.py — impact, full-run-safety
  • tests/test_cross_language_call_resolution.py — impact, full-run-safety
  • tests/test_cross_repo_external_call_guards.py — impact, full-run-safety
  • … and 249 more

non-code file(s) changed (README.md, pyproject.toml, tests/fixtures/new_languages/sample.clj, uv.lock) → running the full suite for safety (a code graph can't see config/fixture/data deps)

changed code file(s) with no mapped test (README.md, graphify/extractors/__init__.py, graphify/extractors/clojure.py, pyproject.toml) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

· 3 grounded finding(s) anchored inline below; 86 more finding(s) on lines outside this diff (see the check run).

Comment thread graphify/extractors/clojure.py
Comment thread graphify/extractors/clojure.py
continue
stack.append((item, bound))

def walk_body(items: list[Node], caller_id: str) -> None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — walk_body()

fans out to 6 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant