Skip to content

[Feature]: Python: from module import symbol across repositories is dropped, not parked (#3152 for Python) #3744

Description

@yupasik

Problem or use case

Two Python repositories. One imports a function from the other and calls it. Each repository is built on its own, then the two graphs are combined with merge-graphs or global add. The combined graph has no edge between the repositories: the imports_from edge ends on a sourceless placeholder, and the call is not in the graph at all. The same two files scanned as one directory produce both edges.

This is the Python shape of #3152. Java, C++, C# and Swift park a member call whose receiver type lives elsewhere (#3222), and #3384, #3386, #3388, #3390, #3393 extend that per language. Python is missing from the series, and its dependency shape is different: not obj.method() bound through a receiver type, but from module import symbol plus a plain call, which symbol_resolution.py binds by (module_stem, symbol) inside one build and drops otherwise.

Proposed solution

  1. Python extractor: when from module import symbol cannot be resolved in the corpus, park it on the importing file node, and park the raw calls that use the imported name (or its alias) on the caller node, names only, in the same spirit as _park_unresolved_member_call.
  2. Merge pass: for a parked module, find exactly one file node in another repo with that module stem; rewire the imports_from edge and emit calls edges for the parked names. Ambiguous stems fabricate nothing.
  3. Add "python": frozenset({".py"}) to _LANG_SUFFIXES so a Python name never binds to a same-named declaration in another language.

Package-level imports (from pkg.sub import x, import pkg) would follow the same rule with the package directory as the stem; the single-module case above is the minimal one.

Related

#3152, #3222 (mechanism), #3384 #3386 #3388 #3390 #3393 (per-language series, Python missing), #3665 (unresolved call indistinguishable from none), #3414 #3415 #3360 (false cross-repo edges from stub dedupe), #3578 (shared type linking for global add, types only).

Alternatives considered

No response

Area

Extraction or language support

Additional context

lib_repo/config_validator.py
    def validator_from_json(data):
        return ConfDict(data)

    class ConfDict:
        def __init__(self, data):
            self.data = data

app_repo/app.py
    from config_validator import validator_from_json

    def load(payload):
        return validator_from_json(payload)
$ (cd lib_repo && git init -q && graphify update . --no-cluster)   # 4 nodes, 4 edges
$ (cd app_repo && git init -q && graphify update . --no-cluster)   # 2 nodes, 2 edges
$ graphify merge-graphs lib_repo/graphify-out/graph.json app_repo/graphify-out/graph.json --out merged.json

merged.json: 7 nodes, 6 edges, 0 edges between the repos.

app_repo::app  -imports_from->  app_repo::config_validator      # placeholder, no source_file
app_repo::app  -contains->      app_repo::app_load               # no calls edge at all

Control, both files in one directory:

$ graphify update joint --no-cluster                              # 6 nodes, 8 edges
app       -imports->  config_validator_validator_from_json
app_load  -calls->    config_validator_validator_from_json

Expected

After merge-graphs or global add:

app_repo::app -imports_from-> lib_repo::config_validator (the file node whose module stem matches the placeholder, when exactly one repo provides it);
app_repo::app_load -calls-> lib_repo::config_validator_validator_from_json, tagged like _cross_repo_call, under the same single-definition guard the in-repo resolver uses.
Where it stops today
symbol_resolution.py::resolve_python_import_guided_calls resolves from module import symbol by (module_stem, symbol) in the current corpus and drops the call when the module is absent. Nothing is written to graph.json, so no merge-time pass can recover it (same root cause as #3152, and the invisibility #3665 describes).
cross_repo_calls.py reads metadata.unresolved_calls, but only receiver-typed member calls are parked, and _LANG_SUFFIXES lists cpp, csharp, java, swift.
global_graph.py::global_add deduplicates sourceless stubs by label among stubs only; a stub is never compared with a file node of another repo. What that dedupe does produce is the false cross-repo edges of #3414 and #3360: on five internal Python repositories, 831 edges between repos, all through shared stubs like Any, Enum, BaseModel, none between two real files. The one real dependency between those repos (a service importing an in-house library in three files) gave no edge; a joint scan of the same two repos gave 25.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions