Add Bazel PyPI manifest extraction (#1324)

This PR makes Bazel manifest creation Python-aware.

This builds on the Maven Bazel work from [#1312](#1312), which closes an inline-declaration gap that exists in `rules_jvm_external`: Bazel can resolve Maven artifacts that do not exist in a checked-in Maven manifest. Python is different. `rules_python` commonly resolves packages from a checked-in pinned requirements or lock file and exposes those packages as Bazel labels.

It works like this: a Bazel Python rule points to a checked-in requirements file. Bazel reads that file and makes the declared packages available as dependencies in the configured pip hub. Future Bazel build targets can then directly declare dependencies on those Python packages.

What this PR does is emit a generated `requirements.txt` that contains only the pinned Python packages reachable from Bazel Python rules. It does not mutate or remove entries from the user's checked-in requirements file. The value is scoping the generated manifest to Bazel's reached package set instead of assuming every checked-in requirement is used by Bazel Python targets.

This functionality does not kick in automatically, since I'm not fully convinced it won't cause more harm than good or cause confusion. It has to be manually enabled with `socket manifest bazel --ecosystem pypi`. `socket scan create --auto-manifest` continues to generate Bazel Maven manifests only.

## Summary of changes
- add `socket manifest bazel --ecosystem pypi` support for whole-repo Bazel PyPI `requirements.txt` generation
- discover rules_python pip hubs via Bazel command output first, with bounded static fallback paths
- keep Bazel PyPI generation explicit; `socket scan create --auto-manifest` continues to generate Bazel Maven only
- add bounded verbose diagnostics for Bazel subprocess, discovery, extraction, and empty-result triage
- document the new command surface and add exact constructed-fixture oracle coverage

Author: simonhj

CHANGELOG.md

@@ -7,6 +7,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 ## [Unreleased]
 - **`socket manifest bazel [beta]`** — Generate Bazel JVM SBOM manifests by running `bazel query` against discovered Maven repos in a Bazel workspace. Closes the inline-Maven-declaration gap that lockfile-only parsing misses for repos like envoy, ray, tensorflow, tink-java, and or-tools. Auto-detects Bzlmod and legacy `WORKSPACE`.
 - **`socket scan create --auto-manifest`** now covers Bazel workspaces in addition to Gradle/Scala/Kotlin/Conda. Repos with `MODULE.bazel`, `WORKSPACE`, or `WORKSPACE.bazel` are detected automatically and their Maven dependencies extracted as part of the standard scan-create flow.
+- **Bazel PyPI extraction** — `socket manifest bazel --ecosystem pypi` now generates `requirements.txt` for Python Bazel workspaces. Discovers custom `rules_python` pip hub names with Bazel command output first, queries `py_library` / `py_binary` / `py_test` dependencies, resolves canonical pinned versions from `requirements_lock.txt`, and emits PEP 503-normalized `name==version` lines. Supports both Bzlmod (`pip.parse`) and legacy `WORKSPACE` (`pip_parse` / `pip_install`) configurations. PyPI remains explicit opt-in for `socket scan create --auto-manifest` until real-world no-lockfile recovery is validated.
+
+### Changed
+- **Bazel diagnostics** — `socket manifest bazel --verbose` now emits bounded subprocess traces with argv, cwd, duration, exit status, output sizes, and failure stderr tails to make customer log-only triage safer and faster.
 
 ## [1.1.98](https://github.com/SocketDev/socket-cli/releases/tag/v1.1.98) - 2026-05-22
 

src/commands/manifest/README.md

@@ -16,13 +16,14 @@ manifest generator. Useful when you do not want to spell out the language.
 
 ## socket manifest bazel [beta]
 
-Generates Bazel JVM SBOM manifests (`maven_install.json`-shaped) by running
-`bazel query` against discovered Maven repos in a Bazel workspace. Output is
-consumed by `socket scan create` and closes the
-inline-Maven-declaration gap that lockfile-only parsing misses.
+Generates Bazel SBOM manifests (Maven `maven_install.json` and/or PyPI
+`requirements.txt`) by running `bazel query` against discovered ecosystem
+hubs in a Bazel workspace. Output is consumed by `socket scan create` and
+closes the inline-declaration gap that lockfile-only parsing misses for
+Bazel monorepos.
 
-> **Note**: This command generates Maven dependency manifests for Bazel JVM
-> workspaces. It does not run reachability analysis.
+> **Note**: This command generates dependency manifests for Bazel
+> workspaces (Maven and PyPI). It does not run reachability analysis.
 
 ### Usage
 
@@ -36,33 +37,98 @@ socket manifest bazel [options] [DIR=.]
 - `--bazel-rc <path>` — path to additional `.bazelrc` fragments forwarded to bazel.
 - `--bazel-flags <str>` — flags forwarded to every bazel invocation (single quoted string).
 - `--bazel-output-base <dir>` — Bazel `--output_base` for read-only-cache CI environments.
+- `--ecosystem <name>` — ecosystem(s) to extract; repeatable. Supported values: `maven`, `pypi`. When omitted, Maven is generated by default; PyPI is explicit opt-in.
 - `--out <dir>` — output directory; default `./.socket/bazel-manifests/`.
 - `--dry-run`, `--verbose` — standard diagnostic flags.
 
 > **Upload**: This subcommand only generates manifests. To generate and
 > upload in one step, use `socket scan create --auto-manifest .` — it
-> detects the workspace, runs the same extraction this subcommand performs,
-> and uploads the result.
+> detects the workspace, generates Bazel Maven manifests, and uploads the
+> result. Generate Bazel PyPI manifests explicitly with `socket manifest bazel
+> --ecosystem pypi`, then scan the generated output with `socket scan create`.
 
 ### Examples
 
 ```bash
-# Generate maven manifests from the current Bazel workspace.
+# Generate the default Bazel Maven manifest from the current workspace.
 socket manifest bazel .
 
+# Generate only the PyPI manifest.
+socket manifest bazel . --ecosystem pypi
+
+# Generate both Maven and PyPI manifests explicitly.
+socket manifest bazel . --ecosystem maven --ecosystem pypi
+
 # Use bazelisk explicitly.
 socket manifest bazel --bazel=/usr/local/bin/bazelisk .
 ```
 
+### Python/PyPI Extraction
+
+When `--ecosystem pypi` is selected, the command:
+
+1. Discovers `rules_python` pip hubs from Bazel's `mod show_extension` output when available, with bounded static parsing of `MODULE.bazel` (`pip.parse(hub_name = "...")`) and legacy `WORKSPACE` (`pip_parse(name = "...")` / `pip_install(name = "...")`) retained as fallback. Hub names are never hardcoded; custom names like `my_pypi` are detected automatically.
+2. Validates each candidate hub by probing it with `bazel query` for `:pkg` targets / `alias(` rules. Invalid candidates are dropped.
+3. Runs `bazel query 'deps(kind("py_library|py_binary|py_test", //...))'` to determine which PyPI packages are actually reached by Python rules in the repo (test dependencies included for whole-repo scope).
+4. Reads `requirements_lock.txt` (the path discovered from `pip.parse(requirements_lock = "...")`) for canonical pinned versions. When the lockfile is unavailable, falls back to parsing `pypi_name=` and `pypi_version=` tags from the spoke `py_library` rules in the hub-and-spoke architecture.
+5. Emits a sorted canonical `requirements.txt` containing `name==version` lines for every reached package.
+
+### PyPI Name and Version Semantics
+
+- **PEP 503 normalization.** Package matching uses PEP 503 normalization
+  (lowercase, then any run of `-`, `_`, or `.` is collapsed to a single
+  `-`). Bazel target names use underscores (`charset_normalizer`); PyPI
+  canonical names use hyphens (`charset-normalizer`). The emitted
+  `requirements.txt` always uses the canonical hyphenated form.
+- **Lockfile pins win.** When the lockfile and spoke-repo tags disagree on
+  a version, the lockfile wins because that is the version Bazel actually
+  resolves at analysis time. A `--verbose` warning is logged for the
+  divergence.
+- **Conflict detection.** When two reached packages normalize to the same
+  PyPI name with different versions, the command fails clearly: a single
+  `requirements.txt` cannot represent both versions, and silently
+  picking one would produce a misleading SBOM.
+
+### Unsupported PyPI Forms
+
+The PyPI extractor is intentionally narrow in this phase:
+
+- **Direct URL, editable (`-e`), and unpinned requirements** are not
+  emitted. Only canonical `name==version` lines from the resolved
+  lockfile are produced. Repositories that rely on unpinned or
+  URL-pinned requirements will see those packages omitted from
+  `requirements.txt`.
+- **Private corpus validation** requires authenticated GitHub access.
+  When credentials are unavailable, the bazel-bench harness's private
+  PyPI case skips cleanly with a distinct reason rather than failing.
+- **Whole-repo extraction.** The initial PyPI implementation emits one
+  whole-workspace manifest. Per-target PyPI slicing is not currently
+  supported.
+
+### Cross-Language Edges
+
+Bazel repos with cross-language dependencies (e.g. `rust_library` →
+`py_library` via PyO3 / cffi / etc.) are **not** traversed by the PyPI
+extractor in this phase. The PyPI extractor only covers Python rule
+dependencies reachable from `py_library`, `py_binary`, and `py_test`
+targets. Cross-language edges are assigned to Phase 4. The bazel-bench
+fixture `constructed/python-pypi` includes Go/Rust sidecars as
+validation context only; they are intentionally not asserted by the
+PyPI correctness cases.
+
 ### Requirements
 
 - `bazel` or `bazelisk` on `PATH` (or pass `--bazel <path>`).
-- Network access on cold cache. Bazel and `rules_jvm_external` own their own
-  retry policy for transient Maven resolution failures — `socket manifest bazel`
-  does not retry on top of them.
+- Network access on cold cache. Bazel and `rules_jvm_external` /
+  `rules_python` own their own retry policy for transient resolution
+  failures — `socket manifest bazel` does not retry on top of them.
 - Writable Bazel output base; pass `--bazel-output-base` for read-only-cache CI.
+- For PyPI extraction: a Python 3 interpreter on `PATH` so the
+  rules_python toolchain can analyze the workspace.
 
-This is the user-visible entry point for Bazel JVM SBOM support; the [beta] label and "Bazel JVM SBOM support" wording must stay consistent across release notes and docs.
+This is the user-visible entry point for Bazel SBOM support (Maven and
+PyPI); the [beta] label and "Bazel SBOM support" wording must stay
+consistent across release notes and docs.
 
 ## socket manifest cdxgen