Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/concepts/hooks-and-overrides.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Hooks and Overrides
===================

Fromager has two plugin systems that serve different purposes:
Fromager has two plugin systems based on entry points:
**per-package overrides** replace default behavior for a specific
package, while **global hooks** broadcast notifications after events
for every package.
Expand Down Expand Up @@ -82,3 +82,13 @@ entry-point group in their ``pyproject.toml``, mapping a hook name to
a callable.

See :doc:`/customization` for examples and argument details.

Configured Wheel Build Tag Hook
-------------------------------

.. versionadded:: 0.99.0

The ``wheels.build_tag_hook`` setting names one callable with a Python
import string in global ``settings.yaml``. Its return value supplies
wheel filename suffixes during builds and cache lookups. See
:doc:`/customization` for the configuration and hook signature.
72 changes: 72 additions & 0 deletions docs/customization.md
Original file line number Diff line number Diff line change
Expand Up @@ -393,6 +393,78 @@ $ tox -e cli -- canonicalize flit-core
flit_core
```

## Global settings

Global settings are configured in the `settings.yaml` file passed via the
`--settings-file` flag. These settings apply to all packages being built.

### Wheel build tag hook

```{versionadded} 0.99.0
```

The `build_tag_hook` is a configuration option that allows you to customize
wheel filenames by appending configuration-specific suffixes to the build tag.
This is useful for creating unique, deterministic filenames that reflect the
build configuration and distinguish wheels built for different variants.

Process hooks register multiple event callbacks through entry points.
`build_tag_hook` selects one callable in `settings.yaml` because Fromager
uses its return value when building wheels and checking the cache.

Configure the hook in your global `settings.yaml`:

```yaml
wheels:
build_tag_hook: "myproject.hooks:build_tag_hook"
```

The module must be importable by the Python environment running Fromager.

The hook function receives keyword-only arguments and returns a sequence of
suffix segments (strings) to append to the wheel build tag:

```python
from typing import Sequence

from packaging.requirements import Requirement
from packaging.tags import Tag
from packaging.version import Version

from fromager import context


def build_tag_hook(
*,
ctx: context.WorkContext,
req: Requirement,
version: Version,
wheel_tags: frozenset[Tag],
) -> Sequence[str]:
"""Return a valid suffix for each configured variant."""
suffixes = {"cpu": "cpu", "cuda-13": "cuda13"}
return [suffixes[ctx.variant]]
```
Comment thread
jlarkin09 marked this conversation as resolved.

Fromager joins segments in order with `_`. If the changelog build tag is `2`,
returning `["cuda13", "torch2.10"]` produces `2_cuda13_torch2.10`.

**Important notes:**

- The hook runs only when the package has a changelog-derived build tag. It
is skipped when there is no changelog entry or the package is prebuilt.
- Return a sequence of strings, not a single string or bytes object. Each
segment must be non-empty and contain only ASCII letters, digits, or dots.
Map variant names with other characters to distinct valid segments.
- The hook must be deterministic and independent of wheel contents, build
environment, or ELF metadata, so that fresh builds and cache lookups
produce identical tags.
- Use `wheel_tags` only to distinguish pure wheels from platform wheels.
The hook must return identical results across architectures for the same
configured variant.
- Installers do not use build tags to select a variant. Keep incompatible
variants in separate package indexes.

## Process hooks

Fromager supports plugging in Python hooks to be run after build events.
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/config-reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -96,3 +96,5 @@ If you prefer managing a single settings file, per-package settings can also be
kept in this file.

.. autopydantic_model:: fromager.packagesettings.SettingsFile

.. autopydantic_model:: fromager.packagesettings.WheelSettings
112 changes: 71 additions & 41 deletions src/fromager/bootstrapper/_cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,29 +86,34 @@ def _look_for_existing_wheel(
search_in: pathlib.Path,
) -> tuple[pathlib.Path | None, pathlib.Path | None]:
pbi = ctx.package_build_info(req)
expected_build_tag = pbi.build_tag(resolved_version)
base_build_tag = pbi.build_tag(resolved_version)
logger.info(
f"looking for existing wheel for version {resolved_version} with build tag {expected_build_tag} in {search_in}"
f"looking for existing wheel for version {resolved_version} with build tag {base_build_tag} in {search_in}"
)
wheel_filename = finders.find_wheel(
wheel_filenames = finders.find_wheels(
downloads_dir=search_in,
req=req,
dist_version=str(resolved_version),
build_tag=expected_build_tag,
build_tag=base_build_tag,
)
if not wheel_filename:
return None, None
_, _, build_tag, _ = wheels.extract_info_from_wheel_file(req, wheel_filename)
if expected_build_tag and expected_build_tag != build_tag:
logger.info(
f"found wheel for {resolved_version} in {wheel_filename} but build tag does not match. Got {build_tag} but expected {expected_build_tag}"
for wheel_filename in wheel_filenames:
_, _, actual_build_tag, wheel_tags = wheels.extract_info_from_wheel_file(
req, wheel_filename
)
return None, None
logger.info(f"found existing wheel {wheel_filename}")
build_reqs_dir = _extract_build_reqs_from_wheel(
ctx.work_dir, req, resolved_version, wheel_filename
)
return wheel_filename, build_reqs_dir
expected_build_tag = wheels.get_build_tag(
ctx=ctx, req=req, version=resolved_version, wheel_tags=wheel_tags
)
if expected_build_tag and expected_build_tag != actual_build_tag:
logger.info(
f"found wheel for {resolved_version} in {wheel_filename} but build tag does not match. Got {actual_build_tag} but expected {expected_build_tag}"
)
continue
logger.info(f"found existing wheel {wheel_filename}")
build_reqs_dir = _extract_build_reqs_from_wheel(
ctx.work_dir, req, resolved_version, wheel_filename
)
return wheel_filename, build_reqs_dir
return None, None


def _download_wheel_from_cache(
Expand All @@ -127,31 +132,6 @@ def _download_wheel_from_cache(
constraints=ctx.constraints,
)
results = resolver.find_all_matching_from_provider(provider, pinned_req)
wheel_url, _ = results[0]
wheelfile_name = pathlib.Path(urlparse(wheel_url).path)
pbi = ctx.package_build_info(req)
expected_build_tag = pbi.build_tag(resolved_version)
logger.info(f"has expected build tag {expected_build_tag}")
changelogs = pbi.get_changelog(resolved_version)
logger.debug(f"has change logs {changelogs}")
Comment thread
jlarkin09 marked this conversation as resolved.

_, _, build_tag, _ = wheels.extract_info_from_wheel_file(req, wheelfile_name)
if expected_build_tag and expected_build_tag != build_tag:
logger.info(
f"found wheel for {resolved_version} in cache but build tag does not match. Got {build_tag} but expected {expected_build_tag}"
)
return None, None

cached_wheel = wheels.download_wheel(
req=req, wheel_url=wheel_url, output_directory=ctx.wheels_downloads
)
if cache_wheel_server_url != ctx.wheel_server_url:
server.update_wheel_mirror(ctx)
logger.info("found built wheel on cache server")
unpack_dir = _extract_build_reqs_from_wheel(
ctx.work_dir, req, resolved_version, cached_wheel
)
return cached_wheel, unpack_dir
except ResolverException:
logger.info(
f"did not find wheel for {resolved_version} in {cache_wheel_server_url}"
Expand All @@ -170,6 +150,56 @@ def _download_wheel_from_cache(
)
return None, None

for wheel_url, _ in results:
try:
wheel_filename = pathlib.Path(urlparse(wheel_url).path)
_, _, actual_build_tag, wheel_tags = wheels.extract_info_from_wheel_file(
req, wheel_filename
)
except Exception as err:
logger.warning(f"could not parse cached wheel {wheel_url}: {err}")
continue

pbi = ctx.package_build_info(req)
expected_build_tag = wheels.get_build_tag(
ctx=ctx, req=req, version=resolved_version, wheel_tags=wheel_tags
)
logger.info(f"has expected build tag {expected_build_tag}")
changelogs = pbi.get_changelog(resolved_version)
logger.debug(f"has change logs {changelogs}")

if expected_build_tag and expected_build_tag != actual_build_tag:
logger.info(
f"found wheel for {resolved_version} in cache but build tag does not match. Got {actual_build_tag} but expected {expected_build_tag}"
)
continue

try:
cached_wheel = wheels.download_wheel(
req=req, wheel_url=wheel_url, output_directory=ctx.wheels_downloads
)
if cache_wheel_server_url != ctx.wheel_server_url:
server.update_wheel_mirror(ctx)
logger.info("found built wheel on cache server")
unpack_dir = _extract_build_reqs_from_wheel(
ctx.work_dir, req, resolved_version, cached_wheel
)
return cached_wheel, unpack_dir
except requests.exceptions.RequestException as err:
logger.warning(
f"network error checking wheel cache for {resolved_version} "
f"at {cache_wheel_server_url}: {err}"
)
return None, None
except Exception as err:
logger.warning(
f"unexpected error checking wheel cache for {resolved_version} "
f"at {cache_wheel_server_url}: {err}"
)
return None, None

return None, None


def find_cached_wheel(
ctx: context.WorkContext,
Expand Down
Loading
Loading