diff --git a/.gitignore b/.gitignore index 7841886..ea9d325 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ logs *.log *.pot *.pyc +.django_cache .dccachea __pycache__ *.sqlite3 diff --git a/CHANGELOG.md b/CHANGELOG.md index eceaee8..c8e70f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,15 +15,20 @@ Don't forget to remove deprecated code on each major release! - Nothing (yet) +## [4.3.0] - 2026-04-09 + +### Fixed + +- Fix a bug where the Django `collectstatic` command could crash when encountering static files that reference a URL containing query parameters. + ## [4.2.0] - 2026-04-01 ### Added -- Added new Django setting `SERVESTATIC_USE_STATIC_ROOT` to allow users to opt in to having `ServeStatic` scan all files within `STATIC_ROOT` at start-up. This is now enabled by default. +- Added new Django setting `SERVESTATIC_USE_STATIC_ROOT` to allow users to opt in to having `ServeStatic` scan all files within `STATIC_ROOT` at start-up. - Add JavaScript and CSS minification support to the `servestatic` CLI command. - Add JavaScript and CSS minification support to the `servestatic` the Django storage backend. - ## [4.1.0] - 2026-03-07 !!! tip @@ -170,7 +175,8 @@ Don't forget to remove deprecated code on each major release! - Forked from [`whitenoise`](https://github.com/evansd/whitenoise) to add ASGI support. -[Unreleased]: https://github.com/Archmonger/ServeStatic/compare/4.2.0...HEAD +[Unreleased]: https://github.com/Archmonger/ServeStatic/compare/4.3.0...HEAD +[4.3.0]: https://github.com/Archmonger/ServeStatic/compare/4.2.0...4.3.0 [4.2.0]: https://github.com/Archmonger/ServeStatic/compare/4.1.0...4.2.0 [4.1.0]: https://github.com/Archmonger/ServeStatic/compare/4.0.0...4.1.0 [4.0.0]: https://github.com/Archmonger/ServeStatic/compare/3.1.0...4.0.0 diff --git a/README.md b/README.md index ce639b4..71b45bb 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ _This project is a [fork](https://github.com/evansd/whitenoise/pull/359#issuecom `ServeStatic` simplifies static file serving with minimal lines of configuration. It enables you to create a self-contained unit without requiring external services like Nginx or Amazon S3. This can simplify any production deployment, but is especially useful for platforms like Heroku, OpenShift, and other PaaS providers. -It is designed to work seamlessly with CDNs to ensure high performance for traffic-intensive sites. It can be run in "standalone" mode, or alongside any preexisting ASGI/WSGI app. A command-line interface is provided to perform common tasks such as multi-format compression, file-name hashing, and manifest generation. Extra features and auto-configuration are available for [Django](https://www.djangoproject.com/) users. +This project is designed to work seamlessly with CDNs to ensure high performance for traffic-intensive sites. It can be run in "standalone" mode, or alongside any preexisting ASGI/WSGI app. A command-line interface is provided to perform common tasks such as multi-format compression, file-name hashing, and manifest generation. Extra features and auto-configuration are available for [Django](https://www.djangoproject.com/) users. When using `ServeStatic`, best practices are automatically handled such as: diff --git a/docs/src/dictionary.txt b/docs/src/dictionary.txt index 2523d4f..3d6687f 100644 --- a/docs/src/dictionary.txt +++ b/docs/src/dictionary.txt @@ -43,3 +43,4 @@ symlink symlinks minification minify +uncached diff --git a/docs/src/django-settings.md b/docs/src/django-settings.md index d554208..8e28b2b 100644 --- a/docs/src/django-settings.md +++ b/docs/src/django-settings.md @@ -16,39 +16,41 @@ Don't use this for the bulk of your static files because you won't benefit from ## `SERVESTATIC_AUTOREFRESH` -**Default:** `settings.py:DEBUG` +**Default:** `DEBUG` -Recheck the filesystem to see if any files have changed before responding. This is designed to be used in development where it can be convenient to pick up changes to static files without restarting the server. For both performance and security reasons, this setting should not be used in production. +Always check the filesystem to see if any files have changed before responding. This is especially useful in development environments to pick up changes to static files without restarting the server. + +When running under ASGI, ServeStatic performs these checks asynchronously. Regardless, keep in mind that this setting adds performance overhead. Additionally, it is [not recommended](https://www.redfoxsec.com/blog/understanding-file-upload-vulnerabilities) to use this setting to serve user-uploaded media files unless you have full confidence in your ability to validate and sanitize user data. --- ## `SERVESTATIC_USE_MANIFEST` -**Default:** `not settings.py:DEBUG and isinstance(staticfiles_storage, ManifestStaticFilesStorage)` +**Default:** `not DEBUG and isinstance(staticfiles_storage, ManifestStaticFilesStorage)` Find and serve files using Django's manifest file. This is the most efficient way to determine what files are available, but it requires that you are using a [manifest-compatible](https://docs.djangoproject.com/en/stable/ref/contrib/staticfiles/#manifeststaticfilesstorage) storage backend. -When using ServeStatic's [`CompressedManifestStaticFilesStorage`](./django.md#step-3-add-compression-and-caching-support) storage backend, ServeStatic will no longer need to call `os.stat` on each file during startup. +When using ServeStatic's [`CompressedManifestStaticFilesStorage`](./django.md#step-3-add-compression-and-caching-support) storage backend, ServeStatic will no longer need to call `os.stat` on each file during startup. This will significantly reduce startup time, especially when you have a large number of static files. --- ## `SERVESTATIC_USE_FINDERS` -**Default:** `settings.py:DEBUG` +**Default:** `DEBUG` -Find and serve files using Django's [`finders`](https://docs.djangoproject.com/en/stable/ref/contrib/staticfiles/#finders-module) API. +Find and serve files using Django's [`finders`](https://docs.djangoproject.com/en/stable/ref/contrib/staticfiles/#finders-module) API. Defaults to `True` if Django's `DEBUG` setting is enabled. -It's possible to use this setting in production, but be mindful of the [`settings.py:STATICFILES_DIRS`](https://docs.djangoproject.com/en/stable/ref/settings/#staticfiles-dirs) and [`settings.py:STATICFILE_FINDERS`](https://docs.djangoproject.com/en/stable/ref/settings/#staticfiles-finders) settings. By default, the finders API only searches the `'static'` directory in each app, which are not the copies post-processed by ServeStatic. +It's possible to use this setting in production, but it will be less efficient than other methods. Also, be mindful of the [`STATICFILES_DIRS`](https://docs.djangoproject.com/en/stable/ref/settings/#staticfiles-dirs) and [`STATICFILE_FINDERS`](https://docs.djangoproject.com/en/stable/ref/settings/#staticfiles-finders) settings. -Note that `STATICFILES_DIRS` cannot equal `STATIC_ROOT` while running the `collectstatic` management command. +By default, the finders API only searches the `'static'` directory in each Django app, which are not the copies post-processed by ServeStatic. Additionally, `STATICFILES_DIRS` cannot equal `STATIC_ROOT` while running the `collectstatic` management command. --- ## `SERVESTATIC_USE_STATIC_ROOT` -**Default:** `not (settings.py:SERVESTATIC_USE_MANIFEST or settings.py:SERVESTATIC_USE_FINDERS)` +**Default:** `not (SERVESTATIC_USE_MANIFEST or SERVESTATIC_USE_FINDERS)` Find and serve all files within Django's `STATIC_ROOT` (file scan is only run during startup). This defaults to `True` if you do not have no other method configured. @@ -58,7 +60,7 @@ This allows users to have their `STATIC_ROOT` directory contain files which are ## `SERVESTATIC_MAX_AGE` -**Default:** `60 if not settings.py:DEBUG else 0` +**Default:** `60 if not DEBUG else 0` Time (in seconds) for which browsers and proxies should cache **non-versioned** files. @@ -74,7 +76,7 @@ Set to `None` to disable setting any `Cache-Control` header on non-versioned fil **Default:** `False` -If `True` enable index file serving. If set to a non-empty string, enable index files and use that string as the index file name. +If `True`, serve an index file when a directory is requested. When set to `True`, ServeStatic will assume your index files are named `index.html`. However, if this value is set to a string, it will use that as the index file name. --- @@ -259,7 +261,7 @@ SERVESTATIC_IMMUTABLE_FILE_TEST = immutable_file_test ## `SERVESTATIC_STATIC_PREFIX` -**Default:** `settings.py:STATIC_URL` +**Default:** `STATIC_URL` The URL prefix under which static files will be served. @@ -289,10 +291,4 @@ This setting is only effective if the `ServeStatic` storage backend is being use Set to `False` to prevent Django throwing an error if you reference a static file which doesn't exist in the manifest. -This works by setting the [`manifest_strict`](https://docs.djangoproject.com/en/stable/ref/contrib/staticfiles/#django.contrib.staticfiles.storage.ManifestStaticFilesStorage.manifest_strict) option on the underlying Django storage instance, as described in the Django documentation. - -This setting is only effective if the `ServeStatic` storage backend is being used. - -!!! Note - - If a file isn't found in the `staticfiles.json` manifest at runtime, a `ValueError` is raised. This behavior can be disabled by subclassing `ManifestStaticFilesStorage` and setting the `manifest_strict` attribute to `False` -- nonexistent paths will remain unchanged. +This setting only takes effect when using `CompressedManifestStaticFilesStorage`, and it works by setting the [`manifest_strict`](https://docs.djangoproject.com/en/stable/ref/contrib/staticfiles/#django.contrib.staticfiles.storage.ManifestStaticFilesStorage.manifest_strict) option on the underlying Django storage instance, as described in the Django documentation. diff --git a/docs/src/servestatic.md b/docs/src/servestatic.md index 0a40f03..a05dc84 100644 --- a/docs/src/servestatic.md +++ b/docs/src/servestatic.md @@ -19,7 +19,9 @@ These can be set by passing keyword arguments to the constructor, or by sub-clas **Default:** `False` -Recheck the filesystem to see if any files have changed before responding. This is designed to be used in development where it can be convenient to pick up changes to static files without restarting the server. For both performance and security reasons, this setting should not be used in production. +Always check the filesystem to see if any files have changed before responding. This is especially useful in development environments to pick up changes to static files without restarting the server. + +Keep in mind that this setting adds performance overhead. Additionally, it is [not recommended](https://www.redfoxsec.com/blog/understanding-file-upload-vulnerabilities) to use this to serve user-uploaded media files unless you have full confidence in your ability to validate and sanitize user data. --- diff --git a/src/servestatic/__init__.py b/src/servestatic/__init__.py index 6fddb9f..e2c3890 100644 --- a/src/servestatic/__init__.py +++ b/src/servestatic/__init__.py @@ -3,6 +3,6 @@ from servestatic.asgi import ServeStaticASGI from servestatic.wsgi import ServeStatic -__version__ = "4.2.0" +__version__ = "4.3.0" __all__ = ["ServeStatic", "ServeStaticASGI"] diff --git a/src/servestatic/base.py b/src/servestatic/base.py index c3b3368..7ad7c11 100644 --- a/src/servestatic/base.py +++ b/src/servestatic/base.py @@ -37,16 +37,8 @@ def __init__( root: Path | str | None = None, prefix: str | None = None, *, - # Re-check the filesystem on every request so that any changes are - # automatically picked up. NOTE: For use in development only, not supported - # in production autorefresh: bool = False, - max_age: int | None = 60, # seconds - # Set 'Access-Control-Allow-Origin: *' header on all files. - # As these are all public static files this is safe (See - # https://www.w3.org/TR/cors/#security) and ensures that things (e.g - # webfonts in Firefox) still work as expected when your static files are - # served from a CDN, rather than your primary domain. + max_age: int | None = 60, allow_all_origins: bool = True, charset: str = "utf-8", mimetypes: dict[str, str] | None = None, diff --git a/src/servestatic/storage.py b/src/servestatic/storage.py index 049a77f..9127d36 100644 --- a/src/servestatic/storage.py +++ b/src/servestatic/storage.py @@ -9,6 +9,7 @@ from collections.abc import Iterator from concurrent.futures import ThreadPoolExecutor, as_completed from typing import Any +from urllib.parse import unquote, urlsplit from django.conf import settings from django.contrib.staticfiles.storage import ( @@ -152,7 +153,9 @@ def post_process_with_compression(self, files: _PostProcessT) -> _PostProcessT: self.start_tracking_new_files(new_files) for name, hashed_name, processed in files: if hashed_name and not isinstance(processed, Exception): - hashed_names[self.clean_name(name)] = hashed_name + clean_name = self.clean_name(name) + clean_hashed_name = self.clean_name(unquote(urlsplit(hashed_name).path)) + hashed_names[clean_name] = clean_hashed_name yield name, hashed_name, processed self.stop_tracking_new_files() original_files = set(hashed_names.keys()) @@ -170,7 +173,8 @@ def post_process_with_compression(self, files: _PostProcessT) -> _PostProcessT: def hashed_name(self, *args: Any, **kwargs: Any) -> str: name = super().hashed_name(*args, **kwargs) if self._new_files is not None: - self._new_files.add(self.clean_name(name)) + clean_name = self.clean_name(unquote(urlsplit(name).path)) + self._new_files.add(clean_name) return name def start_tracking_new_files(self, new_files: set[str]) -> None: diff --git a/tests/test_django_servestatic.py b/tests/test_django_servestatic.py index 2ec0b71..6b8c81b 100644 --- a/tests/test_django_servestatic.py +++ b/tests/test_django_servestatic.py @@ -387,8 +387,6 @@ def test_django_check_accepts_correct_gzip_middleware_order(): ({"SERVESTATIC_CHARSET": ""}, "servestatic.E017"), ({"SERVESTATIC_ALLOW_ALL_ORIGINS": "yes"}, "servestatic.E018"), ({"SERVESTATIC_SKIP_COMPRESS_EXTENSIONS": "jpg,png"}, "servestatic.E019"), - ({"SERVESTATIC_USE_GZIP": "yes"}, "servestatic.E032"), - ({"SERVESTATIC_USE_BROTLI": "yes"}, "servestatic.E033"), ({"SERVESTATIC_USE_ZSTD": "yes"}, "servestatic.E020"), ({"SERVESTATIC_ZSTD_DICTIONARY": 123}, "servestatic.E021"), ({"SERVESTATIC_ZSTD_DICTIONARY_IS_RAW": "yes"}, "servestatic.E022"), @@ -400,6 +398,8 @@ def test_django_check_accepts_correct_gzip_middleware_order(): ({"SERVESTATIC_MANIFEST_STRICT": "yes"}, "servestatic.E028"), ({"SERVESTATIC_ALLOW_UNSAFE_SYMLINKS": "yes"}, "servestatic.E029"), ({"SERVESTATIC_MINIFY": "yes"}, "servestatic.E031"), + ({"SERVESTATIC_USE_GZIP": "yes"}, "servestatic.E032"), + ({"SERVESTATIC_USE_BROTLI": "yes"}, "servestatic.E033"), ], ) def test_django_check_reports_invalid_setting_types(overrides, error_id): @@ -996,6 +996,19 @@ def test_manifest_with_keep_only_hashed(static_files): shutil.rmtree(static_root, ignore_errors=True) +@pytest.mark.usefixtures("static_files") +def test_manifest_with_keep_only_hashed_query_string(): + with override_settings(SERVESTATIC_USE_MANIFEST=True, SERVESTATIC_KEEP_ONLY_HASHED_FILES=True): + try: + # Collect static files + reset_lazy_object(storage.staticfiles_storage) + # This should not raise an OSError (like [WinError 123] on Windows) due to `?` query strings in `hashed_name` mappings. + call_command("collectstatic", verbosity=0, interactive=False) + finally: + static_root = settings.STATIC_ROOT + shutil.rmtree(static_root, ignore_errors=True) + + @pytest.mark.skipif(django.VERSION < (5, 0), reason="Django 5.0+ only") @pytest.mark.usefixtures("static_files") def test_manifest_with_keep_only_hashed_2(): diff --git a/tests/test_files/static/styles_with_query.css b/tests/test_files/static/styles_with_query.css new file mode 100644 index 0000000..f48e5f8 --- /dev/null +++ b/tests/test_files/static/styles_with_query.css @@ -0,0 +1,3 @@ +.question_mark { + background-image: url("directory/pixel.gif?v=4.0.3"); +}