Skip to content

Cache hardening: size limit, total timeout, redirect rules, concurrent .part files #8

Open

Description

@matbrgz

Follow-up of the independent review of PR #7 (spec 004). datasettool.sources.cache (fetch_resource) needs hardening before it is used with untrusted upstreams at scale. Each item needs a test against the local server in tests/conftest.py.

  • Download size limit: a maximum number of bytes per resource (parameter with a default), checked against Content-Length and while streaming; above it, FetchError and nothing cached
  • Total timeout per download: timeout today is per socket operation, so a slow-drip server can hold a download forever
  • Redirects: follow only http and https targets, and refuse an https to http downgrade
  • Concurrent downloads of the same resource: two processes share <filename>.part; use a unique temporary name (or a lock) so they cannot corrupt each other
  • Spec 004 R2 updated with the new rules (minor version bump of the package)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions