You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up of the independent review of PR #7 (spec 004). datasettool.sources.cache (fetch_resource) needs hardening before it is used with untrusted upstreams at scale. Each item needs a test against the local server in tests/conftest.py.
Download size limit: a maximum number of bytes per resource (parameter with a default), checked against Content-Length and while streaming; above it, FetchError and nothing cached
Total timeout per download: timeout today is per socket operation, so a slow-drip server can hold a download forever
Redirects: follow only http and https targets, and refuse an https to http downgrade
Concurrent downloads of the same resource: two processes share <filename>.part; use a unique temporary name (or a lock) so they cannot corrupt each other
Spec 004 R2 updated with the new rules (minor version bump of the package)
Activity