CHANGELOG.md

Changelog

2.6.0 (2026-01-30)

• feat: CORS probe (#359)

2.5.0 (2026-01-20)

• Adapt to the new csv-detective output (#369)
• Better pick latitude/longitude columns when there are multiple options (#376)
• chore: move pyproject optional-dependencies to dependency-groups (#368)
• Fix stable resource URL syntax (#375)
• fix: unlink pmtiles once only (#373)
• Move file cleanup condition to the config (#374)
• refactor: refactor docker-compose files to use profiles (#370)
• Upgrade csv-detective (#377)

2.4.5 (2026-01-12)

• chore(rq): set failure ttl to 10 days (#372)
• chore: clean remaining pmtiles files (#371)
• fix: fix tag_version.sh to work with MacOS (#367)

2.4.4 (2025-11-27)

• Add binary type support (#364)
• Bring tables_index update closer to db insertion (#366)
• Change how we spot tables to delete (#358)
• Handle empty geometry cells (#365)
• Revert "feat: better logs for CLI commands run by crons (#354)" (#355)
• chore: remove parallel tests executors and add minimum test coverage of 80% (#356)
• docs: fix README
• docs: fix README (#352)
• feat: add parsing_table index on checks (#357)
• feat: add resource_id index on tables_index (#353)
• feat: add total resources and deleted resources in crawler status health check (#362)
• feat: better logs for CLI commands run by crons (#354)
• feat: optimize cleanup of stuck status resources (#363)
• fix: resolve async/await compatibility issues with Typer CLI (#361)
• refactor: replace Minicli dependency with Typer (#233)

2.4.3 (2025-10-21)

• fix: fix entrypoint (#350)

2.4.2 (2025-10-20)

• build: add tag_version.sh script (#342)
• build: use uv instead of Poetry for lock file and in CI (#241)
• chore: bump version to 2.4.2
• docs: fix changelog
• docs: fix error in changelog
• docs: remove "current" section in changelog to prepare it for automatic update of changelog
• docs: update changelog for merged PR #328
• Enable to analyse and insert parquet files (#343)
• feat: add performance tests and CLI tools for geographical file conversion benchmarking (#319)
• feat: allow to crawl non-data.gouv URLs (#312)
• feat: benchmarks CI workflow (#339)
• feat: dynamic version in user agent (#328)
• feat: flag deleted resources in CSV DB instead of deleting them (#340)
• feat: store dataset_id in CSV tables_index (#341)
• Fix after parquet (#346)
• fix: fix docker image
• fix: fix release in CI (#349)
• fix: fix user agent string (#337)
• Revert "chore: bump version to 2.4.2"
• Update to version 2.4.2.dev for next development cycle

2.4.1 (2025-09-03)

• Only delete the table if the insertion was not successful #332
• Handle cases of empty table in db when getting the previous analysis #333
• Better discriminate long column names #334

2.4.0 (2025-09-02)

• Simplify CI configuration #290
• Remove unused dependencies #292
• Fill in check throughout the process to send previous steps infos in case of late crash #293
• Fix purge csv tables CLI by using the csv db connection #294
• Better gz files extraction function name #295
• Add more detailed statuses #298
• Handle cases of too long columns labels for postgres #299
• Fix rare issue in /status/crawler/ endpoint #301 #302
• Fix typos, deprecated examples and add emojis in README #304
• Upgrade dependencies and fix pytest-asyncio deprecated code #305
• Handle new cases of geo columns from csv-detective #303
• Fix custom Sentry error by capturing original exception stack trace and add test #308
• Optimize queue priorities for resource processing #311
• Add resource_id info in timer logs and add timer for analysing resource #313
• Add check id in udata extras to facilitate debug #307
• Split csv_to geojson_and_pmtiles function into a new intermediary function csv_to_geojson for better unit testing/benchmarking #317
• Add download resource CLI command #320
• Fix custom exception ExceptionWithSentryDetails to include Python stack trace #315
• Fix deprecated Sentry method #323
• Upgrade csv-detective #322 #324 #327
• Stream geojson conversion to prevent RAM consumption #326
• Gracefully crash if any of the column names is too long for Postgres #329
• Return updated check in handle_resource_exception to send the right payload to udata #330

2.3.0 (2025-07-15)

• Delete tmp files if error in download #277
• Add parsing_table in extra analysis payload #279
• Add a column in catalog that indicates when the status was last changed and delete the status-specific endpoint #276
• Remove F401 (unused import) ignore rule and clean up init.py files with explicit all declarations #281
• Update PostgreSQL version in CI to match Docker Compose (15.13) #282
• Add a CI job to create a release on Sentry #246
• Use pre-cast dataframe from csv-detective instead of re-reading the file #236
• Enable csv to GeoJSON and PMTiles converion if relevant #278 #285 #291
• Add conversions state to health endpoint #284
• Better logs in Sentry #287 #289
• Consistent return types for CRUD methods #288

2.2.1 (2025-06-11)

• ⚠️ resources_exceptions migration from main to csv db. Code is updated accordingly #264
  • Migration should be made manually. See migrations instructions in the migrations files.
• Fix some CLI functions to improve local debug #269
• Upgrade csv-detective and use previous analysis to validate if available #266
• Fix and add new purge tables CLI utils #271 #272 #273
• Increase timeout for exception resources #274
• Fix logs when creating indexes #275
• Fix datetime insertion in db #280

2.2.0 (2025-05-16)

• Better prioritizing #255
• Reset same category udata extra fields on sending #256
• Make sure detected_last_modified_at is set even if HAS_NOT_CHANGED #257
• Fill in new check even if no update #258
• Better change detections and add related tests #259
• Enable GeoJSON to PMTiles conversion #260 #267
• Parquet files are named after resource ids #262
• Upgrade csv-detective #263
• Fix call to a sync function after demo crash test #265

2.1.2 (2025-03-19)

• Add a status resetting mechanism for stuck resources #251
• Update harvest_modified_at on load catalog #252
• Unify exceptions to prevent unhandled ones in Sentry #253
• Get max allowed size even in CLI process #254

2.1.1 (2025-03-12)

• Fix CLI analyse-csv and allow analysis from a resource id #248
• Rework handling of too large files #248

2.1.0 (2025-01-13)

• Refactor ParseError to enrich Sentry with context and to inquire about Sentry errors like #4096 #218
• Remove legacy routes #203
• More explicit error reporting when sending to udata without raising errors for udata responding with a 404 #213
• Minor cleaning: remove unused arg in function #219
• Fix type issue regarding resource_id #220
• Use bump'X #226
• Get actual resource URL in case of 404 (change since last catalog load) #225
• Add CLI util to insert or update a resource into the catalog (change since last catalog load) #228
• Fix deadlocks errors when purging CSV tables by refactoring purge_csv_tables to use atomic transactions #230
• Improve timing of checks depending on changes since last check #163
• Remove bad default value in CLI to insert a resource #235
• Trigger GitLab deployment in CI when pushing on main #186
• Fix GitLab deployment in CI #239
• Add indexes to improve resource filtering and batch selection #240
• Parallelize tests in CI #238
• Refactor analysis logic to remove 5 non necessary queries, using the existing data in the code instead of re-querying it #227
• Use a package-manager-agnostic pyproject.toml instead of a Poetry pyproject.toml #170
• Remove unused setup.cfg #244
• Use a bash file instead of text file to store env vars across CI jobs #245

2.0.5 (2024-11-08)

• Fix minor types issues #204
• Return resources statuses count in crawler status endpoint response #206
• Fix deprecated CircleCI config #207
• Fix Sentry issue #4195 #209
• Clean doctrings for more consistent style #215
• Fix some type hints #214
• Add option to force analysis even if resource has not changed #205
• Fix get all checks CRUD method #217
• Deactivate parquet export for small CSVs #216

2.0.4 (2024-10-28)

• Fix wrong resource status #196
• Fix issue related to empty table_indexes column instead of default {} #197
• Make the last_check column of the catalog table as a foreign key to checks table, in order to fix error when crawling resources with last checks that don't exist anymore #195
• Fix analyse-csv CLI when using by url, and refactor errors for cases when resource or url not found #200
• Fix errors when sending malformed request, and make API error responses more consistent #202

2.0.3 (2024-10-22)

• Save git commit hash in CI and use it for health check #182 and #185
• Add comment column/field to ressources exceptions #191
• Add extra args and DB fields for parquet export #193
• Fix CircleCI config for packaging version not to include commit hash when publishing #194

2.0.2 (2024-10-07)

• Fix typos in README in curl commands examples #189
• Bump csv-detective to 0.7.3 #192

2.0.1 (2024-10-04)

• Refactor function to get no_backoff domains and add PostgreSQL indexes to improve DB queries perfs #171
• Clean changelog and remove useless section in pyproject.toml #175
• Refactor purge_checks CLI to use a date limit instead of a number #174
• Fix resources exceptions routes responses, add resources exceptions tests #176
• Fix CSV analysis CLI #181
• Add a PUT /api/resources-exceptions/{id} route to update a resource exception #178
• Add a quiet argument for purge_check and purge_csv_table CLIs #184
• Fix wrong resource status #187
• More informative error relative to check resource CLI #188

2.0.0 (2024-09-24)

• Use Python 3.11 instead of 3.9 for performance improvements and future compatibility #101
• Refactor and split code from crawl.py into separate files using refactored db.Resource class methods and static methods #135
• Allow routes with or without trailing slashes #158
• Delete resource as a CRUD method #161
• Refactor routes URLs to be more RESTful and separate legacy routes code from new routes code #132
• Display app version and environment in health check endpoint #164
• Use ENVIRONMENT from config file instead of env var #165
• Manage large resources exceptions differently #148
• Add checks aggregate route #167

1.1.0 (2024-09-26)

• Use profiling option from csv-detective #54
• Remove csv_analysis, integrate into checks #52
• Add new types for csv parsing: json, date and datetime #51
• Notify udata of csv parsing #51
• Allow None values in udata notifications #51
• Add tests for udata-triggered checks #49
• Include migration files in package
• Allow to configure a dedicated PostgreSQL schema #56
• Fix typo in handle_parse_exception and schema in CLI #57
• Skip archived dataset when loading catalog #58
• Update resources expected dates in API following udata refactoring #60
• Download csv resource only if first check #61
• Send content-type and content-length info from header to udata #64
• Add timezone values to dates sent to udata #63
• Rename analysis filesize to content-length #66
• Sleep between all batches #67
• Support having multiple crawlers by setting a status column in the catalog table #68
• Add a health route #69
• Make temporary folder configurable #70
• Fix conflict on updating catalog with multiple entries for a resource #73
• Set check:available to None in case of a 429 #75
• Improve conditional analysis logic and readability #76 #80
• Use latest csv-detective version #89
• Compare content type / length to check if changed #78 #79
• Create a list of exceptions to analyse despite larger size #85
• Enable csv.gz analysis #84
• Add worker default timeout config #86
• Return None value early when casting in csv analysis #87
• Ping udata after loading a csv to database #91
• Allow for none value in resource schema #93
• Handle other file formats #92
• Add a quiet option on load catalog #95
• Select distinct parsing tables to delete #96
• Enable parquet export #97
• Update documentation #98 and #106
• Add linter and formatter with pyproject.toml config, add lint and formatting step in CI, add pre-commit hook to lint and format, update docs and lint and format the code #99
• Update sentry-sdk dependency, and update Sentry logic to be able to send environment, app version and profiling/performance info #100
• Basic cleaning: use Python 3.11 in CI, remove Pandas in project dependencies, add type hints, fix wrong type hints, remove deprecated version field in docker compose files, update .gitignore [#102] [datagouv#102] and #107
• Add missing content-type for csv.gz #103
• Remove deprecated pytz module #109
• Refactor project structure to use DB classes for each DB table, with their factorized DB methods #112 and #55
• Add tests coverage feature #122
• Refactor routes #117
• Fix Ruff configuration #125
• Add some API tests to improve coverage #123
• Fix health check endpoint route which was wrongly removed, and add test for API health check endpoint to make sure this endpoint is working as expected #128
• Add basic authentication via API key using a bearer token auth for all POST/PUT/DELETE endpoints #130
• Simplify getting Sentry info by loading pyproject.toml info in config #138
• Add a POST /api/checks/ route for force crawling #118
• Update csv-detective to 0.7.2 which doesn't include yanked version of requests anymore #142 and #144
• Update resource statuses in DB when crawling and analysing, and add resource status route #119
• Simplify save_as_parquet method, and fix type not compatible with Python 3.9; remove unused import #156
• Fix and simplify project metadata loading #157
• Pin Numpy version to 1.26.4 to avoid conflicts with pandas and csv-detective

1.0.1 (2023-01-04)

• Packaging-fix release

1.0.0 (2023-01-04)

• Initial version