Add ignoreHttps & ignoreZipExtension params for GTFS remote file retrieval#7718
Conversation
Specified with build-config.json and only works with remote GTFS files
|
There are some formatting issues in this pr. You need to run |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## dev-2.x #7718 +/- ##
=============================================
+ Coverage 73.99% 74.07% +0.08%
- Complexity 21683 21895 +212
=============================================
Files 2416 2427 +11
Lines 85306 85971 +665
Branches 8410 8567 +157
=============================================
+ Hits 63118 63685 +567
- Misses 19196 19250 +54
- Partials 2992 3036 +44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| * Per-gtfs-source options overriding default validations (http(s) and .zip check), keyed by feed source URI. | ||
| * Only feeds with at least one override set are present. | ||
| */ | ||
| Map<URI, DataSourceOptions> gtfsSourceOptions(); |
There was a problem hiding this comment.
Maybe we should call this httpsSourceOptions and even though the class is now only used for GTFS feeds, I think the documentation here could be more generic that this contains configuration for HTTPS data sources.
| /** | ||
| * Options that override the default validation applied when opening a remote GTFS data source. | ||
| */ | ||
| public record DataSourceOptions(boolean ignoreHttps, boolean ignoreZipExtension) { |
There was a problem hiding this comment.
Same here, I think the documentation could be changed to be more generic and I think this could be named to be HttpsDataSourceOptions. I'm not sure what is the correct package for this though. I can ask in today's developer meeting.
Summary
General
This PR adds 2
build-config.jsonuser-specified parametersignoreHttpsandignoreZipExtensionunder thetransitFeedssection to ignore thehttp(s)and.zipfile extension validation for remote GTFS files. This is particularly useful for developers who rely on third-party transit feed providers, such as government agencies or transit operators, whose feeds may not follow expected conventions or be consistently maintained.Technical
The logical code path is as follows:
ignoreHttpsorignoreZipExtensionset to true inTransitFeeds.javaandBuildConfig.javaHttpDataSourceRepository.javaclass. The class contains URI validations that now have been extended to be ignored if the URI is in the map previously mentioned.build-config.jsonnon-working example:Issue
Issue 1: Malaysia's GTFS feeds - as of December 2024, this feed still neither returns a ''Content-Type: application/zip'' header nor ends in a .zip extension even as the user has written to the government provider.
Issue 2: A 2nd user with the same issue - a 2nd user with the same issue as above
Add GitHub keywords to this PR's
description, for example:
Closes #7512
Closes #6351
Unit tests
The
GtfsConfigTest.javamapGtfsFeedWithDefaultstest was updated to include the 2 parameters. It was successful.OTP was ran with and without the parameters specified. Both ran and were able to construct routes with the debug UI successfully.
Did not measure. Figured that 2 booleans being checked was a negligible change
Yes
Yes
Documentation
Yes
Yes
the configuration documentation updated?
Yes
