Skip to content

Add ignoreHttps & ignoreZipExtension params for GTFS remote file retrieval#7718

Open
johnny-yang-4102 wants to merge 3 commits into
opentripplanner:dev-2.xfrom
johnny-yang-4102:gtfs_source_validation_overrides
Open

Add ignoreHttps & ignoreZipExtension params for GTFS remote file retrieval#7718
johnny-yang-4102 wants to merge 3 commits into
opentripplanner:dev-2.xfrom
johnny-yang-4102:gtfs_source_validation_overrides

Conversation

@johnny-yang-4102

@johnny-yang-4102 johnny-yang-4102 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Summary

General

This PR adds 2 build-config.json user-specified parameters ignoreHttps and ignoreZipExtension under the transitFeeds section to ignore the http(s) and .zip file extension validation for remote GTFS files. This is particularly useful for developers who rely on third-party transit feed providers, such as government agencies or transit operators, whose feeds may not follow expected conventions or be consistently maintained.

Technical

The logical code path is as follows:

  1. The 2 boolean parameters are added in the designated parameter classes (default values are false if none are specified so existing behavior will not change).
  2. The parameters are then used to create a map that contains only URIs that have either ignoreHttps or ignoreZipExtension set to true in TransitFeeds.java and BuildConfig.java
  3. This map is funneled into the HttpDataSourceRepository.java class. The class contains URI validations that now have been extended to be ignored if the URI is in the map previously mentioned.

build-config.json non-working example:

{
  "transitFeeds": [
    {
      "type": "gtfs",
      "source": "http://gov-website.com/rail-gtfs-static-v2",
      "feedId": "TestID",
      "ignoreHttps": true,
      "ignoreZipExtension": true
    }
  ]
}

Issue

Issue 1: Malaysia's GTFS feeds - as of December 2024, this feed still neither returns a ''Content-Type: application/zip'' header nor ends in a .zip extension even as the user has written to the government provider.

Issue 2: A 2nd user with the same issue - a 2nd user with the same issue as above

Add GitHub keywords to this PR's
description, for example:

Closes #7512
Closes #6351

Unit tests

  • Were unit tests added/updated?

The GtfsConfigTest.java mapGtfsFeedWithDefaults test was updated to include the 2 parameters. It was successful.

  • Was any manual verification done?

OTP was ran with and without the parameters specified. Both ran and were able to construct routes with the debug UI successfully.

  • Any observations on changes to performance?

Did not measure. Figured that 2 booleans being checked was a negligible change

  • Was the code designed so it is unit testable?

Yes

  • Were any tests applied to the smallest appropriate unit?

Yes

Documentation

  • Have you added documentation in code covering design and rationale behind the code?

Yes

  • Were all non-trivial public classes and methods documented with Javadoc?

Yes

Yes
image

@johnny-yang-4102 johnny-yang-4102 requested a review from a team as a code owner June 16, 2026 04:18
@johnny-yang-4102 johnny-yang-4102 changed the title Add ignoreHttps & ignoreZipExtension GTFS params Add ignoreHttps & ignoreZipExtension params for GTFS remote file retrieval Jun 16, 2026
@optionsome optionsome self-assigned this Jun 16, 2026
@optionsome optionsome self-requested a review June 16, 2026 09:01
@optionsome optionsome requested a review from tkalvas June 16, 2026 09:01
@optionsome

Copy link
Copy Markdown
Member

There are some formatting issues in this pr. You need to run mvn prettier:write.

@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.60870% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.07%. Comparing base (fe8be33) to head (3153f18).
⚠️ Report is 102 commits behind head on dev-2.x.

Files with missing lines Patch % Lines
...ner/datastore/https/HttpsDataSourceRepository.java 71.42% 0 Missing and 4 partials ⚠️
...planner/gtfs/config/GtfsFeedParametersBuilder.java 66.66% 2 Missing ⚠️
...er/standalone/config/buildconfig/TransitFeeds.java 66.66% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             dev-2.x    #7718      +/-   ##
=============================================
+ Coverage      73.99%   74.07%   +0.08%     
- Complexity     21683    21895     +212     
=============================================
  Files           2416     2427      +11     
  Lines          85306    85971     +665     
  Branches        8410     8567     +157     
=============================================
+ Hits           63118    63685     +567     
- Misses         19196    19250      +54     
- Partials        2992     3036      +44     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

* Per-gtfs-source options overriding default validations (http(s) and .zip check), keyed by feed source URI.
* Only feeds with at least one override set are present.
*/
Map<URI, DataSourceOptions> gtfsSourceOptions();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should call this httpsSourceOptions and even though the class is now only used for GTFS feeds, I think the documentation here could be more generic that this contains configuration for HTTPS data sources.

Comment on lines +3 to +6
/**
* Options that override the default validation applied when opening a remote GTFS data source.
*/
public record DataSourceOptions(boolean ignoreHttps, boolean ignoreZipExtension) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, I think the documentation could be changed to be more generic and I think this could be named to be HttpsDataSourceOptions. I'm not sure what is the correct package for this though. I can ask in today's developer meeting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop presenting URLs not ending in .zip as invalid Allow HTTP/HTTPS URIs that do not end in .zip

3 participants