New feature: validator API service

[themightychris]

Describe the problem

For a recent demo, I needed to be able to run the the validator as a service and execute validations through synchronous API calls.

I initially tried to just add a /validate endpoint to do the job to the existing spring server under web/service but that quickly became a rabbit hole because I wanted to be able to run this without all the Google infrastructure for storing data and doing pub/sub and be able to build it entirely within a container and found that project to be very far from being set up for that. I ended up just building a bare-bones spring server under /api that only exposed the /validate endpoint

Proposed solution

If I end up building this thing for real, I'll want to move away from working out of a fork so there won't be any barrier to the API moving to new validator versions. I'm interested in the community/maintainer opinions on the best route:

1. Contribution to MobilityData/gtfs-validator: refactor web/service so that it can build entirely in a container and so that the GCP services are an optional dependency for the job-oriented endpoints, and add the /validate endpoint to it
2. Contribution to MobilityData/gtfs-validator: add something like the separate minimal API service I built somewhere else in the tree
3. Just build a separate repo and publish it either under my own org or contribute it to the MobilityData org the defines the API service and consumes the validator JAR

Alternatives you've considered

No response

Additional context

No response

[davidgamez]

Hi @themightychris,

Thanks for opening this and for sharing your approach. We’ve been planning along similar lines as part of #1911, since the current web/service project is tightly coupled to the major validator releases. That coupling slows down enhancements and makes it less flexible for demos or lightweight deployments.

1. Contribution to MobilityData/gtfs-validator: refactor web/service so that it can build entirely in a container and so that the GCP services are an optional dependency for the job-oriented endpoints, and add the /validate endpoint to it

I agree with you here. The API layer should be flexible and not tied to any specific vendor. Storage should be configurable, with local storage as the default when nothing else is set. Making GCP integrations optional would make the service usable in both demo and production environments.

2. Contribution to MobilityData/gtfs-validator: add something like the separate minimal API service I built somewhere else in the tree

In order to keep this repository clean, I would prefer to have an API service as an independent repository(option 3 of your proposal).

3. Just build a separate repo and publish it either under my own org or contribute it to the MobilityData org the defines the API service and consumes the validator JAR

This aligns with the original intent in #1911. A dedicated repository would make it easier to iterate on the API layer at its own pace, while the API code would only depend on the published Maven artifacts from the validator. As part of this, we’d also like to take the opportunity to refactor the existing Swagger API
so that current users can migrate smoothly to the new endpoints if they choose to.

So, our preference is option 3: creating a dedicated repository for the API service. We’d like to have this project under the MobilityData organization so that it’s easier to maintain, stays aligned with validator releases, and remains open to community contributions.

[themightychris]

The part I'm interested in wouldn't require coupling with any cloud vendor or need storage. I just want to be able to run a container and then throw GTFS ZIPs at an endpoint and get the validation report back in the response whenever it's done. I think that outside MobilityData's use case of running a global public validator service, other users wouldn't have need for the queue and storage features so I'm not sure that what I'm describing here has any hard need to be coupled with what #1911 is describing.

It wouldn't be crazy to do them both in the same app/repo—the connection to cloud queue+storage services could be optional environment vars that flip on those endpoints. But I don't see the business case for investing in supporting multiple cloud vendors there because no one else would want to run it the same way MD runs the public one and need to engineer for unpredictable global loads

Any mature cloud container runner service provides all the capabilities anyone with simpler use cases would need to autoscale based on long-running HTTP requests per job

[davidgamez]

Thanks for clarifying. A lightweight container that takes a GTFS ZIP and returns the report directly is definitely useful. This fits well with option 3: a dedicated API repo where the default mode is synchronous and straightforward.

[themightychris]

@davidgamez I'm gonna need this soon and can put some hours in. I saw you have an epic going that starts with some maven surgery. Is there anywhere useful I could jump in without knowing maven?

[davidgamez]

@davidgamez I'm gonna need this soon and can put some hours in. I saw you have an epic going that starts with some maven surgery. Is there anywhere useful I could jump in without knowing maven?

Hi @themightychris, we added the epic, but haven't started working on it yet. Let me set aside time next week to define a few issues under the epic. The first one would be reviewing the OpenAPI schema, from which most of the code should be generated. This is a similar pattern to the GBFS Java validator.

[derhuerst]

@davidgamez I'm gonna need this soon [...].

Note that I needed this too, so I built gtfs-validator-metrics-service. Somewhat hacky (written in JS, spawns the GTFS Validator as a subprocess), but works!

Currently, the output is Prometheus/OpenTelemetry metrics only, but adding e.g. JSON support would be trivial.

[themightychris]

Thanks @davidgamez and @derhuerst !

Sounds like in the short term I'm just going to have to clean up and deploy my fork of the validator itself that exposes it as a minimal API service. I'll push that up somewhere and share it here. I'll be keen to get off having a fork by the time I wrap this project and will stay tuned for any opportunity to contribute to that but I really struggle with all the java build stuff and have no business contributing on that front

[davidgamez]

Adding this PR/repo here for visibility, feat: API implementation and Docker artifact publishing.

TL;DR:

• Created a new repository
• Use the core validator's libraries as Maven dependencies
• No MobilityData configuration or specific cloud infrastructure attached
• Spec-Driven Development based on OpenAPI schema

[davidgamez]

Closing this as https://github.com/MobilityData/gtfs-validator-api meets the ask.