v1.2: Migrate to KoaJS

[confused-Techie]

This PR migrates away from ExpressJS to KoaJS, another HTTP server library created by the same team, with a different structure.

I know it might seem rash to switch the library we use entirely, but let's dive into why I think this move makes the most sense.

What Doesn't work now?

Currently, I've tried to as best as possible use an OpenAPI-like specification to declare the backend, meaning our documentation is (as much as possible) our literal code. But there's been huge oversights with this, such as the actual logic of parameters being entirely separate from their OpenAPI schema, or our header handling being a complete afterthought all-together, that I had to make work with messy unintuitive preLogic and postLogic hooks. Or, arguably the worst oversight, keeping the req and res objects away from the logic call itself, or even the underlying function calls that drive the main logic call.

All of this to mean that the actual server's conformance to our specification is nearly an afterthought, with plenty of places to deviate, and even more places where when we want a request to have direct access to the response object, it has to be shoe-horned in. One easy place to find this difficult behavior is in error response handling, which goes through about three levels of obfuscation when leaving the main logic call of an endpoint and reaching the user, making it difficult to refine further. I've accounted for this largely with testing the entire API call itself, but in the cases where we want to change it, it isn't the simplest or easiest task.

What will be very difficult in the near future?

But, forward thinking here, there's plenty of things I really want us to be able to do here, that may be downright impossible with our current setup.

• Support the HTTP Problems Specification Support and Adopt "Problem Details" specification #292 : Adopting this specification makes it easier to get users to documentation to solve their own problems, reducing strain on our support, and increasing user autonomy. But without easy access to req headers we can't determine if a client can support this, and without easy access to the res we can't change what we return, and lastly, with obfuscated error handling we don't have good insight to what a user gets during an error to even try.
• Grouping of GCP Logs per request Instantiate a logger instance per request #294 : Grouping and organizing logs per request within GCP makes error investigation much simpler and quick, but without better control over the per-request context object, it's impossible.
• Implement Server-Timing Specification: I've wishlisted this for a while, but implementing and logging this everywhere means we can investigate and discover where our users have long waits and what takes longer than it should without them ever telling us, but again is impossible without better flow control of the entire res cycle, or having easy access to the res object to create the headers needed, or having easy per-request write access on the context to make a global timer.
• Using Prefer header to return different pagination styles Implement serviceVersion semver aware filtering #310 : The results of this PR make it obvious that to add post-filtering steps we need to migrate the pagination style we use, which if we want to stay backwards compatible, can likely only be done while being able to respond to the req object's header's to understand the preference of yet to be created support in our various clients.

What I've tried instead?

Over in #297 I tried a huge rewrite of this codebase, where we completely rewrote the OpenAPI-like specification we use, created a per-request context object, and a bunch of other features covered in #299 #296 and #298 but even then, I felt like with so much code churn it wasn't as good as it could be, and still exposed blindspots that would come to be a problem in the future.
Really that PR shows a solution for right now, but doesn't feel extensible enough to support us in the future without more disruptive, major changes.

What we should do? (What's in this PR?)

Instead, I'd suggest that we do exactly what this PR does, which superseeds all the PRs listed in "What I've tried instead". Which is migrate to KoaJS and at the same time redefine our OpenAPI-like Specification, and context object.

The difference between doing those other two tasks here or in the other PR, is that migrating already causes huge changes, might as well do everything all at once.

If we migrate to KoaJS, which is a middleware focused library, means we get a couple of advantages immediately.

What this PR does?

• Since KoaJS is a middleware focused library, it means we can take the extreme duplication of this codebase and reduce it. Very often our codebase will look something like: "Check Authentication; Check package; Modify Package" which is then duplicated several times. (I even had to up the Codacy default duplicated lines values to avoid 'false positives' for this codebase). If we switch to middleware handling we can reduce these repeated steps to a single function that's invoked as a middleware function.
• Since one of KoaJS's main features is the shared cascading context object, with support for modifying it, we can add what we need now, and have the ease of adding to it in the future.
• Additionally, by default KoaJS's context already includes the res and req objects, meaning everyone will have access to it as needed.

You'll notice that all the items in "What will be very difficult in the near future" and "What doesn't work now" essentially all focus on a lack of access to the req & res objects, and difficulty in controlling the request flow, or modifying the context object. Which KoaJS's structure seems very likely to fix, and is exactly why I think this is the solution here.

What this PR Accomplishes?

Below will be a list of items from the above lists that this PR should accomplish in order to be considered successful:

• Support Server-Timing Specification
• Support HTTP Problems Specification
• Group Logs by GCP Request
• Use Prefer Headers to modify Pagination Handling
• Use Parameter OpenAPI Specification as Code
• Use Header OpenAPI Specification as Code
• Have shared access to a central context object
• Allow easy access to req and res objects
• Migrate all endpoints to new schema