Compatibility for 1.2.1

[tombentley]

This issue is to track the compatibility requirements for Ceylon 1.2.1. This issue assumes that 1.2.1 will be binary compatible with 1.2.0, so the problem is "just" about making the toolset use the 1.2.1 artifacts, rather than complaining about mismatched versions.

The specific requirements are:

1. We need to be able to run a "1.2.0 module" (that is, a module compiled using the 1.2.0 toolset) using a 1.2.1 distribution.
2. We need to gracefully fail to run a 1.2.2 module with a 1.2.1 distribution.

@davidfestal has a requirement to be able to compile a 1.2.0 module using a 1.2.1 compiler. Specifically his requirement seems to relate to the implicit dependency on ceylon.language that gets added to compiled modules.

[tombentley]

Presumably we also need to be able to compile a module using 1.2.1 which depends on a 1.2.0 module.

[tombentley]

When Ceylon gets support for assemblies we expect them to be able to override declared versions in a similar manner to what's required here. An assembly is expected to be (approximately) a zipped repository, so associating some version overrides with a repository seems like a sensible approach to items 1. and 2. In this case we'd be associating some overrides of the dist modules with the dist repo.

[tombentley]

Doing this using overrides.xml seems to work fine, except for a couple of places where before we checked for version equality, but now cannot (because the versions are no longer equal). One place was the AbstractModelLoader which checked that the loaded module had a matching version. The other was the ModuleValidator which actively preventing overriding of direct module imports (i.e. those declared in a module.ceylon).

I don't particularly like the idea of just removing the checks, but there's no obvious alternative. The more up to date module itself doesn't know that it can replace the older module, so a more sophisticated check isn't possible without knowing about repos, and these points in the code no nothing about repos.

[quintesse]

An assembly is expected to be (approximately) a zipped repository, so associating some version overrides with a repository seems like a sensible approach to items 1. and 2

I don't think I agree with this, I don't think that what defines the needed overrides is "a repository" at all. It's the assembly itself, which in turn is defined by its contents, ie the actual versions of the available modules.

So to me an assembly would be a zipped repository with the overrides stored inside it, separate from the repo, just like a JAR has a META-INF the assembly could have a special folder holding metadata.

I say this because I don't think "a repository" is the correct "level" for handling overrides, repositories are dynamic, things get added and removed, we change repository paths and ordering all the time when using the tools, etc. (which means you could find the same module but in a repository without the correct overrides)

What determines the need for an override is your application or the distribution itself, neither one of those are defined by a single repository (remember that often for our distribution we use both system modules from the distribution folder and the ones published to the user repository in ~/.ceylon/repo/)

So to me an override is something global, it's:

• part of an assembly
• passed on the command line (--system-overrides?)
• part of the distribution

The first two points are easy enough to understand, I think, but the last one, "part of the distribution", is somewhat more involved. We could literally make it part of the distribution, let's say $CEYLON_HOME/lib/system-overrides.xml. That would be easy enough to support.
But the problem comes if we ever want to support downloading a distribution from the Herd using only a minimal bootstrap. In that case what "defines" the distribution and its associated overrides file are the components themselves. So a possible solution for the distribution itself could be to store the overrides.xml file directly in the ceylon-module-resolver-X.Y.Z.car module.

[tombentley]

I'm not saying that repositories in general should support overrides. But the fact is that there is a dist repository. Putting the information about how the modules in it should override other modules doesn't seem like a bad idea to me. Putting them somewhere else in the dist is of course possible, but I don't really see how that's better. But honestly where this file goes doesn't interest me particularly, I'd just like it to end up being consistent with assemblies if possible (and that's hard because we're not really thinking about them yet).

Saying repositories are dynamic things is a generalization. The dist repository is not dynamic.

In that case what "defines" the distribution and its associated overrides file are the components themselves.

Well to my mind the distribution is an assembly, and would be downloadable as a single thing, not as individual components.

[tombentley]

Here's a question: Does the metamodel (specifically ``module.dependencies and `modules.list`) show the runtime versions or the compile time versions?

[quintesse]

But the fact is that there is a dist repository. Putting the information about how the modules in it should override other modules doesn't seem like a bad idea to me.

Like I said, the system components could be retrieved from some other location, it's not exactly advisable to do so (like using -Xbootclasspath in Java), but we do so for example in the test suite. It's the reason we have ant publish in our build scripts. That wouldn't work unless you copy that overrides file to the user repository as well. But the user repository could easily hold several versions of those system modules, so which version of the overrides file is the correct one?

Addendum: That's why an overrides file that gets globally applied makes more sense because it wouldn't matter anymore where the modules come from, the overrides get applied all the same.

That's what I mean when I say that the repository is not the identifying feature.

It might seem like a convenient place to put it but IMO it's just not correct.

An assembly is different because it's a snapshot of a repository, there's no way to mess with the versions or the ordering of the module lookup anymore and the overrides file is bound to that specific snapshot.

Edit: added addendum above

[quintesse]

Btw, it also means that we have to add support for a new overrides.xml artifact in the CMR (not a completely trivial task). And you'd have to find it in a place where nothing gets stored, a non-module folder, the root for example, which also means extending the CMR with new functionality it didn't have before. And all that just to find a single file in a single repository. It just doesn't seem right.
A system to provide an override file from outside the CMR and pass it does already exist and could IMO be extended to include the needed new functionality.

[FroMage]

Does the metamodel (specifically module.dependencies and modules.list) show the runtime versions or the compile time versions?

Since the modules pointed to will be the result of the override, and the original imports may not be available, they should point to the result of the override (runtime version). In the future we may add an API to query the original imports, but those will likely point to a Module? in case (most likely) that it's not available.

[tombentley]

Last night after I posed that question I found that module doesn't work for precisely this reason. I agree we should use the runtime version because that's the only real choice we have. Now I just need to figure out how...