Add add axis transform

[jo-mueller]

Fixes ome/ngff#499

This PR adds a new, versatile transform to broadcast coordinates from N to M dimensionality by adding or dropping dimensions to the coordinate vector.

Affine transformations currently already allow for input dimensionality != output dimensionality. So instead of baking an up-projection into an affine, this can now be declared more explicitly.

I think this transform is super useful, for instance to project 2D images (histology images) into 3D space, where they may have been aligned with some sort of volumetric images (see ABBA tool).

I figured it'd be now or never to slip it in and it adds or takes nothing from the current 0.6 state of the art.

cc @bogovicj @dstansby @clbarnes @LucaMarconato @d-v-b

[github-actions]

Automated Review URLs

• Readthedocs

[clbarnes]

I still can't decide whether this should be defined as a newaxis or a dropaxis.

If we want to co-visualise a multichannel YXC image and its YX labels, putting the label coordinates into a specific channel doesn't seem to make sense, where I think putting a multichannel pixel into strictly YX coordinate system does. Even saying "I have a coordinate in YX space and want all the channel data at that point" doesn't quite make sense in the context of this transform because for a 3-channel image you need to convert your 1 label coordinate into 3 image coordinates (0,Y,X) (1,Y,X) (2,Y,X).

On the other hand, only defining a dropaxis means we can't line up a YX slice of a ZYX volume by adding the axis and then translating in Z.

[jo-mueller]

@clbarnes I'm not 100% sold on the dropaxis transform, mostly because it addresses a problem that viewers seem to get right intuitively? I.e., would viewers generally accept scale values for a channel axis? They definitely exist in the ngff spec, but many readers pop out the corresponding scale values before handing them over to viewers. Attributing a labels image correctly to a particular channel (dimension) seems more like something that collections will be better at solving?

I am thinking of adding something like The corresponding axis in the output coordinate system MUST be of type space or time in the transform's spec, to navigate around the channel dimension/labels issue.

[LucaMarconato]

Thanks for sharing! I wonder if achieving the dropAxis and newAxis should be as part of the specs or as an helper function that returns an affine transformation.

[jo-mueller]

Thanks for sharing! I wonder if achieving the dropAxis and newAxis should be as part of the specs or as an helper function that returns an affine transformation.

Also a good point; Both can already be baked into an affine transform 🤔 But then again, so can a scale, translation or rotation.

[clbarnes]

Discussed in Heidelberg: we do want this to go in, but want to be able to configure it to add or drop an axis.

Open discussion:

• add/drop one at a time, or allow adding/dropping multiple in one instance of the transform?
• add/drop at arbitrary indices, or always at first/ last (may require chaining with a mapAxis)
• add and drop in the same instance?

My maximally flexible, minimally text-y proposal:

// option 1
{
  "type": "newAxis",
  "operations": [
    1,  // insert an axis at position 1
   // THEN
    -2  // remove an axis from the NEW position 2
  ]
}

but it could be made more explicit with

// option 2
{
  "type": "newAxis",
  "operations": [
    {"insert": 1},  // insert an axis at position 1
    // THEN
    {"remove": 2}  // remove an axis from the NEW position 2
  ]
}

Note that this is an externally-tagged discriminated union {"myVariant": ...}, which is different to

• OME-Zarr's common internally-tagged ({"type": "myVariant", ...})
• Zarr's adjacently-tagged ({"name": "my_variant", "configuration": { ... }})

or

// option 3
{
  "type": "sequence",
  "transformations": [
    {
      "type": "newAxis",
      "insert": 1
    },
    {
      "type": "newAxis",
      "remove": 2
    }
  ]
}

// option 4
{
  "type": "sequence",
  "transformations": [
    {
      "type": "newAxis",
      "operation": "insert"
    },
    {
      "type": "mapAxis",
      "mapAxis": [ ... ]
    },
    {
      "type": "newAxis",
      "operation": "remove"
    },
    {
      "type": "mapAxis",
      "mapAxis": [ ... ]
    }
  ]
}

We also discussed having separate add and drop axis transforms, but discounted it because it would be the only case of an invertible transform which returns a transform of a different type (discriminated union variant).

[clbarnes]

Warning: none of the above serialisations imply a dimensionality without knowledge of the coordinate system. We already have serialisations which are this way (identity), but have identified that this can be a bit annoying during implementation.

[jo-mueller]

@bogovicj @clbarnes @LucaMarconato @jni @melonora @thewtex @tischi (who am I forgetting?)

Here comes the overhauled proposal on how to add/drop axes from coordinate systems via a newAxis transform (open to suggestions for name change)

[bogovicj]

Thanks @clbarnes for laying out those options.
I'd be happy with either options (1) or (2), with a small preference for (2).

If this transformation can both add and remove axes, we should come up with a new name (not "newAxis"). some initial ideas:

• addRemoveAxes
• insertDeleteAxes
• editAxes
• rerank
• projection
• axisProjection

[jo-mueller]

If this transformation can both add and remove axes, we should come up with a new name (not "newAxis").

Yeah, agree.

• ´addRemoveAxes´
• ´insertDeleteAxes´
• ´editAxes´
• ´rerank´
• ´projection´ 👍
• ´axisProjection´ 👍
• projectAxis 👍

(copy and add thumb for primitive voting? 🙂 )

[clbarnes]

I think projection covers it best, so long as that's mathematically accurate! Avoiding referring to Axis might be preferable given we're already inconsistent about whether to say axis (mapAxis) or dimension (byDimension). I would also lean towards option 2, where each operation adds or removes a single axis and operations can be chained within a single transformation.

[bogovicj]

I'd be happy with projection too, though the possible confusion is that "projection" is a quite general term. E.g. the camera matrix is a 3D- 2D affine that is also a kind of projection.

The transform we're defining is an axis-aligned projection - all coordinates are unchanged or are set to zero, but I was trying to keep the names short, without too many adjectives:

We'dll have to be very clear in the description no matter what name we choose.

[jo-mueller]

My favorite would be projectAxis:

• indicates operational similarity with mapAxis (which does something very similar)
• Rolls off the tongue a bit better than projectDimension

We are using the terms axis and dimensions a bit exchangeable anyway (may be something to carve this out a bit more in the future).

We'dll have to be very clear in the description no matter what name we choose.

@bogovicj do you think that is currently the case?