fix: array creation in tests

[flying-sheep]

so what I thought happens is that they split up codecs into

1. shards: if present, this contains the parameters for an implicitly created ArrayToArrayCodec that wraps all other given codecs and applies them for each shard individually
2. filters: list of ArrayToArrayCodecs
3. serializer: a single ArrayToBytesCodec
4. compressors: a list of BytesToBytesCodecs

this assumption seems to hold for e.g. tests/test_codecs.py::test_order, but other tests fail, so apparently I didn’t understand this.

[ilan-gold]

Having both of these seems wrong?

[flying-sheep]

Before, it was the following, no idea how to express that the new way.

codecs=[
    ShardingCodec(
        chunk_shape=(32, 32, 32),
        codecs=[
            ShardingCodec(
                chunk_shape=(16, 16, 16), index_location=inner_index_location
            )
        ],
        index_location=outer_index_location,
    )
],

[ilan-gold]

Ok this one specificully is for nested sharding. I think this one is specifically testing nested sharding but I dont know if the way you descibed it is valid, so cc @d-v-b again!

[ilan-gold]

Yeah, I don't even know what this is, I see it's in the type, but it's undocumented:

that shape parameter could mean a two things, as an example of why I don't know how it works: chunks within the shard arrangement or the overall shape

[ilan-gold]

cc @d-v-b

[flying-sheep]

the param docstring for shards needs work:

• the rST type contains only tuple[int, ...]
• the free text contains "auto" and "keep" as well
• the type contains Sequence[Sequence[int]] and that TypedDict allowing to specify index_location

this drift between type annotation and docstring is why I make sure to use the annotations in scanpy …

[d-v-b]

agree this is clunky. we goofed by having 2 flavors of array creation routines (create and create_array) X two function colors, each wrapped by multiple higher-level functions.

The idea behind the ShardsConfigParam is to give you some control of the sharding codec configuration via the shards parameter. If you don't need to specify where the index is, then ignore it and just pass the outer chunk shape you want to the shards param.

[flying-sheep]

OK, so it’s just a docs issue and you should

1. document the exported location of ShardsConfigParam
2. unify the 2 separate locations in the docstring where you explain what’s accepted by reflecting what the type annotation already says.

[d-v-b]

and we should also probably make the ShardsConfigParam model the full configuration space of the sharding_indexed codec (you can't set the index codecs on it, iirc)