What is the expected behaviour when the same dimension occurs in both input_dims and batch_dims?

[boothmanrylan]

It doesn't seem to be well documented what is supposed to happen when the same dimension appears in both input_dims and batch_dims so to try and figure it out myself I started digging through testing.py and test_generators.py and ending up parametrizing test_batch_1d_concat_duplicate_dims to run more cases like so:

@pytest.mark.parametrize(
    "input_dims,batch_dims", [(5, 10), (10, 5), (6, 12), (5, 11), (3, 10)]
)
def test_batch_1d_concat_duplicate_dim(sample_ds_1d, input_dims, batch_dims):
    """
    Test batch generation for a 1D dataset using ``concat_input_dims`` when
    the same dimension occurs in ``input_dims`` and `batch_dims``
    """
    bg = BatchGenerator(
        sample_ds_1d,
        input_dims={"x": input_dims},
        batch_dims={"x": batch_dims},
        concat_input_dims=True,
    )
    validate_generator_length(bg)
    expected_dims = get_batch_dimensions(bg)
    for ds_batch in bg:
        assert isinstance(ds_batch, xr.Dataset)
        validate_batch_dimensions(expected_dims=expected_dims, batch=ds_batch)

Which when run with pytest gives me the following:

tests/test_generators.py::test_batch_1d_concat_duplicate_dim[5-10] PASSED                                                            [ 39%]
tests/test_generators.py::test_batch_1d_concat_duplicate_dim[10-5] FAILED                                                            [ 41%]
tests/test_generators.py::test_batch_1d_concat_duplicate_dim[6-12] PASSED                                                            [ 43%]
tests/test_generators.py::test_batch_1d_concat_duplicate_dim[5-11] FAILED                                                            [ 45%]
tests/test_generators.py::test_batch_1d_concat_duplicate_dim[3-10] FAILED                                                            [ 47%]

In each of the failed cases it is validate_batch_dimensions that causes the failure with the input_batch length of ds_batch being 1 greater than the input_batch length of the expected_dims.

Since I'm not sure what the expected behaviour is, I can't tell if the issue is with xbatcher itself or with validate_batch_dimensions, so I'm wondering if someone would be able to explain to me what is supposed to happen?

[boothmanrylan]

From looking at #132 (comment) it seems like the expected behaviour is to first split the dimension into patches that each have input_dims elements and then to group patches into batches so that each batch has at least batch_dims unique elements, while allowing the last batch to be partial?

This leads me to believe that the above tests fail because of an issue with get_batch_dimensions.

It is also not the behaviour that I would have expected. My intuition was that the dimension would first be split into patches that each have input_dims elements and then to group patches into batches so that each batch has at most batch_dims elements (regardless of duplicate elements from input_overlap) and that a partial last batch would be discarded. For example, in the illustration from #132 (comment) the final batches would be ([0, 1, 2], [2, 3, 4]) and ([4, 5, 6], [6, 7, 8]).

I'm not sure that my intuition is better, but if my understanding of the expected behaviour is correct, I think that it needs better documentation because at the very least it seems to run counter to the default of discarding partial last batches.