Feature: Enable defining custom splits in a splits.json file

[michael-hoss]

With the following code, I can specify eval_set="custom-split-name" in the tracking evaluation.

If I have a splits.json file with the following contents in the nuscenes-version-dir, it will work.

{
  "custom-split-name": [
    "example-scene-belonging-to-custom-split-0001",
    "example-scene-belonging-to-custom-split-0002"
  ]
}

[whyekit-motional]

@michael-hoss can you include more details in your PR description? E.g.:

• A code snippet to show how your feature will be used
• Will this only be used for tracking eval, or it can be used across all the tasks?
• Where will splits.json be located?

[michael-hoss]

Sure @whyekit-motional !

Where will splits.json be located?

Under e.g. $NUSCENES/v1.0-mini/splits.json, so in the directory of the used dataset version, next to the other json files.

A code snippet to show how your feature will be used

In unit tests

I just added passing unit tests for tracking eval and detection eval for demonstration, please look at the files. They require $NUSCENES/v1.0-mini/splits.json to have the following content:

{
	"mini_custom_train": ["scene-0061", "scene-0553"], 
	"mini_custom_val": ["scene-0103", "scene-0916"]
}

In an actual evaluation

Also outside of unit tests, I can evaluate the tracking-megvii baseline submission on a custom split (this time locating the same splits.json under the v1.0-trainval dir):

python nuscenes-devkit/python-sdk/nuscenes/eval/tracking/evaluate.py \
/data/sets/tracking-megvii/results_val_megvii.json \
--output_dir /data/sets/tracking-megvii/eval_outputs_on_custom_mini_split \
--eval_set mini_custom_val \
--dataroot /data/sets/nuscenes \
--version v1.0-trainval \
--verbose 1

ℹ️ Note:

• In common/loaders.py, so far, only the ground truth got filtered by split, whereas the results file was assumed to have exactly those samples of the split on which the evaluation happens.
• In my commit d2e3e09, I also enabled that the results file can contain samples that are not part of the evaluation split. When reading all results from the file, those of the used split will remain after filtering in load_prediction

Will this only be used for tracking eval, or it can be used across all the tasks?

I am only really familiar with the tracking eval. Just did my best at enabling custom splits also in other tasks (see commit 0106acf).

Now, there are only these two occurrences of the create_splits_scenes function left in the code that I have not yet replaced by my newly introduced get_scenes_of_split function. They come with hard-coded checks for split names that I don't feel comfortable enough editing because I am not familiar enough with the code:

• in get_prediction_challenge_split from file python-sdk/nuscenes/eval/prediction/splits.py
  • Would this need to support custom splits for consistency? I haven't worked with the prediction task yet and am not planning to, except for the current undertaking.
• in create_splits_logs
  • To me, it looks like this does not actually need to support custom splits, as it is only used for the KITTI export 🤔

[whyekit-motional]

Thanks for the details @michael-hoss!

I will begin reviewing your PR (but it may take a few days or so, as I'm swamped with other work-related matters at the moment 😅)

[michael-hoss]

Please have another look @whyekit-motional. I did:

• reverted the changes in panoptic and lidarseg
• made sure the unit tests set up and tear down a mocked splits.json file
• parameterized the unit tests to avoid duplicate code (predefined split, custom split)

There was only one potentially unrelated issue: I could only run the tracking unit test in a Python 3.10 environment. In my Python 3.7 environment, I got a pandas error realted to MultiIndex creation somewhere deeper down in the mot metrics evaluation. I guess this is related to some version mismatch, similar to my just opened issue #1055

Should we add some more documentation? What else would be needed for a merge?

[whyekit-motional]

@michael-hoss thanks for the changes 👍

I think we can move the discussion regarding the package version to #1055 (apologies I haven't gotten around to trying to reproduce your issue yet)

Code-wise, I think it looks pretty good - I just left a couple of minor comments and then we should be good to merge!

[michael-hoss]

Thanks for your comments @whyekit-motional! Just implemented them and once more executed an own successful test case.

[whyekit-motional]

Thanks for this nice feature @michael-hoss! 💯

[deo-abhijit]

Thank you for this feature @michael-hoss . I was about to implement something similar. <3