02-metrics

Metrics for comparing two trajectories

Good metrics are crucial for an unbiased and comprehensive benchmarking. Here we test several possible metrics for comparing two trajectories.

Metric characterisation

A first characterisation of the metrics. For each metric we:

• generate some examples to get some intuition on how the metric works
• test the robustness to a metric to stochasticity or parameters when appropriate

#	script/folder	description
1	📄correlation.R	Characterisation of the cordist
2	📄topology.R	Characterisation of the Isomorphic, edgeflip and HIM
3	📄clustering.R	Characterisation of the F1branches and F1milestones
4	📄featureimp.R	Characterisation of the corfeatures and wcorfeatures

The results of this experiment are available here

Metric conformity

Differences between two datasets should be reflected in certain changes in the metrics. This can be formalised in a set of rules, for example:

• If the position of some cells are different than in the reference, the score should decrease.
• If the topology of the network is different than that in the reference, the score should not be perfect.
• The more cells are filtered from the trajectory, the more the score should decrease.

Here, we assess whether metrics conforms such rules empirically:

#	script/folder	description
1	📄generate_reference_datasets.R	Generation of toy datasets used as reference to assess metric conformity
2	📄evaluate_perturbations.R	Perturbing the reference datasets (for example shuffling the cells) to check the effect of this perturbation on a metric
3	📄assess_conformity.R	Using the scores on perturbed datasets, assess whether the metrics follow certain rules
4	📄assess_similarity.R	Assess the similarity between metrics
	📄helper-create_perturbation_images.R	Helper for creating small images for each rule
	📄helper-rules.R	Helper functions containing the rules

The results of this experiment are available here.

Aggregation

Here, we create some examples here for why and how we aggregate the scores.

#	script/folder	description
1	📄normalisation_reasoning.R
2	📄aggregation_example.R

The results of this experiment are available here.

The results of this experiment are available here.