Figure S3 explanation

[atuldeshpande]

We used dyngen for evaluation of SINGE and other methods to identify the regulatory networks from dyngen data and had a few questions regarding Figure S3.
Unfortunately, we weren't able to locally run the scripts in this repository. Can you please clarify what the baseline precision (number of true edges/ total number of possible edges) in the ground truth network in Figure S3 are? Also, could you explain what the multiple points in the plots for each method correspond to? Are those for different hyperparameter settings?

[rcannood]

Hey Atul!

setup

Unfortunately, we weren't able to locally run the scripts in this repository.

Could you let me know what the problems you are having in setting up the dyngen manuscript codebase are?

Normally it should just be cloning the dyngen_manuscript and using devtools to install it:

git clone https://github.com/dynverse/dyngen_manuscript.git
Rscript -e 'devtools::install()'

points in plot

Also, could you explain what the multiple points in the plots for each method correspond to? Are those for different hyperparameter settings?

If you want, you can check out the results yourself. Once you open up the Rstudio project (dyngen_analysis.Rproj), you can load the evaluation results using the following code.

library(tidyverse)
library(dyngen.manuscript)

exp <- start_analysis("usecase_network_inference")

out <- exp$result("scores.rds")

out$aucs

out$aucs contains the scores for each individual cell for each of the 14 datasets (with 3 replicates) and for each evaluated method. Each point in Figure S3 corresponds to one line in out$summ corresponds to the average scores across all cells, also for each of the different datasets and methods. For this analysis, no changes are made to the hyperparameters, it's just the defaults.

baseline scores

Can you please clarify what the baseline precision (number of true edges/ total number of possible edges) in the ground truth network in Figure S3 are?

The baseline mean AUROC given a random prediction is 0.5 (by definition). For the AUPR it should be around 0.01, but this will vary from dataset to dataset. Would you like me to check out the values for this?

Kind regards,
Robrecht

[atuldeshpande]

For the AUPR it should be around 0.01, but this will vary from dataset to dataset. Would you like me to check out the values for this?

Yes, that would be great, thanks!