[WIP] Option for isolated and colony profiles #30

shntnu · 2019-06-26T14:23:31Z

Addresses #29

I pushed all of Greg's changes verbatim

shntnu · 2019-06-26T14:28:56Z

annotate.R

 metadata_dir <- paste("../..", "metadata", batch_id, sep = "/")
+backend_dir <- paste("../..", "backend", batch_id, unlist(strsplit(plate_id, "_"))[1], sep = "/")


@gwaygenomics Do you know why the string is being split?

As we discussed on the brief chat, indeed, it looks like I am splitting the plate ID because I am inputing the plate ID with the sc_type tagged (here).

This is a result of the output from the modified aggregate.R call above, based on this line

This will definitely break in situations where the plate ID has an underscore.

Will need to address!

shntnu

Let's leave the line breaks in there

@gwaygenomics edit - addressed in a3675c9

shntnu · 2019-06-26T14:32:20Z

preselect.R

@@ -98,7 +98,9 @@ for (operation in operations) {
        # This is handled differently because there is no direct way yet to do filtering in cytominer
        # TODO: rewrite this after cytominer has an appropriate filtering function for this
        testthat::expect_false(is.null(replicates), info="replicates should be specified when performing replicate_correlation")
-
+


The print debugging can go

addressed in a3675c9

shntnu · 2019-06-26T14:33:01Z

select.R

@@ -37,8 +37,7 @@ variables_selected <-
    Reduce(function(df1, df2) dplyr::inner_join(df1, df2, by ="variable"), .) %>%
    magrittr::extract2("variable")

-backend_dir <- paste("../..", "backend", batch_id, plate_id, sep = "/")
-
+backend_dir <- paste("../..", "backend", batch_id, unlist(strsplit(plate_id, "_"))[1], sep = "/")


This can probably go (same as before)

shntnu · 2019-06-26T14:33:07Z

normalize.R

 subset <- opts[["subset"]] # e.g. "Metadata_broad_sample_type == '''control'''"

-backend_dir <- paste("../..", "backend", batch_id, plate_id, sep = "/")
+backend_dir <- paste("../..", "backend", batch_id, unlist(strsplit(plate_id, "_"))[1], sep = "/")


This can probably go (same as before)

shntnu

Overall, only the logic of aggregate has changed. Additionally, there's some string processing related to plate id.

gwaybio · 2019-06-26T17:50:35Z

I don't remember why I did this, but I will look into before merging

gwaybio

These updates are not ready to be merged in

A couple changes need to be made first

Specify more options for sc_type flag (conditional statements)
Remove all print debugging statements
Figure out plate ID parsing from single cell profiles

gwaybio · 2019-06-26T18:34:10Z

aggregate.R

-    strata = c("Metadata_Plate", "Metadata_Well"),
-    operation = "mean"
-  ) %>% collect()
+  object <- object %>%


This bit here is also specific to aggregating single cell profiles

It takes code from cytominer::aggregate()

gwaybio · 2019-06-26T18:34:55Z

aggregate.R

 }

-aggregated <-
-  aggregate_objects("cells") %>%
+aggregate_cols <- c("TableNumber", "ImageNumber", "ObjectNumber",


This is also specific to single cells and using it in other repositories will not always work

gwaybio · 2019-06-26T18:37:02Z

aggregate.R

@@ -3,56 +3,78 @@
 'aggregate

 Usage:
-  aggregate.R <sqlite_file> -o <file>
+  aggregate.R -s <sqlite_file> -o <file> [-t <sc_type>]


after reviewing, I don't think this change is ready for prime time. A couple more conditional statements that only behave when sc_type is specified are required.

gwaybio · 2019-06-26T18:42:32Z

annotate.R

 metadata_dir <- paste("../..", "metadata", batch_id, sep = "/")
+backend_dir <- paste("../..", "backend", batch_id, unlist(strsplit(plate_id, "_"))[1], sep = "/")


As we discussed on the brief chat, indeed, it looks like I am splitting the plate ID because I am inputing the plate ID with the sc_type tagged (here).

This is a result of the output from the modified aggregate.R call above, based on this line

This will definitely break in situations where the plate ID has an underscore.

Will need to address!

gwaybio · 2019-06-26T18:43:14Z

normalize.R

 subset <- opts[["subset"]] # e.g. "Metadata_broad_sample_type == '''control'''"

-backend_dir <- paste("../..", "backend", batch_id, plate_id, sep = "/")
+backend_dir <- paste("../..", "backend", batch_id, unlist(strsplit(plate_id, "_"))[1], sep = "/")


gwaybio · 2019-06-26T18:43:23Z

preselect.R

@@ -98,7 +98,9 @@ for (operation in operations) {
        # This is handled differently because there is no direct way yet to do filtering in cytominer
        # TODO: rewrite this after cytominer has an appropriate filtering function for this
        testthat::expect_false(is.null(replicates), info="replicates should be specified when performing replicate_correlation")
-
+


gwaybio · 2019-06-26T18:43:30Z

select.R

@@ -37,8 +37,7 @@ variables_selected <-
    Reduce(function(df1, df2) dplyr::inner_join(df1, df2, by ="variable"), .) %>%
    magrittr::extract2("variable")

-backend_dir <- paste("../..", "backend", batch_id, plate_id, sep = "/")
-
+backend_dir <- paste("../..", "backend", batch_id, unlist(strsplit(plate_id, "_"))[1], sep = "/")


gwaybio · 2019-06-27T20:27:49Z

@shntnu - I am trying to push changes directly to this branch. b/c it is a branch directly from broadinstitute and not a fork, I cannot push!

Can you grant me push access?

shntnu · 2019-06-27T20:31:20Z

All set

…

On Thu, Jun 27, 2019 at 4:27 PM Greg Way ***@***.***> wrote: @shntnu <https://github.com/shntnu> - I am trying to push changes directly to this branch. b/c it is a branch directly from broadinstitute and not a fork, I cannot push! Can you grant me push access? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30?email_source=notifications&email_token=AAJHQPGMKXPCMQGTKWO2DIDP4UPELA5CNFSM4H3S2AC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYYI2XQ#issuecomment-506498398>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJHQPFBUVGIV52RPTG75GTP4UPELANCNFSM4H3S2ACQ> .

(and to also not break without single cell mode)

shntnu · 2019-12-12T17:21:18Z

aggregate.R

  object <- tbl(src = db, compartment)

  object %<>% inner_join(image, by = c("TableNumber", "ImageNumber"))

+  if (sc_type == "isolated") {
+    object <- object %>% dplyr::filter(Cells_Neighbors_NumberOfNeighbors_Adjacent == 0)


@gwaygenomics This wouldn't work when compartment is Cytoplasm or Nuclei, correct?

I originally wrote this (see below), but then realized I am wrong. This may have been my original plan, but I do not remember 100%

you are correct, this will fail as currently implemented! This is because we're aggregating compartment by compartment. If we merge everything first and then aggregate, it should work. It seems like some version of this must have worked at one point (because the call on L72 would have errored out for other non-cell compartments).

Original thought that is wrong

If I'm remembering correctly, sc_type should be specified only when compartment = "cells" (here).

Since we're doing an inner join on L72, this should take care of filtering for other compartments

Adding Greg's commits

8369881

shntnu requested a review from gwaybio June 26, 2019 14:23

shntnu commented Jun 26, 2019

View reviewed changes

gwaybio suggested changes Jun 26, 2019

View reviewed changes

gwaybio mentioned this pull request Jun 26, 2019

cmQTL uses custom cytominer scripts for single cell type profiling broadinstitute/cmQTL#16

Closed

add back empty lines to match repo style

a3675c9

gwaybio mentioned this pull request Jun 27, 2019

Specify isolated vs. colony definition #31

Open

gwaybio and others added 2 commits June 27, 2019 17:09

modify aggregate to work with single cell option

1340d0e

(and to also not break without single cell mode)

Cast variables to double in population data frame

3b72f6c

shntnu mentioned this pull request Oct 24, 2019

annotate.R fails when plate name has underscores in it #34

Open

shntnu commented Dec 12, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Option for isolated and colony profiles #30

[WIP] Option for isolated and colony profiles #30

shntnu commented Jun 26, 2019 •

edited

Loading

shntnu Jun 26, 2019

gwaybio Jun 26, 2019

shntnu left a comment •

edited by gwaybio

Loading

shntnu Jun 26, 2019 •

edited

Loading

gwaybio Jun 26, 2019

gwaybio Jun 27, 2019 •

edited

Loading

shntnu Jun 26, 2019

gwaybio Jun 26, 2019

shntnu Jun 26, 2019

gwaybio Jun 26, 2019

shntnu left a comment

gwaybio commented Jun 26, 2019

gwaybio left a comment

gwaybio Jun 26, 2019

gwaybio Jun 26, 2019

gwaybio Jun 26, 2019

gwaybio Jun 26, 2019

gwaybio Jun 26, 2019

gwaybio Jun 26, 2019

gwaybio Jun 26, 2019

gwaybio commented Jun 27, 2019

shntnu commented Jun 27, 2019 via email

shntnu Dec 12, 2019 •

edited

Loading

gwaybio Dec 17, 2019 •

edited

Loading

		metadata_dir <- paste("../..", "metadata", batch_id, sep = "/")
		backend_dir <- paste("../..", "backend", batch_id, unlist(strsplit(plate_id, "_"))[1], sep = "/")

[WIP] Option for isolated and colony profiles #30

Are you sure you want to change the base?

[WIP] Option for isolated and colony profiles #30

Conversation

shntnu commented Jun 26, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shntnu left a comment • edited by gwaybio Loading

Choose a reason for hiding this comment

shntnu Jun 26, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gwaybio Jun 27, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shntnu left a comment

Choose a reason for hiding this comment

gwaybio commented Jun 26, 2019

gwaybio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gwaybio commented Jun 27, 2019

shntnu commented Jun 27, 2019 via email

shntnu Dec 12, 2019 • edited Loading

Choose a reason for hiding this comment

gwaybio Dec 17, 2019 • edited Loading

Choose a reason for hiding this comment

Original thought that is wrong

shntnu commented Jun 26, 2019 •

edited

Loading

shntnu left a comment •

edited by gwaybio

Loading

shntnu Jun 26, 2019 •

edited

Loading

gwaybio Jun 27, 2019 •

edited

Loading

shntnu Dec 12, 2019 •

edited

Loading

gwaybio Dec 17, 2019 •

edited

Loading