Should backend creation use first image set as a "template"?

[shntnu]

Moved from broadinstitute/cytominer_scripts#12

Trying to make a backend with process.sh, I had issues because my first image had failed QC and my pipeline was set to skip the rest if it did so, so there was only a small subset of the usual amount of data present in that folder- only the Image.csv (with a smaller number of columns), no object CSVs. It therefore for all images only added the common columns, which led to reasonable errors when it went to aggregate the object tables at the end and could not find any.

I hacked around it by deleting all the folders before the first well that actually had been plated with cells, but it seems to me that

1. in creation of the backend we want to pull in all the data that's there, we can deal with NaNs from missing data later
2. there's a reasonable chance of this happening again, as well A01 (and other edge/corner wells) are often not plated into in smaller experiments but in a large fraction of those cases the whole plate is just imaged anyway.

Feel free to disagree though, that's why I phrased it as a question.

End of the error string below, I doubt it's helpful but just in case

(builtins.OSError) /home/ubuntu/efs/{redacted}/workspace/software/cytominer_scripts/.4b21aa7e-6e45-11e7-8ea1-0e60212e428a:1: expected 218 columns but found 568 - extras ignored
 [SQL: 'sqlite3 -nullvalue \'\' -separator , -cmd .import "/home/ubuntu/efs/{redacted}/workspace/software/cytominer_scripts/.4b21aa7e-6e45-11e7-8ea1-0e60212e428a" "Image" /home/ubuntu/ebs_tmp/2017_07_12_Batch1/AU00027623//AU00027623.sqlite']
[Fri Jul 21 16:41:41 UTC 2017] Looking up AU00027623.sqlite on permanent store
[Fri Jul 21 16:41:41 UTC 2017] /home/ubuntu/bucket/projects/{redacted}/workspace/backend/2017_07_12_Batch1/AU00027623/AU00027623.sqlite not found
[Fri Jul 21 16:41:41 UTC 2017] Creating /home/ubuntu/ebs_tmp/2017_07_12_Batch1/AU00027623//AU00027623.sqlite

real    127m40.736s
user    62m50.242s
sys     4m23.562s
[Fri Jul 21 18:49:21 UTC 2017] Indexing /home/ubuntu/ebs_tmp/2017_07_12_Batch1/AU00027623//AU00027623.sqlite
Error: near line 3: no such table: main.Cells
Error: near line 4: no such table: main.Cytoplasm
Error: near line 5: no such table: main.Nuclei

real    0m0.054s
user    0m0.009s
sys     0m0.023s
[Fri Jul 21 18:49:22 UTC 2017] Aggregating /home/ubuntu/ebs_tmp/2017_07_12_Batch1/AU00027623//AU00027623.sqlite
Error in rsqlite_send_query(conn@ptr, statement) : no such table: cells
Calls: %>% ... initialize -> initialize -> rsqlite_send_query -> .Call
Execution halted

real    0m0.993s
user    0m0.603s
sys     0m0.056s
[Fri Jul 21 18:49:23 UTC 2017] /home/ubuntu/ebs_tmp/2017_07_12_Batch1/AU00027623//AU00027623.csv not created / does not exist. Exiting.

[mcquin]

If we wanted to handle CSVs with different/varying columns, we could append columns to the table when a new column in a CSV is encountered, and insert NULL values to the missing entries.

Alternatively, and this might worth investigating, perhaps providing a headers.csv which defines all the columns in a table could be another workaround. I'm not sure what the behavior is when importing a CSV with missing columns, though. Does it cause an error?

[shntnu]

This will be addressed in #100