v4 frequency script

[mike-w-wilson]

Here is the full frequency script. I've run it on the test dataset using the --use-test-dataset arg. The script currently puts out all of the annotations from the split VDSs and any annotation used in correcting for the high AB hets. I plan on removing these once this is ready and have a note before the final write on which fields I plan on removing for now. I also realized this does not account for the downsamplings within the ukb subset which Konrad had asked for so I will work on adding that but figured this is huge so should not hold up review with it. As I mentioned in one of our chats, I copied Tim's freq updates and then updated them here to account for the upstream downsampling annotation. Post v4, I will rewrite the gnomad_methods annotate_freq method. This depends on the code added in broadinstitute/gnomad_methods#565 so will fail checks until that is in.

[mike-w-wilson]

I'm realizing the non-ukb subset needs to have its own grpmax and age hists and this code doesnt currently account for that. Since we are already splitting by ukb, this should be easy enough to add in but I wont until we get through one round of review.

[jkgoodrich]

just a small change, and pinging KC to look at a few lines

[jkgoodrich]

I didn't follow all the discussions on this so pinging @ch-kr to check this (L778-L782).

[mike-w-wilson]

Sorry, I committed a suggestion on naming and it looks like it removed @jkgoodrich 's other comment. @ch-kr do you have thoughts on the grpmax_meta and index_dict? L778-782

[ch-kr]

hmmm. what about removing these globals and restructuring grpmax to have the same structure as the age hists? something like

grpmax: struct{
    gnomad: struct{
        AC: int32,
        AF: float64,
        AN: int32,
        homozygote_count: int64,
        pop: str,
        faf95: float64,
    },
    non-ukb: struct{
            AC: int32,
            AF: float64,
            AN: int32,
            homozygote_count: int64,
            pop: str,
            faf95: float64,
        }
} 

[mike-w-wilson]

I'm fine with that but just confirming we're dropping age hists so it actually wont look like the above?

[ch-kr]

oh yes sorry that wasn't clear. we're dropping the non-UKB age hists, so that annotation looks like this now, right?

age_hists: struct{
            age_hist_hom: struct {
                bin_edges: array<float64>, 
                bin_freq: array<int64>, 
                n_smaller: int64, 
                n_larger: int64
            }, 
            age_hist_het: struct {
                bin_edges: array<float64>, 
                bin_freq: array<int64>, 
                n_smaller: int64, 
                n_larger: int64
            },
        }

[jkgoodrich]

Approved pending KC's review

[mike-w-wilson]

KC approved the structure and schema output via slack, will merge once we resolve the pylint check:#416