long term solution for evaluations

[juniperlsimonis]

the evaluations file is big and will keep growing, potentially hitting github size limits

i think we can avoid this by taking a slightly different tact. whether we have an evaluations file for each forecast or add content to existing forecast files, there are ways to distribute the content among files to minimize individual task overhead

[ethanwhite]

Is it in csv at the moment? We could move to feather, which is the arrow's format. In some other explorations I've been doing the reduction in size is dramatic.

[juniperlsimonis]

yes it is a csv. oooh, that's a good idea. i havent poked around with the feather files yet

[juniperlsimonis]

just looked at what that would do here, and we'd definitely get a lot more space back -- current evaluations file size as csv is 60.5 MB, as a feather file is 13.1 MB
obviously will still grow and eventually hit file size limits, but definitely would give us a longer runway