Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Batch processing: * updates just the declared license in the DB documents using `collection.bulk_write()` * updates denitions using service API `POST /definitions?force=true` _NOTE: Updating the DB makes the fix of the declared license immediately available. When the `POST /definitions` request completes, the full DB document will be updated to be in sync with the blob definition._ Additional changes: * moves global variable definitions based on .env to the initialize() function * adds DRYRUN flag to check what would run and how many records would be evaluated * add estimated time to complete * adds script and function level documentation * includes timestamps to make it easier to estimate how long it will take to complete a run * generate filename based on date range and offset to avoid overwriting output files _NOTE: Azure only supports fetching one blob at a time. Not able to optimize that part of the code. _ _NOTE: Batch size of 500 was selected because that is the max number of coordinates supported in calls to service API `POST /definitions`._
- Loading branch information