You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since the project's inception, we have been maintaining all failed jobs in our log database for auditing and potential reprocessing purposes. Rather than being stored in our active RDS logging instance we should be periodically writing these failed jobs to archive storage and removing them from the live instance. Initially, we should use a long running process which,
Queries failed jobs for a date and exports those rows as ndjson and stores them in an S3 bucket with the key structure year/month/date.json.
Deletes all of those corresponding rows from the table.
Initially this should run in a loop for all dates for a specified range (like Jan-Jul 2022).
After we've done the legacy process a daily cron job should run to execute the same process for any dates > than 2 months ago.
This will provide us an auditable archive of failed granules that we can query and reprocess if necessary while keeping our active production logging RDS instance smaller and easier to manage.
The text was updated successfully, but these errors were encountered:
Since the project's inception, we have been maintaining all failed jobs in our log database for auditing and potential reprocessing purposes. Rather than being stored in our active RDS logging instance we should be periodically writing these failed jobs to archive storage and removing them from the live instance. Initially, we should use a long running process which,
year/month/date.json
.After we've done the legacy process a daily cron job should run to execute the same process for any dates > than 2 months ago.
This will provide us an auditable archive of failed granules that we can query and reprocess if necessary while keeping our active production logging RDS instance smaller and easier to manage.
The text was updated successfully, but these errors were encountered: