feat: upstream sync

[adityathebe]

Resolves: #286

Merge after:

• refactor: context #299
• feat: implement event queue and consumer postq#1
• feat: add push_queue event handler that pushes to usptream duty#294

[github-actions]

E2E -

0 tests   0 ✔️  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit 630d85d.

♻️ This comment has been updated with latest results.

[adityathebe]

Pulling scrape configs

a. Pull everything that was updated since X timestamp

• X could be empty on startup so agent fetches all scrape configs
• or it could be a delta of 5 minutes.

Push config

A. Primary means of pushing to upstream

a. use event_queue to push real time changes to upstream, OR
b. On a schedule X, fetch all the results that were updated in the last (X + delta) interval. (Although config_analysis doesn't have updated_at table so we might need to add it.)

B. Reconcile

Every 8 hours, push all results that were updated in the last 48 hours.

@moshloop wdyt?

[moshloop]

Pulling scrape configs

a. Pull everything that was updated since X timestamp

• X could be empty on startup so agent fetches all scrape configs
• or it could be a delta of 5 minutes.

Push config

A. Primary means of pushing to upstream

a. use event_queue to push real time changes to upstream, OR b. On a schedule X, fetch all the results that were updated in the last (X + delta) interval. (Although config_analysis doesn't have updated_at table so we might need to add it.)

B. Reconcile

Every 8 hours, push all results that were updated in the last 48 hours.

@moshloop wdyt?

Pulling scrape configs every 5 minutes or so is fine

For push I think the event_queue mechanism as the primary is fine + normal reconciliation.

We will probably need to included updated_at in the hash, and then in get upstream details return both ids and updated_at on hash mismatch, and only push the difference

[github-actions]

🎉 This PR is included in version 0.0.140 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀