Skip to content

Commit

Permalink
Queue up the tasks in limit-active-tasks.
Browse files Browse the repository at this point in the history
Provide a DB based queueing system for tasks when
using the limit-active-tasks strategy.

The ordering is insertion-time based: first arrived
first served.

Tasks are assigned different queues when specifiying:
platform, team and tags.

Update package for Squirrel to include bugfix
for `FromSelect` not working properly.

FYI:
I was inspired by this talk: https://www.pgcon.org/2016/schedule/attachments/414_queues-pgcon-2016.pdf
for the design of the Postgres-based queue.

Signed-off-by: Alessandro Degano <[email protected]>
  • Loading branch information
Alessandro Degano committed Nov 29, 2019
1 parent a3baf16 commit 95f8447
Show file tree
Hide file tree
Showing 16 changed files with 1,156 additions and 33 deletions.
6 changes: 4 additions & 2 deletions atc/atccmd/command.go
Original file line number Diff line number Diff line change
Expand Up @@ -596,7 +596,8 @@ func (cmd *RunCommand) constructAPIMembers(
)

pool := worker.NewPool(workerProvider)
workerClient := worker.NewClient(pool, workerProvider)
taskQueue := db.NewTaskQueue(dbConn)
workerClient := worker.NewClient(pool, workerProvider, taskQueue)

credsManagers := cmd.CredentialManagers
dbPipelineFactory := db.NewPipelineFactory(dbConn, lockFactory)
Expand Down Expand Up @@ -763,7 +764,8 @@ func (cmd *RunCommand) constructBackendMembers(
)

pool := worker.NewPool(workerProvider)
workerClient := worker.NewClient(pool, workerProvider)
taskQueue := db.NewTaskQueue(dbConn)
workerClient := worker.NewClient(pool, workerProvider, taskQueue)

defaultLimits, err := cmd.parseDefaultLimits()
if err != nil {
Expand Down
Loading

0 comments on commit 95f8447

Please sign in to comment.