Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gyro executes workflow step incorrectly #297

Closed
tloisel1 opened this issue May 21, 2020 · 8 comments · Fixed by #303
Closed

Gyro executes workflow step incorrectly #297

tloisel1 opened this issue May 21, 2020 · 8 comments · Fixed by #303
Assignees
Labels
bug Something isn't working priority:high

Comments

@tloisel1
Copy link
Member

tloisel1 commented May 21, 2020

We have a multi-step deployment workflow configured in Gyro. The result of which is the replacement of an AWS Launch Configuration and AWS autoscale group through a blue-green deployment. One a specific step, our configuration states to update the old autoscale group. Gyro interprets this step as a replace instead of an update, but skips it.
The workflow stages are as follows (existing resources are defined outside workflow, and new resources are created within workflow):

  1. verify
  • Create new ELB for new autoscale group
  • Create new Route53 record that points to new ELB
  • Create new Launch Configuration
  • Create new autoscale group and associate with new ELB
  • Wait for all instances in new ELB to pass ELB health checks
  1. push
  • Update new autoscale group to be associated with existing ELB's
  • Wait for all instances in existing ELB to pass ELB health checks
  1. push-finish
  • Update existing autoscale group to be associated with no ELB's
  1. finished
  • Replace existing Launch Configuration with new Launch Configuration
  • Replace existing Autoscale Group with new Autoscale Group
  • Delete new Route53 Entry
  • Delete new ELB

The problem occurs at the push-finish step where our configs state that the existing autoscale group should be associated with no ELBs.

        @workflow::update $(aws::autoscaling-group frontend)
            classic-load-balancers: [ ]
        @end

Instead, Gyro tries to execute a replace

    ⇅ Replacing aws::autoscaling-group frontend (ops-test-gyro frontend prod v1 ami-016734320d897e0ea builds/deploy/master 2 fecf06351572fbc59fc869e15f43700d)
    · classic-load-balancers:  - aws::load-balancer cms, - aws::load-balancer frontend
    · name:  'ops-test-gyro frontend prod v1 ami-016734320d897e0ea builds/deploy/master 2 fecf06351572fbc59fc869e15f43700d' → 'ops-test-gyro frontend prod v1 ami-016734320d897e0ea builds/deploy/master 1 7090a1cdf6a0dfc32b447185bff20dbe'
    · tags:  ⟳ { Name: 'ops-test-gyro frontend prod v1 ami-016734320d897e0ea builds/deploy/master 2 fecf06351572fbc59fc869e15f43700d' → 'ops-test-gyro frontend prod v1 ami-016734320d897e0ea builds/deploy/master 1 7090a1cdf6a0dfc32b447185bff20dbe' }
    SKIPPED

This leaves both the existing and new autoscale groups associated with the existing ELB. We expect only the new autoscale group to be associated with the existing ELB.

Gyro should execute the update command instead of the replace at this point.

@deepanjan90 deepanjan90 added priority:high bug Something isn't working labels May 21, 2020
@sshyub
Copy link

sshyub commented May 21, 2020

@tloisel1 can you try it with --verbose option to see why Gyro tried to replace the resource instead of updating it?

@tloisel1
Copy link
Member Author

Updated with an example with --verbose

@sshyub
Copy link

sshyub commented May 22, 2020

The workflow was triggered by the name field change in aws::launch-configuration frontend, where the name field of aws::autoscaling-group frontend relied on.
So aws::autoscaling-group frontend couldn't be updated because the name had been changed.

@sshyub
Copy link

sshyub commented Jun 5, 2020

@tloisel1 Please try feature/update-scope branch with the following workflow configuration changes.

  • Change the configuration to define workflow for autoscaling-group Instead of launch-configuration.
  • Replace $(PENDING) with $(aws::launch-configuration frontend)
  • Fix every place where $(aws::autoscaling-group frontend) needs to be updated to add name: $(CURRENT).name. for instance:
        @workflow::update $(aws::autoscaling-group frontend)
            name: $(CURRENT).name
            classic-load-balancers: [ ]
        @end

@sshyub sshyub assigned deepanjan90 and unassigned sshyub Jun 11, 2020
@sshyub
Copy link

sshyub commented Jun 11, 2020

@deepanjan90 Please ignore #300 for now.

@sshyub
Copy link

sshyub commented Jun 11, 2020

@tloisel1 let me know if you see any issues.

@tloisel1
Copy link
Member Author

Looks like I'm running into an issue on my push step:

Error: Can't replace aws::launch-configuration frontend resource!

Caused by: Error: Can't process the @wait directive!

In prod/frontend.gyro on line 464 from column 13 to 159:
464:             @wait: $(aws::load-balancer frontend).instance-health.InService = $(aws::load-balancer frontend).instance-health.Total -check-every 10 -at-most 600
465:             @wait: $(aws::load-balancer cms).instance-health.InService = $(aws::load-balancer cms).instance-health.Total -check-every 10 -at-most 600

Caused by: Error: Can't use the @wait directive inside gyro.core.scope.Scope!

@sshyub sshyub assigned sshyub and unassigned deepanjan90 Jun 12, 2020
@tloisel1
Copy link
Member Author

Fix in #303 worked with some adjustments to my workflow configuration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working priority:high
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants