Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ResourceConflictException: … An update is in progress for resource #46

Closed
aleon68 opened this issue Nov 22, 2021 · 45 comments · Fixed by #62
Closed

ResourceConflictException: … An update is in progress for resource #46

aleon68 opened this issue Nov 22, 2021 · 45 comments · Fixed by #62
Assignees
Labels
bug Something isn't working

Comments

@aleon68
Copy link

aleon68 commented Nov 22, 2021

We have received for all deploys (new and old lambdas) this error, maybe related to this:

https://forums.aws.amazon.com/thread.jspa?threadID=346005

@gurugutten
Copy link

We have received for all deploys (new and old lambdas) this error, maybe related to this:

https://forums.aws.amazon.com/thread.jspa?threadID=346005

We experience the same.

 - name: default deploy
        uses: appleboy/lambda-action@master
        with:
          aws_access_key_id: id
          aws_secret_access_key: key
          aws_region: eu-west-1
          runtime: nodejs14.x
          function_name: #LAMBDA#
          zip_file: lambdas/#LAMBDA#/bundle.zip
          publish: true

2021/11/23 09:00:17 ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:eu-west-1:xxxxxxxxxxxxx:function:lambdafunction

{
  RespMetadata: {
    StatusCode: 409,
    RequestID: "93137e18-d549-4074-b7da-3c4ef46fae6f"
  },
  Message_: "The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:eu-west-1:xxxxxxxxxxxxx:function:lambdafunction",
  Type: "User"
}

marchof added a commit to marchof/io.javaalmanac.sandbox that referenced this issue Nov 23, 2021
@marchof
Copy link

marchof commented Nov 23, 2021

This is obviously due to the change described here: https://aws.amazon.com/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/

If you specify multiple properties (like image_uri and memory_size) the tool probably internally issues multiple subsequent requests which fail due to the new InProgress status.

@aleon68
Copy link
Author

aleon68 commented Nov 23, 2021 via email

@aleon68
Copy link
Author

aleon68 commented Nov 23, 2021 via email

@merzikain
Copy link

I've been talking with AWS tech support and this is what they have to say:

"I dug into the CloudTrail events for the Lambda service and found that each ‘UpdateFunctionCode’ call that failed with an ‘update in progress’ message was accompanied by a successful ‘UpdateFunctionConfiguration’ call at the same timestamp [1]. Both of these API calls can create the ‘inProgress’ Lambda State [2]. Please verify if the automated update system calls both of these APIs in quick succession. In this case, a loop to check for the ‘inProgress’ state would need to be implemented for the ‘UpdateFunctionCode’ call to allow a delay for the ‘UpdateFunctionConfiguration’ action to complete."

Is that maybe what's causing the issue? If properties being set in the YML are causing the issue it seems like this is a good place to look to resolve it.

@mcuv3
Copy link

mcuv3 commented Nov 24, 2021

Have the same issue, watching cloud trail it's trying to create the function besides being already created, and update the code at the same time.

@Ankit05012019
Copy link

I have the same issue , cloud trail shows updatefunctionconfiguration and updatefunctioncode both events triggered at the same time.

@ogohogo
Copy link

ogohogo commented Nov 26, 2021

same thing here, hopefully this repo is still properly managed. @appleboy @szunami @filipebarros @DmytroSH

@kevouellet
Copy link

The only way to work around this issue currently is to remove all the properties that cause an UpdateFunctionConfiguration API call to AWS. https://docs.aws.amazon.com/lambda/latest/dg/API_UpdateFunctionConfiguration.html

In my case, I had to comment out the description field and it went through.

@jaypeedevlin
Copy link

I noticed there is an open PR on the upstream repo that contains the logic: appleboy/drone-lambda#12, I think if that was merged and this repo pointed to the new version it would be fixed.

@merzikain
Copy link

I noticed there is an open PR on the upstream repo that contains the logic: appleboy/drone-lambda#12, I think if that was merged and this repo pointed to the new version it would be fixed.

Can we get some feedback on if this is being handled or if this has been abandoned? Just want to know if I need to just build my own service.

@tiwarisuraj9
Copy link

tiwarisuraj9 commented Dec 8, 2021

I was able fix this from comment claudiajs/claudia#226 (comment)

i.e update description: "aws:states:opt-out"

example:

name: deploy to lambda
on: [push]
jobs:

  deploy_source:
    name: deploy lambda from source
    runs-on: ubuntu-latest
    steps:
      - name: checkout source code
        uses: actions/checkout@v1
      - name: default deploy
        uses: appleboy/lambda-action@master
        with:
          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws_region: ${{ secrets.AWS_REGION }}
          function_name: gorush
          source: example/index.js
          description: "aws:states:opt-out"

@exalted
Copy link

exalted commented Dec 9, 2021

According to this post aws:states:opt-out shouldn't be working as a workaround anymore (as of "October 01 2021 – End of Delayed Update"). 🤷‍♂️

@aleon68
Copy link
Author

aleon68 commented Dec 9, 2021

Right, but is working

@exalted
Copy link

exalted commented Dec 9, 2021

Well, actually no, my bad: There's an update at the top of that page:

CleanShot 2021-12-09 at 15 47 15@2x

So, apparently, this should've broken as of only few days… 🙄 Maybe this explains yesterday's AWS outage?! 😛

@aleon68
Copy link
Author

aleon68 commented Dec 9, 2021

Again you are right, maybe this explain downtime on services

@aleon68
Copy link
Author

aleon68 commented Dec 9, 2021

Maybe the update was done on services, but the problem persists.

I was doing a test right now, performing a deploy with an existent lambda (previously deployed with aws:states:opt-out label), deleting label and redeploy, and failed the deploy. Restoring the label the deployment finish successfully.

I don't understand the AWS logic, but using the label aws:states:opt-out on lambdas can complete deploy.

@aleon68
Copy link
Author

aleon68 commented Feb 7, 2022

Starting today, the problem appears again. Before I was solved adding on description of lambda aws:states:opt-out, but now this is not working.
Someone can help me?

@marcos-pricefy
Copy link

marcos-pricefy commented Feb 9, 2022

I have the same problem even using "aws:states:opt-out" I have got problems.
Does anyone have any ideas?
This was already updated since when aws recommended it, it was working, but yesterday it didn't!

@thierrysuzanne-cl
Copy link

Same problem here, would really appreciate a fix soon. We had same error in powershell scripts when calling Publich-LMVersion straight after a call to Update-LMFunctionCode. We fixed it by adding a waiting loop (not shown) in between those calls, with a timeout (not shown), something like:

$getFunctionResponse = Get-LMFunction -FunctionName $FunctionName
if ($getFunctionResponse.Configuration.LastUpdateStatus -eq 'Successful') {
    # continue normal execution here
}

@sefasenturk95
Copy link

I have the same issue but only on node lambda's.. My go lambda's are deploying fine..

Alex-Yates added a commit to Alex-Yates/WordleEngine that referenced this issue May 16, 2022
asus4 added a commit to asus4/ml-slack-bots that referenced this issue Jun 24, 2022
asus4 added a commit to asus4/ml-slack-bots that referenced this issue Jun 24, 2022
@94929
Copy link

94929 commented Aug 29, 2022

Anyone solved this issue? Adding description: "aws:states:opt-out" doesn't work for me unfortuantely.

@exalted
Copy link

exalted commented Aug 29, 2022

FYI; Since then we've migrated to AWS CLI (i.e. aws lambda update-function-code ...)

@erikcc02
Copy link

I'm facing this problem in deploy step:

2023/02/10 14:40:33 ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:***:****:function:account-transfers-lambda
{
  RespMetadata: {
    StatusCode: 409,
    RequestID: "11040c97-d69c-43d3-a19a-b"
  },
  Message_: "The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:***:***",
  Type: "User"
}

This is my config:

 - name: default deploy
        uses: appleboy/lambda-action@master
        with:
          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws_region: ${{ secrets.AWS_REGION }}
          function_name: account-transfers-lambda
          zip_file: package/function.zip
          handler: lambda_function.lambda_handler
          runtime: python3.8

@samdelong
Copy link

I wouldn't recommend using this package as it doesn't seem to be maintained anymore. I was able to deploy my go function with the aws-actions/setup-sam@v1 action.

@appleboy
Copy link
Owner

I will take it.

@aleon68
Copy link
Author

aleon68 commented Mar 31, 2023

I wouldn't recommend using this package as it doesn't seem to be maintained anymore. I was able to deploy my go function with the aws-actions/setup-sam@v1 action.

How can I apply this action? I could use same yml or need other configuration?
Sorry by ask about that, I'm not expert on deploy with github actions

@appleboy
Copy link
Owner

appleboy commented Apr 1, 2023

@jjlabajo
Copy link

jjlabajo commented Apr 1, 2023

Hello @appleboy , it's still not working on my end.

@appleboy
Copy link
Owner

appleboy commented Apr 1, 2023

@jjlabajo Please provide detailed information.

@johnwmccallum
Copy link

I'm still getting this error:

2023/04/02 16:48:15 ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:***:741546933029:function:option-pairing { RespMetadata: { StatusCode: 409, RequestID: "d63709c9-7e47-4733-a817-a36d91b95737" }, Message_: "The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:***:741546933029:function:option-pairing", Type: "User" }

@appleboy
Copy link
Owner

appleboy commented Apr 3, 2023

@jjlabajo @johnwmccallum Please help to try the following commit sha:

appleboy/lambda-action@390dab2546e6c97ca3b94fc5f3863d0e15bec0ee

You will see the following log:

image

2023/04/03 04:16:14 Update function configuration ...
2023/04/03 04:16:15 Current State: Active
2023/04/03 04:16:15 Last Update Status: InProgress
2023/04/03 04:16:15 Last Update Status Reason: The function is being created.
2023/04/03 04:16:15 Last Update Status ReasonCode: Creating
2023/04/03 04:16:15 Waiting Last Update Status to be successful ...
2023/04/03 04:16:40 Update function code ...
2023/04/03 04:16:40 Current State: Active
2023/04/03 04:16:40 Last Update Status: InProgress
2023/04/03 04:16:[40](https://github.com/appleboy/lambda-action/actions/runs/4592683642/jobs/8110202047#step:14:42) Last Update Status Reason: The function is being created.
2023/04/03 04:16:40 Last Update Status ReasonCode: Creating
2023/04/03 04:16:40 Waiting Last Update Status to be successful ...

@appleboy
Copy link
Owner

appleboy commented Apr 3, 2023

Hi All,

Please help to try the following config:

      - name: check max attempts flag
        uses: appleboy/lambda-action@390dab2546e6c97ca3b94fc5f3863d0e15bec0ee
        with:
          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws_region: ${{ secrets.AWS_REGION }}
          function_name: gorush
          zip_file: example/deployment.zip
          debug: true
          publish: false

How to resolve the problem

  1. Check the Lambda function states before updating the config
  2. Check the Lambda function states again before uploading the zip file or source code.

See the reference: https://docs.aws.amazon.com/lambda/latest/dg/functions-states.html

@appleboy
Copy link
Owner

appleboy commented Apr 3, 2023

@erikcc02 @94929 @samdelong @exalted @sefasenturk95 @thierrysuzanne-cl @marcos-pricefy @tiwarisuraj9 @merzikain @jaypeedevlin @kevouellet @ogohogo @Ankit05012019 @gurugutten

Please help to try the test version appleboy/lambda-action@390dab2546e6c97ca3b94fc5f3863d0e15bec0eeand @aleon68 has confirmed that 390dab2 fixed the problem in #58 (comment)

I will bump the new version later.

Thanks.

@exalted
Copy link

exalted commented Apr 3, 2023

Hey there, I stopped using this action a while ago. Sorry. 🙁

@appleboy
Copy link
Owner

appleboy commented Apr 3, 2023

@exalted OK. Thanks. Please accept my apologies for the delay in handling the situation.

@exalted
Copy link

exalted commented Apr 3, 2023

No worries. I appreciate your humbleness!

@appleboy appleboy added the bug Something isn't working label Apr 3, 2023
@appleboy appleboy self-assigned this Apr 3, 2023
@nex84
Copy link

nex84 commented Apr 3, 2023

Hello
Thank you, it seems to have solved the issue for me :

      - name: default deploy
        uses: appleboy/lambda-action@390dab2546e6c97ca3b94fc5f3863d0e15bec0ee # old : appleboy/lambda-action@master
        with:
          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws_region: eu-west-1
          function_name: api-demo
          source: lambda-api-server.py

@appleboy
Copy link
Owner

appleboy commented Apr 3, 2023

@nex84 Thanks for your help in testing. I will bump the new version later.

@appleboy
Copy link
Owner

appleboy commented Apr 3, 2023

Hi All, please use the v0.1.9 version.

      - name: default deploy
        uses: appleboy/[email protected]
        with:
          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws_region: eu-west-1
          function_name: api-demo
          source: lambda-api-server.py

@jjlabajo
Copy link

Hello @appleboy , thanks for the update. It's working now, I am using the master.

@appleboy
Copy link
Owner

@jjlabajo I recommend not using the master branch version in your production. The best way is to tag the specific version like appleboy/[email protected]

@sefasenturk95
Copy link

@appleboy works like a charm, thank you!

@markmnl
Copy link

markmnl commented May 16, 2023

Came here to say resolved this issue for me update checking permissions (was missing one or two)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "iam:ListRoles",
        "lambda:UpdateFunctionCode",
        "lambda:CreateFunction",
        "lambda:GetFunction",
        "lambda:UpdateFunctionConfiguration",
        "lambda:GetFunctionConfiguration"
      ],
      "Resource": "arn:aws:lambda:${REGION}:${ACCOUNT}:function:${LAMBDA_NAME}"
    }
  ]
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.