ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda: #226
Comments
|
In a further attempt to get this working again, I added No joy. Fails just as fast. Monitoring with > watch -n1 aws --profile YYYYYYY --region eu-west-1 lambda get-function-configuration --function-name XXXXXXXXthis time showed no other
So something (the configuration?) did change, but the code did not. |
|
Created branches, which only deploy in CI, for the 2 last successful deploys, which worked a gazillion times before. Both experiments get the same problem now. Since |
|
We just ran into the same issue and found the reason here: https://aws.amazon.com/de/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/ |
|
FYI: We upgraded the version of the aws provider in one of our terraform sets to the latest version and that seems to have cleared the problem. |
|
@choeller thx for the response That is consistent with our observations.
|
@spencer-aerian that cleared the problem in Or are you suggesting updating the version of the AWS SDK used inside |
|
Dear Claudiajs, Thx for creating and maintaining this project. We've been using it for deployment for a number of years now (and not other features). This issue is blocking for us. There has been little activity here over the last few months. Can you give us an indication about your intentions with this project? I would like to be able to determine whether it is worth waiting for further progress, or whether it is more appropriate to look for another long term solution. |
No different package but same error. |
|
I just checked out the project, and ran the tests (this takes ages). There are some errors. Notably, there are frequent mentions of so I guess these tests are leaving behind some flutsam in our AWS account now. Also, there are failures with But the heart of this issue is
I am running the tests with both the AWS-SDK defined in the project, and the most recent version, and they fail with the latest AWS-SDK. In other words, updating the AWS SDK is not the answer in this case (it was fairly recent to start with). |
|
@jandppw This pr fixes it for me: zappa/Zappa#992 |
|
Getting this problem here, unable to deploy an update of my lambda in production |
|
v5.14.0 should fix this |
|
@gojko - Still getting this on v5.14.0 |
|
@gojko - Also, still getting this on v5.14.0 |
|
Maybe irrelevant and not applicable, but I solved this by adding the env var to my Cloudformation template instead of during deploy. |
|
Unrelated: I don't use this library, but looks like the code now has to For example, first the function is created and then its configuration is updated — has to wait in-between. log('Updating lambda function');
// Update lambda code
await lambdaApi.updateFunctionCode({
FunctionName: getFunctionName(lambdaConfig, stage),
ZipFile: zipFile
}).promise();
log('Updated code');
// Has to wait for the function update status to transition to "Updated"
// until making further re-configuration.
// https://github.com/claudiajs/claudia/issues/226
// https://aws.amazon.com/de/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/
// https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html
await lambdaApi.waitFor('functionUpdated', {
FunctionName: getFunctionName(lambdaConfig, stage)
}).promise();
// Update lambda configuration
await lambdaApi.updateFunctionConfiguration(lambdaConfiguration).promise();
log('Updated configuration'); |
|
Also just started happening to me today, both 5.14.0 and 5.13.0 |
|
Started happening to me today as well. updating to the latest ClaudiaJs and AWS-SDK didn't help. |
|
Started for us as well. if someone wants to do quick fix until its fixed in ClaudiaJs, use below around Claudia commands: |
|
Upgrade zappa to latest version. Worked for me. |
|
I started making skills for Alexa for about a month. I was doing that using ASK CLI. This morning this error popped out: [Error]: CliError: The lambda deploy failed for Alexa region "default": ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:454044463163:function:ask-reading-buses-default-default-1636455775666 Now I can't deploy any skill. |
|
I all of a sudden had the same problem using the same program I have always used to deploy. I was updating the lambda code followed by a publish version. I would get the error "An update is in progress for resource". I fixed it by calling the getFunctionConfiguration and waiting til State was Active and LastUpdateStatus was Successful. I then continued on with the publish. The State was Active right away, but now it took the LastUpdateStatus to be Successful about 30 seconds or so. |
|
I was in trouble with the same error on Node 12. |
|
It looks like this is only still an issue if updateConfiguration is called with any options. For me, removing the @gojko Something similar to the following should be enough to fix this (just copied the existing wait logic up into the updateConfiguration function) --- a/src/commands/update.js
+++ b/src/commands/update.js
@@ -145,7 +145,11 @@ module.exports = function update(options, optionalLogger) {
},
() => logger.logStage('waiting for IAM role propagation'),
Promise
- );
+ ).then(result => {
+ logger.logStage('waiting for lambda resource allocation');
+ return waitUntilNotPending(lambda, lambdaConfig.name, awsDelay, awsRetries)
+ .then(() => result);
+ });
}
},
cleanup = function () { |
|
This is still broken when using |
|
This is still broken for me as well. I temporarily bypassed the situation by adding the opt out clause (aws:states:opt-out) to the Lambda description field, but this trick will only work until December 6th. |
|
@magoyo Same here, However, i using code-build which is similar. So, the solution is add |
temporary plan |
|
@ambigus9 I didn't see a |
|
I was able to fix this locally as follows:
And making the waitUntilNotPending So now it looks like this: I am not sure if this is the correct approach, but it works without the need to update the description for now. |
|
@magoyo Ok, but i using codebuild on AWS, so for the solution was add |
|
@maltahs Your suggestion worked for us too, thanks! To make things easier for others, I also submitted our hotfix branch as a PR to this project (as you can see above). |
|
@gojko, given the feedback above, shouldn't this issue be re-opened? |
|
Are there any updates on a new release that would fix this issue (perhaps accepting PR #230?)? |
|
+1 for accepting #230 |
|
I'm trying to upload my Node function to Lambda with the vanilla aws-sdk package, and I'm getting the same error for my resource. It must be something to do with my Lambda configuration itself, not Claudia, but I have no idea how to fix it. const aws = require('aws-sdk');
const lambda = new aws.Lambda({
region: 'us-east-1'
});
var params = {
FunctionName: functionName,
ZipFile: zipBuffer
};
console.log('Uploading function code...');
lambda.updateFunctionCode(params, (err, data) => {
if (err) cb(err);
else {
console.log(`Uploaded function code:\n\t FunctionName=${data.FunctionName}\n\t Role=${data.Role}\n\t CodeSha256=${data.CodeSha256}`);
console.log('Publishing new version...');
var params = {
CodeSha256: data.CodeSha256,
Description: `${time}`,
FunctionName: functionName,
};
lambda.publishVersion(params, (err, data) => {
if (err) cb(err);
else {
// continueOutput: Any idea what to do to fix it? EDITMy bad! I didn't fully read through the comments. After adding @maltahs' waitUntilNotPending function to my own script, it works. I'm gonna keep my comment here just in case someone else is confused in a similar vein to how I was. |
|
This needs to be fixed otherwise we need to think about not using claudiajs anymore |
|
I had to find another solution unfortunately now that its been months waiting on the pr |
@cathalelliott1 It's actually not too difficult to manage your Lambda functions manually with the aws-sdk package. If you need a reference, feel free to take a look at the scripts I made for my own project: https://github.com/FIRSTTeam102/ScoringApp-Serverless/tree/master/scripts - the scripts are called with NPM scripts as defined in https://github.com/FIRSTTeam102/ScoringApp-Serverless/blob/master/primary/package.json. Code is licensed with GPLv3 so feel free to use it if it helps. It's highly specific to the one project, but you can use the same concepts to fit your own use. The scripts don't have many comments, but if you are looking into it and want an explainer on how they work, open an issue on our repo and I can answer any questions. |
|
We are facing this issue as well and the code changes are small, so we were able to implement them locally (although it's a pain when you switch workstations and have to update again.) So question - looking at the version history, it seems like after 2019 there have only been three releases (early 2020, early 2021, and October 2021). And there has been no activity here by the maintainers since the Oct 18 changes. Is ClaudiaJS no longer maintained, or is it just sporadically maintained now? |
|
If ClaudiaJS is no longer maintained, then it would be good to know what packages are the easiest to transition to. Anyone has any good suggestions? |
|
@magoyo Serverless Framework is the only option of which I'm aware. I'm sure there are others, I'm just not up to speed. |
|
Thank you @dmackinn, I hadn't used that version yet. It looks like he just patched it 21 days ago (after our commentary) in March 2022, not March 2021 like the date in the README.md says (and indeed there is a follow up commit where the year was changed from 2021 to 2022.) I'll give it a try. |
|
I believe this was caused by AWS sending back |
and others
In our project, Lambda was last deployed successfully by CI with
claudia2021-09-14 ~16:17 CET. There have been no issues earlier.A next attempt by CI, at 2021-09-15 ~1649 CET failed. Retry attempts failed. Manual attempts via CLI
failed. Manual upload, publish, and creation of an alias, worked via the console (but no working version was produced
because there was no investment in getting the package right).
Nothing of relevance was changed (always a strong statement, I know). There was no update of
claudia, or relatedpackages between 2 deploys. A retry of the successful deployed version failed too.
Retries 2021-09-16 ~ 10:10 CET failed again.
The reported error is always the same:
But this happens quick, before the package is build, or after.
We've felt for a while that
claudiadoes some things twice, first checking, and then doing. When the error appearslate, we see several mentions of
lambda.setupRequestListenersResources on the internet are barely any help.
AWS Lambda - Troubleshoot invocation issues in Lambda
mentions
ResourceConflictException, but with a different message, and refers to VPCs, which we are not using.UpdateFunctionConfiguration,PublishVersion,
UpdateFunctionCodeand others mention moregenerally:
Other resources are no help:
Terraform Error publishing version when lambda using container updates code #17153
(Jan. 2021) mentions a "lock" / "last update status", which we can watch during execution using
> watch aws --profile YYYYYYY --region eu-west-1 lambda get-function-configuration --function-name XXXXXXXXThe output looks like
{ "FunctionName": "XXXXXXXX", "FunctionArn": "arn:aws:lambda:eu-west-1:NNNNNNNNNNN:function:XXXXXXXX "Runtime": "nodejs14.x", "Role": "arn:aws:iam::NNNNNNNNNNN:role/execution/lambda-execution-XXXXXXXX", "Handler": "lib/service.handler", "CodeSize": 76984324, "Description": "[…]", "Timeout": 30, "MemorySize": 2048, "LastModified": "2021-09-16T08:19:05.000+0000", "CodeSha256": "zQb6Vss0Zlug46HRjA8+bNe0i1TP6NWfrm70hC6zC90=", "Version": "$LATEST", "Environment": { "Variables": { "NODE_ENV": "production" } }, "TracingConfig": { "Mode": "PassThrough" }, "RevisionId": "9d5f5431-6f2f-4d39-9794-d86778b34446", "Layers": [ { "Arn": "arn:aws:lambda:eu-west-1:NNNNNNNNNNN:layer:chrome-aws-lambda:25", "CodeSize": 51779390 } ], "State": "Active", "LastUpdateStatus": "Successful", "PackageType": "Zip" }most of the time, but we see
LastUpdateStatuschange for a moment before the error occurs.Terraform aws_lambda_function ResourceConflictException due to a concurrent update operation #5154
says, in 2018,
serverless 'Concurrent update operation' error for multi-function service results in both deployment and rollback failure. #4964
reports the same issue in 2018, and remarks:
So, this appears to be a timing issue. Claudia should take it slower?
The text was updated successfully, but these errors were encountered: