Saving Deployment State is extremely slow due to unoptimized glob

[ErlendHer]

How did you install the Amplify CLI?

No response

If applicable, what version of Node.js are you using?

No response

Amplify CLI Version

12.5.2

What operating system are you using?

MacOS Sonoma

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

Irrelevant

Describe the bug

During amplify push, the saving deployment state step of the push is taking more than 7 minutes to complete.

After debugging and forking the CLI I've identified the issue and also have an (unfinished) fix that resolves this issue.

The problem is that when uploading the "studio-backend" files there is an unoptimized glob which recursively scans all the files/folders in the "amplify" folder.

If you have a project with a significant amount of node_modules or other dependencies this adds a massive amount of overhead to the glob.

Adding node_module to the ignore pattern of the glob in our case reduced the time from 6 minutes and 46 seconds to 1.2 seconds which one could say is fairly significant.

This 6.5 minutes saved per deploy (and would be worse in the future once we've added more functions and thus more node_modules)

The below code fixes the problem. packages/amplify-provider-cloudformation/src/utils/upload-current-cloud-backend.ts

/**
 * Publish files that Amplify Studio depends on outside the zip file so that can read
 * without streaming from the zip.
 */
const uploadStudioBackendFiles = async (s3: S3, bucketName: string) => {
  const amplifyDirPath = pathManager.getAmplifyDirPath();
  const studioBackendDirName = 'studio-backend';
  // Delete the contents of the studio backend directory first
  await s3.deleteDirectory(bucketName, studioBackendDirName);
  // Create a list of file params to upload to the deployment bucket
  const uploadFileParams = [
    'cli.json',
    'amplify-meta.json',
    'backend-config.json',
    'schema.graphql',
    'transform.conf.json',
    'parameters.json',
  ]
    .flatMap((baseName) => glob.sync(`**/${baseName}`, { cwd: amplifyDirPath, ignore: ['**/node_modules/**'] })) // added the ignore here
    .filter((filePath) => !filePath.startsWith('backend'))
    .map((filePath) => ({
      Body: fs.createReadStream(path.join(amplifyDirPath, filePath)),
      Key: path.join(studioBackendDirName, filePath.replace('#current-cloud-backend', '')),
    }));

  await Promise.all(uploadFileParams.map((params) => s3.uploadFile(params, false)));
};

Expected behavior

I expected saving deployment state not to take more than 7 minutes.

Below are some benchmarks for how long the different actions took in the saving deployment state step:

Task	Time
Zipping the #current-cloud-backend	1.886s
Uploading the .zip file	7.487s
Deleting the old s3 directory for studio-backend	898ms
Performing the glob on the amplify directory	6 minutes 41 seconds
Uploading the studio-backend files to s3	558ms

Question of the day: What is the bottleneck?

Reproduction steps

1. Have a project with JavaScript lambda functions and node_modules
2. Run amplify push
3. Wait in agony after "saving deployment state" begins

Project Identifier

No response

Log output

# Put your logs below this line

Additional information

No response

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[ethan021021]

+1

[josefaidt]

Hey @ErlendHer 👋 thanks for raising this with the detailed report! I've marked this as a feature-request to improve this behavior, though would you be open to filing a PR for this improvement with the suggested changes?

[ErlendHer]

Hey @ErlendHer 👋 thanks for raising this with the detailed report! I've marked this as a feature-request to improve this behavior, though would you be open to filing a PR for this improvement with the suggested changes?

I'll see if I can find the time for it. Would this require writing unit test for the suggested feature++?

[ErlendHer]

Opened a PR @josefaidt

[ErlendHer]

Any update on this? It would be very beneficial for our team (and I'm sure multiple other teams as well) if we could include the optimization in the next release. Right now, we're pushing using our own fork of the CLI with this fix, but we would rather have the entire team use the official release instead. So if anybody from the Amplify team would be able to take a look that would be great.

[josefaidt]

Hey @ErlendHer thank you for taking the time to dig into this and file a PR! Let me bump this with the team 🙂

[ErlendHer]

It's been another week now, without any further activity @josefaidt

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.