Unable to exclude package from being bundled #1218

huksley · 2022-08-10T22:13:37Z

This is a Bug Report

Description

What went wrong?

I am using the puppeteer package, and it creates inside a .local-chromium folder during package installation.

This package and .local-chromium folder are being packaged together into the zip file which end up being 352Mb.

What did you expect should have happened?

Package puppeteer should not be present in the resulting zip node_modules/ folder.

What was the config you used?

See config and full reproducible example in
https://github.com/huksley/puppeteerless/blob/main/serverless.yml

What stacktrace or error message from your provider did you see?

I am getting very slow upload speed and error that unpacked size exceeds 200Mb.

Additional Data

Node and NPM version you're using: NodeJS v16.16.0 NPM v8.11.0
Serverless-Webpack Version you're using: "serverless-webpack": "5.8.0"
Webpack version you're using: "webpack": "^5.72.1", "webpack-node-externals": "^3.0.0"
Serverless Framework Version you're using: "serverless": "^3.21.0"
Operating System: Mac OS X 12.5
Stack Trace (if available): N/A

Reproducing

> git clone https://github.com/huksley/puppeteerless
> cd puppeteerless
> npm install
> npm run package
> ls -lah .serverless
total 843384
drwxr-xr-x   8 ruslan  staff   256B Aug 11 01:03 .
drwxr-xr-x  19 ruslan  staff   608B Aug 11 01:03 ..
-rw-r--r--   1 ruslan  staff    50M Aug 11 01:03 chrome.zip
-rw-r--r--   1 ruslan  staff   2.2K Aug 11 01:03 cloudformation-template-create-stack.json
-rw-r--r--   1 ruslan  staff   9.5K Aug 11 01:03 cloudformation-template-update-stack.json
-rw-r--r--   1 ruslan  staff   8.9M Aug 11 01:03 fonts.zip
-rw-r--r--   1 ruslan  staff   352M Aug 11 01:03 puppeteerless.zip
-rw-r--r--   1 ruslan  staff    18K Aug 11 01:03 serverless-state.json

The text was updated successfully, but these errors were encountered:

huksley · 2022-08-10T22:23:17Z

Hi, I tried following options (and all of them does not work)

excludeFiles

custom:
  webpack:
    excludeFiles: "**/.local-chromium/**"

forceExclude

custom:
  webpack:
    includeModules:
      forceExclude:
        - puppeteer

nodeExternals in webpack.config.js

externals: [
    nodeExternals({
      modulesFromFile: {
        exclude: ["puppeteer"]
      }
    })
  ]

noInstall: true

custom:
  webpack:
    packagerOptions:
      noInstall: true

this actually causes error

Error:
[OperationalError: ENOENT: no such file or directory, stat '/Users/ruslan/src/s/.webpack/dependencies/node_modules'] {
  cause: [Error: ENOENT: no such file or directory, stat '/Users/ruslan/src/s/.webpack/dependencies/node_modules'] {
    errno: -2,
    code: 'ENOENT',
    syscall: 'stat',
    path: '/Users/ruslan/src/s/.webpack/dependencies/node_modules'
  },
  isOperational: true,
  errno: -2,
  code: 'ENOENT',
  syscall: 'stat',
  path: '/Users/ruslan/src/s/.webpack/dependencies/node_modules'
}

vicary · 2022-08-13T14:52:43Z

The modulesFromFile.exclude option is quite counter-intuitive, their exclude actually means excluded from marking the package as external which in turns is included in the bundle. Also the option values are keys in your package.json which means dependencies, depDependencies, peerDependencies and the deprecated optionalDependencies.

This is also why, in newer versions of webpack, they renamed include to excludeFromBundle and exclude to includeInBundle.

Try moving your puppeteer to devDependencies and use the following config,

externals: [
  nodeExternals({
     modulesFromFile: {
      excludeFromBundle: ["devDependencies"]
    }
  })
]

huksley · 2022-08-14T17:20:01Z

@vicary Thanks.

The problem is not that I get it bundled into .js file but rather that it is included into a resulting ZIP file...
How to avoid that?

It is used by https://github.com/ghostery/adblocker/tree/master/packages/adblocker-puppeteer
as a peer dependency and only used in TypeScript and resulting CJS does not require("puppeteer") at all.

vicary · 2022-08-14T18:28:47Z

If I am getting it correctly, your puppeteer lives inside node_modules in your ZIP. Serverless core, by default, brings everything in your node_modules into the final archive.

Our package installer, i.e. the noInstall option, doesn't matter here because if your adblocker-puppeteer lives in your dependencies, its peerDependencies are in turn installed along with it.

You may try the followings,

Explicitly install puppeteer in your devDependencies, and make sure you do not set false to package.excludeDevDependencies. This may overrides the peerDependencies relation in adblocker-puppeteer and tricks Serverless core to exclude it in the packing stage.
Excluding it via package.patterns !node_modules/puppeteer/**.

andrew-ignatiev · 2022-10-04T06:59:02Z

@vicary it's not ideal to use package.patterns especially when dependencies have a lot of peerDependencies. No one want to review manually peerDependencies in a large number of their lambda functions. For example TypeORM from version 3 started to use ts-node as optional peerDependency which on its own has typescript peerDependency which is huge and now all of it in packaged zip so the size of lamba.zip file has been started to grow significantly and upload of lambda functions have been started to fail due to AWS lambda limits. So I would say it's critical issue which requires urgent fix.

Devs from serverless-esbuild seems fixed it already floydspace/serverless-esbuild#113

j0k3r · 2022-10-04T07:23:01Z

Removing optional peer deps is already supported since 3 years now #542

andrew-ignatiev · 2022-10-04T07:26:52Z

But it doesn't work by some reason. In my example the TypeORM version 3 started to be packaged with ts-node and typescript in node_modules and command npm list typescript says it's due to ts-node which is optional peerDependency.

huksley · 2022-10-04T07:28:55Z

@j0k3r I am not using puppeteer directly, but through chrome-aws-lambda package: https://github.com/huksley/puppeteerless/blob/main/package.json

you think I should go to maintainers of chrome-aws-lambda and ask them to add peerDependenciesMeta ?

j0k3r · 2022-10-04T07:35:49Z

@huksley I guess so, like it's done in typeorm. Even better, you can submit yourself a PR (it's hacktoberfest!)

@andrew-ignatiev regarding the code, ts-node shouldn't be packaged because it's optional and defined in peerDependenciesMeta. None of your deps required ts-node?

andrew-ignatiev · 2022-10-04T08:02:50Z

@j0k3r in unzipped lambda folder I see the next npm list outputs:

npm list ts-node
└─┬ [email protected]
└── [email protected]

npm list typescript
└─┬ [email protected]
└─┬ [email protected]
└── [email protected]

I have checked package-lock.json and ts-node is optional peerDependency here. I don't see any other non optional peerDependencies or primary dependencies.

In package-loc.json in section dependencies:

"ts-node": {
      "version": "10.9.1",
      "resolved": "https://registry.npmjs.org/ts-node/-/ts-node-10.9.1.tgz",
      "integrity": "sha512-NtVysVPkxxrwFGUUxGYhfux8k78pQB3JqYBXlLRZgdGUqTO5wU/UyHop5p70iEbGhB7q5KmiZiU0Y3KlJrScEw==",
      "optional": true,
      "peer": true,
      "requires": {
        "@cspotcode/source-map-support": "^0.8.0",
        "@tsconfig/node10": "^1.0.7",
        "@tsconfig/node12": "^1.0.7",
        "@tsconfig/node14": "^1.0.0",
        "@tsconfig/node16": "^1.0.2",
        "acorn": "^8.4.1",
        "acorn-walk": "^8.1.1",
        "arg": "^4.1.0",
        "create-require": "^1.1.0",
        "diff": "^4.0.1",
        "make-error": "^1.1.1",
        "v8-compile-cache-lib": "^3.0.1",
        "yn": "3.1.1"
      },
      "dependencies": {
        "acorn-walk": {
          "version": "8.2.0",
          "resolved": "https://registry.npmjs.org/acorn-walk/-/acorn-walk-8.2.0.tgz",
          "integrity": "sha512-k+iyHEuPgSw6SbuDpGQM+06HQUa04DZ3o+F6CSzXMvvI5KMvnaEqXe+YVe555R9nn6GPt404fos4wcgpw12SDA==",
          "optional": true,
          "peer": true
        }
      }
    },
    ```
    with 
    ```
    "optional": true,
      "peer": true,
      ```

andrew-ignatiev · 2022-10-04T10:32:51Z

@j0k3r @vicary I've created minimal reproducible repo. Could you please check?

git clone https://github.com/andrew-ignatiev/serverless-typeorm
cd serverless-typeorm
nvm use 14
npm ci
npm run sls:package -- -s dev and wait 10-20 minutes
cd .serverless
mkdir ./res
unzip http-v1.zip -d ./res
cd ./res
npm list typescript

huksley · 2022-10-04T18:49:11Z

Maybe we could have a flag which will treat all peer dependencies as optional if they do not appear in dependencies of the parent package? I allways thought this is a convention.

huksley · 2022-10-04T19:07:11Z

@vicary Answering your post on Aug 12 (sorry, I just noticed it)
I done suggested changes on original reproducible repo and nothing changed.

git clone https://github.com/huksley/puppeteerless
nvm install 16
nvm use 16
npm install
npm run package
zip -Tv .serverless/puppeteerless.zip | grep puppeteer

andrew-ignatiev · 2022-10-05T06:12:08Z

@j0k3r @vicary I've created extra branch with TypeORM v2 typeorm-2. The main branch has TypeORM v3. You can check that in typeorm-2 the size of http-v1.zip is 13 Mb vs 31 Mb in main branch (lambda zip limit is 50 Mb). It's because of ts-node and typescript optional peerDependency.

git clone https://github.com/andrew-ignatiev/serverless-typeorm
cd serverless-typeorm
git checkout typeorm-2
nvm use 14
npm ci
npm run sls:package -- -s dev and wait 10-20 minutes
cd .serverless
mkdir ./res
unzip http-v1.zip -d ./res
cd ./res
npm list ts-node OR npm list typescript will output empty

Could you confirm the optional peerDependency issue in serverless-webpack package ?

andrew-ignatiev · 2022-10-06T09:57:57Z

Temporary workaround for short term in serverless.yml:

....
plugins:
  - serverless-webpack
...
  - serverless-scriptable-plugin
...
  scriptable:
    # add custom hooks
    hooks:
      after:package:createDeploymentArtifacts: 'find ./.serverless -name "*.zip" -exec zip --delete {} "node_modules/ts-node/*" "node_modules/typescript/*" "node_modules/aws-sdk/*" \;'
...

huksley · 2022-10-13T12:26:26Z

I have the same problem now with typescript bundled.
I wonder how one could debug it easily?

Now I do npm list -a --omit dev and when check dependency tree visually.
In my current situation typescript are required by react-intl but both have it marked as optional:

 "peerDependenciesMeta": {
    "typescript": {
      "optional": true
    }
  }

Dependecy tree excerpt:

func
├─┬ [email protected]
│ ├─┬ @formatjs/[email protected]
│ │ └── [email protected] deduped
│ └── [email protected]

huksley · 2022-10-14T06:07:08Z

I updated my project, but you need to remove hook proposed by @andrew-ignatiev in serverless.yml to show the bug

git clone https://github.com/huksley/puppeteerless
cd puppeteerless
npm install
npm run package
ls -lah .serverless

huksley · 2022-10-14T07:08:03Z

I think the better way is to trace all dependencies how NextJS does https://github.com/vercel/nft

huksley · 2022-12-15T21:07:42Z

Why is this marked as "Awaiting reply"? All replies and reproducible repo has been given.

collierrgbsitisfise · 2023-05-01T10:43:49Z

as issue is still open and still relevant, i'll left ref to own one(there is repo with straightforward steps how to reproduce it)

@huksley as workaround you can use packageOptions -> scripts in order to exclude redundant package from final node_modules

// serverless.ts file
.....
webpack: {
    packagerOptions: {
        scripts: ['#define custom script here which will delete redundant files/folders'],
    },
},
....

huksley · 2023-05-13T19:04:02Z

@collierrgbsitisfise thanks I already doing it and use esbuild to trace dependencies and it works pretty well

huksley changed the title ~~Unable to exclude page from being bundled~~ Unable to exclude package from being bundled Aug 10, 2022

vicary added the awaiting reply Awaiting for a reply from the OP label Aug 20, 2022

vicary mentioned this issue Aug 20, 2022

Unable to exclude aws-sdk from bundle #1222

Open

j0k3r removed the awaiting reply Awaiting for a reply from the OP label Dec 15, 2022

collierrgbsitisfise mentioned this issue Apr 14, 2023

broken forceExclude for peerDep on node16(npm v. 8) #1439

Open

pranavburnwal mentioned this issue Nov 24, 2023

Help needed, excluding aws-sdk from Minimize build #1662

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to exclude package from being bundled #1218

Unable to exclude package from being bundled #1218

huksley commented Aug 10, 2022

huksley commented Aug 10, 2022

vicary commented Aug 13, 2022

huksley commented Aug 14, 2022 •

edited

Loading

vicary commented Aug 14, 2022

andrew-ignatiev commented Oct 4, 2022 •

edited

Loading

j0k3r commented Oct 4, 2022

andrew-ignatiev commented Oct 4, 2022

huksley commented Oct 4, 2022

j0k3r commented Oct 4, 2022

andrew-ignatiev commented Oct 4, 2022

andrew-ignatiev commented Oct 4, 2022 •

edited

Loading

huksley commented Oct 4, 2022

huksley commented Oct 4, 2022

andrew-ignatiev commented Oct 5, 2022

andrew-ignatiev commented Oct 6, 2022

huksley commented Oct 13, 2022 •

edited

Loading

huksley commented Oct 14, 2022 •

edited

Loading

huksley commented Oct 14, 2022

huksley commented Dec 15, 2022

collierrgbsitisfise commented May 1, 2023

huksley commented May 13, 2023

Unable to exclude package from being bundled #1218

Unable to exclude package from being bundled #1218

Comments

huksley commented Aug 10, 2022

This is a Bug Report

Description

What went wrong?

What did you expect should have happened?

What was the config you used?

What stacktrace or error message from your provider did you see?

Additional Data

Reproducing

huksley commented Aug 10, 2022

excludeFiles

forceExclude

nodeExternals in webpack.config.js

noInstall: true

vicary commented Aug 13, 2022

huksley commented Aug 14, 2022 • edited Loading

vicary commented Aug 14, 2022

andrew-ignatiev commented Oct 4, 2022 • edited Loading

j0k3r commented Oct 4, 2022

andrew-ignatiev commented Oct 4, 2022

huksley commented Oct 4, 2022

j0k3r commented Oct 4, 2022

andrew-ignatiev commented Oct 4, 2022

andrew-ignatiev commented Oct 4, 2022 • edited Loading

huksley commented Oct 4, 2022

huksley commented Oct 4, 2022

andrew-ignatiev commented Oct 5, 2022

andrew-ignatiev commented Oct 6, 2022

huksley commented Oct 13, 2022 • edited Loading

huksley commented Oct 14, 2022 • edited Loading

huksley commented Oct 14, 2022

huksley commented Dec 15, 2022

collierrgbsitisfise commented May 1, 2023

huksley commented May 13, 2023

huksley commented Aug 14, 2022 •

edited

Loading

andrew-ignatiev commented Oct 4, 2022 •

edited

Loading

andrew-ignatiev commented Oct 4, 2022 •

edited

Loading

huksley commented Oct 13, 2022 •

edited

Loading

huksley commented Oct 14, 2022 •

edited

Loading