Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug?]: yarn patch generated unparsable patch: URL #4711

Open
1 task
lachlanhunt opened this issue Aug 5, 2022 · 17 comments · May be fixed by #6525
Open
1 task

[Bug?]: yarn patch generated unparsable patch: URL #4711

lachlanhunt opened this issue Aug 5, 2022 · 17 comments · May be fixed by #6525
Labels
bug Something isn't working stale Issues that didn't get attention

Comments

@lachlanhunt
Copy link

lachlanhunt commented Aug 5, 2022

Self-service

  • I'd be willing to implement a fix

Describe the bug

My company uses an internal package registry where all npm packages are fetched from. This is configured in ~/.yarnrc.yaml (company name replaced with example)

npmRegistries:
  "https://packages.example.com/artifactory/api/npm/npm-remote":
    npmAlwaysAuth: false
npmScopes:
  example:
    npmRegistryServer: "https://packages.example.com/artifactory/api/npm/example-npm"
    npmPublishRegistry: "https://packages.example.com/artifactory/api/npm/example-npm"

The result of this is that the resolution for the package in yarn.lock looks like this:

  resolution: "@example/eslint-plugin-example@npm:4.0.0::__archiveUrl=https%3A%2F%2Fpackages.example.com%2Fapi%2Fnpm%2Fexample-npm%2F%40example%2Feslint-plugin-example%2F-%2F%40example%2Feslint-plugin-example-4.0.0.tgz"

When I tried to patch this packing using:

$ yarn patch @example/eslint-plugin-example
# make edits
$ yarn patch-commit -s /private/var/folders/path/to/code/user
...

This generated a patch URL in package.json that looked like this:

```json
{
  ...
  "resolutions": {
    "@example/[email protected]": "patch:@example/eslint-plugin-example@npm:4.0.0::__archiveUrl=https%3A%2F%2Fpackages.example.com%2Fapi%2Fnpm%2Fexample-npm%2F%40example%2Feslint-plugin-example%2F-%2F%40example%2Feslint-plugin-example-4.0.0.tgz#.yarn/patches/@example-eslint-plugin-example-npm-4.0.0-bab1aa1587.patch"
  }

}

(Then I fixed the version range in that resolution as described in #4231 (comment) )

Finally, running yarn to apply the patch gives this error:

$ yarn
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: Patch locators must explicitly define their source
    at xue (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:733:658)
    at dE (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:733:978)
    at iO.bindDescriptor (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:736:2747)
    at wd.bindDescriptor (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:391:1082)
    at wd.bindDescriptor (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:391:1082)
    at reduceDependency (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:390:29055)
    at tA.reduceHook (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:392:3584)
    at ee (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:439:6759)
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 23ms

I was able to work around this issue by modifying the patch: URL like this:

{
  ...
  "resolutions": {
    "@example/eslint-plugin-example@^4.0.0": "patch:@example/eslint-plugin-example@npm:4.0.0#.yarn/patches/@example-eslint-plugin-example-npm-4.0.0-bab1aa1587.patch"
  }
}

It appears the presence of ::__archiveUrl=https... breaks the parsing of the patch URL in yarn.

To reproduce

(as explained above)

edit: see reproduction in comment below #4711 (comment)

Environment

System:
  OS: macOS 12.5
  CPU: (10) arm64 Apple M1 Pro
Binaries:
  Node: 16.16.0 - /private/var/folders/78/vv5cyf6570xcqbsy0r2nfyfw0000gn/T/xfs-99d9804c/node
  Yarn: 3.2.2 - /private/var/folders/78/vv5cyf6570xcqbsy0r2nfyfw0000gn/T/xfs-99d9804c/yarn
  npm: 8.11.0 - ~/Library/Caches/fnm_multishells/21303_1659328957074/bin/npm

Additional context

No response

@lachlanhunt lachlanhunt added the bug Something isn't working label Aug 5, 2022
@xenoterracide
Copy link

hmm... I wonder if this is the same as the problem I just came here to report

@xenoterracide
Copy link

I don't get an error though, mine just silently doesn't apply

@xenoterracide
Copy link

linked my bug, seems very related to both that you mentioned, but mine seems to be both of the problems at once...

@wojtekmaj
Copy link
Contributor

I was able to work around this issue by modifying the patch: URL like this:

OMG, I want to kiss you right now. I struggled with this for MONTHS, manually applying the patches.

@yarnbot
Copy link
Collaborator

yarnbot commented Sep 4, 2022

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟

@yarnbot yarnbot added the stale Issues that didn't get attention label Sep 4, 2022
@lachlanhunt
Copy link
Author

I couldn't figure out how exactly to reproduce this issue using Sherlock. Last time I tried to use it, the page failed to load the editor properly, so I gave up and assumed the tool was broken.

I explained the steps to reproduce in the description. I'll need a maintainer to help out.

@wojtekmaj
Copy link
Contributor

wojtekmaj commented Sep 7, 2022

This is my completely blind attempt at repro, as playground is not working, and local Sherlock does not expose (undocumented) methods like yarn or packageJsonAndInstall:

const installPromise = packageJsonAndInstall({
  dependencies: {
    [`@codertocat/hello-world-npm`]: `latest`,
  },
});

await expect(installPromise).resolves.toBeTruthy();

// Use node_modules resolver
// await yarn(`config`, `set nodeLinker node-modules`);

// Set npmRegistryServer
await yarn(`config`, `set npmRegistryServer "https://npm.pkg.github.com"`);

// Extract the package for patching
const patchOutput = await yarn(`patch`, `@codertocat/hello-world-npm`);
const patchOutputLines = patchOutput.split("\n");

// Look for "You can now edit the following folder: …"
const regex1 = /You can now edit the following folder: (.*)/;
const line1 = patchOutputLines.find((line) => line.match(regex1));
const folder = line1.match(regex1)[1];

// Make the edit in extracted package README.md file
const filePath = path.join(path.resolve(folder), `index.js`);
const file = await fs.readFile(filePath, `utf8`);
await fs.writeFile(
  filePath,
  file.replace("'Hello world!'", "'Hello world from patch!'")
);

// Look for "Once you are done run … and Yarn will store a patchfile based on your changes."
const regex2 =
  /Once you are done run (.*) and Yarn will store a patchfile based on your changes./;
const line2 = patchOutputLines.find((line) => line.match(regex2));
const command = line2.match(regex2)[1];
const [yarnText, commandText, ...args] = command.split(" ");

// Run the command
await yarn(commandText, ...args);

// Install repo again to apply patches
const install2Promise = await yarn(`install`);

/**
 * BUG #1:
 * Error: @codertocat/hello-world-npm@npm:1.0.2::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Codertocat%2Fhello-world-npm%2F1.0.2%2F9384a68bd287b859f028683bcaacedd27803f1dd isn't supported by any available resolver
 **/
expect(install2Promise).resolves.toBeTruthy();

// See if the patch was applied
const installedModule = require("@codertocat/hello-world-npm");

const output = installedModule.helloWorld();

/**
 * BUG #2: Patch not applied?
 */
expect(output).toEqual("Hello world from patch!");

@arcanis
Copy link
Member

arcanis commented Sep 7, 2022

Mind trying the 4.0.0-rc (yarn set version canary)? Patches have been significantly improved there, and I'm unlikely to backport those changes to 3.x as some of them are backward-incompatible, and others just difficult to safely extract.

@wojtekmaj
Copy link
Contributor

wojtekmaj commented Sep 8, 2022

@arcanis I did the tests on 2 dimensions:

  • Yarn version: latest (3.2.3), canary (4.0.0-rc.18)
  • Dependency version range: exact ("@codertocat/hello-world-npm": "1.0.2") and caret ("@codertocat/hello-world-npm": "^1.0.0")

Here are the commands I ran:

# 4.0.0 ONLY:
yarn set version canary

yarn config set nodeLinker node-modules
yarn config set npmRegistryServer "https://npm.pkg.github.com"

# CARET ONLY:
yarn add @codertocat/hello-world-npm@^1.0.0
# EXACT ONLY:
yarn add @codertocat/[email protected]

yarn patch @codertocat/hello-world-npm
# (editing the file now by changing "Hello world!" to "Hello world from patch!" & saving it)
yarn patch-commit -s # (dynamically created folder path here)

yarn

All 4 resulted in the exact same crash.

user@WS-Q6JW6PN4MP 3.2.3-caret % yarn
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: @codertocat/hello-world-npm@npm:1.0.2::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Codertocat%2Fhello-world-npm%2F1.0.2%2F9384a68bd287b859f028683bcaacedd27803f1dd isn't supported by any available resolver
    at Bd.getResolverByDescriptor (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:391:1664)
    at Bd.getResolutionDependencies (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:391:1147)
    at TR.getResolutionDependencies (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:435:4101)
    at Bd.getResolutionDependencies (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:391:1176)
    at ce (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:439:7562)
    at /Users/user/.node/corepack/yarn/3.2.3/yarn.js:439:8135
    at async /Users/user/.node/corepack/yarn/3.2.3/yarn.js:439:7666
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 19ms
user@WS-Q6JW6PN4MP 4.0.0-rc-caret % yarn
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: @codertocat/hello-world-npm@npm:1.0.2::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Codertocat%2Fhello-world-npm%2F1.0.2%2F9384a68bd287b859f028683bcaacedd27803f1dd isn't supported by any available resolver
    at ed.getResolverByDescriptor (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:135:1698)
    at ed.getResolutionDependencies (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:135:1177)
    at Fx.getResolutionDependencies (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:190:11001)
    at ed.getResolutionDependencies (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:135:1206)
    at Ce (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:195:7688)
    at /Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:195:8489
    at async /Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:195:7826
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 19ms

Here are all 4 minirepos I created to reproduce this issue:

Yarn patch.zip

In all 4 cases, removing %3A%3A__archiveUrl=https%253A%252F%252Fnpm.pkg.github.com%252Fdownload%252F%2540Codertocat%252Fhello-world-npm%252F1.0.2%252F9384a68bd287b859f028683bcaacedd27803f1dd bit from resolution caused installation to proceed just fine & applied patch just fine.

I have a suspicion that releasing 1.0.3 version of test package would break patch on 3.2.3 as resolution created by patch-commit would no longer be applicable. But that's another story.

@sargunv
Copy link
Contributor

sargunv commented Feb 1, 2023

I found this article quite helpful to understand the bug and how to work around it: https://charles-stover.medium.com/patching-packages-in-yarn-berry-72e4ded29a56

@quisido
Copy link

quisido commented Feb 21, 2024

I'm still getting this in v4.1.0. I wish Yarn would implement some sort of e2e test around the yarn patch command to verify that it works without regression.

@quisido
Copy link

quisido commented Feb 21, 2024

For documentation's sake, I fixed this by changing %3A%3A to :: before each __archiveUrl= and by moving #~/.yarn/patches/... to before the :: in each instance as well.

@BasixKOR
Copy link
Contributor

BasixKOR commented Feb 23, 2024

Here's a simple reproduction I came up with:

const { writeFile } = require(`fs/promises`);

await writeFile('.yarnrc.yml', `
npmScopes:
  types:
    npmRegistryServer: https://registry.npmjs.com
`);

await packageJsonAndInstall({
  devDependencies: {
    "lodash": "^4.17.21",
    "@types/lodash": "^4.14.202"
  }
});

const { path } = JSON.parse(await yarn('patch', '@types/lodash', '--json'));
await yarn('patch-commit', '-s', path);
await expect(yarn('install')).resolves.toBeTruthy();

@lachlanhunt Could you add this into your issue description?

@pniederlag
Copy link

just being a fresh user of yarn I find 'yarn patch' not to be working (yarn 4.1.1 with custom npmRegistry, scoped packages and workspaces). Will now try to dig into it and find a workaround

@pniederlag
Copy link

just being a fresh user of yarn I find 'yarn patch' not to be working (yarn 4.1.1 with custom npmRegistry, scoped packages and workspaces). Will now try to dig into it and find a workaround

it seems to work when I do:

  • edit resolution field in yarn.lock, drop everything '::__archiveUrl.*'
  • start regular yarn patch, yarn patch-commit -s procedure
  • run yarn install

seems like it's doing as intended, while not being the most easy user experience though. ;)

I found some hints in #3012 regardingt the usage of __archiveUrl and non-conformant registry-servers (artifactory in my case), maybe that helps in getting this issue ironed out at some later point

@gozegotushar
Copy link

i am also facing same from last one week.
not able to solve it.

@gozegotushar
Copy link

i was getting issue with installing yarn with homebrew.

it worked after updating my yarn & node version to :
node : v20.8.1
yarn : 4.2.2

to install yarn please follow this link : https://stackoverflow.com/a/78307944/2382964

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working stale Issues that didn't get attention
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants