generator-langium: Transform generated project to npm workspace #1520

kaisalmen · 2024-05-29T15:12:06Z

TODO:

Fixes #1495

TODO: from review:

Update langium-quickstart.md
Add clean script and build:clean to all packages

kaisalmen · 2024-06-05T06:32:57Z

@msujew @Yokozuna59 the generated workspace project builds again. Now, I need to fix the open points above ⬆️

packages/generator-langium/templates/.vscode/launch.json

kaisalmen · 2024-06-14T09:42:55Z

@msujew @Yokozuna59 this is now ready for review. The only thing left is from the list is removing the need for http-server in the web package, because vite preview should do the job. But this is not blocking the review.

Lotes · 2024-06-14T09:44:43Z

For me workspaces were hard to work with in the beginning, because I was switching to the packages folders to run npm run X. Would be good to have some documentation on our website, I think. Like workspace 101, especially with the -w flag to address packages.
Does it change anything for the existing documentation?

kaisalmen · 2024-06-14T09:47:19Z

Does it change anything for the existing documentation?

Very likely 🙂 We need to update it once this is merged.

kaisalmen · 2024-06-14T09:57:39Z

Like workspace 101

The generated project is the blueprint. 👍 It features all facets: TS with specific test configs, VSCode eslint compatibility (root tsconfig.json per package), workspace eslint check, global build and top-level scripts that directly execute package scripts.

kaisalmen · 2024-06-14T11:54:26Z

All points are resolved. I also fix an issue with the web example. I broke the configuration during this update. It is working again.

Yokozuna59

I'm not sure if I should review this PR or not, but since it was mentioned, I reviewed it.

The below review is just comments or questions regarding this approach, which could be ignored if you don't feel they are correct.

One more thing, since you updated the out directory here:

langium/packages/generator-langium/templates/packages/language/langium-config.json

Line 14 in 8931941

"out": "src/generated"

I think you may need to update the .gitignore.txt:

langium/packages/generator-langium/templates/gitignore.txt

Line 7 in 8931941

src/language/generated/

packages/generator-langium/package.json

Yokozuna59 · 2024-06-14T19:11:08Z

packages/generator-langium/templates/packages/language/langium-quickstart.md

I think this file need to modified since some files are moved to different directories.

I would recommend to move the file to the root directory then reference the files from there.

Thank you, good point. I overlooked that.

I have added a TODO for this in PR description. ⬆️

packages/generator-langium/templates/packages/language/test/linking/linking.test.ts

packages/generator-langium/templates/package.json

packages/generator-langium/templates/.vscode/launch.json

packages/generator-langium/templates/packages/cli/src/cli-util.ts

packages/generator-langium/templates/packages/cli/tsconfig.json

packages/generator-langium/templates/package.json

packages/generator-langium/templates/packages/extension/.vscodeignore

Yokozuna59

@kaisalmen I'm not sure if these comments should be resolved yet:

See below.

packages/generator-langium/templates/packages/language/test/parsing.test.ts

packages/generator-langium/templates/packages/language/test/linking.test.ts

packages/generator-langium/templates/packages/cli/tsconfig.json

packages/generator-langium/templates/packages/web/vite.config.ts

packages/generator-langium/templates/gitignore.txt

kaisalmen · 2024-06-16T08:10:29Z

@Yokozuna59 thank you for the review. I still need to implement the two bigger points (added to the description) and then I am done. Any further enhancements should go to a next PR, because this one is already quite large, but there was no way around.

kaisalmen · 2024-06-19T08:50:41Z

@Yokozuna59 and @msujew all TODOs are done. We need to update tutorials and docs on the website as well. This change has some impact.

kaisalmen · 2024-06-21T11:25:43Z

I rebased the branch after Langium 3.1.0 release to resolve the conflicts.

kaisalmen · 2024-07-02T17:19:00Z

@msujew I just updated to the latest wrapper version. Do you have time to review it this week?

msujew · 2024-07-03T12:59:49Z

packages/generator-langium/templates/packages/language/package.json

@@ -0,0 +1,45 @@
+{
+    "name": "<%= extension-name %>-language",


Issue: This package name is built by using the specified extension-name. However, all references to this package (i.e. tests, vscode extension, etc.) use the specified language-id. If I call my extension abc and my language cba:

The language package will be called abc-language.

All imports for that language will attempt to import cba-language.

This obviously yields errors on build. Both should use the same ID.

@msujew then language-id is the better choice. extension-name is only useful for the VSCode extension. Do you agree?

I have changed it like suggested for now. ⬇️

msujew · 2024-07-03T13:04:55Z

packages/generator-langium/templates/packages/language/src/index.ts

+export * from './generated/ast.js';
+export * from './generated/grammar.js';
+export * from './generated/module.js';
+export { default as monarchSyntax } from './syntaxes/<%= language-id %>.monarch.js';


Question: We should only conditionally generate this export - if the web option is selected.

Yes, true, I will adapt this.

This was a bit more work, because I needed to change the workflow, but it looks good now, IMO.

msujew · 2024-07-03T13:05:45Z

packages/generator-langium/templates/packages/language/langium-config.json

@@ -11,5 +11,5 @@
            "out": "src/syntaxes/<%= language-id %>.monarch.ts"


Suggestion: As mentioned above, we should try to make this conditional.

Yes, true, I will adapt this.

Done, see above.

msujew · 2024-07-03T13:08:28Z

packages/generator-langium/templates/packages/language/langium-config.json

        "textMate": {
            "out": "syntaxes/<%= language-id %>.tmLanguage.json"


Suggestion: The textmate grammar is generated into a folder in the language package. However, it is referenced the package.json of the VS Code extension as if it were located inside of the extension package. Unless the adopter changes the generated path or copies the file manually, they don't get syntax highlighting in VS Code.

I added a build prepare step in both web and extension that copies the file over. ⬇️

packages/generator-langium/templates/packages/extension/langium-quickstart.md

kaisalmen · 2024-07-08T08:11:41Z

@msujew rework is complete. Im have not closed some of the discussions, because you should check if you agree with my solutions.

aabounegm · 2024-07-08T10:52:53Z

@kaisalmen I created a project from this branch and it seems there is an issue with the generated CLI. I think the following paths inside packages/generator-langium/templates/packages/cli should be updated:

In bin/cli.js, it should import from just '../out/main.js' (without /cli)
In src/main.ts:13, the packagePath should have one less '..'

At least that's what I edited manually in my project for it to work

kaisalmen · 2024-07-08T11:33:16Z

@aabounegm Thank you very much! Many things changed, it is easy to overlook things. I will push an update.

zeyad001 · 2024-07-18T13:56:22Z

any updates on this?
Will this be part of a 3.2 release?

aabounegm

I just noticed another issue while packaging my extension. Also, npm run lint from root is broken, the path passed to the command should be ./packages/*/src instead of just src.

packages/generator-langium/templates/packages/extension/package.json

aabounegm · 2024-07-25T12:50:40Z

And one more thing: it seems that vsce does not currently support npm workspaces (microsoft/vscode-vsce#777 (comment)), so running vsce package tries to bundle the entire project (with all workspaces) and fails with weird errors.
Until this is fixed, there are 3 possible workarounds:

Add ../ and ../../ to .vscodeignore
Add "vsce": { "dependencies": false } to package.json
Add a note that users need to pass --no-dependencies to the vsce command

I personally opted for the first option in my project with a comment linking to the issue.

And on a somewhat related note (also publishing), I would suggest adding a .gitignore file to packages/extension with at least *.vsix.

spoenemann · 2024-07-26T07:46:35Z

We used solution 2 in our own extension, too:

langium/packages/langium-vscode/package.json

Lines 84 to 86 in 3b007c2

    
           "vsce": { 
        
             "dependencies": false 
        
           },

aabounegm

I got this thought while exploring how a JetBrains plugin would look like, and realized I'm copying main.cjs from extension/out/language rather than from language/out directly.

aabounegm · 2024-08-02T18:52:44Z

packages/generator-langium/templates/packages/extension/src/language/main.ts

Is there a reason this file resides under the extension folder and not under language?

I assume chase it’s specific to the use case as packages/generator-langium/templates/packages/web/src/main-browser.ts is

Except that this one runs a language server regardless of the context as long as Node is available. The browser is the exception because it does not have Node.js or access to the file system.
For example, I installed LSP4IJ in WebStorm and pointed it to this file (node /.../extension/out/language/main.cjs --stdio) and it ran perfectly

The philosophy behind this change is that the language package is designed to be used like a runtime independent library. For example, the CLI package makes use of that as well, without running a whole language server.

Then, every consumer can make use of the library as they like. Some build a VS Code extension, others a web project or a CLI. In my mind all of these packages should provide their own entry point. I think that's reasonable as a default for the generated project.

Got it. I don't see why it can't be both a library and still expose an endpoint, but it's not a big deal, it's easy to duplicate that file. The only (minor) issue in this case is having to make the JetBrains plugin a JavaScript module as well (as in, it will have both package.json and the gradle stuff) to compile the JS file (with the exact same content as extension/src/language/main.ts) instead of just copying it from language/out/main.cjs (for example).

For those cases where a build artifact is required by multiple consumers, we usually just copy it around via some build task/script using shx or gulp. I think the generated build is fine as it is. It's not like we prevent adopters from changing how their build works after yeoman has done its job.

I see, but the only problem here could be in the (admittedly rare) occasion that someone does not generate a VS Code extension, and opts only for JetBrains plugin (for example, once that is ready).
It's not a big deal, I just need to decide whether to copy that language/main.ts to the new submodule and make it an npm package as well, or just depend on extension (VS Code) always existing.

It's not like we prevent adopters from changing how their build works after yeoman has done its job

Fair enough. Feel free to resolve this comment.

- Align template structure with target structure - Integrate a global tsconfig.build.json - Unify how build and compile is done - Make cli, web and extension compile again

- Updated relevant dependencies and aligned version in the repo

…eed for http-server

…roject

…languageId

… review comments

kaisalmen · 2024-08-08T13:30:06Z

@aabounegm thank you for those additional findings/comments. Regarding vsce I selected option 2, modified the gitignore and fixed the prebublish/lint

packages/generator-langium/templates/packages/extension/package.json

…ript

kaisalmen changed the title ~~WIP: Transform generated project to npm workspace~~ WIP: generator-langium: Transform generated project to npm workspace May 29, 2024

Yokozuna59 reviewed Jun 12, 2024

View reviewed changes

packages/generator-langium/templates/.vscode/launch.json Outdated Show resolved Hide resolved

kaisalmen force-pushed the kaisalmen/issue-1495 branch 4 times, most recently from ea748fa to 16211b6 Compare June 14, 2024 09:38

kaisalmen marked this pull request as ready for review June 14, 2024 09:39

kaisalmen changed the title ~~WIP: generator-langium: Transform generated project to npm workspace~~ generator-langium: Transform generated project to npm workspace Jun 14, 2024

Yokozuna59 reviewed Jun 15, 2024

View reviewed changes

packages/generator-langium/templates/packages/extension/.vscodeignore Outdated Show resolved Hide resolved

kaisalmen force-pushed the kaisalmen/issue-1495 branch from 7c811fc to 94ab736 Compare June 15, 2024 16:23

Yokozuna59 reviewed Jun 15, 2024

View reviewed changes

packages/generator-langium/templates/packages/web/vite.config.ts Outdated Show resolved Hide resolved

packages/generator-langium/templates/gitignore.txt Outdated Show resolved Hide resolved

kaisalmen force-pushed the kaisalmen/issue-1495 branch from 2bf7da8 to 7ba0b13 Compare June 19, 2024 08:47

kaisalmen force-pushed the kaisalmen/issue-1495 branch from 7ba0b13 to 84a13c7 Compare June 21, 2024 11:23

kaisalmen mentioned this pull request Jun 28, 2024

langium-development: The vitest configs in generator langium templates prevent the vitest extension to work in the repo #1564

Open

kaisalmen force-pushed the kaisalmen/issue-1495 branch from 40576e4 to bbd095b Compare July 1, 2024 10:57

msujew reviewed Jul 3, 2024

View reviewed changes

kaisalmen force-pushed the kaisalmen/issue-1495 branch from a59cf4b to 3c0b81d Compare July 5, 2024 14:44

aabounegm reviewed Jul 25, 2024

View reviewed changes

packages/generator-langium/templates/packages/extension/package.json Outdated Show resolved Hide resolved

aabounegm reviewed Aug 2, 2024

View reviewed changes

kaisalmen added 14 commits August 8, 2024 12:25

Transform generated project to npm workspace

42554d4

- Align template structure with target structure - Integrate a global tsconfig.build.json - Unify how build and compile is done - Make cli, web and extension compile again

Integrating langauge tests again in generator and enhancing tests

eb78d1f

- Updated relevant dependencies and aligned version in the repo

Move test output to packages/generator-langium/test/genetator-tests

66d1e21

fix: web languageId and fileExt replacement / configuration, remove n…

96bd3ab

…eed for http-server

Implemented a first set of review comments

a57d24f

Implement further review comments

35721eb

Update generator documentation in repo. At build:clean to generated p…

ea179bc

…roject

Updated READMEs once more

23b3283

Updated monaco-editor-wrapper to the latest version

dbb7a40

Implement review comments: tmLanguage location and package names use …

da5562d

…languageId

Fix documentation and fix tests

351915c

fix: monarch is only generated and included if web project us selected

176e78a

fix: correction to cli

782a12e

Rebased branch once more and contained smaller changes due to further…

cff11ba

… review comments

kaisalmen force-pushed the kaisalmen/issue-1495 branch from 42e3a41 to cff11ba Compare August 8, 2024 13:26

kaisalmen mentioned this pull request Aug 8, 2024

Intellij template #1618

Closed

aabounegm reviewed Aug 9, 2024

View reviewed changes

packages/generator-langium/templates/packages/extension/package.json Outdated Show resolved Hide resolved

Do not use "run --prefix" and align eslint+plugin version with typesc…

6f10baa

…ript

		@@ -11,5 +11,5 @@
		"out": "src/syntaxes/<%= language-id %>.monarch.ts"

		"textMate": {
		"out": "syntaxes/<%= language-id %>.tmLanguage.json"

generator-langium: Transform generated project to npm workspace #1520

Are you sure you want to change the base?

generator-langium: Transform generated project to npm workspace #1520

Conversation

kaisalmen commented May 29, 2024 • edited Loading

kaisalmen commented Jun 5, 2024 • edited Loading

kaisalmen commented Jun 14, 2024

Lotes commented Jun 14, 2024

kaisalmen commented Jun 14, 2024

kaisalmen commented Jun 14, 2024

kaisalmen commented Jun 14, 2024

Yokozuna59 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yokozuna59 left a comment

Choose a reason for hiding this comment

kaisalmen commented Jun 16, 2024

kaisalmen commented Jun 19, 2024

kaisalmen commented Jun 21, 2024

kaisalmen commented Jul 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaisalmen Jul 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaisalmen Jul 5, 2024 • edited Loading

Choose a reason for hiding this comment

kaisalmen commented Jul 8, 2024

aabounegm commented Jul 8, 2024

kaisalmen commented Jul 8, 2024

zeyad001 commented Jul 18, 2024

aabounegm left a comment • edited Loading

Choose a reason for hiding this comment

aabounegm commented Jul 25, 2024 • edited Loading

spoenemann commented Jul 26, 2024

aabounegm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaisalmen commented Aug 8, 2024

kaisalmen commented May 29, 2024 •

edited

Loading

kaisalmen commented Jun 5, 2024 •

edited

Loading

kaisalmen Jul 8, 2024 •

edited

Loading

kaisalmen Jul 5, 2024 •

edited

Loading

aabounegm left a comment •

edited

Loading

aabounegm commented Jul 25, 2024 •

edited

Loading