Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: convert to ESM (breaking change) #856

Merged
merged 78 commits into from
Sep 14, 2023
Merged

feat: convert to ESM (breaking change) #856

merged 78 commits into from
Sep 14, 2023

Conversation

kanadgupta
Copy link
Member

@kanadgupta kanadgupta commented Aug 9, 2023

🚥 Partially addresses #801

🧰 Changes

This PR refactors the entire repository over to full ESM. This entailed a few different refactors:

  • Using .js file extensions for relative imports (really wish I had done this after discovering eslint-plugin-require-extensions) 😭
  • Updated a bunch of dependencies to their latest ESM counterparts
    • (with the exception of ora, see the outstanding issues below)
    • Some of our upstream dependencies now use native Node 18 fetch so a test or two had to be updated accordingly. Eventually we'll want to migrate away from nock entirely.
  • Setting module: NodeNext in our tsconfig.json and type: module in our package.json
  • In order to build the single-file executable binaries used in our Docker/GitHub Actions builds, we now compile this entire repository into CJS using Rollup before sending it into pkg 😵‍💫 (see ES modules not supported vercel/pkg#1291)
    • This obviously isn't ideal, but pkg handles a lot of the madness of compiling into single-file executables better than anything else I've tried (see here for more context)
    • A huge benefit here is that our Docker image sizes should be around 25% smaller 📉
Outstanding tasks/issues

Below is a running list of outstanding work — the ora issue is mostly out of our hands, but I'll look into the ExperimentalWarning in a follow-up PR.

(node:92816) ExperimentalWarning: Import assertions are not a stable feature of the JavaScript language. Avoid relying on their current behavior and syntax as those might change in a future version of Node.js.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:92816) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time

🧬 QA & Testing

Our existing end-to-end tests GitHub Actions tests are passing, meaning this should work, but I'll do some more QA once we tag a beta release.

You can confirm this all works locally by doing the following:

# install/build
npm ci && npm run build:exe

# run the JS output (used by npm-based CLI installations)
bin/rdme.js openapi:validate # or whatever command you want

# run the binary
./exe/rdme openapi:validate  # or whatever command you want

@kanadgupta
Copy link
Member Author

kanadgupta commented Aug 10, 2023

Update: while I made substantial progress on this, we've hit a pretty massive blocker 😵‍💫 we rely on pkg for our Docker/GitHub Action build, and turns out it doesn't support ESM at all: vercel/pkg#1291

Jon's work in #857 should make this transition a lot easier, but until we figure out a different strategy for building an executable (or if pkg somehow adds ESM support), we probably have to stick with CommonJS for the time being 😭

EDIT (9/2):

Might think about pkg alternatives, like Deno:

https://deno.com/blog/build-cross-platform-cli
https://deno.land/[email protected]/tools/compiler

Ran into the following error when trying to build an executable with Deno locally:
> deno compile dist/src/cli.js --output deno-test
Compile file:///Users/kanadg/Code/readmeio/rdme/dist/src/cli.js to deno-test
error: Writing deno-test

Caused by:
    0: Reading symlink /Users/kanadg/Code/readmeio/rdme/node_modules/.deno/[email protected]/node_modules/@esbuild/win32-arm64
    1: No such file or directory (os error 2)

Deno version info:

deno 1.36.0 (release, aarch64-apple-darwin)
v8 11.6.189.12
typescript 5.1.6

I also tried following Node's guidelines for building a single executable in Node v20.

I haven't troubleshooted this further but I ran into the following error when following the steps above and trying to run the generated executable:
> ./hello
(node:44464) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `hello --trace-warnings ...` to show where the warning was created)
/Users/kanadg/Code/readmeio/rdme/hello:2
import '../dist/src/cli.js';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at internalCompileFunction (node:internal/vm:73:18)
    at wrapSafe (node:internal/modules/cjs/loader:1153:20)
    at embedderRunCjs (node:internal/util/embedding:18:27)
    at node:internal/main/embedding:18:8

Node.js v20.5.1

Here's my sea-config.json file:

{ "main": "bin/rdme.js", "output": "sea-prep.blob" }

It's weird because running bin/rdme.js works (albeit with a Node warning):

> bin/rdme.js
(node:44508) ExperimentalWarning: Import assertions are not a stable feature of the JavaScript language. Avoid relying on their current behavior and syntax as those might change in a future version of Node.js.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:44508) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time

                  📖 rdme

    a utility for interacting with ReadMe
       .
       .\                          /.
      ’  ‘                        ‘ ‘
      ( nn\    .           .     /  )
      (nnnnn,.MM.          AM   .nn.
       .nnnndMM----_______--M.’’   /
       |nnn/nnnnnnnnnnnnnnnnn\’mmm/
       /nnnn...nnnnnnnnnnn...mmmmm\
      /nn        ‘qnnnP’       ‘mmm\
      /n’   .XXX. \nnn/   .XX.  \mmb
      An   (XOXX)  nnn   (XOXX)  mm\
     /nn   ‘XXXX’.~”~.   ‘XXX”’ .mmmb
     dnnn.      (    )n.       .mmmmb
    .nnnnnn....n.\ ./nnnnnnnnnmmmmmm\
  (READnnnnnnnnnnn’Y’nnnnnnnnnnmmmmmmME)
  REinnnnnnnnnnnnnnnnnnnnnnnnnmmmmmmmAD/
 /MEEnnnnnnnnnnnnnnnnnnnnnnnnnmmmmmmm)E'.
 READnnnnnnn*’             ‘*mmmmmmmm)MEE.
(READ|nnnn’    \../  \.../    ‘mmmmmM)EEE)
 READ(nn*’                      ‘mmm.MEEE)
  ‘READn’  \._./  \__./  \.../     ‘MEE*’
       *                           /*

Usage

  rdme <command> [arguments] 

Options

  -h, --help       Display this usage guide               
  -v, --version    Show the current rdme version (v8.6.6) 

OpenAPI / Swagger

  $ rdme openapi            Upload, or resync, your OpenAPI/Swagger definition  
                            to ReadMe.                                          
  $ rdme openapi:convert    Convert a Swagger or Postman Collection to OpenAPI. 
  $ rdme openapi:inspect    Analyze an OpenAPI/Swagger definition for various   
                            OpenAPI and ReadMe feature usage.                   
  $ rdme openapi:reduce     Reduce an OpenAPI definition into a smaller subset. 
  $ rdme openapi:validate   Validate your OpenAPI/Swagger definition.           

Docs (a.k.a. Guides)

  $ rdme docs           Sync Markdown files to your ReadMe project as Guides.   
                        Can either be a path to a directory or a single         
                        Markdown file.                                          
  $ rdme docs:prune     Delete any docs from ReadMe if their slugs are not      
                        found in the target folder.                             
  $ rdme guides         Alias for `rdme docs`.                                  
  $ rdme guides:prune   Alias for `rdme docs:prune`.                            

Changelog

  $ rdme changelogs   Sync Markdown files to your ReadMe project as Changelog   
                      posts. Can either be a path to a directory or a single    
                      Markdown file.                                            

Custom Pages

  $ rdme custompages   Sync Markdown/HTML files to your ReadMe project as       
                       Custom Pages. Can either be a path to a directory or a   
                       single Markdown/HTML file.                               

Versions

  $ rdme versions          List versions available in your project or get a     
                           version by SemVer (https://semver.org/).             
  $ rdme versions:create   Create a new version for your project.               
  $ rdme versions:delete   Delete a version associated with your ReadMe         
                           project.                                             
  $ rdme versions:update   Update an existing version for your project.         

Categories

  $ rdme categories          Get all categories in your ReadMe project.         
  $ rdme categories:create   Create a category with the specified title and     
                             guide in your ReadMe project.                      

Administration

  $ rdme login    Login to a ReadMe project.                                    
  $ rdme logout   Logs the currently authenticated user out of ReadMe.          
  $ rdme whoami   Displays the current user and project authenticated with      
                  ReadMe.                                                       

Other useful commands

  $ rdme open   Open your current ReadMe project in the browser. 

Run rdme help <command> for help with a specific command.

To get more help with ReadMe, check out our docs at https://docs.readme.com.

@kanadgupta kanadgupta mentioned this pull request Aug 23, 2023
26 tasks
@kanadgupta kanadgupta changed the title ESM spike refactor: esm Sep 7, 2023
Comment on lines +27 to +34
formData.append('spec', {
type: 'application/json',
name: 'openapi.json',
[Symbol.toStringTag]: 'File',
stream() {
return stream;
},
});
Copy link
Member Author

@kanadgupta kanadgupta Sep 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erunion I'd like your blessing with these changes now that we're using formdata-node — does our API care what we name the file? I just set it to openapi.json but let me know if you feel otherwise.

also our API mocks inspect this payload so we should be in good shape.

Comment on lines -18 to -19
- 14
- 16
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with this PR, we're technically requiring Node 18 and above but I'll make that official in a separate PR.

@kanadgupta kanadgupta marked this pull request as ready for review September 14, 2023 15:57
@kanadgupta kanadgupta changed the title refactor: esm feat: convert to ESM (breaking change) Sep 14, 2023
@kanadgupta kanadgupta requested review from erunion, a team, Dashron, darrenyong, RyanGWU82, domharrington, rafegoldberg and ilias-t and removed request for a team September 14, 2023 15:59
@kanadgupta kanadgupta requested a review from llimllib September 14, 2023 16:10
Copy link
Member

@erunion erunion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made it look easy

@@ -24,7 +24,14 @@ export default async function streamSpecToRegistry(spec: string) {

debug('file and stream created, streaming into form data payload');
const formData = new FormData();
formData.append('spec', stream);
formData.append('spec', {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This work seems fine but formdata-node is still pretty shitty. Recommend moving off it and node-fetch as soon as we can. Left a diff in #801 (comment) with the work you can do to this file to move it over to all native APIs.

package.json Outdated Show resolved Hide resolved
rollup.config.js Show resolved Hide resolved
@@ -220,7 +220,9 @@ async function handleRes(res: Response, rejectOnJsonError = true) {
const contentType = res.headers.get('content-type');
const extension = mime.extension(contentType);
if (extension === 'json') {
const body = await res.json();
// TODO: type this better
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as any is probably fine imo. You aren't going to have full documented types for these responses without utilizing our API SDK or anything like that (and even then the types are nonexistent for API v1).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah node-fetch is remarkably bad about typing responses

@kanadgupta kanadgupta merged commit 84b8571 into next Sep 14, 2023
8 checks passed
@kanadgupta kanadgupta deleted the esm-spike branch September 14, 2023 21:52
@kanadgupta kanadgupta added this to the v9 milestone Sep 14, 2023
kanadgupta pushed a commit that referenced this pull request Dec 6, 2024
# [9.0.0](v8.6.6...v9.0.0) (2024-12-06)

* feat!: deprecation notices for non-readme-refactored commands ([#1099](#1099)) ([732e32b](732e32b))
* feat!: drop node 18 ([#1085](#1085)) ([ebc2ac0](ebc2ac0))
* feat!: final v9 touchups ([#1106](#1106)) ([15447c5](15447c5)), closes [#1107](#1107)
* feat!: oclif (take 2) ([#1068](#1068)) ([5e05f93](5e05f93)), closes [#962](#962) [#1067](#1067) [/github.com/readmeio/rdme/blob/d01d76fe3c2e4a98b4f5c415be03e589fbe3b56e/.releaserc.yml#L30](https://github.com//github.com/readmeio/rdme/blob/d01d76fe3c2e4a98b4f5c415be03e589fbe3b56e/.releaserc.yml/issues/L30) [#1067](#1067)
* feat!: remove `validate`, deprecate/hide `open` ([#1072](#1072)) ([f1b46f6](f1b46f6))
* feat!: switch topic separator to space ([#1100](#1100)) ([13eb8ab](13eb8ab))

### Bug Fixes

* add better debugging for fetch failures ([a17f8da](a17f8da))
* add GITHUB_TOKEN ([e106e10](e106e10))
* attempt to use semantic-release/github instead of `gh` ([331d28b](331d28b))
* build `dist-gha/` files with every release ([f42392b](f42392b))
* bump semantic-release versions ([21efc66](21efc66))
* debug properly ([#1078](#1078)) ([5de0a4f](5de0a4f))
* **deps:** bump more deps ([#1042](#1042)) ([786a000](786a000))
* empty commit to trigger build ([51c1566](51c1566))
* empty commit to trigger build ([76efb7e](76efb7e))
* empty commit to trigger release ([3604f2a](3604f2a))
* fixed debug script ([#908](#908)) ([82c3541](82c3541))
* hide `ExperimentalWarning` ([#901](#901)) ([f9b5679](f9b5679))
* **npm:** tweak npm lifecycle scripts ([#1073](#1073)) ([35534db](35534db))
* oclif-friendly error logging ([#1080](#1080)) ([73c5f4c](73c5f4c))
* **openapi:** add spinner catch statement ([#961](#961)) ([4669b29](4669b29))
* **release:** track `dist-gha` assets properly ([b0934cb](b0934cb))
* **release:** track changes to commands doc directory ([78b6554](78b6554))
* swap out ts-node for tsx ([16f64b3](16f64b3))
* Update LICENSE for 2024 ([#963](#963)) ([23cf44b](23cf44b))
* use newest import attributes syntax ([#993](#993)) ([d76c0ae](d76c0ae)), closes [/github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V20.md#2023-11-22-version-20100](https://github.com//github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V20.md/issues/2023-11-22-version-20100)
* **versions/update:** various edge cases ([#965](#965)) ([580e307](580e307))

### chore

* type fixes for tests ([#903](#903)) ([d423b4c](d423b4c))

### Code Refactoring

* remove `oas`, `swagger`, `docs:edit` ([#902](#902)) ([9107d15](9107d15))

### Features

* add `@oclif/plugin-not-found`, other cleanups ([#1074](#1074)) ([6529b03](6529b03))
* add `title` flag to `openapi:convert` ([#1071](#1071)) ([1d71f3f](1d71f3f)), closes [#1068](#1068)
* convert to ESM (breaking change) ([#856](#856)) ([84b8571](84b8571)), closes [/www.stefanjudis.com/snippets/how-to-import-json-files-in-es-modules-node-js/#option-1](https://github.com//www.stefanjudis.com/snippets/how-to-import-json-files-in-es-modules-node-js//issues/option-1) [1#L228](https://github.com/1/issues/L228)
* **fetch:** use undici + `ProxyAgent`  ([#1000](#1000)) ([9da7132](9da7132)), closes [#999](#999)
* **openapi/convert:** support openapi in convert command ([#941](#941)) ([a33bbeb](a33bbeb))
* revamped `oclif`-powered docs ([#1081](#1081)) ([8a2833b](8a2833b))
* sort when syncing by parent ([#973](#973)) ([9795840](9795840))
* support `README_API_KEY` env var ([#950](#950)) ([bcf3f18](bcf3f18))
* support node 18 and up ([#900](#900)) ([a217904](a217904))
* **test:** moving unit tests over to vitest ([#857](#857)) ([36d561b](36d561b)), closes [#870](#870)
* use `actions/checkout@v4` everywhere ([#895](#895)) ([d30b71c](d30b71c))
* **versions:** flag parity with API, copy fixes ([#906](#906)) ([d424d9f](d424d9f))

### BREAKING CHANGES

* `rdme openapi` is deprecated and will be replaced in
`rdme@10` by a command with a simpler flag setup based on community
feedback.
* deprecates commands that are not supported in ReadMe
Refactored. For more information, please visit our migration guide:
https://github.com/readmeio/rdme/tree/v9/documentation/migration-guide.md

## 🧬 QA & Testing

Does the copy in these deprecation warnings make sense to you? Note that
the links will be broken for now since we haven't tagged a proper v9
release yet, but that will be fixed once this release is out!
* the topic separator (i.e., what separates a command
from its subcommand) has changed from a colon to a space by default. For
example, `rdme openapi:validate` is now `rdme openapi validate`. The
colon topic separator will continue to be supported so there is no need
to change any existing commands, but all documentation and help screens
will reflect the space topic separator.

## 🧬 QA & Testing

Do tests still pass?
* dropping support for Node.js v18. The minimum required
Node.js version is v20.10.0.
* `rdme validate` has been removed, use `rdme
openapi:validate` instead.
* `rdme open` is now deprecated

also updates our docs accordingly
* the `rdme` GitHub Actions is now a [the `node20` JavaScript action](https://docs.github.com/en/actions/sharing-automations/creating-actions/metadata-syntax-for-github-actions#runs-for-javascript-actions) rather than [a Docker container action](https://docs.github.com/en/actions/sharing-automations/creating-actions/metadata-syntax-for-github-actions#runs-for-docker-container-actions).
* `rdme` is now powered by [`oclif`](https://oclif.io/). The formatting and content of certain error messages and outputs may have changed. Please continue to only utilize exit codes to determine command successes/failures.

completed tasks:

- [x] a handful of tests are still failing, hoping that
oclif/test#652 gets merged in)
- [x] github actions dry runs are failing (but i got them working in
* **versions:** this flips the `isPublic` flag to `hidden`.
* removes several deprecated commands

* chore: knip cleanup

* chore: type fixes for tests
* removes several deprecated commands
* Node.js >= 18 required

[skip ci]
@kanadgupta
Copy link
Member Author

🎉 This PR is included in version 9.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request refactor Issues about tackling technical debt released
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants