feat(compartment-mapper): Thread commonjs- and module- languageForExt…

…ension options (for TS) (#2625) Refs: #2415 ## Description Toward support for type-erasure style TypeScript support in Endo, `.ts` files may correspond to `.mts` or `.cts` behavior in the same way that `.js` can correspond to either `.mjs` or `.cjs`, depending on `"type":` in the enveloping `package.json` (which is not necessarily the enveloping package, e.g., `src/package.json` beneath `package.json`!). We already implement this machinery for JavaScript, so this change threads additional options beside `languageForExtension` (which would map `.mts` and `.cts` to language behaviors) but also `commonjsLanguageForExtension` and `moduleLanguageForExtension`, which would get folded into the language-for-extension mapping on a package-by-package basis. ### Security Considerations None. Absent from this design is any possibility that a file would be alternately implemented as TypeScript or JavaScript. ### Scaling Considerations None. ### Documentation Considerations Includes an update to README for the new features and the undocumented features it is based upon. Includes a mention in NEWS. ### Testing Considerations TODO ### Compatibility Considerations By design error, `languageForExtension` threaded through `link` before this change, when it should have been threaded through `mapNodeModules`, since that is the machine that is aware of the `type` field in `package.json`. This change moves that internal mechanism and then adds `commonjsLanguageForExtension` and `moduleLanguageForExtension`. By common usage, this move is transparent since the option is taken at the top and simply diverted internally. However, the `languageForExtension` property could be manually threaded through `makeArchiveLite` and omitted from the options for `mapNodeModules`. I’ve elected to treat this breaking change as a bug fix since I find it unlikely anyone has taken advantage of these relatively new “lite” functions. The “lite” functions are part of the refactoring necessary to approach native XS compartment and modules. ### Upgrade Considerations None.
endojs · Nov 12, 2024 · 110ab7c · 110ab7c
2 parents aa6d4b3 + 389de7b
commit 110ab7c
Show file tree

Hide file tree

Showing 37 changed files with 1,500 additions and 177 deletions.
diff --git a/packages/compartment-mapper/NEWS.md b/packages/compartment-mapper/NEWS.md
@@ -2,6 +2,19 @@ User-visible changes to `@endo/compartment-mapper`:
 
 # Next version
 
+- Adds options `languageForExtension`, `moduleLanguageForExtension`,
+  `commonjsLanguageForExtension`, and `languages` to `mapNodeModules` and
+  `compartmentMapForNodeModules` allowing for certain mappings from extension
+  (e.g., `ts`) to language (e.g., `mts` or `cts`) to depend on the each
+  package’s `type` in the way we already vary `js` between `cjs` and `mjs`.
+  These options enter through the high level functions including `makeArchive`
+  and `importLocation`.
+- The new options `workspaceLanguageForExtension`,
+  `workspaceModuleLanguageForExtension`, and
+  `workspaceCommonjsLanguageForExtension` apply like the above except more
+  specifically and for packages that are not physically located under a
+  `node_modules` directory, indicating that JavaScript has not yet been
+  generated from any non-JavaScript source files.
 - Omits unused module descriptors from `compartment-map.json` in archived
   applications, potentially reducing file sizes.
 - Fixes an issue where errors thrown from exit module hooks (`importHook`) would

diff --git a/packages/compartment-mapper/README.md b/packages/compartment-mapper/README.md
@@ -204,37 +204,15 @@ does not exist, to the `index.js` file in the directory with the same name.
 > `fetch` global, in conjunction with usable values for `import.meta.url` in
 > ECMAScript modules or `__dirname` and `__filename` in CommonJS modules.
 
+## Language Extensions
+
 Officially beginning with Node.js 14, Node.js treats `.mjs` files as ECMAScript
 modules and `.cjs` files as CommonJS modules.
 The `.js` extension indicates a CommonJS module by default, to maintain
 backward compatibility.
 However, packages that have a `type` property that explicitly says `module`
 will treat a `.js` file as an ECMAScript module.
 
-This unforunately conflicts with packages written to work with the ECMAScript
-module system emulator in the `esm` package on npm, which allows every file
-with the `js` extension to be an ECMAScript module that presents itself to
-Node.js as a CommonJS module.
-To overcome such obstacles, the compartment mapper will accept a non-standard
-`parsers` property in `package.json` that maps file extensions, specifically
-`js` to the corresponding language name, one of `mjs` for ECMAScript modules,
-`cjs` for CommonJS modules, and `json` for JSON modules.
-All other language names are reserved and the defaults for files with the
-extensions `cjs`, `mjs`, `json`, `text`, and `bytes` default to the language of
-the same name unless overridden.
-JSON modules export a default object resulting from the conventional JSON.parse
-of the module's UTF-8 encoded bytes.
-Text modules export a default string from the module's UTF-8 encoded bytes.
-Bytes modules export a default ArrayBuffer capturing the module's bytes.
-If compartment mapper sees `parsers`, it ignores `type`, so these can
-contradict where using the `esm` emulator requires.
-
-```json
-{
-  "parsers": {"js": "mjs"}
-}
-```
-
 Many Node.js applications using CommonJS modules expect to be able to `require`
 a JSON file like `package.json`.
 The compartment mapper supports loading JSON modules from any type of module.
@@ -252,9 +230,54 @@ As of Node.js 14, Node does not support loading ECMAScript modules from
 CommonJS modules, so using this feature may limit compatibility with the
 Node.js platform.
 
-> TODO A future version may introduce language plugins, so a package may state
-> that files with a particular extension are either parsed or linked with
-> another module.
+The compartment mapper supports language plugins.
+The languages supported by default are:
+
+- `mjs` for ECMAScript modules,
+- `cjs` for CommonJS modules,
+- `json` for JSON modules,
+- `text` for UTF-8 encoded text files,
+- `bytes` for any file, exporting a `Uint8Array` as `default`,
+- `pre-mjs-json` for pre-compiled ECMAScript modules captured as JSON in
+  archives, and
+- `pre-cjs-json` for pre-compiled CommonJS modules captured as JSON in
+  archives.
+
+The compartment mapper accepts extensions to this set of languages with
+the `parserForLanguage` option supported by many functions.
+See `src/types/external.ts` for the type and expected behavior for
+parsers.
+
+These language identifiers are keys for the `moduleTransforms` and
+`syncModuleTransforms` options, which may map each language to a transform
+function.
+The language identifiers are also the values for a `languageForExtension`,
+`moduleLanguageForExtension`, and `commonjsLanguageForExtension` options to
+configure additional extension-to-language mappings for a module and its
+transitive dependencies.
+
+For any package that has `type` set to `"module"` in its `package.json`,
+`moduleLangaugeForExtension` will precede `languageForExtension`.
+Packages with `type` set to `"commonjs"` or simply not set,
+`commonjsLanguageForExtension` will precede `languageForExtension`.
+This provides an hook for mapping TypeScript's `.ts` to either `.cts` or
+`.mts`.
+
+The analogous `workspaceLanguageForExtension`,
+`workspaceCommonjsLanguageForExtension`, and
+`workspaceModuleLanguageForExtension` options apply more specifically for
+packages that are not under a `node_modules` directory, indicating that they
+are in the set of linked workspaces and have not been built or published to
+npm.
+
+In the scope any given package, the `parsers` property in `package.json` may
+override the extension-to-language mapping.
+
+```json
+{
+  "parsers": { "png": "bytes" }
+}
+```
 
 > TODO
 >

diff --git a/packages/compartment-mapper/src/archive-lite.js b/packages/compartment-mapper/src/archive-lite.js
@@ -322,15 +322,11 @@ const digestFromMap = async (powers, compartmentMap, options = {}) => {
     policy = undefined,
     sourceMapHook = undefined,
     parserForLanguage: parserForLanguageOption = {},
-    languageForExtension: languageForExtensionOption = {},
   } = options;
 
   const parserForLanguage = freeze(
     assign(create(null), parserForLanguageOption),
   );
-  const languageForExtension = freeze(
-    assign(create(null), languageForExtensionOption),
-  );
 
   const { read, computeSha512 } = unpackReadPowers(powers);
 
@@ -365,7 +361,6 @@ const digestFromMap = async (powers, compartmentMap, options = {}) => {
     makeImportHook,
     moduleTransforms,
     parserForLanguage,
-    languageForExtension,
     archiveOnly: true,
   });
   await compartment.load(entryModuleSpecifier);

diff --git a/packages/compartment-mapper/src/archive.js b/packages/compartment-mapper/src/archive.js
@@ -47,7 +47,8 @@ const assignParserForLanguage = (options = {}) => {
   const parserForLanguage = freeze(
     assign(create(null), defaultParserForLanguage, parserForLanguageOption),
   );
-  return { ...rest, parserForLanguage };
+  const languages = Object.keys(parserForLanguage);
+  return { ...rest, parserForLanguage, languages };
 };
 
 /**
@@ -61,12 +62,16 @@ export const makeAndHashArchive = async (
   moduleLocation,
   options = {},
 ) => {
-  const compartmentMap = await mapNodeModules(powers, moduleLocation, options);
-  return makeAndHashArchiveFromMap(
-    powers,
-    compartmentMap,
-    assignParserForLanguage(options),
-  );
+  const { parserForLanguage, languages, ...otherOptions } =
+    assignParserForLanguage(options);
+  const compartmentMap = await mapNodeModules(powers, moduleLocation, {
+    languages,
+    ...otherOptions,
+  });
+  return makeAndHashArchiveFromMap(powers, compartmentMap, {
+    parserForLanguage,
+    ...otherOptions,
+  });
 };
 
 /**
@@ -76,20 +81,41 @@ export const makeAndHashArchive = async (
  * @returns {Promise<Uint8Array>}
  */
 export const makeArchive = async (powers, moduleLocation, options = {}) => {
-  const { dev, tags, conditions = tags, commonDependencies, policy } = options;
-
+  const {
+    dev,
+    tags,
+    conditions = tags,
+    commonDependencies,
+    policy,
+    languageForExtension,
+    commonjsLanguageForExtension,
+    moduleLanguageForExtension,
+    workspaceLanguageForExtension,
+    workspaceCommonjsLanguageForExtension,
+    workspaceModuleLanguageForExtension,
+    parserForLanguage,
+    languages,
+    ...otherOptions
+  } = assignParserForLanguage(options);
   const compartmentMap = await mapNodeModules(powers, moduleLocation, {
     dev,
     conditions,
     commonDependencies,
     policy,
+    languageForExtension,
+    commonjsLanguageForExtension,
+    moduleLanguageForExtension,
+    workspaceLanguageForExtension,
+    workspaceCommonjsLanguageForExtension,
+    workspaceModuleLanguageForExtension,
+    languages,
   });
 
-  return makeArchiveFromMap(
-    powers,
-    compartmentMap,
-    assignParserForLanguage(options),
-  );
+  return makeArchiveFromMap(powers, compartmentMap, {
+    parserForLanguage,
+    policy,
+    ...otherOptions,
+  });
 };
 
 /**
@@ -99,16 +125,42 @@ export const makeArchive = async (powers, moduleLocation, options = {}) => {
  * @returns {Promise<Uint8Array>}
  */
 export const mapLocation = async (powers, moduleLocation, options = {}) => {
-  const { dev, tags, conditions = tags, commonDependencies, policy } = options;
+  const {
+    dev,
+    tags,
+    conditions = tags,
+    commonDependencies,
+    policy,
+    parserForLanguage,
+    languages,
+    languageForExtension,
+    commonjsLanguageForExtension,
+    moduleLanguageForExtension,
+    workspaceLanguageForExtension,
+    workspaceCommonjsLanguageForExtension,
+    workspaceModuleLanguageForExtension,
+    ...otherOptions
+  } = assignParserForLanguage(options);
 
   const compartmentMap = await mapNodeModules(powers, moduleLocation, {
     dev,
     conditions,
     commonDependencies,
     policy,
+    languages,
+    languageForExtension,
+    commonjsLanguageForExtension,
+    moduleLanguageForExtension,
+    workspaceLanguageForExtension,
+    workspaceCommonjsLanguageForExtension,
+    workspaceModuleLanguageForExtension,
   });
 
-  return mapFromMap(powers, compartmentMap, assignParserForLanguage(options));
+  return mapFromMap(powers, compartmentMap, {
+    parserForLanguage,
+    policy,
+    ...otherOptions,
+  });
 };
 
 /**
@@ -118,16 +170,42 @@ export const mapLocation = async (powers, moduleLocation, options = {}) => {
  * @returns {Promise<string>}
  */
 export const hashLocation = async (powers, moduleLocation, options = {}) => {
-  const { dev, tags, conditions = tags, commonDependencies, policy } = options;
+  const {
+    dev,
+    tags,
+    conditions = tags,
+    commonDependencies,
+    policy,
+    parserForLanguage,
+    languages,
+    languageForExtension,
+    commonjsLanguageForExtension,
+    moduleLanguageForExtension,
+    workspaceLanguageForExtension,
+    workspaceCommonjsLanguageForExtension,
+    workspaceModuleLanguageForExtension,
+    ...otherOptions
+  } = assignParserForLanguage(options);
 
   const compartmentMap = await mapNodeModules(powers, moduleLocation, {
     dev,
     conditions,
     commonDependencies,
     policy,
+    languages,
+    languageForExtension,
+    commonjsLanguageForExtension,
+    moduleLanguageForExtension,
+    workspaceLanguageForExtension,
+    workspaceCommonjsLanguageForExtension,
+    workspaceModuleLanguageForExtension,
   });
 
-  return hashFromMap(powers, compartmentMap, assignParserForLanguage(options));
+  return hashFromMap(powers, compartmentMap, {
+    parserForLanguage,
+    policy,
+    ...otherOptions,
+  });
 };
 
 /**
@@ -144,18 +222,40 @@ export const writeArchive = async (
   moduleLocation,
   options = {},
 ) => {
-  const { dev, tags, conditions = tags, commonDependencies, policy } = options;
+  const {
+    dev,
+    tags,
+    conditions = tags,
+    commonDependencies,
+    policy,
+    parserForLanguage,
+    languages,
+    languageForExtension,
+    commonjsLanguageForExtension,
+    moduleLanguageForExtension,
+    workspaceLanguageForExtension,
+    workspaceCommonjsLanguageForExtension,
+    workspaceModuleLanguageForExtension,
+    ...otherOptions
+  } = assignParserForLanguage(options);
   const compartmentMap = await mapNodeModules(readPowers, moduleLocation, {
     dev,
     conditions,
     commonDependencies,
     policy,
+    languageForExtension,
+    commonjsLanguageForExtension,
+    moduleLanguageForExtension,
+    workspaceLanguageForExtension,
+    workspaceCommonjsLanguageForExtension,
+    workspaceModuleLanguageForExtension,
+    languages,
   });
   return writeArchiveFromMap(
     write,
     readPowers,
     archiveLocation,
     compartmentMap,
-    assignParserForLanguage(options),
+    { policy, parserForLanguage, ...otherOptions },
   );
 };