Realms API ECMAScript Proposal #542
Comments
leobalter
added
Progress: untriaged
Review type: CG early review
Note that Bloomberg has been funding my (Daniel Ehrenberg, Igalia) work in TC39 generally. |
|
I updated the post to reflect this, @littledan please let me know if this is fine. |
|
Note, I think this makes sense to classify as a "specification review" rather than an "early review", as we have full semantics and a specification with semantics proposed, modulo editorial issues in HTML. I hope that we can discuss this proposal at an upcoming TC39 meeting. Getting a TAG review would be very useful prior to that the discussion. There's one TC39 meeting starting September 21st, and another starting November 16th. Please let us know if there's anything we can do to provide information or context to help the TAG. |
|
I don't think it's accurate to classify the issues with HTML as "editorial"; they cut to the heart of the proposal, and as to whether this proposal is something we'd want to welcome on the web at all, and if so, how it would integrate in a way that deeply cross-cuts fundamental pieces of the web architecture around code-loading, security (CSP etc.), and global objects. |
|
We have tried to make clear what is being proposed, and I don't know of any ambiguity about these details. Either way, I'd love to hear more from the TAG and others about these design concerns, c.f. tc39/proposal-shadowrealm#238. I have no strong opinion on this issue's labels. |
|
@plinss @hadleybeeman and I looked at this in the TAG breakout today. We are generally happy to see this happen, but would like a bit more clarification, like for instance what is exposed to the Realm object? (see below)
So do Realms only have the ECMAScript APIs available? Doesn't this mean that most libraries won't work unless to add its dependencies manually, like realm.globalThis.fetch = fetch. Like we could see people using this even to isolate WebAssembly code, thought that requires you adding the methods needed for that. We are also a bit afraid that regular developers will have a hard time understanding all these concepts (realms, globals, this) and how they relate to each other: realms, like what is a realm really, especially since the top-level realm (like the one with window === globalThis) cannot be accessed as a Realm object. Maybe for consistency sake it would make sense to have an accessor to expose it as a realm, thought currently the only thing exposed is globalThis and import - but we assume that could be extended in the future. The explainer talks about Compartsments (link returns 404 - https://github.com/tc39/proposal-realms/blob/main) but it would be nice with a quick into to that work and how all of this relates. |
Thanks!
Yes! It only exposes a new copy of the built-ins from ECMAScript, but it allows extensions defined by each host.
Absolutely, this is equivalent to what happens to Node VM today as a low level API prior art. As a developer you need to setup the environment to execute code. Ideally the Realms would arrive a clean state, allowing tailoring for what is necessary to be added. This contrasts with the tailoring over unforgeables. e.g. Considering all the trade offs, the clean state seems the best option, in our opinion. It allows tailoring for multiple purposes and comprehends more use cases.
Executed code doesn't need to know it's in a realm, this is designed to be a concern for those setting the realm up. Ideally, code executed in a realm would run seamlessly. There is prior art for this (iframes, Workers, node.vm).
The initial Realms proposal had more content and more ways to access things. We tried to build a MVP and hope we can explore expansions of the API in the future.
Thanks for catching that up! The correct link is here: https://github.com/tc39/proposal-compartments. Compartments is a more complex API that offers tailoring over aspects beyond the global APIs but with modifications to internal structure such as module graph. The Realms API just offers immediate access to what is already specified in ECMAScript as it's already structured to distinguish different references from realms. I'm looking forward to continue this conversation. Thanks for the feedback! |
|
One thing to understand here is that Realms are generally intended to be a sort of metaprogramming construct, which would be used by frameworks and libraries to build emulated JS environments for developers. I understand the feedback that this concept may be difficult for JS developers to understand; probably an introduction in the explainer to show how múltiple globals in JS already work would help make this document more accessible. Either way, it is an underlying primitive in the platform.
Trying to understand this comment--what would this accessor be? What does "it" refer to--are you suggesting making an accessor named Does the TAG have any thoughts about tc39/proposal-shadowrealm#238 ? |
|
I copied my answers above to the current explainer. I'd be happy to set more additional topics there or expand anything there, please just let me know. |
|
Pinging @torgo and @plinss, Hi! I'd appreciate a lot if we could have a follow up before the next TC39 meeting in November 15th 2020, is there any chance to fit this into the schedule? I know we have TPAC coming in and I understand all the time constraints, so I understand if we time limits and constraints here. Thanks! |
|
Note that this proposal is on the TC39 agenda to be discussed next week for Stage 3. Further TAG feedback would be welcome, especially if you're available to review before the meeting. I've also iterated on the HTML integration proposal at whatwg/html#5339 to try to resolve the issues that @domenic mentioned. |
Could you give any examples of what host extensions you expect hosts to add? I assume a host here could be something like node.js or the browser. |
|
@domenic I'd be curious if there's a summary somewhere of your current set of concerns with the proposal. Seems like you had a clear list of concerns back in March, but it's not clear how many of them are current. Is it still the last one there that's the major issue? |
|
@leobalter or @littledan what are the new communication channels that will be possible with having this new capability? Having read the explainer and security questionnaire I couldn't find a clear answer if they will be less or more compared to what is available today. |
Thanks for the ping. Currently my concerns, in order of largest to smallest, are:
|
|
hey @domenic thanks for reinstating your concerns, I will try to do my best to address them:
We (the champions) have been very clear about this for a long time, Realms are not a security boundary. If you want a security boundary, you go async, where Realms can be complementary if you decide to slice and dice the evaluation of code inside that process. We have addressed this concern to make sure that when we use the term "3rd party code" we say "trusted code". This proposal is simply trying to formalize something that is available across all platforms that are implementing the language. Which in many of them, it is extremely difficult. Let me list them here yet one more time:
In each of them, it is harder and harder to achieve the same, for no particular reason. This proposal attempts to normalize this across the board, and fix the drawbacks of using the same domain iframe. For these reasons, I will disregard this concern as subjective.
As we have debated in the past, going async is not an option for an architecture that attempt to provide any virtualization between trusted sides in a language that is primarily sync (a good example here is Google AMP DOM virtualization project). As I mentioned above, we see Realms as a complement of the architectures that you listed above.
This concern is a solid concern, it is not our place at TC39 to dictate what the web platform should do, or not. My personal opinion is that this ship has sailed a long time ago with the surge of nodejs, and the respective native platforms exposing V8 and JSCore to developers. The de-facto distribution model for JS code via NPM already highlight this issue extensibly, and developers, and more important, the tools available for developers, have helped to mitigate this in some extend.
This is not accurate, you do not have to enable eval to use Realms, nor it will encourage to do so. How will a program (running on the web) evaluates another piece of code? what are the available mechanisms to do so? You have script injection (considered legacy at this point), dynamic import, and eval. In a Realm, you will have a subset of that, you have dynamic import (via
I'm not an expert on this subject, but this has been extensibly debated by other folks, and as far as I can tell, they believe this is not as complex as you think it is. I will let others to counter this argument. |
|
To clarify about hosts adding properties to Realms' global objects: this is currently not planned for HTML or Node.js. The specification recommends against adding properties, and actually we're considering prohibiting it, in tc39/proposal-shadowrealm#284. Instead, Realms contain only the JavaScript built-ins, but you can add more properties from JavaScript code. |
You can try to be clear about it, but it's not working. E.g. there is a separate proposal, titled "Secure ECMAScript", which uses realms as the basis of its "security". Or there are people trying to use realms for security boundaries, and getting burned, as seen in e.g. https://www.figma.com/blog/an-update-on-plugin-security/ . If a feature encourages writing insecure code, you can't just say "but we told you not to write insecure code" and use that as justification for adding it to the platform anyway. That is why I think that people who want integrity via multiple globals should continue to use the power tools that are available in their environments, and should not get support from this footgun-laden API being baked into the platform.
I strongly disagree with this. It is "hard" (e.g., environment-specific) for very good reasons, which I've listed above.
This concern is my strongest one, and certainly not subjective. Adding something which encourages buggy and insecure code to the language---not just the V8 API, or the Node.js |
I would like to make the counterpoint that at the least it would be unified "buggy and insecure code" that could be built upon and is a improvement from current hacks to achieve the same goal which people are failing at anyway. Now I am not a expert on this proposal (and could someone tag me with a correction if I am wrong.) but it seems to achieve similar results to what is described in this post. As can be seen in the post it is a mess that most people (including myself) don't actually understand. Does it make a eval that is more secure? Yes. But if only 1% of programmers use it it is useless. It doesn't matter if Realms is only partially secure if people are using a i-frame, (or just a normal eval,) anyway. Using this at least makes it clear what and why the code is working in the way it's working. As I understand it part of this groups remit is to improve readability. The solution may be as simple as renaming the stupid thing to make is sound less like a sandbox.
If I understand correctly, using realms in a web-worker would solve many of these issues (oh, and you can't have i-frames in web workers, but you could have a realm in a web worker). Additionally, there are varying levels of bad idea, there is "running plugins in a bank app" all the way down to "Modding a single player web game". |
|
@kenchris @LeaVerou and I looked at this during a breakout this week and we have some questions (I have some follow-on questions I'll post in a separate comment):
We didn't have the time to delve into compartments properly, but what's the functional difference between a Realm and a Compartment under the new proposal? |
|
Follow on personal questions about state management and crossing Realm boundaries. I understand that if Here's an example, consider the following (off the cuff, untested) code being imported into a Realm: I then do: What gets logged? '1 1', '1 2', or ? |
|
Also, I've had a use-case for Realms for several years now and I'm not sure the current approach still allows me to do what I want to do... A while back I designed an extension to Home Documents for HTTP APIs that adds JS function bindings to HTTP API endpoints. The idea is that a browser can load the document describing the HTTP API, auto-generate a class that implements the API as JS methods, and return an instance of that class ( a 'remote object'). The consumer of the API just sees a regular JS object, whose implementation comes from the server (each method returns a promise that does a Fetch under the hood as defined by the Home Document). The Home Document can also carry internal state exposed as properties on the remote object (in addition to private state), and that state can be manipulated by the HTTP API by returning a JSON-Patch. I also built a polyfill that implements this (this repo is a bit out of date, I've been using a more recent version in production code, but it gets the idea across): https://github.com/plinss/remote-web-objects (the live-demo is no longer online) One feature I wanted to add is to allow the Home Document to also carry raw JS code that implements synchronous methods run entirely client side. That code should be restricted to interact with the remote object and nothing else. I was planning on being able to create a Realm that's scoped to the remote object. Doing this would require the ability to share that object's internal state between the code running in the realm and the Realm's parent. It's not clear if this kind of thing can still be built with the current proposal. |
|
Thank you so much for the review, @kenchris, @LeaVerou, and @plinss!
await importValue('./file.js', 'default');
We had this question in other channels. At first glance, we think about setting much of the control directly from Incubator Realm to Child Realm, but the modules injection can also be controlled with a module in between. The example below uses // ./inside-code.js
import { start, getTapReport } from 'test-runner';
import './test-file.js';
export default function(cb) {
start()
.then(getTapReport)
.then(report => cb(report.toString()));
}// ./main.js
const r = new Realm();
const log = console.log.bind(console);
const runTests = await r.importValue('./inside-code', 'default');
runTests(log);It's good to note that, anyway, consecutive
Yes! If we don't have a full usage picture, that's our initial intuition. Although, this demands more implementation details I was looking for in // ./main.js
const r = new Realm();
async function importValues(realm, specifier, bindingList) {
return Promise.all([].map.call(bindingList, bindingName => realm.importValue(specifier, bindingName)));
}
const [ padLeft, padRight ] = await importValues(r, './str-tools.js', ['padLeft', 'padRight']);I'll answer the next questions in follow up comments here. |
Only primitive values are fully transferable. Any try to transfer Non-callable objects will cause an abrupt completion (thrown exception). Callable Objects are internally wrapped into a new exoctic callable, called "Wrapped Function Exotic Object". That means, if the importValue or the result of This means, if I have incubator Realm A, and child Realm B, and I inside A I run When Realm A calls this new exotic object, it synchronously calls the evaluated arrow function from B, captures the return value, and return it in Realm A. // Realm A
const B = new Realm();
// Realm B creates an arrow function and returns it
// fn is a Wrapped Exotic object that as an internal [[Wrapped]] containing the arrow function
const fn = B.evaluate('x => x * 2');
// This call will internally call fn.[[Wrapped]](3). The result is a primitive, return it.
fn(3); // 6There is no function unwrapping in user code, and this is a hard requirement to avoid leaking identities. This means each time I evaluate something that returns the same callable, I always receive a new Wrapped Function Exotic Object. // Realm A
const B = new Realm();
B.evaluate('globalThis.fn = x => x * 2');
B.evaluate('fn === fn'); // true
const wrapped = B.evaluate('fn');
const wrappedAgain = B.evaluate('fn');
console.log(wrapped === wrappedAgain); // false
wrapped(3); // 6
wrappedAgain(3); // 6, they are both callables from Realm A connected to the same function in Realm BThis also means we wrap functions the other way around, with no identification: // Realm A
const B = new Realm();
B.evaluate('globalThis.fn = x => x * 2');
const wrapped = B.evaluate('fn');
const compare = B.evaluate('callable => callable === fn');
compare(wrapped); // false
const verify = B.evaluate('callable => callable(7)');
verify(wrapped); // 14In the example above, The |
You're not alone. This functionality reflects what we've been trying to push forward for so long until we had to find an alternative with this current callable boundary API that still resolves most use cases, but - as you point out - not all of them. It is hard for me to tell you that you could try using a membrane framework that gives a better sense of object identities and injection, but I feel this has a very steep learning curve and cost for initial implementation. The membranes systems works for us in this callable boundary realms API and many other orgs and projects already using membranes. Unfortunately, we faced pushback - as you can even find in this thread - about giving object access cross realms. Although, the object access already exist today in the web platform through iframes. I believe @gwhitworth and @caridy might wanna say more about this. |
Yes, this is analogous to import, which means you can call that method multiple times for the same specifier, and you get the same module. And yes, you can use Promise.all, etc. to try to construct the object in the incubator realm that contains access to various exported values.
As @leobalter mentioned, it is a hard stop, throwing.
Yes, you get two different exotic objects, both bound to
You can't return an object, that will throw, but if you return another callable, yes, every time you return that ref, the other side gets a new exotic object. Basically there is no identity preserving semantics here, that can happen in user-land with a fancy membrane. Another good example here is when you pass a function to the other side, and the other side calls you back with the same reference that they have received, you get a new wrapped exotic object. Double wrapping can occur at any given time, while implementers will be able to optimize this to avoid going over multiple jumps to evaluate the target function.
It will log |
|
Thanks for all the (quick!) feedback. I'm going to have to think this through some more. One more quick question, what if the callable returned from a realm is a constructor? e.g. what happens if have: and I do: |
|
In this example, const rArray = realm.evaluate('Array');
try {
rArray(); // would return a new Array from the other realm
} catch(e) {
e.constructor === TypeError;
}This happens because the low level code abstraction just observes the existence of a There is a curiosity for this case using |
|
Adding more for the curiosity, if you call this specific const Foo = realm.importValue('module.js', 'Foo');
try {
Foo();
} catch(e) {
e.constructor === TypeError;
} |
|
Thanks, I suspected it would fail, I was mostly curious how... |
|
@leobalter Thank you for the quick responses! How does one import named (or default!) exports with different names than the ones they were exported with? That's super common with modules. Also, how can someone do the equivalent of |
|
Hi @LeaVerou!
In this case you just import the value contained from a binding name. const someDifferentName = await r.importValue('./file.js', 'originalName');
// roughly equivalent to:
import { originalName as someDifferentName } from './file.js';
As pointed out in the last example in this comment, user land code would need to serialize multiple names. I fully understand the concern and that's something we considered. Although, there are some differences for operations between realms. First, for
For most of the Realms use cases, relating to controlled execution of a code inside a child realm, you can rely on communication channels which can translate to a few functions. The other ideal, which I believe match your concerns, would be a direct access to the imported module namespace. That's actually part of the previous and "original" Realms proposal, where we had a |
|
Just to make sure we get this point across, the goal is not to allow you to do all the things we do with dynamic import, the goal is to provide a low level api that allow you to implement such behavior in user-land. A good example of it was discussed in the last tc39 about the enumerability of the exported names, and how a program might want to iterate over them, and how you could do the same with this Realm API. Our answer to those questions is always the same: You will be able to achieve that with a little bit of preparation work if you know what you're importing inside the realm, in principle you can do that by creating a wrapping module that does the work, but in the future, proposals like module blocks will provide a lot of flexibility to describe this behavior inline. |
torgo
added
Progress: propose closing
|
Hi all, Our understanding is that TC39 is contemplating taking up an alternative proposal, in which the only method of cross-Realm communication is via callables. We'd like to review whichever alternative wins out. Please do come back to us when a decision has been reached & we'll take another look. We hope that whichever alternative wins out, the result is easy for developers to work with. |
|
@hober it would likely help us choose an alternative to get the TAG's opinions on one versus the other. |
Saluton TAG!
I'm requesting a TAG review of TC39's Realms API.
Realms are a distinct global environment, with its own global object containing its own intrinsics and built-ins. The Realms proposal provides a new mechanism to execute JavaScript code within the context of a new global object and set of JavaScript built-ins. The Realm constructor creates this kind of a new global object.
Further details:
You should also know that...
We'd prefer the TAG provide feedback as (please delete all but the desired option):
(We are also fine to reuse this current issue)
CAREFULLY READ AND DELETE CONTENT BELOW THIS LINE BEFORE SUBMITTING
Please preview the issue and check that the links work before submitting.
In particular:
¹ For background, see our explanation of how to write a good explainer. We recommend the explainer to be in Markdown.
² Even for early-stage ideas, a Security and Privacy questionnaire helps us understand potential security and privacy issues and mitigations for your design, and can save us asking redundant questions. See https://www.w3.org/TR/security-privacy-questionnaire/.
The text was updated successfully, but these errors were encountered: