Agoric SES, 28 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SES-mtg: 28 October 2020 Part 2

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

um uh I think, let's just unambiguously refer to that as the load hook.

B

Yes, where only.

A

Is the term import when it also losses.

B

That's right, yeah. I I feel pretty strongly about that too um yeah this this this uh you know in order to not presume that consensus existed on that issue. Already, when I read wrote the slides I um I left it as import hook, so that there was some clarity, but I do think so that we should does anybody object to renaming import hook to load hook um in the future are going forward, at least in the scope of this conversation.

C

I think it's clarifying.

B

Okay, cool, um uh I will point out and replace import hook on these slides, live uh and I'll go back and edit them later.

D

You you said something that was a bit confusing to me. You said the referrer is different.

B

D

Redirect, yes, that is not currently how we implement it.

B

The um the the purpose- um I don't wish to overload so bradley you're, referring to the distinction between um you're you're, referring to the distinguished the a distinction between the module specifier, is requested and the module specifier as obtained.

D

Well, I guess, but really it's about the uh nature of. Is this an atomic operation?

D

um If, if we do this route of redirects, where you intercept each redirected is not an atomic operation.

D

So currently, whenever we go through a load uh in node, um we must resolve to a full path. We can't resolve to a redirect that is re-entering on the loader.

D

Is this, uh as I understand it, uh able to be re-entrant.

B

It's not. The expectation is that you return uh that you must return in your from from the load hook must return, not just it must return, both the information of the redirect and the uh and the static module record of the module that you were redirected to it. Can't it cannot result in a re-entrance into the load hook.

D

Okay, so that that was my primary concern and then it seems like if you supply compartment, you cannot supply specifier.

B

No, they are not mutually exclusive um and- and I only had compartment for completeness- we didn't need it.

D

Because if, if you specify both that seems a bit confusing.

B

I see what you're saying yeah um that the specifier in the target compartment- I I um I will uh find out what the escape button is on this computer.

B

And I I think that this.

D

Yeah, that seems fine. We actually had an older uh spec for loaders. That effectively did this, so that's actually kind of nice to see. uh We had specifier be optional. Originally in that and then after designs, we made it not optional. We never shipped that api, but uh yeah.

B

How do you solve the problem without uh how do you solve the problem of nodes redirect um for in I mean there are other cases I know like symbolic links or http redirects, but just for example.

B

How do you do the name to name slash index.js today.

D

uh So that's done during resolution and resolution is done uh prior to evaluation prior to parsing. Even and so here you would perform resolution, and that is how you obtain your static module and then you must provide the absolute uh url in our case, uh which is effectively the full specifier.

D

um And then there.

A

D

Normalization that node does on that. Actually, when you provide a uh the absolute url, it does real pathing by default.

B

Yeah and real path is io, so it has to be well, it doesn't have to be async, but it should be.

D

uh It is currently not async.

D

When we benchmarked it, it was actually faster, of course, of course,.

B

Yeah, I I remember having a conversation with brian cantrell um uh at a conference and and one of the points that he made was well at that time it was still. uh There was still. It was still raw.

B

The common js was adopted by node and the the downside purportedly was that common gs at least appears to be a synchronous api and by node is implemented as as a synchronous api, um and the downside of that is like sure some there's some web portability issues, or at least it was perceived as a time to be limiting web portability and brian's uh brian's.

B

Take on this because brian's coming from a systems programming pro this was ryan bell's boss at the time um and uh at giant and uh his point was well you're- never going to asynchronously dynamically load, uh um uh a dynamic library. So what's the point in worrying about it,.

D

Yeah, so we actually do have an outstanding issue with upgrades to move stuff to asynchronous uh forms, but uh the cost of crossing the c plus plus barrier. Until recently, it was just too drastic to ever think of using asynchronous programming, because you cross the barrier more often.

B

Yeah, oh it I. I think that that is fine as an optimization. It certainly is not contrary to the spirit of the thing. As long as the compartment api allows you to make that choice by um allowing for the load hook to be async, the the problem is that the resolve hook must be synchronous, but I don't think that that's actually a problem, because node the node resolver, is something that takes place underneath the locate phase within.

D

The workbook, so our resolve actually has an asynchronous contract. Yes, and you can do asynchronous operations, it's just the default. Implementation is always synchronous.

B

Yeah uh well, what I'm suggesting is that the resolve hook as proposed for in the compartment api is synchronous, but it is not the same thing as the the as the resolve function in in node, correct, yes, um and that if you there's, certainly no harm in doing a logical resolution before doing a I o based location um like I, I would call nodes um resolve hook to locate in the context of the compartment api.

B

A

B

A

What are the arguments.

B

uh Import hook only receives a full specifier: okay, yeah um yeah, okay. uh So it's it sounds like this is this is um it sounds like there wouldn't that this would not meet objections? Introducing this the this aliasing feature.

B

All right, we'll uh I'll, create the issue and and um and eventually get around to writing a pr for this. uh This change to the api.

B

um Okay, global lexicals, this one is going to be long and tedious. I suspect.

B

um So the motivator, the motivation for this proposal is that at agoric we needed a way um to introduce lexically scoped variables into a compartment that were not uh that were not discoverable by enumerating the properties of the global object, and the reason for this is to in is in order to create metering in. In our particular case, the metering is uh metering is a similar use case to um instrumentation um through a code transform.

B

So so then, because that's more familiar I'll describe that, instead of metering. um So uh if you were using a tool like nyc with istanbul to do code coverage analysis of a program, you would do that by doing two things with the compartment api. You would do that with two things one would be by introducing um by for one uh introducing a global, uh introducing an object in the scope of every module.

B

That is a dumping ground where uh coverage information can be recorded by the text of that module, and then you would introduce a transform that converts every expression and statement into um first first record. I got here on that global object um and then execute the the expression that was actually intended um and like using the comma operator, for example, and.

B

That is, that is sufficiently that particular use case is sufficiently served by adding a coverage object to the globalness of the compartment. Metering, on the other hand, is not is very similar in the case and because that there you're adding you are in a transform code, transform introducing statements that record. I got here so that uh so that those so that execution can be halted um if it runs too long, um the but the tr.

B

The problem with that approach is that for metering, that global object must not be tamper, discoverable or temperable by the program that is under metering, and in order to do that, it was necessary for us to introduce it as a lexically scoped variable that is not discoverable on the global object, and we did that by creating an enter scope around the global. This.

B

B

I say on the slash bullet point necessary for rebels, uh which is not actually correct. um This is this is no longer true. What's necessary, for rebels is a similar feature, but not the same. um Repels uh have this rule that if you do successive evaluation in sloppy mode, um that any constants that are declared by a previous eval are preserved for a subsequent eval.

B

This was previously discussed. I believe uh pr people in in previous conversations with tc39 that was called the global contour, which is a name that everybody hates, but is this the handle that sticks for now and is distinct from global lexicals.

A

Well, you say it is distinct from global obstacles. It is potentially okay. What we're proposing uh is only one concept: we're proposing to support global lexicals by supporting the global contract, correct.

B

um Possibly it is the the diff, uh the shim.

B

There is I, I haven't fully reasoned, whether they can be reconciled.

A

Okay, I know that the shim right now doesn't support the global contour, but everything about the way it supports global lexicals, I believe, is all necessary.

A

uh You know uh in order to support the global contour, and I would certainly object to introducing two um two such odd concepts. One such odd concept is unavoidable, because it's in the language as it exists right now, in order for apartments to fit with the full language outside of sets. We need to support the global countries contras and really you know, I would consider it really really quite terrible if we found we could not do what we needed to do with global lexicals, using the global contra mechanism.

B

I agree completely. I I just cannot propose that at this time, since I have not reasoned through what it would take to reconcile the two um to get there. I think that we need to figure out um for all of the possible modes of evaluation and for all of the possible modes of uh of loading of module contexts.

B

um Is there a single concept that that spans all of those things, and so one of the one of the problems is that the global contour is mutable by eval, um which I think is undesirable.

B

It would be undesirable for the for that glo for the for this mutable global contour to also be used in a module namespace.

B

um I I don't think that we want to create a system where introducing a const by running compartment, evaluate introduces a value in the scope of modules in the same compartment um and that that, in particular might put a wrench in using the same object for both.

A

Okay, that's a good point. That's a very good point uh is that the global contour, as currently specified, does have this horrible hazard that we that we cannot allow that module a.

A

Adding a variable to the global contour changes, the behavior of module b, because module a's additional variable can shadow what would have been a global binding of the same name check your muted chip, your muted.

B

I think chip is deliberately muted. um Yes,.

C

I would I was saying something to my son.

C

So I'm I'm not clear on the first claim that um that putting something on the global object would be discoverable in a way that uh introducing lexical values is not because eval eval inside the compartment could still discover the lexical values. Could it not.

B

uh No because we do this because the transform that we use sensors use of that name um uh basically.

C

By overwriting evil, among other things,.

B

uh It does does it.

A

It would have to it's it's um since the transform the metering transformer as a mandatory transform. uh It can't allow the if the source code fed into the transform says, evaluate computed string. uh Obviously, the enforcement that only transform code runs cannot allow that call to eval to be an end runner around the transform.

C

Right so so the transfer, the code that is introduced that references, the uh the introduced lexical, would have to have some kind of privileged access that the that the transformed source code does not have.

C

It has basically like the the transform code would need a restricted eval. Basically, that is um where, where a less restricted eval is available to the introduced code, correct.

A

The uh I'm I'm getting a little bit lost in the in the different standards. Okay,.

C

So so the example we're looking at here: it introduces like const foo and, let's just let's just run with that. So if I have source code that says uh eval type of.

A

C

String, concatenation, f, plus o plus o like I, I can now access this lexical value.

B

Yes, the the the the work around for that is that every all code that is in this compartment is subject to the transform that is supplied at component compartment construction uh construction, including the computed code that you feed back to eval.

B

um So so the sensor, even though you have computed a value, the sensor would still still still see the text and forbid it.

C

Right because you're overwriting eval.

B

Yes, because compartments have compressed, because every compartment has an overwritten email. Yes,.

C

But that also means that, in order for the introduced code to access foo, it has to have something that isn't available to the uh to the transformed source code.

B

A

Which code is the introduced code? That's what that's! What I'm going to do.

C

uh The the metering code- let's say the the code that should be able to access foo, yes,.

B

Must have access.

C

To something that the the transform source code does not have right, but now it seems circular because that's what you're trying to do in the first place.

B

Yes, the transform does two things one: is it sensors the input text and produces output text that contains the named variable, um but.

C

But if you can do that, then you don't need this.

A

To understand the circularity that.

C

A

The the problem is the difference between uh computed code versus computed property lookup. uh We can, we can like we can afford in these scenarios to replace eval with a transforming eval. uh We cannot afford to in that rewrite, replace uh all square bracket computed property lookups uh with something that checks the property.

C

No, I I get that, but if you, if you replace the eval yeah with a you, know with a a form that will do the censoring yeah, then what is? How is the other code accessing the new lexical.

C

So I was using um transformed and introduced. Does that work.

A

B

uh Transform introduced is the one that's provided by the by the client and transformed is provided by the metering transform got it.

C

Okay, no that's actually backwards, then um how about privileged and unprivileged.

B

uh You, okay, so privileged is the code that is produced by the metering. Transform unprivileged is the input to the metering, transform.

A

Yes, I think I think we have three kinds of code: there's the code that goes into the metering transform. There's the code comes out of the meeting meter and transform and there's the code that uh runs directly without having gone through. The metering. Transform can say things that you couldn't have said simply as a result of going through the transform. That's the privileged pen, the metering transform itself is producing code that uses privilege. But it's it's doing that in order to impose a constraint not in order to exercise privilege.

C

A

C

Constraint that I'm getting at so the.

C

Yeah, we need terms here.

A

Yeah yeah, okay, so the the code that goes into the meter and transform. uh Let's call that user code. Okay, um the code that comes out of the metering transform, let's call that transformed code and the code that runs without having gone through the metering transform.

A

Let's call that privilege.

C

A

C

User code transformed code privileged code and what actually gets evaluated is the union of privileged code and transformed code. Yes, okay, so the the privileged code can access foo, no, no problem that that's its whole, that is its privilege. uh The transform code is prohibited from accessing food, but it does have an eval.

A

The user code is prohibited from naming through the transformed code, uh the the transformation inserts a reference only for the purposes of doing what the transformation is doing. So the the idea is that the the where the you know where the foo is the the the the metering.

A

So so the idea is that the transformation introduces a reference to foo, but it's one. That is um not one that the user code could have mentioned.

C

Right, so if the user, if the user code mentioned foo, that would be, you know censored away or somehow fail or or redirected, you know the the user code, referencing foo doesn't get the same. Foo definitely doesn't get the same foo as the privileged code.

A

Right, let's just deal with the simple case: it's not fully transparent, but it's similar to reason about and adequate uh where foo is spelled very funny. You know dollar each underbar or whatever, and that if the user code mentions it, then the user code is statically, rejected.

C

Sure so, uh but the user code can also reference eval. You know: combination, string, concat of f, o and o, for instance, that's right uh and in the transformed code, the input to um what was in the user code. Input to eval is subject itself to uh to dynamic transformation right.

A

Right, the eval that the user code has available to it is a. It is an eval that accepts user code, transforms it and then causes the transform code to evaluate.

C

Perfect, so so, given that um if the user code references either the.

C

The foo variable directly or the the post transformation. Well, actually, we said it rejects it. So if user code references who is just rejected outright and that that protects it enough that only the only the privileged code has access to it.

A

Right: okay, the key thing there is that for a global, I'm going to use global, lexical and variable global contours synonymously here, um but as previously mentioned, there is an issue we need to come back to them, but the key cons.

A

The key invariant, is that for a global lexical variable, a evaluate and evaluated source code text can only access it by naming it so you can statically censor it, whereas a global variable I.e, a variable, that's alias to a property on the global object, can be accessed by computed property name lookup and to censor that by transformation would require rewriting all computed property lookups, which we definitely want to avoid.

C

Agreed, I definitely see you know the need for uh lexical injection. Okay, thanks.

B

Okay, um I think that we've covered this I'll create an issue, but it sounds like in general. There are no objections to this.

A

Well, there's there's: the open issue is.

B

A

Of reconciling global lexicals with global contour and global contour, the weird hazard that I haven't paid enough attention to is that module a adding a global lexical can cause module b dynamically even to change its behavior. Because now the the introduced lexical can shadow uh a global variable that module b had been referring to.

B

Yeah and there's also the matters of the global contour. um The global contour in a module um does not have the same.

B

It might be necessary in order to do this, it might be necessary to introduce another another con layer of contour in the scope of modules, because it is simply not possible to have an evaluate expression, say const foo that may or may not collide with an existing const foo in any of the modules in that system.

B

um So, in order for that to be sensible, it would have to be injected in a scope that is higher than the global than the scope in which consts are are are placed inside of inside of a module uh chip. You had your hand up.

C

Yeah, I was just just before you moved on. um Is there some significance to the dagger character in the uh bottom code? Exam bottom line code.

B

Example, let me see.

B

um There almost certainly is, but that's the problem with not putting the footnote uh globe global. True, uh the dagger might be a reminder to me to um uh to come back to this and make sure that global true is a a sensible thing in the context of the current spec. I I think that I don't. I haven't checked whether it's spelled that way.

A

So right now it's only spell it's only an explanatory concept in the spec there's no other place where it shows up as a name in an api that needs to be explained, uh and so one of the things we need to be aware of is that when we turn a internal expository, spec concept into an api name, the understandability of the name choice becomes a much higher priority, and I think that the global code versus eval code, which is the terminology station in the spec, is just terrible.

A

It's just not an understandable piece of terminology for this distinction.

B

Yeah I was, I was more concerned about the clarity of the slide yeah, the the intention on this yeah. I clearly need to expand upon this dagger.

B

Escape should apply to the slide not to zoom.

B

A

Yeah one one opportunity for simplification here that is politically complicated uh is that um in a session machine like mautable's building, there is no stopping mode. uh If we also don't support evaluation as global code, we only support evaluation as eval code, then, with both both sloppy and global code off the table. There is no global contour.

B

Okay, um I I think that clarifies chip is a yeah.

C

Yeah, okay, cool: it's not part of the source text. There yeah.

B

Okay, very good. um I think that that's all we can do today so we'll uh we can pick up on this next lovely issue um in a future presentation, so the uh make compartment constructor make compartment making, because I can consolidate compartment options, bag, uh okay, so there's only. There are only two more topics: um oh yeah and oh, there are three. There are three more topics um and we can pick up here next time.

A

What do you think about um doing this presentation at tc39 coming up.

B

um I think that it is not sufficiently polished for that audience um and, given that it took this much time, we would have to construct a much more polished and shorter version of this presentation. To do that. um What I would like to do for the next tc 39 is uh not. I don't think that we can call for advancement. um uh I think that we could give a status update and, and we could do a status update.

B

That is a very compressed version of this um that pertains just to refocusing on the module, loader facet of compartments, um yeah and may maybe uh maybe briefly cover the the structure of sex proposals up front so that it's clear that this is a thin slice of of what we're proposing for cess.

C

um And and focus on the areas that are um soliciting feedback or for which well and um the areas that refinement to the presentation is probably warranted, because that will the presentation will get feedback from tc39 that can be used to refine it for the one that ultimately requests advancement.

B

Yeah yeah yeah. um I think that the the upshot of this presentation is that there are a number of unresolved questions, um I'd l, what I would like to do over uh before before we even present progress is uh like it gets some to arrive, arrive at a local consensus of what what uh path forward we collectively recommend um and and and still present the alternatives, because I know the the audience will be interested in the alternatives as well and uh and again I do. I do expect that there will be.

B

That it will be useful to discuss a drastically different straw man than the compartment proposal um in order to uh in order to to uh address concerns that are that are brought up by I mean we've had conversations with michael hunter about the compartment api, uh where michael proposed that we have an inside out version of this um which which matched my design intuition going in.

B

But I think that I'm, I'm convinced that the compartment at least is useful um because it gives us a uh a beach head for further host virtualization hooks um that the counter proposal would not, um but there might be a middle ground between those two things. That's that's more uh um more amenable um and, of course, gus. Kaplan has a proposal that more closely resembles v8s api or pardon more closely more closely resembles the vm module as implemented by node, which itself closely resembles the v8 api, which closely represent closely resembles the tc39 specification.

B

As written um and the there there will probably be a contingent that wants uh whatever module. Loader api gets added to the language, to closely resemble that um and and that might put pressure on the proposal.

D

There's also counter pressure there um in the last modules working group meeting for node. We explicitly said we don't like the current vm api, uh because it's too hard to reasonably and so people were literally copy pasting, the source code of our module loader, to make it usable.

D

That is a serious issue with it.

B

I'm I'm glad to hear that we'll have support on the counter side too. That's especially given that the vm module is a better foundation for realms than it is for compartments um and and doesn't give you all the things that we need all right, I'm going to stop the recording.