Agoric SES, 9 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SES Meeting: Promise.allProperties and security invariants for lockdown shims

Description

This week, Alex discussed his idea to add Promise.allProperties to the language, concluding that a Promise.allEntries might be a suitably general foundation for shallow object and Map resolution. We discussed invariants that the language must or should preserve to avoid introducing zero-day exploits to lockdown shims in the wild.

A

All right welcome to the assess meeting uh today's agenda is that we're going to briefly discuss uh whether this this group has anything to say about promise.all properties, we've scheduled where we will schedule a conversation about uh realm revocable for a future meeting. You.

B

Mean all settled, I don't recognize a promise, not all properties.

A

Yes, all properties is what uh what what alex has put on the agenda so so we'll we'll start off with clarifying what that is, um and uh next week we have uh a packed agenda. The first half of our meeting is uh we have invited guests, uh the proponents of uh module assertions to discuss um the enforceability of module assertions um and, uh and then the second half we're meeting with jordan, and uh hopefully we can send a reminder so that uh all of that stays on the rails.

A

um So today, let's start with discussing alex's topic of promise.all properties alex. Can you start with recapping what the pro, what the proposal is.

C

I certainly can um so, basically, I'm sharing an email thread that I started a couple months ago in the ses strategy- email list.

C

Basically, um when I was looking at, I saw some code that came across my screen, um where we would, we would be defining an object in an asynchronous function, but we would have to await individual properties and that bothered me it's not how promises are should be used, it's legal, but it's not the way that asynchronous operations should happen. What would happen is you'd have to wait for the shape property in this example, and then the color property and then the mass property.

C

We have the ability, through promise.all and promise.race, to run multiple functions simultaneously x, using multiple promises simultaneously so that they don't block each other.

C

So I I got to thinking um it'd, be a really good idea to add a promise.all properties function, a a static function to the language, wherein, if you wanted to await all of those in a promise to all like context,.

C

Having that built into the language would probably be a good idea, the one thing that stopped me from running a straw man is that this really isn't a um security kind of thing. That's why I was wondering: is this even relevant for ses and um in the email thread um later on?

C

It turns out that other people, bradley jordan and mark, have all looked at this kind of idea and said yeah. We kind of do the same thing.

C

The other thing worth mentioning is that it's, although it's easy to do a work around for this, it's not something that a relatively new javascript engineer might even might even imagine- and I just wanted to have an open discussion and see what people thought about this.

C

A does ses even care, or should it be proposed to tc39 outside ses, which is perfectly fair and b. um Is this something that people are interested in exploring and in terms of adding it to the language, or should this just be dropped as hey there's an easy workaround.

C

That's really all I had to say on it. I just wanted to hear what people thought.

D

E

Go ahead, um we do have a fairly sizable kind of proxy that we're using to do something similar, actually because uh the developer vigilance in order to await everything properly is difficult to maintain in a larger code base.

E

So basically, what our proxy does is something similar to explain it. Instead of doing it as this does where it does a shallow uh traversal, we actually do a very deep synchronous, traversal and then coalesce into kind of a mirrored object. After that.

E

I think this is very useful, particularly if it is done deeply because then you know everything available within the entire object that is traversed is available synchronously, so that's my only opinion is: maybe it shouldn't be shallow.

A

In the queue ecosystem, I did implement both shallow and deep versions of this as analogs for promised at all or q.all, um and- uh and I put I put something like all- your all properties in a utility library until I could figure out whether it was um useful enough to integrate into q proper, um it turned out to not be uh it turned out to not be popular enough to put inside of the payload of queue, but it certainly is popular enough that I think that it would be beneficial for it to be part of the language.

A

um uh I don't think that this is a secure. It's not a security related thing, but it is promises and promises were brought in for security-related reasons um so it'd be this. Is this is a right audience to talk about promises? I think very much. um I don't think that it needs any. uh It certainly doesn't need to be blessed by this group to be proposed.

D

I agree with that last point, um but I disagree actually that it belongs in the language and it's for a practical reason that this is so analogous to the object copy problem. There are numerous dimensions that are going to be flexed in different ways for different use cases. Things like deep versus shallow, eager versus lazy, um innumerable versus uh all owned, um let's symbol, properties versus string value only like there's, there's so many things that that this kind of thing basically becomes um use case specific and it's practical.

D

It is impractical for a capability in the language to support all of them and therefore it usually ends up not existing. That's what it's the same reason that there's no deep copy in uh in the language.

B

Yeah, I'm I'm I'm very much with richard on on this point. The um uh in endo uh we're you know where endo is ces, plus our distributed object semantics. uh We take a very, very specific stance on what a passable value is and within that very specific stance.

B

We have, we do have an abstraction called all comparable that takes a um a promise for a passable. I'm sorry takes it. Passable, which includes promise for past, takes passable, which may have promises at the leaves and returns a promise for a comparable where a comparable in the distributed object. Semantics is something whose pass by copy superstructure terminates only in in uh atomically comparable leads, which is primitive values, and uh I I and uh remotable objects which are objects with unforgivable identity.

B

uh So the all comparable is the deep generalization but the deep generalization there is well defined for exactly the reasons that richer grazes, which is it's within a whole set of decisions that characterize what's effectively a new language, semantics embedded in the javascript semantics that have made very particular choices.

C

uh May I respond sure, um first and foremost, um I'm not familiar with the object copy problem as you've described it. Maybe I am, but not as that under that name, I am familiar with the with the fact that recursively diving in is a complicated problem that is generally left out of the language, and I'm fine with that.

C

um Where was that going? I.

A

Think I think that I can finish, for you is that uh it is a much much lower burden to introduce all properties as a shallow. uh A shallow copy of this of of innumerable promises on an object.

C

Thank you for that. um I would love to see even a shallow capability for that. um Now again, I'm just throwing the idea out there.

B

um Question already, um uh maybe this is direct more chris, since he just said innumerable. Why innumerable.

A

um Because the common pattern, uh the pattern that this is for is, is, uh or is likely to be most useful for is an object. Literal um I say innumerable is perhaps too strong. Certainly own.

A

um The shallow copy of with with all of the immediate values resolved is is, is approximately as useful as promise.all on arrays, but for objects used as structs.

B

So the the thing that there's two analogies that uh that I would make for this- that pull and opposite that that make that pull in opposite directions with regard to yet another distinction.

B

uh The thing that this seems to me to be most like is an eventual form of object.keys or object.entries and object.keys and object.entries uh believe it or not, are specific to innumerable own string, valued and the other thing that's analogous to is triple dot, which is specific to innumerable own, not restricted to string value.

B

And the fact that those are different uh is something that is, you know incredibly unfortunate.

C

um I have thought about the possibility of having promises on keys, not on the values, and I do not like that idea. um I'm just saying why would.

B

C

Why would you have an object where the key was a promise.

E

We have it because we need to coalesce jobs that are asynchronously resolving to data.

A

um Matthew we uh I'm not going to call out for hands, um so just just please do try to get in get.

F

In a word image, I wanted to basically say the same thing as mark. uh There is two precedents: the uh object, construction and object that entries or object.keys, uh which behave only on and variable, but there yeah anyway, that was already yeah.

B

I was very surprised recently by the way and had a bug in my code due to not realizing that uh object.entries did not include symbol named properties. That was, I actually had a security hole in my code as a result of that.

A

um Alex, I think, in summary, that you can antici that if you were to bring this forth to tc39, um that it would, the the shallow with the narrowest profile would be a shallow version of this, and I think that you that we've visited some of the arguments it'll immediately run up against. I got a choice about what what kind of all properties it'll be um and yeah I'd. Welcome the conversation.

C

Personally, okay um yeah: I can uh try to draw something up, uh but then I've promised to draw things up in the past and I never got around to it. um But this is relatively simple. I think compared to my other ideas.

A

Are you in a position where you can champion this yourself.

C

No, I am not. I do not work for a company that is a member of tc39.

A

Okay, so um so you'll be looking for a volunteer.

C

That would be much appreciated um before we close this topic out. I do want to raise another part that I thought about when I took this a step further.

C

um When we look at this in terms of maps and entering over iterating over the values of a map, it gets quite a bit uglier, um I'm thinking for an initial straw, man.

C

We would want to leave out maps but call it out as a danger will robinson moment, so to speak, um just as an initial stage: zero, hey here's, an idea just.

B

C

B

So uh what with regard to so that I think maps is actually uh a a good way to understand better what stance you have in mind. uh In what way would the idea you're talking about? Do you think it would generalize to maps.

C

I haven't thought of through fully yet um I've got my am I still sharing my screen here. um Yeah.

F

E

C

E

I can walk through what we do with a shallow version of this behavior on our link phase in nodes, yes, module loader that helps.

C

Yeah go ahead. If you want me to stop sharing my screen just say so,.

E

um And you can keep sharing it, I'm not going to show code because it's just gobbledygook, um basically in node's module loader, we have the concept of a module map that we've discussed multiple times at tc39, and the module map is actually represented by a javascript map structure and that map structure does a few things, but in particular it can have pending resolutions and it can have pending jobs.

E

So for any given module loaded and each of its dependencies in particular, you try to resolve it asynchronously and then you populate a job. The job is: what's going to be used to store things like the source text that it loads and hold on to the module. Namespace object for some other operations going on.

E

When you do this and a module has dependencies, we actually have to wait on all those dependencies to get their module job finished before we link or evaluate- and so it's this kind of big for loops everywhere, where we do essentially uh a big loop over everything. um That is basically just an object.

E

We're storing it as maps for reasons. But basically the idea is not all keys are strings. For this module map, this is increasingly becoming the case with the html integration spec for import assertions, and it's likely going to become the case even more so in the future.

E

It looks like um our original intent was to allow for this module map to have direct first class values as the keys, rather than a string or a url, but that's likely never gonna come to fruition, um but right now we do essentially this on a map object where both the left and right hand, uh the key and the value are both promises. Essentially, usually the right hand is not a promise it's already resolved, but we keep it wrapped.

E

F

All pretty creepy.

A

I have many questions that are completely irrelevant to the topic.

C

um Well, if I think about how I would implement this, how I would implement promised all properties based around the object.entries method and I keep maps in there because maps are iterable by by nature.

C

um Your point is well taken bradley. um I'm going to have.

A

E

This video afterward one thing that might be constructive is we have a lot of different entry um methods um they're on sets they're on arrays maps everything it might also make sense to, instead of work directly on an object to just accept an array of pairs since that's what they always return for entries even sets, which we have.

C

Yeah, I was thinking the same thing.

E

It would essentially be like a flat map and promise all with a limited depth.

A

Yeah, it could be um promise that all entries could be the core primitive that would unite um unite each of these concepts.

F

So that there would just be a shorthand for an awaits uh object from entries, promised.all.

E

um That entry is promising.

A

Not exactly uh it's a it's, it's uh you, it's yeah, it's similar there. They would be waiting for the resolution of the uh of all of the values of the underlying array.

A

Instead of or in addition to waiting for the resolution of the entry itself,.

F

Oh, you mean uh if there is a promise in either of the key or value uh of the entries. Wait wait on that. I see yeah yeah. That would make it a lot easier. Then.

C

All right, um yeah.

A

um Alex, uh let's, um let's make a note that you're looking for a volunteer to co-champion this idea with you um and uh and uh if you find time for it, uh take it from there. The um the next topic on our agenda is: um what constraints should we, as uh as champions of cess, um impose upon other proposals to tc39?

A

um We by we I mean mark blocked a proposal. I don't recall which one at the last tc39 plenary since it was going to introduce a new hidden, intrinsic iterator, prototype.

B

Yeah I can, I can, I can say very clearly which one it was. It was something in intel internationalization.

F

B

And the new and the new intrinsics it was introducing were completely unnecessary.

B

They were uh and the the reason why they were introducing it revealed that there were previous proposals that I should have blocked on these grounds that I didn't because of inattention um uh every time. I think that there's some big area, like dates or regex breakx, was a specific one in this case or internationalization that well surely. This is just um about this specialized area that doesn't have any security implications. Every time I I drop my guard on that uh something that's fatal to security slips in that we then have to clean up later.

B

So this is one of the reasons why it would be very good to have to try to start writing down more of these invariants so that uh we don't have to to be so on guard and other people can start to notice when they're about to break some security property.

B

The the problem is not that there was new api that returned an iterator or an iterable. The problem was that the way in which the api was introduced purely based on preserving tradition from from what was seen as the tradition of previous apis, was to introduce essentially a new iterator class with its own primordials. Its own prototype object in particular to represent the new iter, iterable or iterator class, uh which was cut, which is completely unnecessary, um and the intel thing is not blocked.

B

It is that they have to revise the api in a way that a hundred percent preserves their intended semantics, but doesn't further project. This particular broken precedent.

A

Let's um I I want to uh I've been thinking about this, and I've been thinking about what is the? What is the hard security invariant um this? This has been a concern of a concern of mine from a future future compatible or ensuring the future security of the lockdown shim that new language syntax would introduce the new language syntax like, for example, if we were to add observable functions to the languages has been proposed in the past as a as a dual to iterator functions.

A

um That would introduce syntax from which you could obtain a reference to a prototype uh that did not previously exist in the language and uh and existing code. That is trusting a lockdown shim um to ensure that there's no mutable state accessible from to client code that could be used as a side channel or a.

A

Beachhead for interfering with with code in the same compartment um could be would be unpreventable it would. It would basically be assuming that that the lockdown was not maintained. It would be a zero day, exploit uh upon release in a particular engine and uh and invalidate the lockdown shim.

B

Yeah and we've we've had that situation before with the old kaha scs, that was when uh engines started rolling out, implementations of async function. Async function made new primordials, reachable only by syntax. Our white listing mechanism was therefore not able to detect or remove them, and we had a zero day.

A

Yeah, so I think that the strong language invariant that we need to propose to tc39 at some future meeting is uh the invariant that the language, if it does add additional syntax that reveals intrinsics, that those intrinsics also be accessible by property access, discoverable from uh by discoverable by walking properties starting at global. This um and then uh mark- and I were bantering yesterday about what the shape of that would look like.

A

Would it be something like um like, for example, to backfill the principle onto proposals that have already violated this in order to establish a pattern for future fixes? um Would we introduce something like realm.async iterator prototype, not realm, but um something with something something done. You think iterator prototype accessible from global um and then suggest or mandate that any future proposal that adds syntax also provide it, uh provide access to that prototype in that location.

F

That's a minimum.

B

Yeah I like that, uh for one thing, I've seen a number of libraries uh independently try to just try to enumerate all of the anonymous intrinsics. uh Jordan wrote one that's independent obsessed.

B

I have not compared these to each other for coverage, um but it's unfortunate that people keep need needing to try to do this and that it's accident-prone- and there was even one case for which there was a primordial enumerated by the language that nobody could figure out how to reify using code in the language, uh and it was a big mystery for a while. It was a security. uh You know it was potential security vulnerability. It was the sink to async, iterable, adapter or something the.

B

So. I think that not just making new primordials findable by name-based traversal, starting from the global uh but taking all of the old ones that were not discoverable in this way and making them discoverable naming all of them, uh I think, would be good starting from the one that was introduced in es5, which is the um uh the uh uh type throw type type thrower. I forget what it's called um the thing that was representing the poisoning of function.caller and function.arguments.

B

The getter that always threats.

D

Yeah, it's called something like throw type error or thrower.

B

A

um So if I, if I may, the in summary the low bar is that the language must not introduce new language that the language in uh when adding any any any syntax added to the language must uh uh reveal any of the hidden hidden uh hidden objects that it would would make reachable by syntax, so that they can also be reached by a name.

A

D

Think we want to rely upon alias uh in that way or so broadly.

B

D

Oops um I misspoke apologies proceed. Oh.

A

Other conversation in parallel yeah.

F

You can't see the other people but they're here. Sorry.

A

They're there we believe.

F

You richard.

D

B

D

B

Cat, so I would say: that's it's that uh it's not just syntax. uh It's all new primordials should be discoverable by name-based traversal. So.

A

So, that's that's why I specifically specifically start with the lowest bar okay, the one, the lowest bar, the one that is firmly necessary in order to secure to ensure the invariance of existing lockdowns continue to be uh to be in invulnerable to future zero days. That is the minimum okay. I agree.

B

I agree: that's the move.

A

The the the next step above that would be obviously to satisfy the desire for the the the language as it stands, to establish a firm precedent for for future editions by backfilling that principle to other things that have been added in the past, like async, iterator um and then the next more firm above that would be I which is like uh I. I do not think that um I do not think that the intel proposal actually violated either of those two invariants um it it it uh it.

A

It's it's violating a softer invariant um that any api added to the language must also reveal any of the prototypes or constructors of the things that it's going to return from functions.

B

um Well, it's it's. When you say return from functions, it's any any refinable primordial uh the um you know. Any new reifiable primordial should be discoverable by property. Traversal um uh and uh refillable means it's not necessarily just that it returns. It might pass it as an argument to a callback function or a zillion ways in which something might be reified. I don't think we need to to distinguish return values.

A

Yes, I agree um the any any any object that is revealed by a new api to to client code should be discoverable now any any.

B

Primordial object.

A

Any primordial, my my. I think that I think that.

A

I think we need to frame the argument in terms of what we need, what we would like and what would be nice and the last one would be nice and we're blocking a proposal based on a would be nice at the moment, um and- and we should make that clear in order to not only.

B

The intel proposal.

A

The until proposal, specifically, yes, I.

B

Think that's more than or would be nice.

A

Well, at the moment, lockdown, it does nothing to invalidate lockdown, because intel itself isn't exposed and also it also does not provide any, and even if it were exposed, um we would be in a position that any new api, they add, would not be automatically revealed to client code via lockdown, because it's not in the white list, um and so therefore, there's no zero day exploit that is created by this behavior either.

B

Okay, okay would be, nice still is too so I I get your distinction. I would like a phrasing that sounds stronger than would be nice.

A

F

Yeah, it's yeah.

B

How about not completely fatal.

A

Yes, non-fatal is good. I I can go with that. um Yeah all right um I'll, make a note in a minute so that there's an action item to.

B

Okay, so some other, so some other things along these lines uh is that um uh uh no introduction of hidden of primordial hidden state or primordial. I o abilities.

B

Especially no introduction to of such state, or I o abilities to any primordial object that currently does not have them.

B

And I think that we can bundle this name based enumeration with another issue that that um that we've been facing, that the that the controversies over their own proposal makes yet more urgent, which is um a uh we've. We've talked about the need to have some kind of platform uh provided enumeration of the safe standard, globals um the safe standard, javascript globals, uh I.e, um omitting the hosts but including safe globals, introduced as the language evolves, so that old lockdowns can potentially honor new, safe globals.

B

The object on which these safe globals reside could also be an object that from which all of the currently undiscoverable primordials can be.

B

B

uh Chris, you seem to be muted.

A

For sure um yeah, I I think that we have taken this topic to a satisfying conclusion, um the and and the and the queue is empty for so um I think that we can reclaim some time.

A

All right, thank you. I'm gonna stop recording.