Agoric SES, 9 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SES Meeting: Shadow Realm Serialization Boundary

Description

Wherein Leo proposes a graduated plan to add serialization at the ShadowRealm boundary.

A

Now welcome to the cess meeting today's date is march. 9Th. We have a very light agenda this week, so it will probably short be short. um Leo. Has a couple of topics he'd like to announce, if not discussed, since we don't have enough of a quorum to give it a proper conversation, but um please take it away leo.

B

um Thank you, so I'm gonna reverse the order of the topics here to just start discussing with the small one uh first um and this small one is an issue that was raised uh this week by jack works, and I think it's a reasonable request to consider at least part of the partial uh part of these. uh Their request.

B

This request is about restricting the second parameter of import value. Did we? Oh people are still here zombies kind of messing up the windows, um the restricting the second parameter of import value to be string only to only.

C

Are you, are you think you're sh? Are you sharing in this uh slides or something because I don't see anything.

B

um No, I just wrote some quick notes in the uh agenda.

C

It's fine, it just sounded like you, the way you're talking nevermind carry on. Oh.

B

Okay, okay, uh sorry about this confusion, so I don't have anything like as a presentation. It's more like. I think it's a quick topic. The example in the agenda might be uh good enough for what I'm telling um and the reasons so import value. Today. Those have two uh parameters: the first parameter to be the specifier and the second parameter to be a name. There will be parsed as a string um saying what name do you want from the restore module name? Space object.

B

An import value will return a promise that resolves into that name value.

B

So the first step for this work for jack. What check works uh proposed is to have um a restriction that that that was actually a suggestion by matthew, hoffman, sorry, I just remember here uh and given the the names of who actually uh did that that suggested.

B

I like that a lot, because if we restrict uh the second parameter of import value to be a string only that avoids uh people trying to use import value without a second uh argument to have a parsed, uh undivided undefined parse to a string and try to capture like under undefined name from the module in space object, um a string undefined name from from that.

B

That is good enough, and that also opens a path to later as a follow-up to this proposal. But I don't intend to expand this proposal more than just this restriction.

B

We can have as a follow-up like considering import value, to receive iterable values that are not exactly a string such as an array um that gives let's say, multiple names to be resolved. So when the promise is resolved, it returns an array of of the values with the from the given names.

B

So my intention is to bring to tc39 a normative change to the current stage. 3.

B

That only means a restriction, the import value. The second parameter- must be a string if type of the second parameter is not string. It trolls a type error.

B

I'm going to write this down in the agenda just to make it clear.

C

Just a thought, if I may.

A

If this restriction.

C

Is too much meaning if it turns out, we need symbols in the future, for names it'll be easier to loosen it rather than to tighten it in the case of we're too loose initially,.

B

D

Like and symbols are currently not, and I don't ever see them being allowed as import exports, because they are, you can already add them to module. Namespace objects uh as user lens, which is weird, but that's what it is.

C

B

Yeah one of the reasons I'm not working on uh the second part, which is which is actually uh considering iterable values, because I think that adds a lot for a stage three proposal. So I think it's worth having it. That is as a separate proposal to avoid.

B

Waiting for longer for shadow rounds and that's actually a little bit poly fillable as well, you can monkey patching or import value and do some uh word there and work there in user land. Also there's the second topic here, which is a little bit sensitive because I know not. Everyone is happy about it, uh but I think we we should do some work to incubate a pragmatic approach.

B

We can you could incubate some pragmatic approach for some structure serialization across other realms and by incubating I mean definitely not bringing oh yeah thanks for opening and sharing it already.

B

So that means some work that will consider what we have today from shadow realms, what is evaluated, uh how things are evaluated across realms and what is the just like structured serialization, which is not the desirable part and what is a like? What are the changes that I would want on top?

B

um This is a thing I just briefly discussed internally. You need some expansion, um but that's what I mean like. I want all sorts of feedback like uh if this can move to a workable and desirable state.

B

Of course, this work should be done once again as a follow up work, not.

C

B

In stage 3 of shadow rounds, so the levels level zero represents the current status of what we have today like primitive values can be transferred, as is including symbols. Callable values are connect via wrapped exotic objects and in the uh any non-caliber objects, will throw a type error.

B

If you go through the levels they're like there's one thing that we want and we discussed in the past- and I know we still have contentions on that, but that's like I think this is better seen as a whole package, so shared buffers uh shared array. Buffers are one of the steps, but also bringing some other serialization there. um I think, which is represented by level 1b.

B

Can we thank you chris um yeah, so for every level, like all the bold parts, are the new additions for each one of these levels here, um so with that, we can have some uh do some deep serialization of array, objects or ordinary objects.

B

uh One of the things that I know of is of a concern is how we handle proxies, and so we have some considerations as well. I think it's um down below this level is like from uh where I'm starting this work, I don't mean like we should have separate the implementations of each one of these levels. I'm just saying like how I'm progressing like from applying the structure serialization to all all the the other parts. My uh desire is to go all the way to the end. Can can we just scroll down more? Thank you.

B

um So this serialization here would actually lead us as well to wrap promises and iterators as we do wrap up functions today. That would be helpful. If we resolve any validation into a promise, we can do some wrapping uh of these resolved promises as well in iterators.

B

uh This is a sort of like complex work, but I think it's uh the way to resolve that, and uh this work with uh proxies is one of uh I think it's one of the things that is sensitive like how we do resolve proxies as structure serialization. Today. The known issue with that is just like um ignore proxies, and I don't think that's that should be the approach.

B

We can actually just capture the value, but I think, in my opinion, proxy should be just resulted in a serialized static value that would have no communication or we could do a wrapping of the proxy to grow up in being much more complex.

B

Go ahead and match your thoughts.

D

I think the issue with proxy is that, if there's a proxy in front of something else that is recognized uh with internal state such as a date or uh something like that, it needs to work uh the same as if you had actually uh sent the date uh and the only way. I believe this kind of work is uh somehow the proxy creator is able to say like this is how you're uh you should be serializing. This object.

D

uh This is how you should be sterilizing it uh and that's a mechanism that structured cloning does not offer, and that's a mechanism that implicits cloning in general doesn't provide.

B

A

I think that one thing that's fair to say is that the scope of your ambition with this is um high, but less than hours like what what what agoric wants to build or what we've already built, provides a communication channel. That goes all the way up to level the partner. It creates a means for for inter-realm communication.

A

That's at least level two, but then goes on to also do eventual send so the um or at least pardon matches the scope of the ambition up to level two and then does eventual send and varies in detail, um and I can't speak to that myself. But what I can say is that um in the long term, we need to have something like this. We we agree that we need to be able to communicate with this level of fidelity between.

D

What is shared between the realms is highly dependent on the application and whether you want to sterilize just data like objectives, uh it's just a record that you serialize or whether you reify something on the other side. That's a proxy that can do live.

D

Communication to the original object is, is another approach.

A

May I propose that as a litmus test for whatever whatever designs we come up with here, we would want it to mostly be the same as inter-process communication right, so that you could largely do the same thing between two agents that you can do between a shadow realm and it's among shadow realms or between a shadow realm and the incubator realm. Is that fair? So.

D

I guess there's one question: we need to solve ipc and uh eventual send and all those things are asynchronous uh over asynchronous communication channels. The interaction between uh shadow realms is actually synchronous, and it is a valid question whether whatever cloning mechanism and sterilization happens, like should work the same for both and it's actually a valid question for what platform too. Currently web platform only has um serial full serialization, because um it is an asynchronous communication channel.

B

Just to bring some of the signals here, uh our initial goal was, um as you know, heard, was to create a channel for uh shared raid buffers, because that's not a possibility today and we cannot take advantage of buffer cross realms.

B

When I mentioned that, I got the signals from implementers where they could do shared rate buffers if we had structural serialization.

B

But I'm well aware that structural sterilization, as this is not desirable either by other stakeholders like people in this group.

A

When they say that, do they mean that in the sense of trading horses or uh physical limitation.

B

More like trading horses like it's, not desirable, for them to just create a channel for shared array buffers only and if we want to create a channel they rather uh they're, they would be very positive. That's one of the signal like that would be very positive to a sap structure. Serialization that has shared a radar first bundle together, uh but structure serialization, as is it's also not interesting, and so uh the idea here is like this document is to become a sort of like expanded structure sterilization like or enhanced structure.

B

Serialization is like a structure sterilization that does the things that it is already doing, as is today for many of the values, but many of the things that he errors today. That is we're gonna fill some of we're. Gonna fill these gaps like what we do for proxies. What we do for promises, what we do for not iterators.

D

D

I think you can probably decouple I'm wondering if it would be worth decoupling the promise and iterator uh stuff um from this. I I I somehow always consider them like somewhat orthogonal, and I believe you can actually just do the um promise and iterator handling through a very thin membrane type uh approach on top of the callable boundary.

B

I think promises in iterators are very um requested feature and that's why I also have have it in a different level in that document.

C

B

D

But putting is a level like a true behind the rest, like I'm saying you might want to approach it uh entirely uh orton like is there a way to solve those things without sterilization.

B

It can be one of the things that I can tell it's. Unfortunately, it's not unlike a as a a hind back feature for uh me, so I will not be able to chant on that part uh as I for many things I I'm like, I have since I joined the product, uh I started as a product uh manager.

B

I have so many things on my plate right now and I'm having to say no to so many uh so many things that are like I'd love to, but it's unfortunately not on my priority uh list like array. Buffers are one of the things that I uh that will create more effect for me, so that uh that would make me like. If I divide them, I would probably go with the part where array buffers are living.

B

Now promises integrators are nice to have in this case.

C

But to revisit chris's question what what we're talking about with all of those are special cases, special treatment that that would not be possible in their process. For instance,.

A

I promise an async iterator async iterator. At that level. I can imagine a possibility of having the same behavior between realms, regardless of whether they're in the same process, but iterator for certain could only be done with a synchronous.

C

With well I'm not even talking about synchronous versus asynchronous, but it's like it's a it's a privileged treatment uh that what comes over the channel is more than one could express generically through serialization right.

D

What's different is that you promise async iterator iterators, you can build all those things in the user lands where shared array buffer sharing across the callable binary. uh You can't you have.

C

D

A

Yeah yeah and in general, I'm in favor of um starting with this doing the work at this at the tc39 level. That allows more to be done in user space before before, paving account pass in user space.

C

Right, I agree.

B

So that means putting promises and iterators apart and not working on that right now or the other way.

D

I I think at some point one of us should probably build a user land implementation of it. It shouldn't be too hard uh and show people how to do it. Yeah. I just don't have time.

B

Yeah, as I mentioned, like the functionality that I don't have today, that I cannot afford to have today is to share the radar first. This is what effects and like uh virtualized environment, so my hope is to provide us some like if the implementers are positive, if we bring them structure civilization.

B

What is also like the modifications that we do with search structure, civilization that wouldn't suck uh that wouldn't bring like problems as it brings you to proxies today.

D

I don't know if it's possible.

D

B

Yeah, that's the the that's the the challenge and that's why.

C

Right, that's, that's all privilege, that's all things that you know is working behind the scenes in observable ways. You know with observable consequences.

B

If I- and it's still to my understanding that, like structural civilization, it says today for some of these things or not still remain undesirable,.

B

Right like without any special treatment needs viruses.

B

Yeah, okay, yeah um yeah. That gives me some sense of like.

B

The size of this work.

B

um Good, I appreciate.

C

A

All right all right, that sounds like a wrap to me. Yes, is there anything anybody else wishes to discuss on the agenda all right all right, let's I'm gonna, stop the recording. Thank you for coming everyone.

A