IPFS IPFS þing 2022 - IPFS and WASM, 10 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Autocodec - @expede - IPFS and WASM

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

All right: let's talk about auto codec, which is actually maybe a good segue after uh ipvm, uh because we're going to talk about shipping codecs in wasm into ipfs, uh so self-decoding graphs, essentially so uh at a high level, uh because if you can treat a wasan blob as a black box pure function right, you can get to all the cool stuff that we were just talking about uh with it. So dynamically load, codecs instantly discoverable travels with the data, and you can signal content with these okay, so yeah we'll talk through all of these.

A

Don't worry, okay, so um there's a couple different ways of doing this right, so a a challenge that we've had we've heard from a few other people, and even just like in the category of from from others in the category of like, wouldn't be great. If is what, if I didn't have to pre-deploy my codex? What if I just got some data- and I knew how to translate it automatically. Hence auto codec, roughly there's two ways of doing this right.

A

You can ship up some kind of a blob that you can interpret and run over the ipld, or you can write a dsl um for those of you who were in the um session talking about graph sync a couple days ago, um graph sync got pretty complex and dsls over for complex cases, often get very complex and deep and difficult to reason about. So.

A

Basically, uh even though dsls the nice thing about dsl, is they scope the problem down to exactly your domain, like to the point that you can eliminate entire classes of error like just completely um now, people have to learn the dsl. They have to learn all these other things right.

A

um So what if we did the opposite direction? What if we went with the messy version, which is I'm going to ship, wasn't blobs, which yeah are turning complete? They could potentially run forever, but we can mitigate that and let people write whatever they want in there. It's fully general, and uh now it does literally everything we want. We never have to upgrade a dsl again because we're literally just operating on data, so this is sometimes called worst- is better.

A

um There's a great uh article from the early 90s about this. um That's basically look the the thing that has momentum. The thing that's getting mind share. It might not be better today, but it will be better in 10 years and I've talked to uh a bunch of companies. um You know shopify um cloudflare fastly, others like. What's your experience with wasm we're thinking about turning everything into wasm uh and the resounding response is uh wasm is the future.

A

The tooling is going to be just extremely good, uh we're investing in it now so nice thing about doing this, this worse version than a dsl is it is adoptable familiar. You can write it in any language like if you want to write your codec in ruby, great, go for it, uh the tooling's good and only getting better. uh You don't have to say well, you know, like you know, describe it in this dsl. It's like no, like do exactly this thing, I'm not relying on somebody else's.

A

I mean, I guess you are depending on web assembly, but we all understand how that works you're, not depending on somebody's potentially buggy implementation of the dsl interpreter, and it's openly extensible. If we, if we later add a bunch of stuff into the ipld data model, that's fine, you start writing uh new, wasn't blobs that handle the new things.

A

Layout options so couple ways of doing this right. um You can add a new field into the ipld data model uh called codec or something similar, um and the nice thing you get with this is all of the deep linking uh stays intact. You can point to any part of the structure and be like ah okay cool. This is git right done, simple.

A

um You get some redundancy because you have to write that sid out on all these blocks. um So that's a trade-off. This is not my favorite layout, but, like you, you could do this and then anything backwards compatible.

A

When it didn't have these, then you run it through. You know you, you leave it up to a built-in codec right, the other one which I like, because it's very upgradable um and has a few other advantages, even though it's technically osmorphic, um it adds an extra kind of node in here.

A

So this is our auto codec wrapper every entry point into a graph gets one of these at the top and if you're, if you have, if you switch so if you have winifest and inside of winnifest, is a git repo inside of that git reposin is a unix fs and you need to switch between them. You can use these as the bounding boxes for those.

A

The downside is now you have to like you know you have some sub graph and let's say it's like you know highly linked from multiple places. All of those entry points need to have this wrapper on it, and but they could have potentially different wrappers. So if you want to interpret it in a different way, you put a pointer in and say don't use that one use this one or if you want to upgrade the webassembly, that's getting pointed to simple, you take the node below it and you place it underneath a different wrapper done.

A

You've, upgraded.

A

Oh yeah, but the downside well upside down side is this would break all the existing codecs, so you wouldn't be able to run dag json on this because it wouldn't know what these ac wrappers are right, but then you would run it with wasm, so it wouldn't matter, but anyway um they should be self-describing.

A

So yeah there's a abi on this, so just kind of like in the last presentation right we have this nice tree. You have the api, the wasn't blob and some other metadata.

A

You must include a way to go from your serialization format: your uh dag pb, your dag cbor, whatever down into a base data model that is represented in wasm, and you lay that out specifically, then you must include one or more codecs with a default to run. So if you have some data like say winifs as it exists today, you can interpret that as manifest, but you can also interpret that as unixfs, so you can ship all of those into the wasa module right and then you put in the api and then in the metadata.

A

Here's the things that I can do and you may include traversal functions so once I go into this data structure, maybe I have a you know. Maybe it's a tree give me the links out of the more semantically rich structure right, not the intermediate ones, only those right, so you can add extra traversal functions inside.

A

uh These obviously have to terminate right. uh We don't want these to run forever, so we need some kind of bounding on them right, so we're gonna go from some, you know leaves on a tree through awesome and then output, some kind of file, and that has to happen in finite time and not just that you need to have it run in some kind of a bound.

A

Some problems here are that every machine is slightly different, so setting a timeout like a global timeout. You know it's gonna, be very different on my m1 mac than on a low powered android device, so to get consistency, and so that people know that when they ship something that it'll actually finish and that people can actually like you know, whoever is on the other side of this- is actually going to be able to run it.

A

um You can do ahead of time cost dynamics which is a form of static analysis, um but really realistically you'll use gas, and we already have experience with gas bounded wasm from the fem work. So it's actually baked into a lot of implementations these days so uh who gets to encode these encoders.

A

So going back to um how we upgrade things, how do we make things better? Well, uh this is just a tag right. It's just it's just data. We don't have to run that data, but we do know it's content id, and that gives us a signal for the kind of data that's inside right, so we can instead of saying the wasm.

A

It's like ah you know what I just see so much unix of s and I'm running wasm all the time, I'm going to upgrade that to my native rust code and then execute that inside of my node instead right and then time goes on, you're like actually people are using a lot of unix fs, I'm going to turn that into you know, hand tuned assembly, and you can just upgrade that by reading the thing and saying these are equivalents, I spent the time to make sure that they are.

A

uh They have the same behavior um and I'm gonna run it like this. I can always drop back down to the wasm. If I'm worried that there's a bug.

A

I was chatting with a couple people yesterday about in-place upgrades and fixes. I actually think this is a red herring so for for two reasons: right: uh it's a deterministic codec in a most well, okay, so wasm's not fully deterministic. Today, it's mostly deterministic, um but I I believe in the next version of the spec which they're currently working on to be totally deterministic, and we can also use the deterministic subset um that I believe, is what's used inside of svm correct me. If I'm wrong, um so you can just check it.

A

You can just run it locally and be like. Do I get out what I what I expected? Yes, no right and because it's uh in place, whoever runs, it will also get the same result. So you don't need to upgrade, because you know what it's going to do. It's not dynamic.

A

I also think that this is at the wrong layer. If you need to upgrade something you put a mutable pointer somewhere else, dns link ipns somewhere else, you put a different, auto codec wrapper on it. You point out to it to your upgraded, um upgraded wasm and you stick that in ips right like it's, it's the wrong layer for the mutability, I think so um yeah.

A

If you really wanted to do this like if you had to instead of pointing out to um to the awesome blob directly yeah, you point out to ipns, um but now, uh if you wanted to replace it well, do you put in a ttl? How often do you update this? You have to do this extra lookup. You know this extra translation. All of these things right. What? If you get a malicious update? What if you used to be able to decode this thing?

A

And now you can't, I actually think that's like that's, potentially pretty dangerous right, because you're, starting from you know, I'm gonna like update this thing and, like you know, it's gonna get better and better and better, uh and then next thing you know it's not so good anymore right and so now, like. What do you have a flag to roll back like it gets complex, really quickly?

A

The thing with persistent data is you should keep it persistent right. It's it's static under here. It's easy to think about. You don't have to do anything. Yes, you want a immutable pointer because the world is mutable. Do that add separate that out do that at the top layer only so to wrap up.

A

What I want something highly accessible, dynamic concrete, by which I mean I can just run this myself and see. Did this give me the thing that I expected yes, okay, great, then I'll share it with people a pure function, that's potentially optimizable, so that we're not stuck with you know this specific thing. If you want to get native performance, you can thank you.

A

Oh- and I should also mention sorry- I think I missed a point in here somewhere along the way. Another reason that we want this so from from lived experience in with a web data file system, we encoded an entire extra part of the tree. We called the pretty tree so that the gateway could handle it just in unix, fs, regular. So every time we would do a write, we would walk the tree and build this latest version index right, which is finds a little bit of extra compute.

A

It's fine whatever right, but it's just mainly annoying. We had a workaround, it would have been nice to have that just be completely natively handled underneath and getting that codec loaded in uh into the node is not as simple as I send it to sorry into our node. We could have done it, but then it wouldn't have worked on ipfs.io or on pinata or anybody else's thing. So having this come with, the codec means that we could have just shipped that in 2019 and not thought about it ever again.

A

Anyway, questions we may have covered some of these in the last session. So it's okay.

B

I wonder about like the specific kind of uh interfaces that are possible for the autocodec. I can see there being like an interface for finding new links which is like for different applications. I always say where finding links is something you would want to do. If you want to pin something, for example, whereas there is maybe another application that wants to actually read some data and interpret it as like. A file system.

C

B

Is then something you could use on the gateway which would solve our predictory problem? uh So I wonder what your thoughts are on like? Is there one single, multi-codec or autocoding, sorry uh for your data or is? Does there have to be like a set of multi-code, auto codecs or so.

A

Yeah, this is part of the worst is better it. There is no global registry for interpret this data. This way it's I don't know what kind of data is under here run the wasn't blob and I'll find out, and I might later know I wrote this thing here. It has the same uh sid for the wasm section. Therefore, it's probably the same kind of data right at least the data is laid out the same, so I can interpret it this way, so it actually doesn't matter what's in the ipld.

A

Actually all that really matters is how what am I going to hydrate out of that right and if you want to come with a completely different, codec and point at it. So you know we have the old pointer. Now we have a new pointer here. That's also fine. It's just a different tree.

A

It's actually it's, it's almost a subs. It's a subset of the stuff that we were talking about in the last presentation where you have uh interplanetary linked invocation, wasn't blob arguments, but the arguments are implicitly the tree or the dag underneath it.

D

Mm-Hmm, I love the idea of progressive enhancement in place and because we're dorks about perf like that uh trying to think ahead about that scenario, and I just want to make sure I have everything correct.

D

If you wanted to say, drop down to your rest code, you or some other thing that is going to execute in a higher performance context.

D

We would be detecting the sid itself and keying on that, or do we key on the codec of the sid uh or sorry the multi? You know the multi-codec of the set.

A

Right um so I mean I, I guess you could do either right.

D

A

Even with multi-codec and again please correct me: if I'm wrong, it seems like for from what I've been reading and participating in some discussions over there.

A

There's a distinction between here's, how the data is laid out as json versus here's, my application, specific thing right and so a lot of that's getting pushed into adls, and so this essentially takes both use cases.

A

So we're not like yes, you could go into the multi-codec, but for a lot of use cases, many use cases um you want to get down into the application. Specific data type right, so probably keying off the sid would be my guess, but let's try it out. Let's find out.

D

Yeah and my only concern there is that you now have to sort of like you fall a bit into the mutability problem following the sit changing over time and knowing when you can kick in your optimization.

A

D

More I'm just imagining it's like okay cool! I hit the I I under the block, and I see all right- I'm I'm I'm to use the following: wasm tube uh the sid. This sid points to oblong that I'm supposed to use to decode this subgraph the so that sid is my indicator on what to key on. So, let's use versioning numbers for a second, that's version. 1.0 of that thing later on.

A

Oh yeah, no, no, no, no versioning period.

D

A

It's like you're, going to run this thing over that that's the user expectation, if you as the maintainer of iro, wants to make that faster and do it in uh and have a specific one in rust. It has to be that's.

D

It isomorphic yeah.

A

So you don't have to run that sid right.

D

No, you can run you.

A

Can load in your code, but that has to the guarantee that you're giving the end user is that it'll behave. The output will be identical exactly.

D

Yeah, no I'm totally clear on that and trying to just make sure that what we're effectively doing is tying that sid to a behavior.

A

Yeah you're, essentially uh replacing everywhere you see that sid with a rust call.

B

Yeah yeah yeah.

E

So yeah we we thought about this, a lot in the ipld team over several years. um One thing that we kind of came to is that if you ever take the view layer and put it into the graph next to the data, you have to make sure that the read layer always has a way to overwrite it, because eventually you just want something else to be there.

E

um Yesterday there was a uncommon session about the ipld scheme and what we might be able to do with pathing and stuff, like that, and I very successfully was like yeah, let's get all of our good ideas, but let's make sure that we reconcile with this and whatever we do in terms of pathing here can always be described in a system like this, so that we always have that, like nice, modular layer, where we can layer all this stuff into it, but that would be really nice to like kind of work together between the uri pathing scheme for ipld, and this so that every time that you define something like this, we make sure that there's always a way to overwrite it.

E

In that read layer.

A

Just to just to clarify, though, by saying uh to override it, you mean to do like an upgrade on the codec like a force upgrade right.

E

Like the reader of the data, naturally- and they know that there's like.

A

Yeah totally, but so the and we can get into this like way more detail after as well, but so part of the idea here is: let's say that that's the case it is a directed acyclic graph. I should be able to map over it and just do a find replace basically like every time.

A

You see this one replace it with this, this other blob, and that essentially gives you a you think of it as a function running over the data to give you a new ipld and then run that right and so and then you can also persist the new, updated one and say update this pointer.

A

So I like, I really don't want to do that at a at read time. I want this to be at right time and that, if I know there's something wrong with it, I'm going to fix it in place as immutable data.

F

So sometimes that's what you want to do, especially if you are like upgrading something, then that's clear, um and sometimes there are other user stories too like if you want to take the same data and look at it through two different lenses, then maybe that's actually like a read time choice.

F

The um the only other thing I want to offer is: we have a vocab word for this- that I want to toss out.

F

um Signaling problem so like there is the nature of we must have implementations and wasm can be an implementation, and there can be like replace the wasm with native code and all of these implementation selection problems and then signaling that something must be done here at all can be a slightly different problem, and I think there will be many mechanisms for that too, so like having something in the cit is one way of solving the signaling problem. Having wrapper jumps is one way of solving the signaling problem.

F

Useful vocab word.

A

Awesome thanks so just uh like, so even in in the case where you were saying you want to read it in a different with a different codec, so this this top ac wrapper here. Why couldn't I also? Let's call this bytes link directly underneath it you know, block a. I can point to that from another wrapper.

A

A

For this to work, it has to start at the very top of any region. That's going to change like as a hard requirement.

C

So so one problem with putting the wrap at the top isn't because you can't leave dynamic types. um You can't sorry. If you have dynamic types uh like you might have something: that's dynamically typed uh internally and you don't know the code. You're gonna need there or sorry like you. You have a list of things that have different code. Does that make sense like that.

A

uh Sorry so walk me through it much more slowly.

C

Sorry so like like, um ideally at the top, you would say like for this dag like use these these types- yes, but some of you might be in the middle and you need to switch between different types that make sense. Yes, yeah.

A

So so yeah the the idea is at each of these ac wrappers they're a different kind of node as you're, so in the codec you're going to start walking down.

B

A

And you hit one of these and you decide. Am I going to respect this or not? Am I going to switch and you can write the code that does this that respects switching or not and at right time you can make that decision.

A

Okay right, these can be. These can be specific specifically to one dag and never used again, because it's awesome.

C

Okay, I'll have to read up on this more okay, so.

A

It like yeah, I think, from talking to a few people right. It's not that we have. This is a schema that we're going to interpret this way and then we have a registry. It's run this exact wasm. Yes, over this exact thing and the the, if you think about.

C

It that'll give us the bytes yeah.

A

If you think about it in types, it's ipld arrow x what you know, what, whatever you want back out, bytes from the perspective of the the machine right.

C

Yep yeah yeah, okay, that makes sense uh the the other side is like um like I, I do kind of agree on the read side thing: the problem is like: quite often you will have some immutable thing. You don't want it to change ever you don't want something to be able to control with a key. So that's why it is important. We have able to have read overrides. The saving grace here is like it doesn't um like when I produce the data.

C

uh Sorry, it's hard to exercise like a security voltage because like when I make something I'm already locking in that data and that code, uh so, like chances are like. If someone wants to exploit something, they would need to provide new data. But if the data's already content addressed, they can't uh so you actually don't always need to upgrade this. It's only if you find a bug and you really really want to replace something because, like it turns out some produced, a bunch of data that we just had like was bad.

C

It didn't didn't work in some cases, um but yeah that can be fixed right side. The other thing I wanted to note was uh specifically in the signaling. We've talked a lot about, so that was part of my presentation as well about having like these fat pointers um like this wrapper way is the other way of doing it, but one way is literally just like two pointers together, um uh where you start at the top, and then you can submit your type train your data tree. um If that makes sense.

A

Right so in in that case, you have some dsl that describes what's inside.

C

uh You know like the factor is like a cid special type of cid that has two cids inside or one points of code and one points to the data trade. So we kind of glue code onto that.

A

That that's what this is, but.

C

That's a spec yes, but this puts in the cid itself.

G

C

So like it makes it like this, no.

G

No, no go back once take the ac wrapper and lift it into the make it a link and then lift it into the bytes link. Above.

A

Right, that's well, so this one is every block, but you mean just to have it at the the edge basically of the graph.

G

Include include the uh yeah.

B

Okay, if you can treat the cd as a pointer.

G

Coming into the subject, they're saying take that cid and add another pointer into that same tid. Sure it's like a cfd that has additional data that points to the wasm. So so you would have the purple arrow and the pink arrow together from one point right so.

A

You take the the tuples that are inside the links and you make them triples.

C

No is it tuples.

A

E

Kind of cid the cids already have like multiple things in the database.

C

So links have names and cids. My point is like in the cid itself you make that two pieces where one piece is the code.

A

So your cid is like two cids today.

C

Yes or like you have like, like one link to.

A

C

One the nice thing about this means like I, I could take your link and I can replace the code yeah and it also has small objects. Now, if you're willing to use small objects, yeah, you just have the intermediate blocks: you're fine.

A

G

A

Yes- and you end up with smaller.

C

E

G

E

And you, you can also do stuff like when you have beautiful references you can put the pointer and the musical. You can like always go back and resolve that. It's also about.

D

The upside of the double cds in the brook in that one yeah, your cid of ac wrapper, is always going to change because it contains bite length, and so you have to you can't lazily look at ac. Rapper go ah that's going to be so. You have to reaction.

G

Yeah, the one with that pointers is that there's a lot of places where you want to put in a link, and it's too, the the link length is not big enough to to allow two very large updates. Yes,.

C

That is a big problem. There yeah internally inside the data structure, you're, usually fine, because, like that's what I say, that's trying to get across with dynamic versus static, where, like sometimes you'll, have a link and you'll statically know like the code. You want to apply there, but that can actually be applied by the parent code so like when you decode the object, you can sort of like explode. That out means the demo pointers. That's when you need to do the ac wrapper stuff, but.

G

Make it work with objects and then solve in line at all, because we can call it lining in a compiler step, yeah.

B

Like you can still treat it, a separate object not have to do in.

G

The cid and then be able to point to objects, and in line all these things as you're, generating I'd like the.

B

Idea of inviting multiple different options of interpreting the data, because you may have like a binary class that you want to like as text or as like raw binary and like that's two different, like green views.

D

uh And we're gonna serialize that to the deck or we can do that. Let's just let's just point.

B

To the work, something what.

C

Are just multiple.

B

C

Thing about a lot of discussions is like definitely the layer uh so like the problem is like the ipld is complicated and confusing to a lot of people, because it has a lot of stuff uh where like but like at the bottom like it is, you have an object and you get a set of links. The next letter up is, you have an object and you get a set of name links that you can resolve the next layer up is it goes to a data model so like like.

C

If you create these kinds of layers, and you can let people opt in to what they need and then you can at the minimum, get like bitswap a graph sync over like just the links yep.

A

Yeah, but so, like you know uh doing it with uh in the link or the wrapper, I mean as they sound isomorphic it's just maybe a more efficient way to lay it out happy with that.

C

On your use, the reason we talk about that a lot is like there.

D

Are two things.

C

That help them like one, if you want to swap things out really fast yeah uh and two like small blocks, can be problematic for storage all that kind of stuff. If you get that fixed, the thing is: that's a transport issue that we're leaking into the data layer we shouldn't. um uh So ideally we fix the transports, but it's like the problem's.

F

Like okay, what if I advertise that small little block.

B

Now I have to average.

C

All these additional blocks, which is like yeah.

A

Well, I mean you could also you know for changing ipld really in this version or well less.

C

The other version.

A

Yours is actually more like transport layer compatible right than.

F

This, where there's.

A

Differences in the tree right, but you could lay this out where um we know that one disc or I mean actually everywhere, we're just going to inline the wasm into the wrapper, and now it's larger right like we have options for that too. It's just yeah, where exactly you put it but yeah I mean I'm happy either way. Those both sound great.

F

Oh, that's right.

C

A

Really quickly, one other thing: when you have this broken out like this is: yes, you need to have at minimum these things. You can stick whatever you want in there. So let's say that I had encrypted data and I had extra fields I need to pass in. Like I don't know a key, I can do that.

A

That would be amazing so anyway, yeah. That's all thanks.