wasmCloud Community, 15 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: wasmCloud: k8s Applier Demo, wadm Discussion, Roadmap Update, Community Callout! - 12/15/2021

Description

wasmCloud is a platform for writing portable business logic that can run anywhere from the edge to the cloud, that boasts a secure-by-default, boilerplate-free developer experience with rapid feedback loop.

A

Welcome to awesome cloud wednesday for december, the 15th 2021 as usual, we'd like to get started with the demo taylor I'll hand it off to you thanks liam. So hopefully I already have the magic sharing powers um like I do.

A

Aha, look at that everyone, if you're a familiar watcher of these you'll notice that liam assigned me the screen sharing permission, so we have successfully been this demo has been more successful than many demos so far.

A

Well, we'll see if that holds um this is demoing, something that we will soon flip the open source button on that I have not done yet, because I've been in meetings this morning for everyone to see this is some work we've been doing at cosmonic um and it's work that we feel would be valuable for the community around providing basically a bridge to the future for people who are currently running in cooper minis.

A

So what we have here is um a a set of hosts that are running inside of kubernetes. um This is a dashboard that I, if you're familiar with kubernetes um I've port forwarded it I'm going to be using some kubernetes terms. So if you're not familiar with it, it's okay. This demo is probably not for you so right here we have several different hosts that are running now.

A

The problem is, is you need some way for traffic to be able to get into the lattice, and so a common way to do this is to expose something using an http server, no matter what it might be. So what we've? What we created are two different, um uh two different, uh an actor and a provider. I should say that are to be that are meant to be used inside of kubernetes. The first one is the kubernetes applier interface. This is one that we're open sourcing. This interface um has one provider implementation.

A

I don't think it will need more, but theoretically someone could implement a custom one for their own, crazy stuff that they do it at their own company. However they've customized it, um but this kubernetes supplier basically acts as a tube control, apply, dash f and a kube control delete dash f. So this allows for the for you to apply resources from within an actor now the service, applier actor is an actor that is, um will watch for link definitions that are for http servers and then automatically create a kubernetes service.

A

So right now, you'll actually see one in place. Let me go ahead and show all hosts so you'll see we have the echo actors running on some of the other posts here and then we have the http server definition. We have.

B

A

Server running on four different nodes and I'll show what this infrastructure looks like after I show it working and so.

C

A

Now I'm going to go ahead and delete this link desk just so, we can see that this works and I'm gonna come here and hit the enhance button.

A

There we go enhance um so now we have this. um This uh thing running. We have two control service, oops, sorry, future get service.

A

Okay, so right now we just have some some things running inside of kubernetes, but we don't have any link definition. So you'll, you won't see any service, but right now this service isn't accessible because we haven't linked it. It wouldn't work anyway, so let's go back here and actually define the link, so we'll link it from the echo to the http server with hdb server and we're going to give it an address of 8081..

A

So we're going to go ahead and submit that so now we have this link here and if we go back and do two control get services, you'll see that um a service was created. Listening on the port that I defined for 8081., and so now I actually have also address just.

B

To show that this is working.

A

Because I'm running this all locally, I'm using an ingress contr, an ingress thing and I'll output it in yellow, so people see I'm not doing any funny. Business you'll see that this is pointing at the service name with the path of echo. So now I should be able to do curl.

A

Echo and I'll get a response back.

A

So what you just saw in in uh in summary here is that I was able to automatically create a service that links to the newly exposed port on each of these hosts and to allow traffic to come into lattice, and so at this point now that it's in you can do whatever you want like these, you could have hosts running outside of kubernetes along with hosts running inside of kubernetes, and then your your things are able to talk into services that are currently existing, even if those are running external to kubernetes.

A

So, like I said this is meant to be a bridge to the future. um We believe that, with in the wasm cloud project, we all believe that wasm is the future. In particular. Wasn't cloud is a very good way to express that future, and so this is a way for you to go from kubernetes, which is the thing that most people are used to doing at this point and bridge it into something when you start something new with wasn't cloud. So how exactly do you set this up?

A

Let's go ahead and look at this is the there's a kubernetes, applier and once again like this is going to be the repo that I expose in just a second as soon as I'm done with this demo.

A

This has all the different parts and components inside of it, but inside of the the actor I created a nice little diagram of an example, uh architecture which is exactly what's set up in this example that I've been doing so you could have. um What you're supposed to have is a set of router hosts. Okay, and these hosts need to have this label right here. Wasn't for the for this actor?

A

You could technically customize this actor to make it do whatever you wanted to, but I mean it'll have this label value attached to it, and so what happens is on these router hosts on every single router host?

A

We spin up an http server provider so that way, anytime, a link definition is created that port will be exposed on all of those http server providers across those router hosts, and so the kubernetes service will point at those router hosts when it gets when the um service supplier gets the the link definition, it creates that service for you and it will automatically route through and then be able to access wherever the actors are running on inside of your lattice.

A

So that is the basic overview here. Are there any questions.

D

Taylor, what was that um like? How did you determine the service name in kubernetes? It's like a super long string. Yes,.

A

This is the uh actor id lower case, so it fits kubernetes dns rules. um So it's going to always be the actor id. um If you do some sort of you can actually you the intent of this. This actor is actually to use it as a template for wash nuke wash new actor um as well, so that you can basically because it's going to be the same kind of code that you're going to use to do this.

A

If you wanted to apply multiple resources, you're going to have to come up with a different naming scheme which might mean you have to have like a key value store available to be able to like map some sort of randomly constructed id to the link definition it's tied to, but because this is only doing one type of link definition, the actor id is enough to know to basically reconstitute the id based on the link, def being deleted, and so the nice thing here is it will clean up for you I'll show that to you I mean I kind of have showed at the beginning.

A

But if I go ahead and delete this and go back and do a queue, control get services, we'll see that it's gone. So this will do the management for you and it's just using that. So the id of the service is just the actor id which should allow you to be able to say. Oh, this is pointing at this actor. It kind of gives you a nice way to to go back and see which thing is pointing at which actor.

B

So is the capability provider keeping track of the node ips of all of the potential target, http servers, or is it just creating a service that uses the the selector.

A

To find the route servers, it's using the selector, so in this case this requires that that label, which is the wasmcloud.dev route 2 label- um that's that's a requirement to use this actor as it's compiled, but honestly like if someone was using. If someone wanted to customize this with their own use case, they could pretty much fork this, as is come in here to the to the service, and then um right here apply so um right.

A

Here is where I put in the label, um you could have any sort of selector that you put in for those labels, and so, if you have some other set of labels that you want to match on, you can change that inside of here. um That's that's a completely viable option. You can. You can.

B

A

This is the the simple bare minimum, so people can take this. It's completely usable 100, as is this, isn't a toy it can be used. But if someone wants to really customize, they could create a custom version of the applier, where you could say, use these set of labels or use these very special, configs or or types of resources to create like if you're, using sdo or or link linker d or any of the meshes, and you need to like connect something you could throw something in there as well.

A

You could connect create that in your actor, you could have that be part of your applier. It doesn't really matter. um The complexity can go up from here, but this does provide a a very much a viable solution for someone who just needs to create services, which is the base unit that pretty much any of these things interact with anyway.

A

uh That's awesome, taylor and um I look forward to getting some feedback um in this on this. I know we have helped some folks get this going. um uh You know to sort of enable um the community- and I think, there's. um I think this is super exciting between this and the helm chart and um ultimately, some of the stuff that we're working on long term.

A

I think that we have a really good story here to help people you know start using the power of webassembly on their existing infrastructure and investments that they have today um great, hey, justin, um so I guess a quick around the table. um I know we've had a few open source uh issues um closed and uh matt's has been contributing uh a bit. Is there any road map stuff to run through today, steve or um uh uh or do we we on hold for anything along those lines.

E

uh Well, as you, you saw taylor's demo, um we've gotten some community feedback that there's a lot of interest in running on top of kubernetes and although we're not dependent on kubernetes, that's what people are using so you've seen uh the helm charts a couple weeks ago and uh the uh the kubernetes supplier that taylor just demoed.

E

So we we continue to listen to to what people what people are doing and really want to make it work. For you I'll also say there are no no patches required for log4j vulnerability in in wisdom cloud. In fact, uh we like remote code execution uh we want to. We want to run even untrusted actors, so um good job y'all, and uh so that's that's. Our roadmap update.

A

That sounds great um uh brooks. Do you have any um good first issue callouts this week or some couple open.

D

I do um I've got two new ones for this week. We can do that uh right now. Maybe if I, if I only had okay, so I have, I have two new um good first issues. These are things that are two new issues that are actually, I believe, they're they're. Both um one of them is submitted from a community member.

D

One of them is from taylor, um but when I see things like this that are like relatively well um specked out and and like we're still working through like our prioritization, for what we're doing for each sprint, I try to tag them as good first issue, because it'd be a great place for somebody to come in and plug in um I'll zoom in a little bit just so, it's probably easier to see the first one is in wash there when, when using the wash rag commands for for ping or pull or or push even if you properly use the insecure flag, if you submit an oci reference that has like a url scheme, so http or https this causes, I guess an unwrap on an error value in in wash, and so the specific line number is here, but it looks like this is something that you know when you're when you're using an oci reference, and you include http or https, which is pretty easy to do on accident.

D

I think that wash has a good opportunity here to provide a better, better feedback for the for the user. We should be able to tell them if their oci reference is invalid, because this is something that you know I've personally done before when like helping to work on the oci distribution crate. This is something that you know. If you mess up the way that you're specifying a reference, it could easily happen.

D

um So this is a great um great good first issue to get in on, especially if you're looking to just do a little bit of rust work and the second one is actually also a great issue to get on.

D

If you just want to do a little bit of rust work for those of you who have developed with wasmcloud when you're, compiling actors and providers and signing them, you may not even know because our goal is to make it really seamless for the user, especially with our like make commands but wash when you're, generating or when you're signing an actor or a provider we automatically generate in keep a pro sorry a in key key pair for your actors and providers.

D

So this is how we're securely signing our actors um and when we do that, we store it in your home directory at dot wash keys, and then we can further look that up in the future, and this is really just so that you don't have to deal with managing a bunch of keys when you really just want to get started working with developing mods and cloud.

D

So the tldr of this issue is that when we automatically generate these keys, it looks like we, um they have a a set of permissions of 644 for read, write, execute um and that should be something like 600 um and the directory should be 700. So this is something you know if you're familiar with some unix file system things um and have maybe poked at this at rust before this can be something that we may want to test on windows, and I personally have when windows, linux and and mac environments.

D

So if you're submitting an issue for the or a pr for this- and you only have a mac or only have linux or something like that- I'd be happy to help test that out once you submit fix so yeah. Those are our two good first issues for the community call out. um These are great ways to get involved.

A

That is awesome brooks. Thank you so much for that uh just open floor. Are there any other? um You know discussion items we would want to bring up today.

F

I do actually um with elixir that are probably questions a little bit outlandish, which I've been chasing around for a little. While um can I share my screen? Almost there you're on.

F

I think this is the right screen all right, um so I've been chasing around stuff with muslim or whatever. um So this is a before I get here, but basically this is a supervisor that exists that we want to have the monitor as a gnat connection, and then this is a test that I will actually show that tries to actually hydrate the state. But I haven't figured out how to do that. um So I went down a rabbit hole, um that's actually to do with the connection state disappearing.

F

So this is the same thing on the other side, which now opens a named connection um here and then there's a supervisor tree and it starts an initialized supervisor and you actually call it through here. So if I go back to this guy, we're not seeing your screen sorry, I missed the share button, all right, my bad uh sorry.

F

So um so I've been working on the integration between what trying to work on and the integration between the kubernetes operator and observing the lattice and using oem to get the state of the lattice up from running as well as observed.

F

What I will get to eventually is actually showing um well trying to hydrate the state of the lattice using uh the application, that's written by kevin, so the lattice observer, um nuts observer, etc. I did end up going down a rabbit hole, which is why I am late on this. um One of the things that I actually noticed was that the gen server actually loses its pid, sometimes not always, um and I've been trying to trace that through, and I don't necessarily know whether people have seen that problem before.

F

But what I have done here is to basically copy the code out into a different area. It still has the same supervisor, that's running on this side, um and this so and it still has the same name. These guys are the same and then in the test. It actually just tries to call these two routines here.

F

So what I actually noticed is that, depending on what's actually going on that, this will actually uh blow up sometimes and not blow up again. This is because I didn't save it.

F

So in this case, it's actually saying that the pid disappeared in gen server, and I don't necessarily know why that is the case, because on the opposite side, if I do the same thing, which is running line number seven, which is this guy here- um this will actually pause, but that's actually still going to the server and getting connected to the gen server. That is under the name gnat.

F

So I don't necessarily know about sets happening, and if I look at something magic like this, this is using pry to actually see what's going on. So I'm going to start mix using that method and then, if I do um make stop talk, stop doing something which is up here.

F

I'll actually cry, and then if I do gen server thought: where is.

F

That actually returns the process id, um so this has happened to me spontaneously or like randomly a couple of times. I don't actually have an explanation for why this actually happens and happens only a few times uh so, but so that's, I kind of left that problem around um brooks. I see your hand up.

D

Yeah I just wanted to um our resident elixir expert kevin. I I guess dropped off, but I wanted to um essa's disciple, try and try and help you out um this one if it only happens to you occasionally, especially since you're able to do that, pry and like check the the process id and it exists or like check that atom and you find the process id.

D

um I'm curious if this is just like a race condition, type of thing where the the supervisor is started as a as a child and it just doesn't end up initializing all the way by the time you get to that part. In your test, I see you have like a timer dot sleep there yeah you try that out.

F

Start link actually is supposed to make everything synchronous, so it's actually not supposed to return so yeah that the timer hasn't. Actually, I did try it, because that was my first thought as well, but then I was reading through the documentation and looking at whether this was going to make a difference. It didn't actually make a difference um so yeah, it's just a rabbit, hole that I've been chasing through. I don't necessarily know why that is the case um and it happens randomly in different places.

F

But seems like it actually works in wadm, one of the things that actually had so, and this is back into um using the lattice observer and getting the distribution manager, I think, or deployment manager um working. So here I do start with supervisor, which is the same one that I was showing before um that one monitors the connection and that will actually connect here.

F

If I go back um so this is now starting the observer, which is just the one with the connection, and if it goes down, it will actually maintain the name and this dumps out. What I am not sure about is how dehydrate the state- because all of these are basically returning.

F

There is nothing in here right now. That's actually defined that lets you hydrate the initial state inside of the deployment manager. So if I have another test that actually runs, which doesn't even start the connection, then this is there. One of the things I actually did notice in otp is that there is a colon of a um well that's. Basically, this hello, which I think kevin is using to hydrate the state. Is that something that we're planning to do here?

F

As well, and should I write that part or is there some other mechanism that I haven't seen that I am unaware of.

D

Yeah, so this is something that we have on our backlog for the lattice observer. um You know in the otp host and the wasm cloud dashboard, and then I guess you can also like add wadm into this. Now we have initial like a a problem where, if you start them before any events, start coming across the event stream, um then you know you can supervise, not supervise. You can watch for those events and you can update state accordingly.

D

But if you start something like your lattice observer or the or wadm after things have happened in the lattice, then you have to, I think you're using the phrase like hydrate your state, um and we just don't have that now um so.

F

Before as well- and I haven't actually noticed anything coming from the ladder, so there is something that I'm not doing right well, I don't actually see the correct apis.

D

I I think, you're, I think, you're right.

D

I think that wadm um probably doesn't have this logic to to listen for the events, the same way that the host is um or- and you I think we we have it in the lattice observer now um to we have it in the lattice observer now to like observe a lattice for events, um but in terms of hydrating from the start like if things have already happened in a lattice, we're talking about doing something like creating a stream or like an interested stream in gnats in jet stream, that, like registers, interest in the wasmbus events topic, and so when you create a lattice observer for the first time, you can do things like create an ephemeral consumer for that stream.

D

um Essentially, you get to replay all of the events that have happened in the past um and and that's the method that we're planning to use for like for for hydrating, our initial state with the lattice observer.

D

um So the lattice observer as it is today should have the it has the mechanism to listen for events and and populate a state um and that's what the wadm is using under the hood right.

F

um So I guess what I'm trying to do is actually hook everything up together. Yes, they work in terms of like the functional tester there, but they don't actually tie all the ends in. So that's why I'm actually observing that there is no way to rehydrate the state but yeah the.

F

If you had the cloud events come in then that's where it actually starts, because that's all mocked out, um but nothing to actually listen to the cloud. Events on the wire as a mistake.

D

Yeah, all of that should be in the lattice observer side of things and not on the wadm side, um but I'm actually not super familiar with where wadm is as of right now. So I may not be able to answer the question for how far that is.

F

Yeah so I mean okay.

F

Yeah, like the test we have here for reconciler and stuff like that played around with before it works. If you get the right cloud event in and that's what I did as an initial demo, but now I'm actually trying to hook the whole thing up, um and I was just trying to see because what I actually have is.

F

uh I have otp running and I created an actor after I started everything up and I was hoping to see things come across, because one of the things that's actually mentioned is that I need to define uh this guy here with the stage change event which actually goes to this guy, which is an observer which is defined in the lattice observer and it's a callback uh but it's magic.

F

So I'm trying to figure out what the magic in.

C

D

I thought that that state logic was already implemented on the lattice observer side, like with.

F

D

F

So I guess in the readme, um this is the reason you're basically saying that, once you actually do a connection um from whatever application is actually using it, then you need to define a module which will get the event eventually. Hence the handle info question. It's gonna have to send it somewhere. Somehow.

F

um And I was hoping to just see a bunch of messages dumped out, but I know I'm not wearing it correctly, because I'm trying to figure out how to wire it.

D

But you don't see even even though it's stubbed out and has those logger.d bugs you don't see. Events come across the wire.

C

F

Okay, I mean like when I look at the gen server. For example, there is no handle info, which I guess is where the messages actually come from right now from, and that part is.

F

Yeah, so you have all the handle, calls that actually do get prefixed get lattice, and this is all based on the messages that are already there for when you actually start the lettuce, because that just starts as an empty dummy. So first question. Hence the quest question is like how do you actually hydrate this and then once you actually get messages across um either through the callback or not to the callback?

F

uh Something needs to work too and, like I thought, dude decay was kind of there, but no because it's still not really connecting all the way out.

F

um And then handle event with another one, which also is trying to update the state with the new state as opposed to the old. But like the.

C

The actual call out so to speak is missing. If I can.

F

Call it that, um because yeah- I don't I mean since the well, I mean nobody's really. This is not a pub sub thing right, so it's actually basically something actually happens. It sits on that and then you try to get the messages or intercept the messages that they come through yeah. I guess jet stream is the cubing mechanism, so you can actually go and empty out the entire queue and then go from there, but that seems I think, that nobody's actually consumed the messages so far.

F

Because I think once it's delivered, it disappears because at least not the way to understand it to work.

D

Yeah, I think, even though we're we're using jet stream, then the nats guarantee is still like at most one or for message delivery. So you, if you, if you fire off an ax message and there's no interested party, I don't even I think nats is actually smart enough to not even try to send it. If there's nothing subscribed, so it um you know it well. I guess it's probably being delivered to other parts of the otp host, but um yeah it's not guaranteed to come through, um okay.

D

Well, I think that this is something that we can probably touch base with kevin on, because it seems like there's just like one piece of connective tissue: um that's missing for wadm and and its connection to the lattice observer to actually get the events.

F

Yeah yeah, because I think you need to be able to- I think sequencing here is probably important, because you definitely need to be able to get. You need to be able to hydrate.

F

I think in my opinion and then go from there and then actually because I think the rest of it with the applies and all that sort of stuff are actually designed to go, uh do things and then reconcile those actually managed to get things back, um and that I mean that part once you get the events isolated and yes, that works, but it's actually just hooking up the whole thing together.

C

F

And as far as gen server disappearing, I have no idea what's going on because it's actually happened. I went down that path because I was trying this stuff initially and it actually blew up in my face and I tried, I tried all sorts of crazy things. I haven't been solution or answer for why it actually goes up.

D

And that one is interesting if it's, because it's intermittent, I really thought that it was going to be. Like a I mean, the the nat, the gnat um consume uh connection supervisor can take a little bit of time. After, like the gen server start length, like actually finishes, um you can probably find in our otp host, where we like wait for it to be available at one point.

D

um But if you were sleeping for two seconds it shouldn't take that long, because I think the convection supervisor uses like a it will tell you that it's done initializing and then continue to do more. Work like in a handle continue in order to actually be ready, um but I would think that the gin server would be available, even if the nats connection is not yet finished. um It's a lot of words to say that I don't know that seems strange and it's unfortunate.

F

Yeah um yeah, I just tried, like debugging or building up elixir, to actually see whether that actually works, but that eventually takes you down to erlang and I don't know how to make the two things together to actually debug it all the way.

C

Or actually it just, I.

F

C

It flew up in my face so kind of stopped chasing that whole.

D

Well, one thing I you know I I didn't use to do this, but one thing I'm actually a really huge fan of is using pull requests as a way to go back and forth and collaborate. um So anytime.

D

You run into something like this feel free, even if you don't feel like whatever your contribution is, or whatever is done just to like, submit a pr and tag kevin or tag kevin, and I on it and like, if you do that, then we can always like clone down the code that you're working with and try to reproduce.

D

I'm happy to use like a pull request as a even if you just mark it as a draft to indicate that you just are stuck somewhere. We can always use that as a collaboration mechanism.

F

Yeah, I think that I did that once and I forgot to sign things and that created a bit of a mess though, but this one I mean I just was chasing it because I was interested in how it actually worked. But I I think now, I'm just in the case that there's stuff in here I'm sure.

C

A

All right, this is awesome um yeah, and I think um you know trying to set up a couple paired programming sessions is super helpful as well.

A

You know, so we can really help to keep you enabled and unblocked here, because I think that I love all the contributions that you're really driving here, and this is going to be a really cool feature when we get this completed. So um that's great, um you know brooks. uh Is there anything else? You guys want to chat about on the call, or do we kind of pick this up offline?

A

We can pick it up offline, okay, super well! Thank you so much uh for uh uh leading this work and uh it's great to see all the progress that you're making um open floor. Any other call outs today that we wanted to chit chat about.

D

I have a game that I like to play with everybody. Oh I love games, it's a little game of. Where is wasmcloud. I think I dropped this in the slack, but I just love the idea of doing this. So did anybody see where? Where wasn't cloud is at this tiny little map I can I can zoom in a little bit, it might make it a little easier.

D

I can't enhance to the point of the normal terminal enhancing, because then you couldn't see anything.

D

um This is really just a fun way to announce that we, you know wasn't cloud has been donated to the cncf of the sandbox that stuff we've already kind of talked about for a while, but we finally landed on the landscape page.

D

So if you take a look under scheduling and orchestration right under kubernetes and nomad, you'll see wasmcloud and you can click on that see all of our github stuff and our tweets and everything- and it's just I I wanted to you- know kind of show that off on the call, because it's pretty it's cool being a part of the the big cncf landscape, so yeah feel free to play. Your own game of uh where's was on cloud at home, but it's right here under orchestration and management.

D

If you take a look at it, we kind of felt like that was the best category for wasn't cloud, because we're really you know we're not only concerned with just running web assembly modules or like running containers or something we're really in like the bigger space of orchestrating. All of that so yeah, that's that's live on the cncf landscape.

A

I love it, it's great um one, more thing to deliver, just transfer the trademarks and we are done and we already qualify for the incubating tier, which is a six-month process to go through so we'll go ahead and kick that off and start lining up corporate partners for interviews and all that and just be on the road map to or on the path to being incubating.

A

I would call out that we actually hit 90 contributors recently, which I think is incredible. That's in a year you know, we've been doing these. um You know these uh uh weekly meetings. Now we did one every single week we didn't miss a week for every year. I think that's just incredible. You know when you think about the growth, and I think that the message is really starting to resonate.

A

I can absolutely attest that on the on the corporate side, you know we really have a lot of excited and engaged potential large community members that we're hoping to transition in and help.

A

You know bring uh uh bring into the community um so open floor here uh and then I've got maybe a couple community call-outs anything else to chat about.

A

Okay, um uh kubecon uh submission deadlines is friday december 17., so pull together. Your was in cloud submission, um even aspirational. Barcelona is still five months away uh and um get in the queue because we'd love to have more users more valencia. Thank you so much. I think I tweeted barcelona today. I should know I'm gonna, be there for web assembly day. Of course, now webassembly day has not opened up our cfp.

A

Yet so, if you decide you'd rather do a more focused presentation, there's still time for that, and we've got a lot of fun stuff in the queue for that as well. I really look forward to the collaboration and we'll pick up and go from there.

A

I guess that's it for today, as usual, I'll go ahead and stop recording and we can continue to hang out and just chit chat a little bit, but if you're watching at home have a great day.