Istio Community, 29 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio April Meetup/ WASM filters for data observability

Description

#Istio #Data #CloudNative

In this video, the complete meetup that includes the talk given by Nilesh and David is included. Their talk describes how they built a custom filter using web assembly and demonstrate its application for sampling service traffic, mapping dependencies, detecting schema changes, enabling anomaly detection and more.

A

Thanks for inviting us, we are layer 9. We have been working on data observability using steel as our foundation, so we are very happy to be here uh telling you what you have done. um We have used webassembly as a core technology to do a lot of these things that we're doing so we'd love to share our experiences that building webassembly filter.

A

Now our vision is not very different from studios, broader vision, which is uh provide observability of security privacy out of box, uh based on a deep understanding of the data planes.

A

um So our kind of the key uh you know the key technique that we're using is actually going pretty deep in terms of understanding uh the network traffic that services are using to communicate and use that to derive pretty deep insights into how services are performing from a security privacy observability point of view, and let me actually give an example to describe um the kind of differentiation that we have.

A

So you know if you look at services, communicating with each other. uh The apm tools today look at the requests and responses, and you know out of the box, uh be able to provide metrics like request volume, latencies and response codes.

A

So still today, you know comes with built-in dashboards out of outer box that can tell you between any pair of services. What's the traffic volume what's the latency in and what response codes are we getting?

A

But you know by kind of looking at data itself, we can actually draw. You know much deeper insights into what's happening. For instance, services often have logical endpoints, so everything all the dashboards that you see in stu could become a lot more powerful if you can actually split them by endpoints. uh That does require us to understand.

A

You know in some more depth what's happening with the request responses. What are the url patterns use some intelligence to understand what are the logical endpoints?

A

uh We should be able to look at what kind of authorization is present and what are the api keys? Are they being rotated? Are they shared between environments? So we can understand authorization in a much deeper level. Now, endpoints have you know schemas? They have specifications in terms of what kind of requests they accept. What kind of responses they give.

A

So we should be able to look construct your entire api and down to the to the field level uh and understand the types um and be able to kind of detect when those change uh you know people are very interested in understanding services from privacy and security point of view, so understanding like looking at data plane. You know we want to actually look at what kind of sensitive data is present in each field and be able to tag your end points and services with what kind of data are they exchanging?

A

um And then you know a lot today. A lot of application metrics are manually injected, but looking at the data plane, we should be able to look at pretty rich data metrics at the schema level at each endpoint.

A

So our the vision at layer 9 is to really understand uh the network traffic deeply to understand how services are communicating, look at the payloads and then derive a lot of very deep, rich insights in terms of how they're performing um you know, we, the core foundation we are using, is basically a way to understand payloads, and you know we have an onward filter that lets us asynchronously sample a bunch of requests between within your entire mesh.

A

So the rest of the talk is about how we, our experience, is building an actual web assembly filter. uh I will send over to my colleague chandler, who is going to describe our website filter and then at the end, I would actually give a live product demo to show how this actually works in in real life.

B

Thank you, nilesh and I'll be calling on nalesh uh to change the slides, because if you try to change who shares the screen, clearly it's 20 21. We can't do it so this slide uh saying we need to capture the data. So the natural istio extension point is an envoy filter for us, because we do care about things like the http request body.

B

We care about tcp as well. So if you're talking to kafko layer9 wants to see what did you write to kafka? What did you just read from kafka and if you're going to have a an envoy filter webassembly is you know exciting, because it's shiny and new? It's not generally available. It's my understanding still so many times along the way we felt like we might be the first people trying to really put an assembly script. Web assembly filter in production uh assembly script was uh the second language.

B

We chose the beauty of webassembly. You can write in any language. You want, except not really, but you know a few years from now. Maybe any language you want right now, probably tiny go rust, c, plus plus and assembly script, because if you choose a language other than those you're going to write your own proxy runtime, so your web assembly.

B

uh In order to look like a beautiful c plus program, it needs to rely on somebody else's library. That's going to speak to the application binary interface. That envoy is presenting to the webassembly virtual machine.

B

So um when you compare webassembly to lua, which is uh the first thing we tried before webassembly in envoy in istio, existed um in lua you're, going to have more trouble saying complex things because of the language lua and the libraries available to you and um blue is a little a different way to think about filters. It's not the way. An envoy filter, implementer from the c plus plus native c plus plus world, would think about it. It's a little different and it's going to be http.

B

Only webassembly works for tcp http, both uh udp theoretically um next slide.

B

uh So they're yeah, so I mentioned like their native c plus plus envoy filters, but then you're, maintaining an eternal fork of the fork of envoy that istio uses for istio proxy and then how do you get that to?

B

um How do you deploy that to production? uh Well, at the very least, you're gonna have to you know, build your own fork of envoy and build a docker image around that until istio use this docker image for istio proxy, not the standard one.

B

So if you want to try your filter and then stop trying it not so easy compared to webassembly where we can really just tell kubernetes, here's an envoy, filter, custom resource and istio tells every envoy run this web assembly.

B

There's this other mechanism, the external process, but it's http only it doesn't seem to be very polished. I don't know if it's generally available, uh where you can do grpc to an external process.

B

Lua is you know the first thing uh people tried and apparently they didn't like it, because now there's web assembly, uh it just seems like a kind of a dead end. At this point, no one's ever gonna put uh tcp support into lua, because there's webassembly it's it's the way you would do it from scratch. Next slide.

B

So now you're going to hear what happens if you actually try this when we tried this, which is over the past year, I think the first experiment with blueway was somewhere around isco 1.3 or.

B

B

B

uh So lua, when we first when nalesh first tried it, uh our filter needs to make a http or grpc call to the layer 9 service.

B

uh If you're a customer, you install two helm, charts one for the envoy filter, one for the layer, 9 service and to make that call at first in lua it was a synchronous call which was highly sub-optimal, but nowadays the the call is asynchronous. So lua gets the job done for http well enough for us and we have a filter that uh it just observes.

B

Basically, it doesn't modify things it just observes things. Lua can modify at least some things, maybe all http things you'd want to next slide.

B

B

I was um the first one at layer 9 to try webassembly and I tried c plus because a dozen years ago I was uh very good at c plus I don't know rust, I know go, but the tiny go support it's like well, tiny goes not really go and it just people. People had written some things in the docs. That made me think I don't want to do this in production. Not really I'll use c, plus, plus battle tested, it'll work, and it will.

B

uh I now know how to make it work, but at when you're just starting out. The first thing I tried was basically the add a header example and everything I tried was an instant segmentation fault.

B

uh Once once you could figure out the plumbing to get the web assembly by code to envoy envoy would crash segmentation fault and it wouldn't say here's the problem. With your c plus program, here's a stack trace from the c plus plus program. You wrote it would give you a stack tricks from envoy and then you'd say what did I do wrong?

B

It was not very friendly, and this could have been because I was using istio 1.5, even though that wasn't the latest thing available at the time, but we had a bold vision of let's just write. One web assembly filter that'll run on all versions of istio ever because we want.

B

uh We want more customers, not fewer, so the more versions of istio we support the better it turns out at the time you the very latest and greatest also would not give you webassembly back traces. So you'd have to hope that your c plus envoy back trace would clue you in as to what you did wrong next slide.

B

um So, let's try assembly script so why assembly script, um I don't know rust uh tiny go seems like people. Don't really use that I'll. Just try! Assembly script, so nalesh wrote an assembly script filter that didn't segmentation fault. I mean it did it first, because anytime we would call an abi.

B

Let's say we passed in a a null for an argument to this uh function. uh Envoy would crash that just seemed to be just doesn't matter what function you call envoy's just going to crash, but eventually he made it work for an http call. So that's the rubicon. For us. We need a filter. It needs to make an http or grpc call.

B

So we said well, this is a little scary, because eventually this needs to run without you know crashing the sidecars in your service mesh. But let's go so when you're writing an assembly script. We started at assembly script. 0.9 assembly script is a new language.

B

That's made just for webassembly, it's not a subset of typescript, but it kind of seems like it um and it's moving fast, the language so assembly script, 0.9 the you had some very ugly typescript because it just didn't support a lot of typescript language features, but that's fine because we have this nice proxy runtime written by solo.io.

B

That would hide a lot of that from us. The version of assembly script. uh If you changed your version of the proxy runtime, you were going to change your version of assembly script. They were married.

B

We noticed this isn't much. Bytecode c was generating a lot more bytecode in the wasm file, so assemblyscript is making something tiny and c plus was a docker image to compile it ran in scriptin and who knows what else to generate the webassembly by code and it took forever and long enough that I thought well, I'm gonna have to set up a build machine for this.

B

I tried to cache the standard libraries and things like that. It was slow. Assembly script was fast. It didn't use the docker image at all. You npm installed assembly script and you, you built your web assembly by code, so that was nice next slide.

B

So, as I say, if you try to call a function on seo, 1.5 say the first version that had webassembly support.

B

If you're not perfect, then you get a segmentation fault. You don't get an error message. Your webassembly virtual machine doesn't get terminated and the sidecar soldiers on none of that this is different than today.

B

uh It's not as different as I would like, and maybe that's part of why, as far as I know, webassembly's not generally available just yet um next slide.

B

So why why the focus on 1.5? So we don't need to do much. We like the original abi. So if you run istio 1.5, it's got a different webassembly avi than istio 1.6, which is different than 1.9, which is different than so these apis they are versioned. They change they're, not um perfectly versioned and they're, not very well documented. So you have to basically read the source to figure out what has changed between these versions, but they are versioned, but we we're fine with 0.1.0.

B

If we can, we want to write just one webassembly program for every seo version ever made, and our hope is that you know istio 2.0 will still support the abi 0.1.0, which is where istio 1.5 was um so we're going for universality, um and if we don't have that, then there would be some challenges, because that would mean.

B

So if we had to have forks of our program for different istio versions, we would have to have different versions of the proxy runtime, and that would mean different versions of the assembly. Script, compiler and so we'd probably have to change our program to work for different versions of the compiler, so we'd, like universality for many reasons, next slide.

B

um So this is about assembly scripts uh runtime. um It had a bunch of a bunch of functions for tcp udp. The um the network filter support as opposed to http filter support, um except it didn't have all of them connected, which was another clue for us like. Maybe we are the very first people trying to to actually put an assembly script web assembly filter into production.

B

um It only took a dozen lines to add the other half of this bridge and that's been merged upstream, and things are good unless you want to do grpc. Grpc is still not good, because the the original abis for grpc they sort of require that your program can create and manipulate protocol buffers which, if you're in assembly script, no, you can't unless you write your own protocol. Buffer library and assembly. Script was written by the same person who wrote the most popular javascript protocol buffer library and that person has not written anything for assembly script.

B

So you know uh maybe it's impossible. It's not it'd be a lot of work, so it doesn't exist. So if you want to do grpc, you should probably use c plus- maybe you can do it in bruss? Maybe you can do it in tiny go. I wouldn't do it in assembly script, because you'd have to write your own support for that, but tcp. It works very well. Next slide.

B

uh So when we, when it was working for the easy cases, it was time to see, you know: okay, let's shake all the bugs out. So let's load test it. Let's, let's stress test it. Let's chaos engineer: let's try to break it, um so you do things like take down the layer. 9 service, just it doesn't exist on the cluster right now make an http call to it. What happens?

B

Segmentation fault cry change your code uh file, the bug. So just there was a lot of time. You know trying to get this uh to the point where we could. We could actually ask customers to install this filter um a lot of time and there are some things we still have to work around for certain versions of istio.

B

So we got to a point eventually where we like it. We like it a lot it does. You know we carefully monitor our test beds for crashes and it doesn't crash. It hasn't, crashed in a long time, but this was a painful process in part, because when you have a crash, you don't have a back trace from your language, because we're doing this work before the back traces for webassembly existed, which I think was histo 1.9.

B

So without that back trace, how do you figure out what's going on? Well, you have to figure it out from the envoy c plus backtrace, and we tried to get core dumps and poke around in gdb. That turns out to be hard to get line numbers there's a script like stackdecode.pi.

B

Well, it turns out you have to build your own envoy, docker image. If you want to try that, so even that is a lot of work, so just to figure out what went wrong a lot of work.

B

Yeah, this will definitely be easier in the future. My hope is that in the future, not so many segmentation faults and instead the virtual machine uh essentially gets uh deleted. If, if your program does something that upsets one of these functions, you can call like. Please start an http call next slide.

B

So once you have your line numbers and your c plus plus on voice deck trace, just just to figure out exactly the source code, I was looking at uh well. That definitely depends on the istio version right so for 1.5. I think they were using github.com envoy, which, if you look at it today, the blurb about it says this is just for developers for some testing. But I think it was like the real repository for cutting the releases of 1.5 and 1.6.

B

So you have to find all the repositories the webassembly support was upstreamed somewhere in the middle between 1.5 and 1.9.

B

So nowadays, in modern code, it's in proxy wasm cpphost is a repository that has much of the interesting code, and so just figuring out, like the right set of code. To look at uh is a challenge, and if you want to build your own binary so that you can really really get to the bottom of that crash, so that maybe you could, you know, make make envoy hardened against this, submit a pull request. Like that's a lot of work next slide.

B

And then a function that didn't segmentation fault but returned essentially like not found when it should have found something uh we just wanted to get some properties and if you read the documentation for that, you'd probably be surprised to learn that in webassembly you have to insert some null characters in your strings to dig into protocol buffers. So if there's nested messages like a nus b, which nests c then you're going to need a null b null c, and uh so even the things that you can do with it.

B

Not everything was was easy and documented properly next slide.

B

Oh yeah and after we thought it was finally it stopped crashing it does. What we want. Can we release it? Can we ask customers to run this uh even at that point? It's like well, we've invested significant time and we still may not may not be able to release this until a future istio version, because we've got a memory leak.

B

The memory leak, if you, if you just you know, do the harshest load test you can and hammer envoy again and again then in something like a few minutes, uh it would go a gigabyte of memory up uh which is not good, so the webassembly virtual machine in istio. Today, it's all v8 and there's no way to customize anything about the memory there's. Also, if you have a memory leak, one proven solution is we'll just restart the program every now and then you know every monday.

B

We need to restart this service because it has a memory leak. Well, you can't do that because there's no abi for quit. I quit I give up. So your filter is going to run until the envoy filter. Kubernetes object is removed um or changes so even doing something like uh reverse uptime, which sounds like a fancy way of uh avoiding solving the real problem.

B

Even that would require, like a kubernetes operator, to mess with envoy filter custom resources. So we were very motivated to fix the memory leak.

B

uh So we we already had a fork of the proxy run time for assembly script, so we just updated it to use a a bleeding-edge version of the assembly script compiler that had an entirely new garbage collection system, or so it seemed from the docs it didn't take many. It was just a few lines of changes to the proxy runtime to use that no more memory leak.

B

B

All right next slide.

B

uh So once you have webassembly bytecode that works very well for the versions of istio that you care about. How do you get it to them?

B

So when we were first getting into web assembly for envoy there was the solo.I o had wasmey w-a-s-m-e, which had a kubernetes operator that would help you deploy your bytecode and we were hoping to avoid that because that's another thing, another dependency. Another thing you have to ask customers to install another company. You have to ask them to trust.

B

There is some work in recent versions of istio to just give envoy a url for some bytecode, but we want to support some older envoys. So um back in the lua world, you can put your lua script inside the envoy filter kubernetes object.

B

You can kind of do the same thing with webassembly. You can put the bytecode inline uh you're, going to run into like a one megabyte limit. So if you've written a c plus plus program that uses the full protocol buffer library, that's going to be over two and a half megabytes. So you can't do that.

B

But since our web assembly by code is less than a megabyte, that'll fit in kubernetes, so we just in line it. That's. It seems to work uh pretty well no extra dependencies, but we we do. We look forward to uh the future where, where the envoy proxy can just grab it on every every version of istio that we care about, which is probably like every version of istio after 1.4 next slide.

B

So testing your webassembly program depends on your language. If you're in rust, rust has webassembly in the blood there's a core rust team all about web assembly, I'm sure it's very easy to test your web assembly program. The only problem is you're going to have to mimic envoy all these abis. How do they work? How does how does envoy call into your filter?

B

You've got your root context, your stream context, and it depends on whether it's http or tcp, and so you might get it wrong when you're mimicking envoy in your tests, but it would be straightforward for assembly script.

B

There is a s-p-e-c-t aspect for unit testing. It's like javascript's, just framework stylistically.

B

We use it a little bit except uh it's. It's such a pain to try and mimic envoy and just just to mock these functions. I think I'd have to spend another another day with aspect to really get there. So testing is a challenge.

B

Ga is generally general availability. um Someone please correct me if I'm wrong, but I don't think we're we're there yet and part of that might be because of the edit compile segf cry cycle. I hope that's part of it. I hope that envoy, just uh soldiers on and politely deletes a virtual machine uh which gets to fail closed or is it called fail open anyway, there's a setting where you can say my webassembly filter.

B

If it has a problem, just just keep going envoy, which is what we would want to do, some people, don't it doesn't work the way we would like it to, because most of the problems we would have would end up in a segmentation fault in the c plus plus, which means there's the you know, envoy can't run the code that gracefully recovers because convoy's not running it, has crashed.

B

The malicious webassembly byte code- uh this I was reading a github issue where um an istio or envoy developer was saying well in the future. We want to support. uh We want you to be able to run even malicious web assembly without having to worry about it, and I, when I read that I was like oh, I hope I hope I hope I hope this is right, because right now, benevolent programs, edit compile segfault cry.

B

So if you can, if in the future, you can handle even malicious web assembly, then uh that bodes well for trying. You know to write your own benevolent envoy filters, um and I will say nowadays, if you're using the latest uh istio, then yes, you get your back traces from your assembly script program or from your rust program or time you go depending on their language's runtime, but hopefully in all all languages. So that's nice, um I actually don't know the next slide.

B

I should probably know the next slide.

B

ah So now is when uh nalesh distances himself from my remarks and gives you a product demo.

A

Thank you chandler, um I think web assembly. It was hard work, but we felt it was totally worth it. We now love web assembly. uh We have stood on the shoulders of some pretty great communities around istio on y and solo. So now we were able to actually kind of get to the point where we're happy uh shipping this for customers uh who are running it.

A

You know in pretty much all versions starting from 1.5, so I wanted to quickly show you: how does our end product look like and in terms of delivering some of your promises and in getting deep visibility in your services?

A

So this is our ui, um so we have webassembly filter that ships samples to our local data processing pods, which kind of convert those into some metadata and that powers our dashboard. So this is the end result of it. uh We have our own sandbox, showing bunch of services right now, you're, just seeing a list of all the services in our cluster.

A

um I mean this so far might look very similar to um a dashboard you might see on istu. There are a couple of differences. I would point out. One is that a lot of our customers want some kind of cataloging capabilities where they can actually curate a lot of content around this uh people who tag uh their services with teams and owners uh ad run books. Wiki pages basically use some kind of central database to get all kinds of information in about the services.

A

We have built some cataloging capabilities here and then a lot of our customers want to get a lot of external tools signals about the services into central place.

A

Here you can see a ci integration that tells us when was a service last deployed, and we also get in you know, alerts events from tools like your sentry and cage duty and other kind of devops tools. You might be using just get like a central place where you can see all your problems in your services uh from multiple tools.

A

uh This is kind of the same view graphically which is basically clearly, um except that this kind of like plugs into some of our detailed views um like in the future. If kiali has an api that lets us actually build applications on top of it, we'd love to you know, bank, on those, but for today we have something that uh displays this that can hook up to the rest of our app.

A

um The other difference here is, you would see. So since we look at the data layer, we can actually go slightly deeper in terms of understanding dependencies like here, you are seeing a service, that's writing to a specific kafka, topic, uh product payment action and then someone else reading from it. So a lot of these asynchronous dependencies through either cache caches or queues or or even databases, uh we're able to kind of highlight those uh based on understanding the data.

A

A lot of our customers have have dependency graphs, which are you know, pretty much driven through asynchronous computations, and uh they you know they really want this kind of dependency graph. That can take you through some of your asynchronous edges as well, and then we um highlight services where any of the you know, our own system or third party uh integrations might indicate an alert. So let me actually go deeper into a service.

A

And now here you would start seeing kind of the differences in terms of how detailed we can go in terms of uh observability. So, first you will notice endpoints, and there are current points that we have discovered based on some clustering of all the requests and responses, including an endpoint that has some kind of url pattern, and then we are able to kind of show throughputs and latencies uh segmented by your endpoints.

A

uh We have an activity feed that can tell you what's happening to the service uh and we pull in kind of events from um tools like duty, ci kubernetes.

A

uh We can tell you overall has status from multiple sources and can local dependency graph. Now here it says that the throughput and latency looks fine, but there's anomaly in the data itself. um So let me actually click to this endpoint.

A

So, first of all, you would see that this endpoint we're able to pull out the entire schema of how the requests look like uh including kind of the structure in it. The field names the types and what kind of personal data is attached, which type so the pretty deep api spec of what is endpoint provides.

A

uh What responses it gives and what are the structure of the responses we are able to actually go and look at metrics around each data element. Here. There's one element, that's red and if I look at it, there's a uniqueness metric, um so emails very kind of generally pretty unique and then it has dropped down to zero, which means you know you are basically seeing the exact same email. It probably means it's like null or empty string, and we detect this this anomaly at a data level which you're flagging out.

A

So these are kind of like deep insights. We're able to of pull purely out of the box with, like you know, using your istio as a the backbone, uh and we do have detected authorization based on the beer token here.

A

So this kind of tells you uh some of the capability of what we built and then let me quickly show you another example uh which is.

A

A cloud service, so here there's endpoint, which has a classic um kind of http error, and if I look go into it- and I can see here that this used to return 200 and uh it has started returning 500s.

A

So what we can do is we can actually overlay events, uh events that includes deployments changes in other services, kg duty alerts and we can rank them by their correlation to this incident and can blank order them, and here it says that the most likely event is um a schema change and if I actually look at this event number one, it shows me that um the requests have changed your schema and there are two new fields added and that correlates perfectly with the finders you're seeing so it makes it very easy to understand why you're, seeing something anonymous, um and even if that change came from some downstream service, um you get this kind of visibility here, so you can go and uh look at the dependency upstream that caused it.

A

uh You can go to that service. You can look at the owner of that and maybe contact that person.

A

uh That is a kind of worldwide venture of the product. um Let me get back to the slides, so yeah, that's our demo. um You can try it out uh if you are running in steel cluster and would love to see if you have any questions around either web assembly or the.

A