solo.io Hoot Livestream, 13 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Rust-based Ztunnel for Istio Ambient Mesh?

Description

When Istio ambient service mesh was announced on Sept 7, 2022, the ztunnel (zero-trust tunnel) was implemented using Envoy proxy. What were the thought processes when evaluating Envoy vs Rust based ztunnel? What were the key considerations? Join us to hear an update on the ztunnel and bring your questions!

0:00 - Begin
0:07 - Livestream introduction from Lin Sun
1:11 - Introduction of Kevin Dorosh
1:45 - News updates
2:50 - Why did we start with Envoy-based ztunnel?
3:51 - Challenges of Envoy
7:10 - Kevin was convinced that Envoy was the right approach for ztunnel, but what’s changed?
9:27 - What is work-stealing?
12:10 - Work stealing diagram
13:40 - Envoy’s threading model
15:09 - Why is work stealing so important for ztunnel?
19:10 - Special characteristics of gateway within work stealing
20:50 - Connection balancing
23:20 - Rust-based ztunnel
25:00 - Live demo of Rust-based ztunnel
35:43 - HBone explanation from Lin Sun
37:32 - Resumption of live demo
39:45 - Delta workload discovery service
42:40 - End of demo
43:00 - Questions
43:17 - How to connect with Lin Sun and Kevin Dorosh
43:42 - Closing remarks from Lin Sun
--
Join Solo on Slack: https://slack.solo.io
Follow us on Twitter: https://twitter.com/soloio_inc
Follow us on LinkedIn: https://www.linkedin.com/company/solo.io/
Past episodes: https://github.com/solo-io/hoot

A

Hello, hello, welcome to hoot episode 42. today we're going to talk about raw space z channel for istio ambient service mesh. If you recall, when it's your ambient service mesh was announced in September 7th uh this year, 2022 the sea tunnel, which stands for zero trust tunnel, was implemented using the envoy proxy. So what would a thought process um when evaluating Envoy was raspace zetano and what are the key considerations? I am so excited today.

A

Kevin is joining me to explain the thought process and we will give also give you guys an update on the channel latest development. So as always, uh thank you so much for joining us and please let us know if you have any questions in the chat um with that I'm going to pass the mic to Kevin. uh Why don't you introduce yourself to the audience.

B

Hi, yes, uh thank you for having me this is uh I, can't believe it's already our 42nd, who uh you could call me a long time, follower first time uh host over here. I'm one of the tech leads here at solo.io. I did some of the some of the testing work.

C

Initially, for ambient.

B

Mesh on Amazon, um you know but definitely involved in the project, and you can talk about why certain decisions were made recently so excited to share the insights. We have.

A

Awesome, thank you so much uh Kevin. So, let's briefly get to the news. Unfortunately, I didn't have a lot of time to prepare for the news. uh The first thing I want to mention in case you miss it. uh We have a g tunnel project in the istio.org and this is the new raspace z-tunnel. Essentially, the Israel Community is investing very heavily in the rough space uh Z tunnel and recently I I believe it's last Monday.

A

Yes, uh Kevin and I actually published a Blog um the thought process of how solo as a company deciding and join the community on Raw space zetano for istio's ambient service mesh. So Kevin, do you have any news in mind: yeah.

B

um The only thing that comes to mind, uh yeah I, saw recently that Amazon had getting back into service mesh space. They had app mesh and now they've got this new VPC lattice thing um so you're still working on the details, it seems pretty new, but it's exciting check it out. I love, seeing surface mesh being adopted kind of across the industry.

A

Yeah, it's definitely a very exciting area, thanks for sharing that news Kevin. So with that I would love to kind of start a discussion with you. uh The first question I have Kevin to you is: why did we start with Envoy based zitano, initially in uh in September, and also before September, right.

B

Yeah well, I think. The the first reason is that it was you know, starting ambient mesh. We had that high level designs the objectives, you know what we thought it would look like, but in the implementation days it was very much like a almost, not POC level, but you know we knew it was Alpha. We wanted to get something that people could test and and look at quickly. um To that end we thought you Envoy was probably the quickest way we could get there.

B

We already had the infrastructural Investments around cicd, you know familiarity on both the developer and the uh you know operator side, um and in doing so we would discover things that we maybe didn't, anticipate um and better to do that in something. That's like battle hardened and we know well uh so yeah it felt like Envoy was like a natural First Choice. Until we we discover otherwise.

A

Yeah and now that you are talking about this so for folks, if you guys remember uh in this blog, we actually talk about uh some of the challenges of envoy. uh Maybe I'll pass down to Kevin to talk about that uh next.

B

Yeah so um I think the the first part that you've got highlighted here, we're talking about some of the the reasons we liked Envoy. Initially right, we covered you.

C

B

In the long term, as well things like uh cves and shared code, you know if we could use Envoy again. You know we have to do extra work. uh You know the customer onboarding I just talked about. We could be ready in production faster, um but yeah in doing so. In this Alpha phase, we knew that there would be complications too uh some difficulties. uh The envoy implementation for ambient mesh um is pretty complex under the hood.

B

um This is largely because of envoy's design, but you know decisions early in the projects where it was designed to be multi-tenant. Each thread be separate and we'll get more into that. uh You know separate memory, we'll get more into that in a bit. But the kind of result here is that the envoy configuration for z-tunnel is actually quite complex.

B

If you ever did like a proxy config dump on Envoy and the alpha release of uh ambient mesh you'd see you know, N squared plus, you know configuration for you know per pod uh abstractions that you know the cluster listeners. Those common Envoy interfaces are very much meant to be per tenant, but here Z tunnels is really kind of a multi-tenant proxy.

B

um It's responsible for proxying traffic for tons of different identities, anything on the Node, and so when you, let's take an example like the envoy cluster, you know you got to pick your mtls certificate to use. You might change the cert depending on the destination, um so you have to have Envoy pre-configured with every cert that it might need, and also, if we use uh you know, an internal listener in the implementation to defer the load. Balancing decision based on whether we need to use H bone or not so another example is, is exactly that.

B

We need listeners listening for whether it has the protocol or not, and each one of these kind of decisions that we have in the implementation, which is, you know, explained in more detail in this blog and in some of the Articles underneath but uh kind of. In short, the con configuration gets very large. It needs leads to lots of CPU and memory usage and NSB software. Perhaps some performance issues as well.

A

Yeah I remember at one point before Ambien was launched, whoever we need to debug our Envoy based z-tunnel. We come down to like the control plan for zetano becomes so complicated. It's just so hard to troubleshooting any single type of problems, and then we really have to type um people who are really really expert level and Envoy, either as Solo or sometimes at Google. So it was super super hard um and like Kevin was mentioning I guess the config dump was complicated.

A

I think the logs was also pretty complicated, like the access log uh for Zeta no Angry base. Titano was a little bit hard to read because of the internal listener because of the traffic jumping a bunch of Hoops, even with things that I know uh yeah, it was definitely complicated, so Kevin. The next thing I want to ask you I. Remember at one point, even though we talk about complexity of envoy right, we talk about a lot of things. We love about angloy as a as a company.

A

Solo is entirely built on top of our way, initially with our glue Edge product right. So at one point you were convinced that Envoy was the right approach for zetano, um because a couple of reasons I'm just trying to understand you know what's what's changing your mind,.

B

Yeah, so it kind of to dig into that thought process. Further um yeah I was I was pretty convinced that Envoy would be the right choice going forward. I know that we talked a little bit about how envoy's multi-tenancy kind of assumptions in design uh were problematic to the envoy, config dump and likely the processing and the number of listeners clusters. Everything like that, um you know part of this evaluation was kind of evaluating.

B

How could we have solved those problems if we wanted to use Envoy right um and we did, there was an exercise you know done. You know public in the community, Google contributed heavily solo was was involved and, and part of this is you know we could have used special custom listeners and and routing filters. They were uh we detailed ways that we could simplify Envoy to make this to make this work without a configuration explosion, um and you know looking at that.

B

It seemed pretty reasonable, like the end result with a custom listener, a certificate provider to let you kind of live, swap the cluster mtls configuration seemed pretty reasonable. It felt like Envoy, was pretty naturally doing what we wanted. As a multi-tenant proxy, in that imaginary, final State, um but the and we would get the the cves common, you know code we'd already familiar with it, we didn't have to redo cacd.

B

uh You know, developers in the community don't have to learn rust or a new language, um but the the question really came down to a concern brought up about uh work stealing. So this is kind of what really changed my mind. Personally, um you know. The really short argument is that we wanted to support I'll talk more about what we're stealing is, but basically, if we didn't do it, we didn't think Z tunnel could perform as well as it could, um and it's not really easy to bring work stealing to Envoy.

B

So I don't know if you want me to dive into that yeah.

A

So yeah, so for the folks who are not familiar with works Sterling. Can you first explain what it is.

B

Yeah, so the idea behind work- stealing is um you know if we think about uh just computer architecture? In general, you have a certain number of Hardware threads operating on cores on your machine. This is true on your laptop or out in the cloud um now. The idea is that they're, probably not all threads, are busy. You can do a lot of work simultaneously. If one is not busy, you can steal someone else's work and be more efficient. That way, none of the threads are, you know, you know, lazy or not doing anything.

B

um The you know assumption baked into this is that memory can be shared across these threads um or else it can't properly steal the work and do it. uh The problem with envoic threading model is that it's not. It does not share memory across threads. It was kind of intentionally designed to delete copyright to copy memory to thread local storage. Each worker thread operates.

B

You know, on its own copy of the memory um they're not shared intentionally, which makes the code a lot easier to write, especially for complex, like L7 policy like implementation filters, things of that nature become a lot easier. Perhaps even more performant, but in a multi-tenant case, where we want to support brick stealing does not work.

B

um We've got some diagrams here to.

A

Yeah, we would love to see them, so just a clever one thing for our audience. When you talk about work. Stealing you are talking about.

A

um One thread starts a job but hasn't fully finished the job right and then some other thread can comes in.

B

It lended we lose you or did you lose me.

A

Yeah so sorry, I went away for a second I.

A

Died for that uh Kevin you were sharing the.

B

Screen yeah and then I thought I broke things, but here let me present share my screen screen again. I think it'll.

A

Help you yeah yeah, because the question I was trying to get it I think it's important to understand. What you are saying is uh for the jobs, that's in the middle right and then for another thread to kind of take over that existing job right and then be able to continue uh with that job without like restarting everything, because it was able to reuse the memories from a different thread. Is that a good way to put it yeah.

B

I would I would kind of separate it exactly it's a pretty good way and I think this diagram pictures it pretty. Well, it's it's in the diagram here. You've got this thief who's. You know, Q2.

C

A

B

C

Go to slideshow mode yeah.

B

uh So yeah here we have, um you know Q2 is is empty and it you know on a runtime, can steal the work of of job three and put it over here. It kind of operates at a different level requires a runtime scheduler. So you know in this context we might have uh you know a single thread pool trying to share all these cues and jobs. I just want to be clear that this is operating a different level of of the kernel, scheduler and processes and threads.

B

You know the kernel is, is limited to you, know number of Hardware threads that you have and each context switch is very heavy. So we try to avoid those very aggressively um and to do work stealing properly um and be highly performant. We want to keep things very light.

B

You know, keep the jobs quick um and let the runtime layer on top of you know a process handle that for us um it can scale to you, know more threads and workers, then we have Hardware threads and we can avoid contact switches by having a runtime do scheduling on top but kind of in our ideal scenario. If we want to maximize performance, we want to have a single thread, pool shared by all of our you know, kind of workers and then be able to steal work when they're not busy.

A

Okay got it, can you also just to contrast things? Can you talk about how Envoy works today, I assume it's without the work scheduling.

B

Yeah, so it has none of that that runtime scheduler can be done in in Rust. It's got the Tokyo Library, there's simple, plus equivalence like boost, but um out here in Envoy land. The way it's implemented today is kind of pictured. Here you know on the left. We have the main thread which handles XDS stats, admin, Process Management. The main thread does everything but handle you Downstream requests. Each worker thread, you know, has the listeners connections, it has its own copy of everything, including memory, um and so you have your main set of memory.

B

This is thread local storage, TLS, and so when you're writing Envoy code or you're in a filter. Here, whenever you want to get memory from Fred local storage, it copies it to its own copy of memory. You know it gets like a readlock copies. The three in whatever's in memory bucket three and each worker has a copy of the memory that may or may not be completely up to date, but it can always use its copy of the memory further.

B

Each worker threaded Envoy has its own thread pool, which means they can't be shared across, and if we go back to our ideal scenario, we prefer you know one thread pool and to have that shared have memory shared versus the way Envoy works today, um so just kind of as a necessity of of supporting work. Stealing um on waste is really kind of far from that.

A

Okay, so I guess the next question would be. You know. Why is world wolves dealing so important for Zeta note right because we've been using Hawaii for a long time, I mean it's still project right, we use Zomboy for psycha I mean we're going to continue to use Envoy for Waypoint proxy and also gateways too. So why is it so critical to zetano yeah.

B

So it really all boils down to engineering trade-offs. So you know I'd like to highlight how important connection reuse is in Z tunnel in a second, but um I think you know if we think about the ideal for work stealing here. Other assumptions built into doing this properly. When you implement a work, stealing runtime or scheduler, there are still locks. You still need to lock on lots of the memory that's shared, and it's only performant if the jobs themselves that we steal are very quick. You know if they're, very small, so that we can.

B

You know stealing that is effective and things are waiting if we think about complex L7, Envoy logic, that's probably less quick than something at The L4 level, um so work stealing is a better engineering trade-off. You know the locks will perform better. The simpler the proxy is so number one Envoy being complex with L7 can be to its own detriment of needing work stealing, but there's also some. uh You know other reasons that we might want to care more about Rick stealing.

B

um So if we think about the connection, pool and connection reuse, um I kind of have two different diagrams depicting similar request flows here, we've got L4 with sidecars and L4 with ambient. So here we're just talking about um you know TCP, packets and encryption.

B

um You know, let's say we have two kubernetes nodes here in the sidecar model and sidecars running with like two threads as configured. If it wants to connect to Applications bar and Daz with replicas on two different nodes. You know it makes a total of 20 requests per second five per connection. We've got four different connections here.

B

um You know if, if bar and bass are the same service account, so we could also just up the replica count and not do different services, but in the Z tunnel model uh you'll be connected to Z tunnel and we can reuse that connection connection four is really kind of the Crux of it here. If we look at the Z tunnel threading model, it's going to primarily connect to other Z tunnels. So here we have a sidecar that needs to create four connections and that might scale very large with each pod.

B

But z-tunnel is going to scale more per node and each connection is going to be more aggressive, um so kind of. In short, the the request pattern is a little different and it's more important. We can reuse connections.

B

um Does that yeah? That's.

A

Really interesting, I think you highlight an important point about in psychop. We don't really share the connection between different applications right, but in the context of Z tunnel we open a new connection with this non-exist for a particularly source and destination service, account pail right so in your example: application bar 2 and bats two. They when they share the same service account. So it could be. Oh, it could be the same connection tunnel from the sources Eternal, which is really cool.

B

Yeah and with that connection, reuse, we're saving the you know, TCP handshake, and by leaving that open, we can really bump up performance and connection reuse with work. Stealing it's another thing that we saw just in the community. uh Cloudflare uh had success with that their pingora proxy as well I think they got connection reuse from 87 to 99, something so um there's a lot of also just real world examples where you can see the impact that this can have.

A

Yeah totally now you talk about uh psycha right. uh What about Gateway? Any special characteristic of Gateway make work stealing not so interesting for the Gateway.

B

Yeah, so I didn't do a separate diagram, because the the reasoning in rationale is pretty similar. um You know the reason why Z tunnel it matters more is because we can reuse the connections and the targets are are mostly Z tunnels and things threat local to that node. But in the sidecar model we could have tons of connections to tons of PODS. You know at Scales per pod rather than per node, and the Gateway is very similar. The request pattern is is very much the same.

B

um In fact, it's also. You know a good reason to keep Envoy as L7 Waypoint, because if you think about the Waypoint proxy enforcing L7 policy and ambient, it basically is a gateway. um It's basically a service account specific Gateway. That applies policy on on egress or Ingress, um and so its threading model very much models. Sidecar, regular Gateway, which works pretty well with Envoy.

A

Yeah I guess the other side of the right, so Gateway you, you can potentially run multiple replicas same as Cyclops but Detail. No, you can't right. So that's number one number two is Gateway. There's no way to guarantee that Gateway only process layer for traffic right, a lot of Gateway actually does layer.

C

A

Also, so that's another advantage to continue use Envoy for the Gateway. Do you agree with that?.

B

Yeah yeah I tend to agree.

A

Okay, cool because I just thought about it: uh let's see what else. uh Okay, there was uh uh Niraj had a Blog and I personally kind of will ask you about. You know how uh what's mentioning, in the blog, through a onboard filter, to balance the connection different than uh than what we are trying to do here uh with uh with work threading model.

B

Yeah so there's um you know it's a lot of different angles that we can. We can talk about. We spot spoke a lot about. You know we're stealing a connection, reuse, um I'm sure those of us who have spent a lot of time working at Envoy have thought about. You know how can we just distribute the work equally, you know, and is that going to be enough? Is it a similar problem? Is it a different problem?

B

um It really is kind of tangential and we we do obviously want to balance our connections um but to highlight the blog that you're talking about um that Niraj had written. um He kind of had this identified this performance problem that you can see in the left here, like we said before, each worker has its own copy of memory and serves requests um based on Downstream client. uh You know request patterns.

B

You can see that it's possible that they all end up on the same worker thread um and then he was noticing that this caused it bottleneck um in the blog linked before, and the solution here was to go ahead and you know: do a collect connection balance, Envoy, filter on the the inbound and outbound listener sidecar, uh and the result of this is that, if you, you know, you can see the metrics here, you would get connections distributed equally and not you know vastly improve performance and you see improved performance and connections, and you might think naively.

B

This is the same problem where we can just do this, um but kind of just going back to what our ideal state is. um We want to avoid the cost of initiating a connection. um We want to reuse ones that are already open. We don't want it to you know if we distribute this equally we're going to need to initiate a connection perhaps to the same destination numerous times when we might be able to get away with doing it once um you know if we could share a thread pool.

B

So um it's a little bit different.

A

Yeah yeah, that makes sense, so this is more for new connections with uh reusing the existing connection and then kind of will steal the job and finish it up. Yeah.

B

A

Okay, cool um with that uh I guess we're going to talk about uh the rust based seat handle. uh So, as you all know, uh if so Kevin I'm going to remove your string here and I'm going to quickly share um my screen a little bit so I'm going to first uh share the browser first yeah. So, as we mentioned, the raspace zetano is a it's a repo in istio.org and I. Actually just got a build myself.

A

So one key cavia is if you're using Mac OS, which I am uh you won't be able to build uh the fips stuff for long uh Linux, X 8664, so make sure you uh use uh disable the flips um and then there's actually really good instructions on here too. So you can follow the instructions and how to also deploy uh ambient uh using the raspace Z title So. The instructions are out there uh with that I'm going to see if I can show a demo um uh share.

A

A different screen and Kevin feel free to ask me if there's any questions that I didn't thought about, so I'm going to pray the demo Gods. Hopefully, everything works.

B

A

All right, great okay, I'm going to change the.

C

A

Little bit smaller all right, so let's go ahead, get started um so the first thing I want to say, is I have a kubernetes cluster, um so it should be two nodes: um I have a ambient worker ambient work 2 and also a control, plane nose and in my closer right now, I don't have anything. Actually. Let me launch um the K9s on here and also on here. So I have a little bit more contacts, all right so um and in here I am going to jump into here.

A

The first thing I'm going to do is uh I already have the images built like I mentioned to you. I was following the instruction to build different images, so this is the raw space c tunnel, I build it locally on my Mac, so the first thing I'm going to do is actually push these images to my doctor to my registry in the kind cluster, so I can use them to install ambient.

A

Now the next thing I'm doing is install ambient, so I'm in the experimental branch and I've set my Hub and tag to use localhost 5000 and uh with the ambient value as the tag so notice. Here. I'm specify using Z tunnel as the image so I'm using the Z tunnel. Raspberry zetano image I just built yesterday and I'm also continue to run the ambient profile, which essentially enables uh edgebone for for the proxy and the Gateway um all right. So ambient is uh installed uh with the raspberry zitano.

A

So what I'm going to do here is on the first one, um I I'm going to show you um so I. Have uh the Z tunnel installed right. So it's the Z tunnel is a demon set like we mentioned before. So it's installed on one replica on each of the worker VM, which I have three I. Have it's your control playing and Gateway installed? I have also the CLI installed, so I don't have any application at the moment, so everything is just uh playing installation of ambient.

A

So the next thing I'm going to do is label the default namespace to be part of the ambient.

A

You could potentially do it after your parts are deployed, but I think the current code base have a little bit issues, so I might just just doing a label first before I deploy app as we need to work through that issue, Upstream, so I'm going to deploy the demo application, which is the sleep and the hollow word so hopefully on here. We should, if I jump into um just show.

C

The default namespace, hopefully I'll, be able to have them up learning, um maybe I'm. Let.

A

Me double check this under if I have the right. Okay, so I wonder: why is this maybe I need to slash at the end, all right that was it? Okay, um okay, so I have the demo application coming up now, so uh it's basically sleep and hollow wood version. One and two you guys, probably all familiar with that. If you've seen uh Christian's uh talk about Ambience, so essentially a hollow World version. 2 is running on the same node, which is ambient worker 2 asleep, which is our client and the hello version.

A

One is the cross node right all right, so I have these uh deployed and uh let's see uh it's still trying to come up. So let's give them a few seconds all right, almost there, okay. So uh the next thing we want to do is we want to be able to call the sweep uh color from sleep.

A

um So now it's his version two. So that's the same. Node I heard version one. That's a different note. Now, if we go to the Z tunnel logs right, so we should be able to see uh some logs, that's interesting. So, for instance, this is the hollow version. Two um I'm. Sorry, hello version 1C tunnel, which is uh ambient worker okay. So if we go to that log, uh you will see a connect request. Okay, so first of all, you'll see the certificate right.

A

This zitano can represent hello, world and you'll, see the edge phone connect request to send to hello version one. So uh there's that and if you go to.

C

I guess we should have another zetano uh MBM worker.

A

Okay, ambient worker 2 yeah. So if you go to this c channel log, uh let me send some traffic, so it sends to hello version, one right so yeah you can see it's trying to send to the destination 1.4, which is hollow version, one using Edge ball.

A

um So yeah, it's established the outbound traffic to uh the destination zitana uh any questions, Kevin.

B

And no questions for me here.

A

Okay, great, uh let's continue that now. The next thing I want to do is figure out the parts for Z tunnel on on the first walk of the, um and also the parts for the tunnel on the second World Cup again, and uh the reason we're doing that is we're going to look at the config dump right uh so similar as on boy. You can also do config dump for uh raspberry CT tunnel.

A

So what we're doing is uh we uh dump the config dump of z-tunnel for um the node on the first node and second node? Now, if we do a format the config down you can see, this is actually readable, right, I remember when it was on way based uh zitana. There was literally like three to five people can read the configuration because of the Hoops jumping through the internal listener. It was multi. uh uh Multi-Air tendency may represent different identities. It was really really hard to read.

A

um This is actually human readable right so because it's essentially this captures the entire zetano config, um and you can see it has the workload information right. So this is our hello world. It speaks the edge ball. It's not H1 native, uh which psycha should be actual native, uh because this is leverage Edge phone through the Z tunnel.

A

uh So this is hollow version 2, and it also have some of the istio system uh system pod, and this is our sleeve and you can see some of the system parts they are not in ambient right, so they don't speak Edge bone. They can only speak TCP right, even though it's your D is there and we also have Z tunnel and it's your English Gateway. So all these are speak playing TCP.

A

Besides the workload information you also see work out to with right uh Kevin early on talk about part of the challenging with the envelope c channel is to be able to resolve from the web to resolve to the actual endpoint. So it's important for us to understand how the width maps to the workload and this this map would show you that information.

A

In addition to that, there's also a map for whips how, from a whip you can map to.

A

How firm, a whip, I think you can map to the actual, endpoints and I think that's pretty much it that's important and then just typical version and config right. This is a lot simpler from the on body based uh config, Kevin.

B

Yeah, it's it's infinitely simpler, I think even the the best case, Envoy config. This is probably simpler. um It's one.

A

Of the many benefits.

B

Of moving to a net new proxy, so.

A

Yeah I remember it was even just for like a 203 app, it was like tens of thousands of lines, multiple tens of thousands of lines yeah all right, so the next command I'm going wrong is a proxy config and uh examinate uh the zetano config that we just reviewed.

A

um So this would show you a little bit easy to read format of the of the workload. The information that the Z tunnel is aware of. So you can see um uh what we actually just reviewed in Json, but this is a nicer format uh that you can see these uh three application deployments uh in ambient, which can speak the edge phone protocol and the others couldn't all right.

A

uh So this that we review the config we reviewed the log- and the last thing we want to do- is actually uh pull out uh a traffic capture, so just making sure from the sleep to hollow wood. We're getting meter to us I believe this should be neutrality as 1.3, uh upgraded connection and.

B

A

C

B

One quick question: not enough of the comments, but perhaps uh do you mind just a quick second talking about h-bone for those who may not have heard of it before.

A

Yeah, so hpong is a new uh I I think it stands for HTTP connect uh overlay for networking environments, it's the new uh connection tunnel, that's being um that's, been using istio to connect for inter-service communication in SEO ambient service mesh, so essentially um the the traffic from Zeta mode to the Waypoint proxy or the traffic from zetano to Z tunnel. uh It's speaking uh it's going to the edgebon tunnel and we typically open edgebone Tunnel per uh per source and destination. uh Service uh service account payout right.

A

So if you, uh if the Zeta notices are existing connection between the particular source and destination, which is a slip to hello wood here, it would try to reuse their connection. And if you know, if this nothing exists, it would create a new connection. Do you want to add anything Kevin.

B

No just that it's it's relatively new and brewed for the situation and the the purpose of doing an HTTP connect, I believe it's HTTP 2 and it will be HTTP 3 connect, but it allows us to pass additional metadata So. That's its purpose about the connection.

A

B

The bite raw bytes.

A

Yeah, that's a great point, all right, uh so I have uh this thing in uh we're using uh this uh debug container right attach this debacle container to the zetano Pod uh ambient worker one and uh we're going to uh use uh term hack uh to capture the traffic on Easter zero. Now the next thing I'm going to do is sending some traffic so um so come to a different terminal. Just so we can send some traffic. So hello version, two, uh that's the same node. So essentially um it doesn't run cross node.

A

So this because the traffic from the Pod hello version 2 to the tunnel is plain text and also the traffic forms sleeve too zetano is also plain text, so it's less interesting in that case, but we should have some interesting traffic now. So, let's see.

C

A

um So, for instance, this all right- this is the end, so the scope up a little bit so this uh to is 1.2 I believe is back to the control plane, uh so 29 Dot 254, that should be the istio control, plane, I, believe it's your D um I think it's the control planes uh SC. Is this service and it's zero to see yeah. So that's seod's uh control, plane, uh whip address 254..

A

um So so this is uh from The Zeta node to the ecod control plane. Remember, uh Z tunnel is served as XDS client. It also serves as a CA client right. So it needs to talk to Ico D and that's secured through uh TOS 1.2.

B

A

Really wish go ahead, I.

B

Was just gonna say that XDS is a new custom, brewed workload DS the idea is going to be a Delta workload, Discovery service, which it helps Eternal scale further, as well like an on-demand set. um Rather than sending larger resources, we can send a customized special Spa workload, Discovery service with just what we need and if we can do it with Deltas, then the request to load on pod endpoint changes are very small, um so.

A

B

A

Yeah, that's a great uh comment Kevin, so.

C

A

So basically, the workload configuration that we reviewed together early the conflict dump right. So those are the configuration XDS and workload configuration the Z handle gets uh from ecod yeah it's much reduced and plus uh Delta uh is I. Think Delta is even Implement here. It's.

B

Not existing yet, but it's planned to do. Delta and On Demand.

A

Okay, I because I here are from John that might already be implemented, the the Delta for the sea tunnel. It was certainly not something I have tried uh to uh confirm yet, okay, so you see the request to slash hello comes in right, so that's from our source, uh 1.2 to destination 1.4. So if we go to the.

C

A

Can type so 1.4 is the hello word right, so that's the destination and the source is 1.2. I suspect is the zetano uh yeah. So it's the Z channel on the ambient work.

A

And uh you'll see uh the application data, so this is uh from 1.4 to 2.3 so 1.4, which is thank you.

A

um Okay, 1.4 is version one and two point: three: is uh the ambient worker 2 to the Z Tunnel right? So that's the most important pass because that's cause uh node and that's uh encrypted using to as 1.3 so very cool. um Basically, you get uh automatically Mutual tiers traffic across different nodes.

A

um Just by adding your pods to ambient I think that's the demo I have uh Kevin. Is there anything else you think might be interested in showing.

B

I think you did a great job covering this and I think the the audience does as well.

A

Okay, great well, folks, if you guys have any questions uh we would love to, uh you can send them through the comments uh we haven't seen any questions, but we really appreciate uh you guys, join and send us to your questions.

B

Thank you for joining us.

A

Yeah well, if, uh if there's no questions Uh Kevin, where can folks find you if they have further questions.

B

At work, we've got uh just drop into the public slack uh solo slack um on all any of our docs. There should be a link at the bottom, I think believe. It's just solo, Dash slack I'd have to look at the link itself, but anyways we're very active and responsive in the join the community and uh chat there I'm also in the istio slack, if you're in that and uh under the name, Kevin dorosh, so um just drop a line I'm happy to to chat about whatever's. On your mind,.

A

Okay, great uh I, want to take a minute to thank you Kevin. Thank you so much for getting on the show, and it's been really nice chatting with you. I feel like I personally be learning a lot from you um work, stealing connection rebalance, be able to steal jobs uh and always ready model. So thanks for all the education you have on me and our audience.

A

So folks, uh if you like this, uh please give us a thumbs up and if you have any future questions or any topics you want to see for any of our future, show please uh open up a GitHub issue or send us a request to uh through istio also looks like and with that I would say: I wish everybody a happy holidays. Our next episode would be in the new year, so see you in the new year thanks so much bye. Now.