Kubernetes Network Gateway API Special Interest Group, 30 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Network Gateway API meeting for 20230130

Description

SIG Network Gateway API meeting for 20230130

A

This meeting is being recorded, hi everybody thanks for coming to January, 30th's Gateway API meeting.

A

um Just a reminder to anybody who has not been here before or a reminder in general that we are governed by the community's code of conduct boils down to please be nice to one another. um We do not. We have a fair number of people here, but we don't have much on the agenda.

A

So we'll start with what we do have on the agenda, but then we'll open it up for General discussion and if we don't get much it's okay, if we close early today, but if you do have something you really want to talk about, please run in into the dock and put it on the agenda.

A

I will start sharing my screen. We can go over what we got.

A

A

So actually see who put this here, but somebody wanted to talk today about stunner.

B

Yeah sorry, it was me, I'm got one and I forgot to give my name. uh I love this uh gaping fermentation, sometimes like at the middle of December, and then people are generally interested, and then we discussed that maybe we could give a very short introductory talk on this. I think what is maybe the most exotic implementation of the Gateway API, because this one is for webrtc. So it's all about audio video, RTP, UDP I'm, not entirely sure, but maybe we are maybe one of the main users of udprout um around.

B

So uh if you, if you, if you give me like 15 minutes, I, try to very quickly walk you through. Why we need a media gate paper about this. Whatever DC is, after all, and if you have questions I would be really happy to discuss.

A

Sure is this something where you want to share your screen or.

B

Yeah it would but I guess I don't have the rights I can give.

A

You the rights, uh it's probably a better way to do it, but I think this will let you do it go ahead.

C

uh Let me see, oh.

B

I guess you you're supposed to see this now. No.

A

Yeah, we can see it or I can see.

B

Okay, yeah, so welcome uh I'm, uh actually Grabber so I'm this guy uh yeah Video Connection is is is like regard until I do today, but I'm like looking like this uh and I apologize. This is midnight over here in Central Europe, so I'm, not the sharpest uh imaginable, but I try to be very quick and not bore you to hell.

B

I'm uh I'm, gaborate, buddy I hold a PhD in uh algorithms and systems, uh science mostly doing academic, and it does research for like 20 years now, I'm senior researcher at the University, and also I'm working for a major Taco vendor, so I'm mostly doing algorithms and system stuff for for telcos uh and I'm, also the co-founder and the CTO of s7mp. uh This is a small startup.

B

We did just to to do Consulting work and web artists in kubernetes Professional Services, for uh mostly for this Gateway that I'm going to talk about uh I'm, mostly a high performance software, networking person so in in our academic career, uh with quite a lot of work on Fast software. Switching like imagine, 20 million packets per second on a single core. uh We still hold the record for uh for this.

B

uh Of course, you can imagine that when you are with the Telco and like serving millions of users uh from like two or three servers, then you really want to churn out all the the the last bits of performance from a funeral from your Hardware. So so we are doing lots of lots of low level dpdk and Driver uh hacking stuff, but also we are doing agreements um just a short note on. What's uh what's on the agenda?

B

What's webrtc I I guess most of you might have heard about this, but maybe a quick reminder will will be uh useful, but the main question is why we need a webrtc media Gateway for for kubernetes and why this thing might be important into you for for the Gateway API and the stick network, uh if at all, uh some some lessons that we learned in the end and then q, a so like webrtc is web, real-time communication. So this is an open source API for browser-based, real-time Communications.

B

What what we mean by real-time communication is real-time. Communication is exactly what we are doing right now. Okay, so so um my my microphone and my camera is producing the the video and the audio streams right now and you want to consume it right away.

B

The end-to-end latency should be around like below 200 milliseconds, because uh when we actually just passed that that magic deadline or like like latency bound, then people start to crosstalk, um but it's like so webrtc is essentially a JavaScript API, it's a fairly usable and the consumable one, but but under the hood, it's basically just voice over IP uh from the 2000s hidden behind this modern API. So basically voiceover I'd be repackaged uh uh uh to look like a modern service, but underneath this this whole thing, it's still the Voiceover IP.

B

Now the the first drafts and then the first rfcs came out then I think Google was not a thing and we didn't even have a cloud so this this this word has not been invented. Yet uh Sun Microsystems was still a big company. uh Everyone wanted to do Java back in those days, so today it's mostly for video conferencing, uh now Zoom infamously, doesn't use the RTC, but everyone else from Google meet and Discord and Microsoft teams. Everyone is doing that.

B

So, if you look at this, if you see this little pop-up in your browser that an application wants to use your microphone and your camera, that's a test sign that webrtc is uh uh actually just doing its work uh in the background. uh But there are lots of lots of new use cases. We have lots of uh users from desktop streaming and the cloud gaming area live streaming is becoming a thing, so media ingestion, but mostly mostly video conferencing.

B

So if, if you want to to have like the the typical application for webrtc, then then that's exactly what we are doing right now. um The main idea is to do this peer-to-peer. So this was the original Voiceover IP use case. uh uh You have two browsers and then you actually just directly connect the two.

B

This thing is starting to go out of fashion today because, like you can imagine that doing peer-to-peer connection between two users, which are typically behind several layers of Nets, is a very, very difficult, difficult thing to do, but once you actually want to do something more complex uh processing on the media streams that are the exchanged between the clients like, for instance, you just pass the two particles deadline and start or like the boundary and then start to do uh multi-part video code or a cloud gaming session on desktop streaming.

B

Then you will definitely need a media server in the middle um that will do some media processing and, like everything that you do on in a latency, sensitive matter, then manner. Then you have to process live audio and video streams.

B

um That's extremely CPU intensive, so kubernetes is a perfect fit to host this media. Server like there are pools of literally like tens or uh dozens of media servers uh running on this very, very high-end servers uh that actually just do light, transcoding and and trans sizing and then re-packetizing of media streams, uh uh so that uh we can enjoy this sort of mousse media conferencing.

B

Experience that we are doing right now now, when you actually want to minimize latency, you have to design everything from scratch for latency, so this means no HTTP and no TCP. So this is the first thing that we drop and then we introduce this super exotic media transport protocol stack, which is basically real-time protocol. This is our layer, 7 protocol uh on top of UDP and UDP is this red herring. So when you see UDP, you start to feel the pain.

B

Udp is connectionless, okay, so uh and actually RTP is sort of like so the same. So the problem is that when you actually set up a media connection at the media server the client there is no connections there. You do not mean there is no like nothing like the TCP and check or something like that when the clients in the server just enter into this negotiation and when they discuss the the various parameters and the ports and the IP that they are going to use for the connection. There is nothing like that.

B

So the very strange fact is that the only way to associate a packet like in a server or a client we get a packet, and then we want to know which color to be sent or like which video stream this this packet belongs to. The only thing we can use to decide is the source IP and the port.

B

So that's the only two uh header fields in in the protocol stack that you kind of like can expect to point you to uh to a particular color, and then you have to do this Association this decision package by package, so that there is no connection State uh maintained there.

B

Actually there is- but this is this is beyond discussion here now, when you actually try to just associate the user variety and Port is how this happens is that before you actually set up a call, there is a signaling session where, with the associate with the assistance of a signaling server, the client and the and the media, server discuss or negotiate the the IP addresses and Port that they are going to use to send package.

B

Okay, so the like I I, guess I'm, not behind the net, so I guess my computer uh told Zoom servers that I'm going to just send you packets from like 10.0.0.1, that's a private it uh any middle of the box in the media past and then I bet. There is going to be at least one video box in any media in today's internet. Anything that modifies the layer, 3 and layer 4 headers like Network, address translators and load balancers. This basically kill the RTC media connections. You cannot do that.

B

uh So there is a very complex protocol, so you mostly stand and turn. These are two protocols in the very complex workflow ice and Ice light which help webrtc clients to deal with natural version.

B

But the basic problem that we are trying to solve is that you know that kubernetes networking is effectively built on Nets and and load balancers. Okay, so just sending a web artist is a media stream uh uh via cluster IP. Kubernetes service will actually break the connection because the cluster IP is essentially a destination, so you kind of like like change the uh the destination I think that's enough to kill webrtc. So just ingesting UDP and RTP media into a kubernetes cluster is already a major pay.

B

Okay, because because whatever you do, there will be several layers of net from the client side on net to the the cloud load, balancer and then the note port and then the cluster at the end whatnot, and until your packet reaches the media, server that's deployed into a kubernetes pod. So the main reason for the existence of some stunner is to help clients uh or help servers ingesting media traffic webrtc media into a kubernetes cluster.

B

uh uh But once we actually have a Gateway that the processes webrtc traffic uh and does not travel that we can bolt on several very useful features. On top of that, like the usual Gateway uh uh functionality, mostly routing so request routing uh security and monitoring. So these are the uh the typical Services the standard does for you, but this is just a tip uh like specifically for webrtc media. The data plane is a standard, compliant turn server. So this return is a natural as a protocol from the idea.

B

All clients know how to talk to a turn server, so they know how to talk to stunner uh and the control plane is, of course, the Gateway API. Okay, so basically, stunner is a is a gateway that exposes uh this. This very strange protocol uh uh code turn to clients, but inside uh the cluster uh it actually just does UDP.

B

um The workflow is, like I, think the standard stop in the Gateway API Gateway class. Okay, you register uh your gateway class with the controller.

B

um These points the uh to Gateway config through the parameters the Gateway config gives the the general configuration for turn, which is basically mostly just the log level and the couple of authentication credentials currently in plain text, but there are a lot of lots of different uh uh authentication methods that we support, so this is only just the simplest, so this is the usual Gateway config stuff. uh Now this is the way you define your turn server listeners.

B

In this uh uh example, we have a UDP listener, so a turn server that listens on top of UDP the standard, turnpone turn port, and then we also have a TLS listener, because turn also runs on top of TLS. This is in order to allow clients uh connecting uh from behind very, very restrictive, Enterprise uh firewalls, to still be able to talk to to turn servers, so we can also terminate the uh TLS connections, but it's so. This is also the usual stuff.

B

uh A media, routing is, is actually just done through udpirat object, so you define a UDP route like usually uh you attach it to the to your gateway, and then you name the media server Service as the back end and this this is basically everything you have to do uh to expose your media servers to your clients in a way so that they can still be able to load the media server.

B

Despite that, I don't know at least four or five layers of nest on the media path automatically exposes the Gateway listeners over the load, the answer service. So let's call this the IP address like standard IP uh and then the only thing you have to do on the on the browser side is to Define this very short snippet of JavaScript that you actually Define your RTC peer connection.

B

This is the main uh uh API code to set up a video or an audio connection from a browser to a server, and then you give this ice config the parameter and the Ice copy is essentially just uh telling the server uh telling the rtcp connection to connect through this turned protocol listener, uh which we've just exposed from kubernetes and basically that's all. uh This is fully compatible with webrtc. uh uh The the web artist is stacked.

B

This you don't need to do anything uh uh apart from these two or three uh lines of JavaScript on the client side to support this, and it's fully uh compatible also with servers on the server side. We did zero uh modification, and this just just works.

B

uh The status now is that we have a custom turn server, which is basically, this is written, written in goal and if you, if you actually uh have to work with webrtc, I I, can only just uh recommend or just just uh yeah use Bion. It's amazing one of the best and code bases that I've ever worked with. It's performant, it's it's nice, it's readable. So it's good. uh The control plane is a partial, Gateway, API implementation, I'm going to just tell you very shortly, but why it's a partial implementation?

B

We have like an active Community, a small startup backing uh um the Gateway, and we already have a couple of users deploying standard into into production. We are deeper, uh I think GA will will happen sometimes early summer.

B

um The future is performance. Optimization, like I I told you, we did like 20 million packets per second on a single core. Now we are doing 30 000 packets per second on a single core. We know how to make this super fast. uh We know how to do an ebpf candle upload for for handling turn, uh so we are going to enter into the multi medium packets per second VM. This is doable for turn because turn is UDP.

B

uh Evpf is much and no float is much reverse fit for TCP, but for for udpa this at least this thing is simple. The lessons and I'm going to close this with uh with these short slides, the Gateway API we found is a very, very comfortable and nice Target to uh uh or a nice API to Target. So it's it's definitely versatile. You know to abstract even this very, very exotic use cases.

B

So, even though it's it is not TCP, it is not HTTP, it's still somehow very nicely abstract, uh UDP and rtpbs webrtc media problem is like, like we are doing two things.

B

At the same time, we are teaching our users to to use Tanner and Meanwhile we're also teaching them to use kubernetes because most of the people from the on the webrtc community, they are mostly new to kubernetes and the the one of the reasons that uh of the one of the reasons of the of the behind most of the the issues that we see at GitHub is that they they they really struggle on handling the complexities of the role-based model in the Gateway API.

B

um So we try to make their life easier and that's why we simplified the Gateway API as much as possible, so a user difficulty do not use role-based model I'm super happy if they at least use kubernetes they're, really not in the in the in the situation where you know they, they just you know, have these in-house teams uh and and and- and so we won't read in these sorts of complexities to allow them to work together. There are no reference grants we used to have them. We don't support them anymore.

B

We can in any time, but but this is the one- was one of the major reasons why people got confused. They forgot the reference um some minor controversies. I mentioned this at a slack and we decided. Maybe these are not the major problems, but I thought I would just bring this up uh uh again. Maybe if someone has a comment, I would be really interested. Turn is a super strange protocol in that it's layer, 7 protocol.

B

uh This is what we expose at layer 7, but it works basically over any layer for transport. Udp tcpts DTS, you name it, uh but the problem is that what to write in The Listener into the listener protocol field? Okay, we currently set this to the transport protocol that underlies turn that people actually expect this to be turned, and then they get confused.

B

What is now why why this is this is like you, you just expose the term server, but you never really mentioned the string turn in any way in your gateway specification, so they they they they got confused by this. So maybe I think it would be nice for us at least to have a separate app protocol or something to to just name of this MBK the layers Tab and protocol issues. It's not important.

B

They I I'm, really surprised by how fast most of our users grasp on the main ideas of the Gateway API they get confused, but but they're still very efficient in working today. The other is the UDP route semantics. So that's the other thing that that usually confuses our users.

B

uh The problem is that so this is I know. This is super strange, but by the time our Gateway sees the media traffic so that they see the actual turn allocation request.

B

The client already knows the IP of the Pod it once involved to, and this is because in webrtc there is a signaling plane so before any media connection and an audio video connection, the client and the server negotiates uh the connection parameters so like they, the stereo just tells the port IC it's actually just just assigned to to the to the client and the client actually just asks stunner. Okay, I want to talk to this spot that Port. So there is no load balancing installer.

B

Okay, we only just do perform at the admission control. So when is there is a back-end service in the udpira. That basically means that there are not. The client is allowed to to talk to a particular board via a UDP route, but there is no load balancing there. They always thought that somehow inherent to gateways and HTTP routes, there is the there is the the assumption that load balancing somehow happens.

B

uh So if I don't know, I've used this deal quite a lot and there it's it's somehow very expensive that you are doing a load balancing. So the details very often comes up at this Square, so they are just inquiring, but what what what's Happening Here? What? What is the I expect some loading going around here, but then I I know that this this is not the case, so this is another source of controversy, and these are our minor.

B

This list, uh if you can add dtls, this is the datagram TLs version that runs on top of UDP to protocol type and be very happy if our backend object references contained an end port field. That would be really nice because then we could handle the port ranges. Of course we can always write a custom row, uh but maybe, if there's an intention to do that, I know there. Is this this long standing, kubernetes issue about uh protocol ranges in services and I, don't know.

B

What's what the plan around that also I'm I'm I'm super interested in in the fate of udpiras. Will it uh uh uh get promoted to to Beta or vv1? Or so? What's the plan around here and basically that's all so um um thanks for listening.

A

That was extremely interesting. That's cool! Oh happy! Does anybody have any questions.

D

Yeah I was curious. um You said that your users were confused by the roles I'm wondering to what extent they ended up having to be exposed, because I think we tried as hard as possible to make it so that the default you kind of don't need to mention any of it like, for example, reference Grant.

B

Yeah sure uh so one thing that uh that is typical uh in our setup, if you uh that the media servers are deployed in the separate namespace. So this this is fairly usual and then every backend wrapping stunner is basically a cross namespace uh reference uh from stunner uh to the to the actual media, server pool which is deployed other. So so.

B

Theoretically, you would need a reference current there, uh but but I think that the the thing that really confuses them is is the attachment of UDP routes and and gateways and basically routes to Gateway. So there are lots of dancing around uh until these things, like originally I used to have a quick implementation where we actually introduced all the Gateway side uh rules and also the the the the the routing side uh uh parent traps with the with the full semantics. But but then even even my own Google got confused by that.

B

So we removed everything from the gate. Phase. Gate phase are now running with the default uh uh preferences, so everything from the same namespace can attach to our gateways. That was a major simplification and I think since then, we get much less trouble with that, um and parent traps are also super simple, so only just from the same name space and this this kind of like helped us simplify enough, but still just just the last week, uh the the trash the bug there.

B

Some are misconfigured this uh and then you know the problem is that, like I, have to tell them that that status is in UDP routes and gateways are a thing so.

A

Okay, that's what I was actually my first question when we even stopped this was. Is it like a problem where they like they did? There are no statuses being made in that implementation yet, but it sounds like there are, and it's just hard yeah to get to them.

B

Yeah, it's it's somehow like the common knowledge that the statuses are so much interesting. In this whole research.

A

Statuses are kind of a I would say just a mess in general across a lot of different things that actually are implemented on kubernetes, but um it's interesting, so you just turned off the reference grants entirely: basically don't Implement them at all anymore yeah.

A

B

An interesting thing, I.

A

Think we'd want to I think we would want to come back around and try to think about your use case and your experience as a way to see if there's improvements that we could make there to make it so that that isn't the result that we get um but I.

B

I definitely understand why reference grants are there and and when you actually have the role-based, you know organizations. This makes a lot of sense, but it's so simple to forget.

B

It's so simple to to, or even just just you know, have a reference going and completely forget about that, and it's so I, don't know what to do about that and if you're good in kubernetes, you will easily just find the status that tells you that okay, this reference is missing, but but somehow I still feel that, but uh so I know why this is and I don't really have any better ideas how to handle this. But somehow it I still feel that that this is this. This might be a major source of Errors yeah.

A

Just to add on to what Bowie said in the chat the cap to have reference, Grant move under Sig auth and become a standard thing across all of kubernetes and not.

B

A

Just merged today, so there is a lot of momentum to basically make reference Grant a a core thing in kubernetes, so this feedback definitely is helpful and we should. We should keep talking about this. You know after this meeting and talk about uh uh areas of opportunity to make improvements. There.

B

Yeah, it somehow confuses so people who are really new to this. This whole thing: it's just, maybe a too much to take it's- it's surprisingly, nice and, and and and and and it's nice to see how quickly they start to understand, gateways and rounds. So this is, this is really no. These are abstractions which are somehow very convenient, yeah. Somehow reference, I I, don't know these are but I. So anyway,.

E

Can I comment on on the load, balancing, confusion, sure sure um I don't know this than protocol or but, um like the back end I'm sorry, the UDP route that you have up right now, uh the GDP route? Sorry, so it points a back end right so that back end, usually fronts like a pool of pods right yeah, so I can kind of see how that leads to that confusion of like ocean. That service be load balancing between the pods. But it sounds like you said.

E

The stun protocol establishes the relationship, and so there's no like. Oh you know this. The the service will load balance how's. That is that something like the difference between the configuration of Gateway, API and the ways the sun protocol works.

B

No I think this has nothing to do with Gateway apis. It's much rather the fact that so just imagine an HTTP service, no client connects to your gateway and tells this is like you know the service that I want you know like and and then they get load balance to whichever pod and then report you perfectly this, you know in the position to serve the request. Now. What happens in webrtc is completely different different because we, the the major thing here, that we are doing out of band signaling.

B

So before anything happens there is this application server stuff, which is basically a web server. But but the first thing that that you do is that the browser opens of apps okay to the application server and they start um discussion there. I'm I'm, a client I want to register with you and then I would like to join to this and that room, and then my I'm I'm, decent that user and I want you to set up a media uh processing pipeline for you. That's your media.

B

Server that hand handles this room, uh and then please attach me to this conference. These are my references and give me a turn credential or a turn server that I can use to connect to this media server and the application server just just sends down instructions to the media, so it already knows which media server this uh room is assigned to, and this is a stateful media server.

B

The only like there is this video server pool that media servers are not created equal in the sense that they maintain in internal state with respect to the rooms that are currently assigned to them. So when the client actually wants to join a room, it has to connect to that particular media server pod that is actually just hosting the the room the client wants to connect to. So it has to ascend to a particular pod, uh and this is this- is completely different from HTTP, where everything is. You know, restful and even done here.

B

Everything is non-restful fully fully stateful and out of band uh one of the one of the main goals of of the start, I mean one of the visions that we have in this small startup is to slowly try to move this.

B

This use case closer to the cloud native principles: X externalizing stays from media servers, this to some extent already happening, but but so that media servers would be equal so that any media server could process any uh uh any room and any client, and so so this sort of thing, but this will be a very slow progress. uh Currently I think webrtc is anything but Cloud native, uh but but we know I think we have some good ideas how to make this uh uh more Cloud native. In this sense,.

A

Would you mind going to your wishlist page for a second, oh yeah sure.

A

So I think there's potentially opportunity for the first one I would say put in an issue for that and we can kind.

C

A

Discuss that um it's gonna be a little rough, if you're, literally the only person who wants it but.

B

A

Or an issue or a discussion is still fine to like get the conversation going and it may attract people that aren't coming to the calls to say: hey actually, I need this as well. um The second one.

A

Well, it looks like there's an ongoing conversation for that one.

D

So I think um as I remember. This was something that we temporarily tabled because adding endpoint end port was not a backwards, incompatible change, so we we basically said hey well. We can just resurrect this at any point.

A

Again, getting additional support would help that, but you know bump that thread and then for the last one. This is actually close to my heart, so T I'm I'm.

A

Currently one of my charges right now is to focus on the progression of uh layer, four in Gateway API, so TCP route, UDP route to some degree, TLS route and so I have every expectation that you're going to see the progression that you're looking for there I, don't know how fast it's going to be, but getting this kind of like feedback, and this kind of like a presentation uh showing that it's being used helps that it definitely helps to to move things along, and so thank you for kind of coming and showing this to us today.

A

This definitely made me feel good, because I was hoping that there were like some. You know more layer, 4 stuff. We don't talk about it a lot- and this was definitely um one of the more intricate implementations I've seen despite not actually using a lot of Gateway API, but I I think it was very cool. So.

C

A

Yeah does anybody else have any more questions.

D

If we had questions, what's the best way to send them.

B

Yeah uh I will share the slides and give you the uh and then I I have all my pointers there, but I'm also Gabor on on the slack. So you can write me anytime, and there is also the uh so just just look at stunner and uh on GitHub and uh if you find all the necessary pointers there uh happy to have many times.

A

Cool, thank you very much all right. Sorry, I, muted, myself. For a second, um we have another agenda item, uh that's minor, but you kind of picked. Oh, we got two gen items now kind of picked a good day for a uh for it because we didn't have much agenda, so you got to kind of do the whole thing there.

A

um We do have just a a heads up, I I, think Phillip put this in, but like a heads up, there is a conversation going on in slack right now about egress. There will likely be a actual, like more involved um uh bullet point item for us to talk about it next week, but not quite this week.

A

If you're interested in egress jump into the gateway, API, slack, Channel and jump into that thread, it would be very helpful to make sure everybody who's interested kind of uh gets in there and starts talking um and then and I think go.

C

Ahead, one more thing to add to that: it's going to be raised at the next um um I, always blank on the name of it, the gamma. Thank you.

B

C

Meeting so you know we'll start looping them into, and this will end up probably at least temporarily broadening out.

A

Roger that Thank You Phillip uh Dave Studio. You wanted to talk about a TCP route example for iot.

F

Yeah, just so since you're looking for sorry I'm, just cooking dinner, so I was but I was like I heard and I was like: oh I got I got a layer four example: we have um a number of people using Emissary who you know they shift over donville Gateway one day, uh but it's I think it'll be a high priority for us.

F

You know um who use TCP rats because they do internet of things like there's a trucking company that like tracks trucks across the country and every one of their devices pings through and it uh they purely use, TCP routes um or our equivalent of it, uh TCP mappings. uh So yeah, that's, but that's another. uh Like it's low level.

F

You know, uh I was on a really interesting call with one of them trying to figure out how to do something because they were trying to route and they're not allowed to Route off of when you're on on pure TCP um but yeah. You know so I I, don't want to take up 18 minutes talking about it, but um that's another thing: I could bring more details if you want, but since you were saying I love hearing examples, I was like I've got. A I've got a good one.

A

Yeah no I would absolutely love for you to bring a agenda item For an upcoming upcoming uh thing, maybe a a demo in like a brief kind of presentation or something. uh Let's talk about it. uh So just so we're clear, they're they're uh using V1 Alpha One TCP router in production.

F

um I mean so they're going through it through Emissary right now, so they're not going through through Gateway API specifically. But we're going to need to to shift over at some point and um and that's a really critical piece of of Emissary for a lot of people is the ability to send through pure TCP cool.

A

Yeah definitely would like to hear more about that, thanks for bringing that up.

A

All right, we still have 16 minutes left. um We don't need to use them. We can close this if everybody's done, but if anybody has any last minute thing that they'd like to bring up and talk about now would be the time. Otherwise we can uh I'll head off a little early today.

A

All right, thank you all and cheers everyone have a good one.

C

Bye have a good one.