Antrea LIVE !!!, 17 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Antrea LIVE: Episode 4 (Antrea 1.4.0, AntreaProxy, Tanzu feature gates, Carvel.dev integration)

Description

Join Jay and Grayson for another Antrea-LIVE episode - this time - we're going to dig into Antrea 1.4 and test it out by running AntreaProxy without kube-proxy enabled. In addition we'll look at how Antrea enables Feature Gates for new knobs.

- serviceAccount selectors for networkPolicies !
- egress IPs https://antrea.io/docs/v1.3.0/docs/egress/
- Installing Antrea 1.4
- AntreaProxy Enablement
- https://github.com/antrea-io/antrea/pull/2632 doc updates for AntreaProxy
- Running Antrea w/o kube proxy - does it work ?
- VMWare Tanzu feature gates for CNI providers and the Carvel tooling (kapp, ytt)
- kapp-controller
- BONUS ! looking at @aojeas Conntrack fix
- https://github.com/kubernetes/kubernetes/issues/105657
- https://github.com/kubernetes/kubernetes/pull/106163/files

A

All right folks, we're live. Welcome to the andreas show. Luther and grayson are here luther. What's going on, I'm here.

A

I said it's wednesday: it's been a busy week, yeah, it's wednesday you're right wednesday, I'm looking forward to a long weekend coming up here soon uh wait. Why is my.

B

There we go my downstream stuff's there, okay cool, let's get started, so um this has got a lot going on today. In this show, so we've got, uh we were going to talk about, egress, ips came up and then there's also a new incoming dock on yeah.

B

There's that bonus, there's antonio's contract fix that we're going to talk about, but then there's another there's a dock on um there's a pull request that just went up from zhenshang today and uh let's see here- and I think june- oh john's here from the carvel team, so we're going to talk a little bit about ytt today and look at how it pokes in with the rest of the tanzu framework.

B

Stuff john has helped me a lot doing some of the ytt stuff for for windows and the entry installation, so hi john thanks for coming and of course june jen is here june jen. We are always very lucky to have jun jen he's our he's our andrea expert. Well, we've also got grayson too, though so we're just really lucky. Today. We've just got a lot of a lot of friends here. Okay, um so there is a poor look: grayson you're, the last you're, the last guy.

B

You're the most recent- this is a good day for you to be here because you're, the last person in the entire planet earth to commit any code duane.

C

B

C

B

Aliens like came over and took over the blow up the world right now, you'd be the you'd, be like the last person. Okay. So what's this, why don't you tell us what this is.

C

uh This is kind of a little bit changed to our end-to-end test framework, to make it better just a little bit improvement. I guess yeah.

D

Okay, so you've.

C

Got the last one, did you.

D

Adding testing I'm sorry the last.

C

Commits you're, adding testing, that's it and not adding testing. It's like make the test framework better, like.

B

C

We decided the connectivity based on like a lot of probes, like maybe some props is connect, some problem, it's a timeout. How we decided it's, uh we can't connect it or dropped it.

D

We need we need features, we need features grayson. We only ship features, man, yeah.

C

D

uh Working on another.

C

Features, this is he's.

B

Got features going on too all right, so let me kind get clusters so we're going to install 1.4 today, so let me kind delete cluster dash. Dash name equals andrea. Let me delete this cluster while we get started so that looks a lot like our network policy upstream tests grayson. So um okay, cool grayson is adding tests.

B

That's good news: okay, 1.4, okay! So we're going to install 1.4 we're going to go through there and then there's another thing. I wanted to show folks that lizzie shang was working on to add um the andrea proxy uh documentation. I think, are you familiar with this uh pr wilson, because I think he design document here it is: is this it yeah?

B

This is a. This is a cool pr that, I think, just I think was. I think it's recent or maybe it was recently updated to have a full design dock for andrea proxy, so, like uh andrea proxy luther's excited about it because someday they're going to run andrea over at rancher. It's just take that day.

B

So, okay, andrea proxy, is using encap mode. We're going to run entry proxy today, so andrea proxy is used in end cap mode. Android proxy is enabled and supports all types of services uh whoa. Well, what looks like a little bit of a.

C

This is a kind of like typo, I guess yeah.

B

That's a title.

C

I think he wants to see the coupe proxy. It's disabled, coop.

B

Proxy is disabled, so we need to fix that yeah ip6 disabled. All the nodes are linux nodes? Okay, um we can run andrea on windows, though we know that right, so we can run android proxy on windows. In fact we do um so. I don't know if this last one is really required right.

C

Yeah uh yeah, I guess so, but I'm not sure if the windows has a support proxy all feature.

B

I think it does now, I think, as of 1.4, I think windows does support. All of the features hey what's up. El elmiko is here: oshitosh is here, he's got a picture of an airplane el mico, el mico, my friend from red hat, is here and he's work. I think he's still working on cluster api over there. What's up el mico, what are you working on nowadays like, like you know, he's an open he's?

B

Oh he was he was he used to work on openstack a long time ago, and then he was open shift and then I don't know, and then he was doing cluster api stuff. I think he was working on machine health checks or something so um I just pushed a 1.4 update to the little kind, local upscript. So I'm going to 1.4 update for andrea local upscript, so I'm going to get push that.

E

Oh, you all see my secret things I'm doing here. Okay,.

B

Get push origin. Let me push this okay, so if folks want to play at home, uh I think I've shown folks this before, but you can go to this script here. This kind local upscript, is what I use for installing and playing with different ci's.

B

So you can grab this script and you can set the oh yeah there. He is there's lebron, what's up lubron, um so you can run this script.

B

uh K8S prototypes, blob kind kind, local up, and then you send it the andrea argument: why won't it get full dash, dash rebase push this up so and so the way we do the andrea stuff on this is we we just patch the intra image inside of the? uh So, if you clone down andrea, you there's a ci script. So what we do is we sort of we sort of leverage that uh under the hood and we just patch the patch the image so now we're going to test 1.4.

B

So here we go um so I'm going to do. Cluster equals android kind, local up and oh, I got to go to the right directory here, spread.

D

The conviction, what's that, it's quite the config gram, your xyz over here.

B

Oops I screwed up. I screwed up luther. It.

D

B

D

B

So what tell us about the feature you've been working on grayson.

B

F

B

Tell us about the features yeah.

C

We're working on.

F

C

Yeah, currently, I'm working on the service account selector card. uh It's kind of like you can use the service account as a as a selector in the in the untranslated network policy, because.

G

C

You want to like the service account is kind of abstract to define a group of pods you, you define access to external surveys or something like that. So maybe you want to apply some network policy to these groups without this, maybe you have to define a lot of network policy to select all these parts, and after this you can just use service account. So all the parts with this service account will be selected like you. You cannot talk to some specific part or something.

C

B

G

That that how long has that been in the works.

C

ah Maybe one or two weeks we just got the design dock. It's uh under review after that we'll start to developing this. Hopefully it will be in the 1.5.

B

Okay, yeah. I remember we were okay, so I remember these. These whole service account network people were asking for these right. So I remember I'm gonna search.

B

I don't know if I have it, but I remember folks were asking for these at one point and we didn't. um We didn't have them. So that's exciting. I think other cni's have them like. I think calico you can do that. Yeah.

C

Kyle had that.

B

um That's cool, yeah, um okay, cool! So is there a design dock for that.

C

uh It's a still in a draft condition, I guess- and it's.

B

C

uh Not in the yeah, not in public.

B

C

Yeah, it's in the.

B

It's just a google doc somewhere, okay cool, so I'm gonna spin up 1.4. While that comes up. uh This was interesting. I saw this and I know scott was excited about it. Scott rosenberg um was telling me about this, the other day, uh egress ips, so um so now this is. This is not 1.4. This was around for I think, before this is 1.3. This was around, but it allows you to uh so. If I normally, uh how do I do this? Where am I.

E

B

If I go over here- and I say like so normally, if I've got up here's one of my diagrams- you ready here- we go, I'm gonna, do it, I'm gonna draw okay. So normally um you have a node and you have a pod and that pod lives on that node. Let's say right and um that pod is gonna say go to say. Www.Saydominos.

B

Okay, so we're going to domino's, okay, we're ordering a pizza okay. So when it reaches out, you all probably know this when it reaches out. If the node's ip is say, 1.2.3.4.

B

And then the pods ip is say: 3.3.3.

B

Right then, the incoming ip here is going to be one two, three four, that's usually going to be the way it works right now that ip's gonna be so it's like, so it's gonna get uh what is it called source netting or destination? Adding this is source matting right, um so uh it's gonna get an added out. So that way, when this sends the traffic back, it goes to the node and then the node sends the traffic to the pod in general. That makes sense, but this is a problem.

B

If you are in an enterprise- and you have you have you you, for example, you want to maybe maybe you're an enterprise, and you want to only allow traffic from say. You know four dot, something I don't know, I'm just making up ips here that aren't real. So maybe I only want to allow things from the four whatever you know, so four dot zero, zero, zero.

B

Whatever slash! Sixteen, let's say: that's, that's, let's say that's what's allowed. Well, then this traffic would be disallowed right. So what the egress ips allow you to do. Is they allow you to not use the node ip and instead sort of use your own ip, that you choose as the outgoing ip address, so that allows you to very be very, very sort of discriminate about what ips you might be accepting or like you know, you can have all your ips that are all the traffic that's leaving.

B

Your pods can go out of a very well-defined ip range, as opposed to just leveraging the node's ip address.

B

um So this is real good for, like you know, if you're a sas- and you want to allow certain things and don't want to allow other things, but you don't have a very good control over the ip addresses of the nodes in your cluster and so on and so forth.

B

That's my understanding of the new egress feature. I don't know is that accurate grayson or am I missing anything june jen uh-oh text is blurry.

B

uh Yeah, uh let me make it, let me make the text bigger here we go zach. Let me know when it looks better.

B

ah There we go cool um okay, so that's uh another one of these new features that we're not gonna be able to test it. This came out in 1.3 but anyway, so now I think we should be legible and, um let's see if our cluster okay, cluster came up and our andrea agent is running down here and let's confirm it's. The right version.

B

Cool um there we go all right so we're at 1.4, so we've got 1.4 running now. I've got a coupe proxy. Let me kill the coop roxy delete diaz dash and coop system proxy okay. So let's get rid of that proxy so that we're not proxying any traffic anymore.

B

uh Okay, so we're not yeah we're, not proxying any traffic. Okay, so um there may be some iptables rules that might have been written by that proxy. It doesn't matter that we're gonna make a new um we're gonna make a new plot here. So so we've got so. We've got the um we've got the andrea proxy uh grayson. Is there a diagram of anterior proxy anywhere?

B

Let's see gian gen still not super clear, okay make it bigger.

C

uh Yeah, I guess it's under the under the dock, in the github.

E

C

B

C

B

You leave that link in the stream.

C

B

Join it: okay, cool.

B

Luther I heard by the way, vivec I thought I think vivek was going to come, but I don't know if he you.

E

Said he was yeah.

B

um He had a question he was. I was talking to him earlier about um what's interesting, cross playing in k3s.

B

Do you know what the difference is? I thought.

D

Cross playing was a you were talking about cross plane.io right.

B

I think he said there was some relationship and I was like yeah. I didn't think there was. I didn't. I didn't.

F

B

From my perspective that it had anything, I thought they were totally different. I thought.

D

I never even heard about it until you mentioned it this morning,.

B

D

Universal, I leave the comments.

B

D

Running across.

B

Playing on k3s, that was a question he had and I didn't know how to answer it.

D

Don't think so I mean I've been in that code base. I don't remember seeing anything talking about it, we're going to talk about it, but, uh okay, let.

F

D

The code base real quick, so I don't sound like a completed yet on the internet. Let's see if this is oh there's movement, we don't even use the word cross plane in the entire code base.

F

Explain yourself, what did I do.

B

Vivek's here um I don't know he's in this he's in the uh he's in the channel, so he may have a some. He may be able to add some details, but we were gonna.

D

He's never different. He said that I am right and you are wrong.

D

I need you to repeat that very loud, very clear.

B

Yes, if I, if I do yeah we'll we'll get it out of them hold on so if I do coup ctl create dash chef, I'm going to create an nginx pod, so I'm going to create an engine x in a busy box. So I have this smoke oops, I'm a cd 80 smoke test, coupe.

D

Let me answer vivek real, quick, I'm gonna, I'm gonna, poke it real, quick, okay, vivek um I'll see. If that's possible, that's interesting, though I don't know what crossland completely does right. Now, though, so, give me a second and let me see if I can go in there: okay,.

H

B

C

On the show, by the way,.

D

I'm not sure if it isn't over here.

B

Just ask if there's a way to orchestrate k through these cross playing, oh grayson, what happened? I think.

C

Oh yeah yeah, I just missed click the button. uh Could you see the I sent the I sent the doc in the comment.

B

Yeah, let's see where's the doc where's the doc. Where is the doc? Oh, you left in the. I don't see.

A

G

B

You can leave it in the youtube comments if you want okay I'll, do it, okay, cool! So there is the lvs. Okay, there's our little visual aid. Thank you! Grayson, okay, so I'm gonna make it big zach. Is that looking better.

B

Okay, let's see if we get a diagram in here, I think we've used this before, but I don't think this has the proxy in it or does it and it's just I'm looking in the wrong.

C

Actually, entry proxy is uh based on the obvious pipeline.

B

Yeah, so it's just really another implementation detail of how ovs works. I see what you mean.

B

Okay, let's is there are the rules in here here we go so okay, so here we go so here are the so is the?

B

Is there a set of rules that are managed by the entry proxy here or a set of tables that are managed by the proxy? Is that how this works.

C

C

For example like when you proxy the service, uh the service traffic, uh it's for the table, 41 42.

B

C

uh Yeah uh yeah, for example, the traffic is targeting the targeting the clock, cluster ip of the specific service, and this table can do the d-net to the endpoints of this of this service.

B

C

The load balance.

B

Oh okay, um let me make sure the resolution on the stream is correct. Thanks for that feedback, let me see, um I think it should be yeah. It should be 1080p uh zach, but I'm going to make the text bigger, because I think we just have to have we're not we're not my monitor's 4k and this stream is only in 1080p and if I did the stream in some other resolution, then half people wouldn't be able to see it anyway, so, um okay, so 41 and then what's the other one 42 42, okay, yeah.

B

So these are the is and are there other ones that are that are made that get written to buy the proxy? Or is it right to other tables, or does it do things in other tables, uh or is it able to just edit these tables and then that's enough to enable the proxy.

B

I guess that's a good question for next time.

C

Yeah, what do you mean, like you, mean other people, doing the proxy job.

B

So so we know when I think of android, I think of things that have to edit these tables in order to do their job right. So.

G

B

Of like android network policy has to edit like, for example, I know that it has to go and it has to, I think well, it has to modify table 85 right um and it has to touch table 45. But uh you know I'm assuming that and android network policy is spoof. Guard has nothing to do with it right um and I'm assuming it's not doing anything to the contract table right.

G

But I'm assuming.

B

The and I'm so so so that so that's the answer in network policies, but then, when we're doing entry of proxying you're telling me that these two tables are two of the tables that that the andrea proxy or components use. Are there other tables that those components also use like? Do they have to write to the other tables? I guess what I'm wondering is you know: does every andrea controller have to write to all these tables or do controllers generally like work on different tables.

C

uh Is this the the proxy is on the data path, so uh you mean, while initialize the controller will write which table to do the proxy.

B

Can every controller write to every table or does every controller write to every table.

C

The controller can write to the table, but um like tables 770 or 80, they are doing the forwarding this this table are existing, no matter the um no matter the entire proxy is enabled or not, but uh the entire proxy also rely on this table to do the forwarding.

B

Okay, so when you say it relies on it, does it have to add entries to these tables or remove entries from them.

C

C

G

C

I'm not too sure, but I think it's uh when you add the, for example, if you add a another endpoint to a service, uh it have to add some forwarding entry changing this table.

B

Okay, yeah yeah, so that would be an example of like yeah, so so, okay, but in general jianjin's mentioning that yeah. So it sounds like you're both saying similar things, which is that, like in general, the the tables kind of are bounded to to the features, okay, cool um and so all right.

B

Let's go back, and so we looked at these some last time and we it's not like we we're going to understand them after today, but I like to look at some of this stuff every time we do this because well maybe someday we'll all we'll know what all the tables do. um So when we did the metrics, we were looking, I think at the uh I forgot we were looking at one of these tables.

B

I think it was ingress rule table 90 um and how that was changing when we scaled up the number of pods in in coordinates. So I created a couple of pods, so I want to see if android proxy works, I killed coupe proxy right. So, let's see if we've got a pot, we've got an nginx deployment pod here, and um so, if I do, coop ctl get service.

B

The actual oh wide luther is it going to work. I guarantee it again.

B

I've never seen you so sure of anything all right, coop cpl.

G

Exact dash t-shirt busy.

B

Let's get in here come on well.

D

If you can get in the pod, I think it'll work.

B

Yeah, if I can figure out how to oh, I what oh it's not in coop system. Okay, I made that busy box plot, so I'm making a.

G

B

Why do I have two containers in here.

B

I think it's just this here we go so I'm in here and then I'm gonna ctl get service dash- oh wide, okay, so I'm in here and I'm going to try to get to this so um quick, quick, visual aid here. So I've got my service and my service is on 1096 right, okay, so this is on 10.96.

B

Now I know that here we go I'll make this big. Here we go 10.96 dot, whatever right dot, 85.206, okay! So that's where this lives all right and then I've got um I've got. I've got some ovs rules somewhere right. I've got some ovs rules. I've got some ovs rules somewhere so that when I'm inside of a pod, I'm going to hit this service and then that service is going to go through one of these tables that we've looked at over here. So this is a big deal right like I've.

B

I've never really run um kubernetes without the coop proxy right. So one of these tables here then- and I guess we'll- I guess grayson- is saying it's one of these here- some magic 41.

C

B

Yeah, so we're going to have some magic happening over here where, if luther's right, which I don't know, there's going to be some magic there's some going to be some rules here right that uh allow us to then go back out to our nginx pod right, okay, so and instead of coop proxy right. So there's no cube proxy. Okay, no more coop proxy cool! Let's see if it works.

B

Okay, um now I'm gonna w. Can I curl from this like w get? I can w get 10.96.

B

Dot 85.20 and it worked okay, so I'm inside of a pod and nothing up my sleeve here we have no group proxy running and we're able to curl down a cluster ip, okay good. So so entry 1.4 enter a proxy enabled by default.

B

I think it's been able, but enabled by default, maybe before that I don't know, but um okay.

D

Cool, uh if you already have a if I already have a cluster, though, and let's say I installed something else first, um will you guys do cleanups and stuff um on the way in or do I have to create the cluster with actual proxy in mind.

B

Well, I mean so like you know like it depends how native your integration with tkg or with with I'm sorry with with with andrea, is right like so, if you're running like if you were running, for example, like in tanzu right and vmware tanzu we're going to get into that in a minute, anyways right and look at how it's installed. So, let's just look at that now, so um that's what you call a segway yeah. So, let's that's why it's a live stream. So let's look at the way we configure it.

B

I mean john came all the way here from.

D

B

On the internet ah waited yeah some, but there can be cases of other tables. Okay june jen says in most cases one future yeah, okay and then oh zach had zach mentioned yeah. That's what I should show uh zach. That's a good point. I should probably do.

F

My entire stream on.

D

When I do master classes, I always set it back to like whatever 1080 or something like that. I always set my whole entire screen. Then I share everything.

B

Yeah, I'm going to start doing everything in 1080 from now on.

D

um Or just get another box to do it on yeah, I'm sitting in there.

B

So um now, if let's go look at, let's look at how we install andrea, so so, if we go here, um if we dig into actually had the code open anyway, so if we look at how we install android so so in vmware tons of when you install andrea, here's our ytt- and you know since john's here- maybe he can tell us what we did wrong here. So uh so here you go john here's how we do it so so when we install android clusters, we have all these feature.

B

Gates turned like so like what we do is like. If we want things on or we want things off, we define all of that in ytt and this all gets generated at run time and so like in in tanzu. Let me make it big hold on. Let me get it big here. We go okay in tanzu. ah How do I do it? Okay in tanzu? Right, if you like, you know we'll turn on android proxy. True now it turns out that we we haven't fully disabled coupe proxy, yet so like.

B

If there was a bug in this, you still get coop proxy. You could still run it, but um you know like like for a distribution, I think it's kind of now with andrea proxy sort of becoming a thing that sort of does everything for us now like with the like.

B

You can run the proxy all option and if you run the entry proxy all option, then um then you really don't need coop proxy, and- um and so you turn this to true, but you would also be able to delete the the coup proxy um daemon set now like to do that, at least in our distro.

B

um Where do we define coop proxy?

B

We define it uh for for windows, your favorite luther. We actually run coupe proxy dir inside as a post, boot step right.

B

So um now the current version, our current version of um of konzu, doesn't use this version of android, so we still run coop proxy, but like we're going to delete this code pretty soon right like so this stuff here, we're just gonna get rid of it and then at that point we won't be running coop proxy as an nssm service anymore and we'll just be running andrea proxy with the proxy all set to true.

B

I think it's andrea proxy all is the actual way you you do that and then it'll proxy all traffic um and then yeah. You won't need it. um So it's you know it's just like a it's like a thing like yeah we're gonna have that native integration in tanzu, but, like I don't know, once we get in rancher, maybe we'll be able to do the same thing.

B

um Where is so yeah, um so it doesn't work, blah blah, blah, okay, so yeah, so the feature gate, so we kind of skipped ahead here, but um so the way this all works. So if we look in uh tanzan, we look at how we install so the last thing we're going to go over is um well. We have two more things I wanted to get into. One was go over how we actually use carvel.

B

So for folks that don't know carvel.

B

Where is carmel where's the? If you google it you get all these other. You get ice cream, carve carvel, kubernetes.

B

Here we go carmel.dev there. It is john, so john, what's the latest news in the.

B

What's the latest news in the carville like world um for tons of we use cap controller, we use ytt, we use k up. We use these three. We use all this stuff but like for when it comes to the life cycle of how entry is installed, we really heavily rely on those three. So um the way it works is um we have we have.

B

We have a thing. We have cap controller right and then we have k app right, and uh so, uh when you install.

B

When you install uh when you install andrea, it's actually installed as an application instance right and uh cap, then reconciles all of andrea right and then, but actually the the entire like ytt for that right.

B

um So, like the ytt that I just showed you over here, which was here um all the configuration of andrea, is done in ypt right. So this is the config that gets passed down and then this config this config here ah here this config uh like there's this thing called framework right, which has all of vmwortonzo in it right and um once we once framework substitutes all of the android installation stuff using ytt that the users sort of defined um then android gets installed, and then the life cycle of antra is entirely managed for us.

B

Using this thing called cap controller right, so we have a so what we have is um like in a condo cluster. You have you have like a management cluster right, so you have like a management cluster and you have a workload cluster right and you have a thing called a cap cap controller here and k. App is a um is a thing that can manage your applications for you. It can basically make any application have operator like behavior and k.

B

App is just a way of defining a whole bunch of kubernetes resources as a um as a single bundle right, and so what you have, then is you have the cap controller going and here's my cap instance? So this is how all of tanza works actually for all of our add-ons right. Is you have cap controller going and looking at cap instances and reconciling them and it can reconcile. So these are separate clusters right. So you have this one operator and it can reconcile stuff on its own cluster and it can.

B

It can manage the applications and reconcile the stuff going on on other clusters too right, um and so that's kind of how we manage android is we package it package it all up using ytt to define what parameters we want and then cap for the actual, like entire payload of like the deployment and the config maps and all the other stuff for it right, and so that's like a pretty reasonable amount of stuff that you have to manage right.

B

um So it's like really nice to have carville to help us with that um so like on a live cluster.

B

B

We can see these are all the components right. We've got andrea, config, um we've got this android cluster identity, we've got this andrea ca um and then we've got coupe ctl get deployments, wait, that's just the config maps and then we've got coop ctl get crds right, and these are all the anterior related crds that you wind up having to manage. Now I don't know how the logic will grayson. You may know this do. Is there a specific logic for like how does andrea when you, when you upgrade an andrea version?

B

um Where is the logic for managing upgrading things like do, network policies or other crds and stuff have to get upgraded to the new version like when I upgrade from android 1.2 to 1.3 or 1.3 to 1.4? Is there logic for how api things have to be migrated.

C

uh I'm not familiar with this part.

B

C

B

C

B

I'm not either so I'm not either, but I I'm assuming maybe there might be something in here that, like maybe one of the integrate controllers, is able to go in and make decisions about stuff like that. um But I don't I don't know right like and then it also makes me kind of wonder. I guess we have more questions here than answers like what, if I upgraded from android 1.0 to 1.4 like would that work and how would that work?

B

um So, um let's we can see, I know junior's there, so maybe he has some. Oh here's, john. What's john, let's see what's john about to say: okay, john and then june jen, I assume you have an app cr for andrea. It might be useful to show that because that's how yeah.

E

So uh the app is uh where's the app cr.

B

So the if we, if we go to, let me go to tanzu community edition, so we can show people where this all really originates. um So if I go, oh that's the wrong, so t. So the way this all works is tanzu community edition, which is on github github. All right. Let's go here so tce we'll do a walkthrough of how it all works on tce one of these days, but but um oh june jen mentioned something as well.

B

So if there are any crd changes across versions, we do auto conversion of old crd new to new versions of crs okay cool. So now I'm wondering like what, if I I assume that there's a limit on that right um like I assume I can't go from one dot o to 1.4 without any problems- or maybe I can- I don't know- but okay, that's good to know so, um and that's kind of how the kubernetes api works too right like it can um so.

B

Okay, if I go in here so in tanzania, community edition there's these things called add-ons and then each of these add-ons um like has like a package associated with and amim, is here, but I don't know if a meme can stream with us right now, but he's kind of the expert on this because he's actually packaging all of andrea for up streaming for tons of framework.

B

But um so, if I dot this, let me see we can go and we can look at what's in the bundle right, so here's android, 1.2.3 right and so let's go in here. Oh look at this josh josh made one of those like video embedded things in the readme. That's cool! Okay! So if I go in here and I go in here, there's an image package bundle right and this defines um the uh the actual shaw of the actual image that we that we're using.

B

So this uses the carville image package toolkit right and that allows you to sort of create any turn anything into an image. Basically right. So we use that and then we package this whole all this yaml and stuff. We package all that up uh into like an image right and.

D

When you say image, you mean like a like a container image. This is your burrito.

B

Yeah yeah into an into an oci image right, so the ammo artifacts. Are there here's a meme, so, um okay, I mean so now we use image package to package all of the ytt, all of the yaml as a as a base, andrea, like set of yaml files right and then um here is the core like.

B

If you run an entry upstream like if you just grab the yaml from upstream, that's this, and so you could see that this is just raw right, and so this is the same thing that if you're just gonna run the defaults. This is what it would be. So you start with this and then we package up this overlay, and this is a ytt overlay. So what this does is this then patches um these feature gates right. So now, where these feature gates come in, I mean: do you know when where these two triggers are?

B

These are part yeah.

H

There is a values where you set this configuration and then there is a velocity of a.

B

Yeah, I guess it gets directly put into this config map and then that config map is the input. So this is what, since this android agent.conf is the entire confiamol, so that comes from this function right so so this is the okay cool, so yeah. So this winds up being the uh way that we generate the and then we and then, as john mentioned, there's like somewhere in here, there's an there's a cap config here it is.

B

I think this is what john was talking about, um and so the cap config defines rebase rules, path, data kind, where's, the where's. The cap is this: the cap cr, maybe it is, is this the application for the um let's see where is june, gen?

B

Okay, that was him earlier? Okay, okay, so you have a package cr, that's the place to see how it's all integrated cool, um so, okay, yeah.

B

Where is our well the the package? Cr, though, only has the the yeah it has all the it has all the um it yeah it has all the it has all the ammo that gets put into the to a container, and then we unzip that container or untart or whatever you call it. um And then these are the default values that you get.

B

If you don't, if you don't patch any of this stuff, so then what happens then, later on in in tons of framework right onto the framework right, which is the code base that we're in over here? It's time to framework then relies on.

B

We have this android add-ons and we have an ad cni. So in the add cni you go and you um you, where is it yeah, so they add cni, it decides what add-on to use right and then that winds up being like a secret and then that secret um winds up having all the information that was bundled uh over here. So all this stuff winds up being like a secret and then that secret winds up getting substituted with this uh andrea add-on data lib stuff right, but from an end, user's perspective.

B

You hack up these parameters and then tanzu does the rest. So you set these parameters and then tanzu goes and does a bunch of ytt magic under the hood to make sure that your android cluster actually has those running in the wild. So then, coupe dtl gets cm system, andrea.

B

B

All right so then, that's it! So then you wind up getting this right. Whatever the situation is right, um cool uh all right, so that's that and then I think that's most of like we're kind of we have a. We have a lot of stuff today that came up so me and me, and uh me and amie, were looking at this earlier. It's really interesting, antonio, has another cool pr, that's worth looking at. um So this is a another bug in coop proxy that was fixed around uh stale contract entries.

B

So this just came out, and um so it turns turns out that- um and I don't know how people found this, because I don't know who who digs into stuff like this, like who's who's able to how do you even find a stale udp entry? So one of the things I didn't understand- and I still don't fully understand- is why it is that contract even uses udp, but I guess it's related to um or why why contract tracks udp, because I thought it was for pcp, but I guess it's for natting and stuff, but anyways.

B

It turns out that if you have a um in coop proxy so now we're switching over to looking at a new bug in coupe proxy, it was just fixed, so it turns out that with contract in cube proxy, there um was a situation where, um if you had a an it pod and the emit pod was running but an endpoint wasn't up yet then um that thing could get tagged as like.

B

You might wind up, unfortunately, getting tagged as having a stale, endpoint and then contract would get cleaned up, and then you would lose the ability to ever route to that connection again, and so we were just drawing a diagram of this earlier. We were looking at it here. It is remember this one I mean.

B

Do you, okay? Here it is so we were just looking at this, so it turns out so like say I have an endpoint.

B

um You know: here's my user over here, okay and the user's accessing that endpoint through this service, so that endpoint then, is for some period of time going to be um not up yet right and then eventually it's going to become ready right and so what this fix does is it makes it so that if that endpoint is not yet ready, nothing is marked as stale and the reason it does.

B

That is because, if you mark this end point as stale and then eventually that end point comes up, then that end point is now just going to be like a black hole right. So this sort of adds some nuance to the coupe proxy's ability to interpret whether or not it needs to clean up connection or not for a pod that isn't just just makes the basically it just makes the coop rocks a little more patient with the.

H

Planet I see I seen this black hole comes from a new contract entry created without the iptables rules, so this is. This is a invalid contract entry and.

G

H

After you create the iptables rules and the services up the endpoints right, it doesn't clean up the old wrong entry, so you need to.

B

H

B

Yeah, so it doesn't know so yeah the whole pr winds up being so this is a good way to learn about contract and it's a good way to learn about the coup proxy. If you want to look at experience, 106163 and- and um so this part here wound up being helping me to understand it, a lot right because what's added here is it used to be that it would add so this stale service names, this object is managed by. I want to make this bigger.

B

Oh john has something we need to go back to uh the ytt stuff, so john, we might want to do a whole show about ytt and andre and look at this together next time, um but I'm gonna try to get back to that in a second. um So this stale services name. This is a function in the cube proxy and what this function does. This detects dell connections is it's just whenever it runs right. It writes things to this.

B

This data structure right here and so what this pr is doing, is it's making it so that it writes less to that and it's like okay, I'm not going to call this a stale service unless we now have new endpoints and the the old and or if the, if the old, endpoints, weren't ready and the new endpoints are right.

B

um So it adds that little bit of nuance, so you have less of that less of those things being flagged as stale services um and anyways. Evidently, this fixes this, but to understand this is tricky because, like you have to read this whole issue and we only read some of it earlier today, we spent like we probably spent like 45 minutes trying to read through this entire thing. Earlier I mean I mean um so uh this is an interesting issue.

B

I'll put this issue in the show notes here: cool, okay, so uh yeah there we go so I mean we're already at the long mark. We had so many things to talk about today, so to be clear, so let's go back, we got 10 minutes. Let's look at john's thing so to be clear, the values.yaml is ideally the way to define a configuration api.

B

um Okay, we should look into that. I I don't. um I don't know what you mean, but but I want to understand.

B

um So, john, how about I don't know what you're doing next week, but why don't you do a show with us and we can actually look at the config, how we use cap and ytt like sort of inside out and and kind of walk through the entire thing.

B

um I think that would be like really cool um to get some of the carville folks on here live and do it together, um because does he work on the car belt he's a carvel yeah, so he's up so next week, we'll get john on and john can do a show with us, and he can show us how he how we might, what what would be a better way of sort of modeling this uh this whole android configuration thing um and also maybe he can tell me what to do with this wall of powershell.

B

That's in here. I kind of think there's a better way to do this too, but I haven't decided how to do it yet.

F

B

F

B

Okay cool, so uh all right, john, so we got that's cool, so john's gonna come back next week and he's got some some questions about this which which I knew he would which so I'm excited, because we're going to get some expert advice. So if you all have ytt or ammo configuration problems or other stuff, maybe related to some of the stuff that we solve with carvel um come next week, because we're going to have john here- and he can probably give us um a lot of advice on how we can do things better.

B

So um cool, I mean that's kind of all. I got grayson anything else going on in the entry world that you want to tell us about, or um any other news updates um before we close up for today.

C

I'm not sure, maybe we can. uh If you are interested, we can go back to check the proxy like the table. 40 41, like the exact flow or how it works.

B

You can try.

C

B

I don't, I don't see any reason why we can't. So what do I do? I just exec into an agent right.

C

Yeah you go into agent and dump the flow from table. 41 42, to see.

B

Let's get pods dash a grab agent, so let's get into an agent. So that's a good idea.

B

Jason has got us.

B

We're back we're decking into the agent okay, so here we go so now. What do I do? How do I check this out? Do I do ovs ovsctl or something? Is there a bad shell.

E

B

B

C

B

B

They can stuff up over here. What do we do.

C

Oh as uh I sent a link like a command to you,.

B

Oh and cuddles, oh june, jen mentioned ant cuddle cat cuddles on the agent. I didn't even know that, oh here we go. Okay, I got something here we go. You can leave that in the uh comments too. Here we go. Oh we're gonna dump the flows.

B

I don't think we've ever gone this deep before this is. This is.

G

Why I always like it.

B

Because then I learned something new okay. Here we go all right here we go ovs, ofc, cctl, okay, all right, so we've got a cookie. We've got a duration, okay! So what's going on here?

B

What did we do wilson? I can't I don't know what we did, I'm blinded by the okay. Here we go by the beauty of this, I'm a separated string. Let's see what we got in here. We got a duration, we got a table.

B

Okay, we got a number of packets. That makes sense to me. I think it makes sense.

B

B

Okay, so you want to talk us through these fields and then this is another file. It's like.

C

Yeah yeah, like the important thing is like you, can see the network destination match. That's the like lion, 30.

B

Okay, so this is the service. This is the k-8s api.

C

Yeah, if you have your like, for example, your engine service, uh this will be the cluster ip of that service and it will do some uh uh load, some register use for entria, and then it will go to the group one um yeah. Maybe you can choose the the the flow that for your engines, cluster engine service.

B

Okay, so I'll go back here, so how would I do that.

C

B

Do you mean like grep for nginx, I can't grab for the service.

C

Oh, no, maybe the the cluster ip.

B

Okay, like actually.

C

B

E

Okay cool: I got it. Okay,.

C

E

All right so now, let's.

B

Look at this one instead yeah that one didn't make much sense, because it's forwarding us back to the node or something yeah. Okay, so.

C

So the the service to disk the traffic to this service will hit this flow since it's the destination ips the desk service, cluster, ip and also the port and after some load some register. It will go to the group six, as you can see at the very end,.

B

What's group six.

C

uh Group six is a a group, is a definition in the ovs. You can also dump the group to see.

B

Okay, so this is the definition inside obs and what.

C

uh It's a kind like uh entity. I think you can.

D

Add some group.

C

You can yeah dump groups change the dump. The groups. Oh, it's cool- is.

D

The grouping are you blocked off.

C

Traffic for the different uh groups, it's kind of like, if you uh each group, can do several operations and, for example, first operation is change the destination ip to some endpoint ip. Maybe the second one is another endpoints ip, and this called the bucket. You can put a lot of operation in this bucket and you can assign weight to this bucket, and this is how we do the load balance.

B

B

uh I don't like that. You want the bridge. What's the point.

C

Yeah bridge is the br br you're.

B

In okay, so I do so I do br and six.

C

B

Cool okay, so I can give it the group. So then I can run this and then this will give me all the rules.

C

Yeah, this is a group and uh you can see uh this is a group a six and you also, you only have one endpoint so.

H

C

Yeah, it only have one endpoint: it will store the endpoints ip to the register. Three.

B

C

Yes, like the line 39 you we store the ip to the register. Three, uh probably you can read.

B

C

What is this, what is this? Oh, this is ip like 10 like yeah, it's the ip address in.

B

C

B

Oh okay, okay! So that's the okay, so okay cool all right, yeah.

C

If you have a lot of endpoints, it will have uh like the bucket will have another bucket. You can see the line. 37 there's this only one bucket id 0 level. 1 will have a lot of bucket.

F

C

Yeah, so it will, you can define the weight you can see weight. A hundred different bucket can have different weight, so it will be as like distribute load balance to the different ip.

B

Wait now, which one is the of this bucket? Okay, so.

C

B

It so like tell me about these buckets so, like you have.

C

Like this, one group can have a lot of bucket that you only have one so the id 0 weight is 100 yeah.

B

So that's an ovs thing.

C

I guess we're going to have an obs.

B

G

C

Yeah, a lot of uh other.

B

This is a generic open research thing: okay, cool yeah,.

C

So like after this group load the endpoints ip to the register and it will resubmit to the endpoint dna table, this is the table. 42.

B

That, okay, that goes back to the pipeline here, okay, so.

G

B

Okay, so that's here: okay, all right!

B

Okay, so I've got table 42.

B

Let me make sure this is legible. Okay, cool, so I've got table 42, which is.

B

Here endpoint dna table.

C

Yep, so this table will do the dna thing, so your your origin, original traffic, will change the cluster ip as a destination to the endpoints ip.

B

Okay, so resubmit. That means. Does that mean it like what is and what does resubmit mean again.

C

It's just send this traffic to that table.

B

Okay, so like maybe I'm here and if I said resubmit, then it sends me back to here, no matter where I am in this whole flow yeah.

C

So yeah you will go to this uh yeah. It's really sub me to the uh endpoint dna table. Just the this flow will. This packet will go to okay.

B

So then I'll go back here and then I go like this, so you so is it possible to build an infinite loop in ovs, where you continually derive.

C

uh No, we only use we only submit to the to the backward table.

B

So if I submit this to this, isn't that a loop that goes on forever or I can't submit- I can only submit forwards.

C

uh I'm not sure if obs allow you to do that, but we don't do that.

B

Vivec, what is the nginx plot ip? The nginx pod ip in this case, was.

B

So grayson we're gonna have to do a whole show on nginx next time.

D

Can you do a whole show on debugging obs too? That was interesting.

B

Yeah, I think that's yeah, I think we're gonna hurt you because.

F

B

Really curious geojet: can you make an infinite loop, an ov open v switch.

B

I think you can looking at this table. Groups can be dumped through ant ctl through ink, okay, cool and cuddle.

D

Get ovs flows wait. So can I just have some questions here, just because I don't deal with it, you guys haven't the obs tools inside of that anterior container that I just saw you go into. How often is obs actually bumping stuff what's their cycle, and how often are you repinning those? When do you guys have to do? You have to do any releases of andrea to coincide with obs releases? I'm just curious how comfortable it is.

D

C

uh I'm not familiar with the update to the you mean like how often we update our obs version.

D

Yeah like how like what does obs do like is obs just like a done project, and they don't update anymore or is like are they doing? Are.

C

They doing yeah, they.

D

Just do they're still updating, but so.

C

Have you guys had to re-pin.

D

All your obs, stuff and bump and use new features and stuff is ovs making stuff that you guys still yeah. I I don't know.

B

C

um Yeah yeah, I.

B

Don't know the frequency.

C

D

I'm just curious, I mean we because we've got some pulling that we've embedded and then, like all of a sudden, some new features wholesale right into your project. You got to get them all up there, just just curious how.

G

D

Yes, it is to deal with just interesting yeah.

B

Oh, is it oh, I got it ovs, let's find out right now.

B

Oh god, I just started it. I didn't mean to do that. Why.

F

Did you start the damon.

B

Okay, so we're at 20. Let's see we're going to find out.

D

I think that's current.

B

So what version of ovs github? If I go to ovs, I guess I don't know whether they release in I think they do so.

D

B

Go 21 tags; okay, oh grayson,.

G

We're behind you if.

D

B

D

B

Oh no, oh dude. We do not change, obviously, but mostly when a new major ovs version is released or when we need a fix or enhancement. Obs is super active, okay cool. What is your brand to do? Well, I think you all are doing great, because guess what I was just able to curl down from my cluster. If you notice, we should probably do a more focused episode next time.

B

We just had so much to talk about this week, um because so many people wanted had some questions and stuff so next week we'll do a more focused episode. Maybe we'll do ovs or no next week, we'll do ytt with an and the carvel stuff. The channel.

F

Episode, what's that where's my promised crossplane episode? I guess we're doing crossplaying too so we're doing here. Let's do carvel and then let's do crossplane and then let's do.

B

um Carville cross plane and then what's the next point after that.

B

Then we'll do lvs with grayson.

D

There you go over here with the grayson, I'm gonna be there that one's gonna be sick. Okay,.

B

Cool, well all right everybody, it's we've been in here, we've been so it was a long show today, so I'm gonna, I'm gonna, get going and then um we're gonna see you all next time see you later so thanks. Everybody leave some comments on the street on the show. If you all want- um and we can answer them there or whatever- and give us a like on the video. So we can keep doing this. Okay, see ya, see you.

C