Flatcar Container Linux Tech talks, 2 Feb 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kinvolk Tech Talks: Introduction to Kubernetes Networking with Bryan Boreham

Description

This talk was recorded at a Pre-FOSDEM Brussels Kubernetes Meetup on February 2nd, 2018.

Kubernetes has two simple but powerful network concepts: every Pod is connected to the same network, and Services let you talk to a Pod by name. Bryan will take you through how these concepts are implemented - Pod Networks via the Container Network Interface (CNI), Service Discovery via kube-dns and Service virtual IPs, then on to how Services are exposed to the rest of the world.

Speaker: Bryan Boreham
Bryan has spent decades building and operating large-scale distributed systems, focused on performance and dependability. His current role is Director of Engineering at WeaveWorks, an Open Source technology company.
Twitter: https://twitter.com/bboreham
Company: WeaveWorks, https://www.weave.works/

Meetup: Brussels Kubernetes Meetup
Event: https://www.meetup.com/Brussels-Kubernetes-Meetup/events/245974093/

Video by: Kinvolk
Web: https://kinvolk.io
Twitter: https://twitter.com/kinvolkio

A

I I wish to assure you, I'm not a networking expert, I'm a programmer. My.

A

Name is brian boram. I I have worked on a certain network product. The this is the.

A

One advertising slide really. What we do is monitoring and ci cd tools, and things like that, so we've got works. If you want.

A

Any about it's all focused on kubernetes, so it's really cool. What am I going to.

A

Talk about today, I'm not a topic, the exact same stuff that mike and michael.

A

Talked about, I don't know why you didn't read the agenda, but I'm gonna. Do it with.

A

Pictures instead of command lines, so hopefully, hopefully that appeals to.

A

Different people in the audience- and I I helped a lot of people out at least I.

A

Think I do on like on the on the kubernetes lack like cig network. I'm.

A

Getting a lot of blank looks okay, so there's nobody here that I've personally.

A

Insulted on on the kubernetes slack, that's good! That's good!.

A

All right so yeah, I really my main aim is a selfish one. I I want to give you.

A

Enough knowledge that I don't have to repeat the same explanation over and.

A

Over again, on slack, at least for the for the 70 odd people in this room, there are.

A

Some words on the screen we'll get to those words: okay, show of hands. I think.

A

We did kubernetes before, but I was at the front so I didn't see so who's like.

A

Kicking the tires, I heard the name I ran at once so a couple of people, maybe a.

A

Couple of people whose who's full on everyone's full on ok, so you don't.

A

Really need this right, you know, you know no show fans who's heard of my employer.

A

Lots of people.

A

So so I wanted to put up. This is the the high-level picture this is. This is.

A

Really, what kubernetes is I am for those of you who know this already. I apologize.

A

I'm gonna go through. Oh, maybe not that diagram. I'm gonna go through there. Maybe.

A

I'll go quite quickly, but just just to make sure we've got all the words and.

A

All the concepts don't so so the basic idea is cuba. Nietzsche is going to run.

A

Some work for you, the work is the light blue bits, cobra.

A

Dc's has this thing called master, which is the thing that knows everything has.

A

All the facts and it's running some software on each of the nodes, the nodes.

A

Or computers, those are the things they're actually executing your work. You.

A

Might have one of those you might have a thousand bit of software on each node.

A

Running bits of kubernetes bit of software in the master that has all the.

A

Facts, that's kubernetes, kubernetes runs services. What is a server service is.

A

Something that does something, and you can call into it- you can call into it.

A

From somewhere else in the cluster, or you can call into it from outside the.

A

Cluster, get all the words straight, take a little segue into ports. Everything.

A

Runs on a pork right who doesn't know ports.

A

So if you want to run two of the same things, you can't run them both in the.

A

Same fork right, so so what people typically do? They pick a different port.

A

Number, maybe you know 9 o--, 9, o--, 9, o-- 9. One great problem. You've then got to.

A

Solve is, let's add some add some tech here, so I don't have to walk around so.

A

Much any cats in here the problem you got to solve at any anytime anytime.

A

You've got someone making a call into your service in this kind of environment.

A

It has to know which port to use and people solve that problem, usually with.

A

Some kind of service registry- they put them in some kind of database. They look.

A

Them up, and so the downside of that is every bit of software that you want to.

A

Call into this environment has to understand this relationship where you.

A

Stored the ports and how to look them up and that's a non-standard thing. There's.

A

There's no, you can't expect any random bit of software to be able to do that. So.

A

So that's a bit annoying so first thing: we do. We give every service its own ip.

A

Address so people write blog posts called things like there's. No such thing.

A

As container networking- and I think what have I been doing for the last three.

A

Years, if container networking means nothing else, it means giving each of your containers its own ip address, and now we.

A

Can use the port number that was baked into the thing in the first place, we.

A

Don't have to go changing the port. No, that's why we do this right. This is the.

A

Big feature give every service every instance of the service, its own ip.

A

Address: okay, now the problem we had earlier, which port knew I use now.

A

Becomes which ip address do I use? Okay? Oh hang on I'll. Come to that in a second.

A

Let me just make sure we got some more words. Pod is a collection of containers.

A

Containing docker whale a collection of wales's pod, okay, it's where we get that.

A

From the ip address in kubernetes goes on the pod, so we collect a bunch of.

A

Containers together and put the ip address on the pod.

A

And we saw note earlier, containers are inside the pod, okay, you're fast right.

A

Yeah, so what I was saying: how do we? How do we find the right ip address? Well, it.

A

Turns out, there's already a way to do that which is called dns, so this this.

A

Question, which was a real problem, a couple of slides back, is no easy, because.

A

Every piece of software in the world can do this. In fact, it doesn't even do it.

A

Calls into a library it just it just says, give me the address of a thing.

A

Called foo and underneath somehow it calls dns. So so this thing that was a.

A

Problem is no longer a problem. Okay now add scaling and redundancy. So we got not.

A

Just one of these things we got multiple running, and so we set.

A

Ourselves a basic requirement: what I'm calling into this thing? I want to hit.

A

One of the endpoints I want to I want to hit one of these and I don't care which.

A

One because they're all replicas for redundancy or failover or scaling or.

A

Whatever reason I did that, so that's my that's my basic requirement. I'm gonna.

A

Start here, I'm gonna call one of these things. I don't care which one dns can do.

A

That who thinks dns can do this, even though I've got a slide up the set.

A

Account we've got, we've got some, we've got some hand waving, so so the the kind.

A

Of boring reasons why we don't do this are, are the first two? Let's go back to.

A

The tom's taking a picture so the the.

A

Kind of boring thing that can happen is, is this guy is going to query dns every.

A

Time it wants to make a call and if you start doing thousands of calls that. But.

A

That's kind of annoying it's kind of painful on your dns server.

A

The worst thing that happens is if somebody decides that's expensive and.

A

Cash is the result looking at you java. Well, the default up to java 1 6 was to.

A

Cash it forever for you and then somebody said well. Maybe that's maybe.

A

That maybe forever is a long time so but yeah. So if we have this requirement that.

A

For failure like if one of these things dies, I'm gonna go to another one and if.

A

I've looked it up a few seconds ago and I've cashed the result, I'm gonna, I'm.

A

Going to still go to the one that just died, and that is that is a real problem, so we we don't get a great result of of using.

A

Dns for that particular purpose and and there's a third thing that goes wrong.

A

And if you want to know the details, google dns happy eyeballs. I am not.

A

Making this up there's a really deep, intricate technical reason why it just.

A

Doesn't work and lots of people will argue with me about that one. Maybe, let's.

A

Do that over a beer later, just doesn't work. Okay, so kubernetes does not do that.

A

Because it just doesn't work and because tim hawking is a god he knows he knows.

A

These things, what cobra neat so everything I do.

A

Everything I did from let's talk about ports is generic up to now. Now I'm back.

A

In kubernetes, nothing that I said about ports has anything specific to do with.

A

Kubernetes, I'm just motivating this okay, so kubernetes does a little two-step.

A

Dance, instead of all the other things I said you couldn't maybe do in order, I'm.

A

Kind of blocking out the light I mean, let's go back to the cat thing so in.

A

Order for or my client to reach my service, the.

A

First thing: it does, it says where's that service, because all it knows is the name.

A

It just knows foo and what it gets back is fake, fake news, fake ip's. It gets back.

A

An ip address that does not exist, it exists only in the mind of kubernetes, so.

A

This is called a service ip or a virtual ip, and if you call it a cluster ip, then.

A

You'll get confused, but that is sometimes what kubernetes calls it. Let's.

A

Not worry about that, so we first of all, we asked cube dns. We.

A

Get this address we start talking to that address. This are a bit of software.

A

Here it just totally believes that fake address and then this other bit of.

A

Kubernetes, which is called coop proxy, sets things up so there's a mapping to.

A

The real pod address, okay. So that's how services work in kubernetes a couple of.

A

Points of detail, so, first of all, every single piece of kubernetes is.

A

Replaceable in general, I am talking about the most common way that people.

A

Run things if I make any specific statement about a piece of kubernetes.

A

It's always false in the sense that you could have taken your kubernetes and.

A

Replaced that piece that does that with a different piece: that does it differently, so I'm the most common way that people run this. It works like on.

A

The picture it could be replaced. Another thing that is really important or really.

A

Interesting, we're really good to remember this mapping takes place on the.

A

Client side of the conversation it takes place on this node. That is useful when.

A

You're trying to debug this thing and trying to figure out where are you where your network traffic went where the packets went whatever, whatever? Whatever.

A

This conversation will get mapped to the real address before it leaves that node.

A

Okay, eight questions. At this point. I don't go deathly quiet. Oh yes,.

A

Space: okay, how do we arrange things so the virtual addresses and the real.

A

Addresses don't clash. Well, there's those two settings, one of.

A

Them is the service side er and the other one is the cluster cider cluster.

A

Cider means the pod addresses, which is why I said: don't use the term cluster id.

A

Because it's for service, because it's a confusing thing to use sorry, so this is.

A

What happens if a pod does a name resolution for a.

A

Kubernetes service, if, if it does a name resolution for something like google.

A

Comm then that'll get bounced up to the upstream dns and it will get the right.

A

Answer back from there, it is only applies to kubernetes services within.

A

The cluster hi we got a, we got a question over there.

A

Okay, okay, did you miss the part where I said.

A

I wasn't a network expert. I have no idea what you're talking about.

A

Sorry, I I have a little idea: what you're talking about.

A

But what's the question right so this is the this is exactly the way it works.

A

Could proxy sets up ip tables rules that map this address? A width plus the port.

A

Which I've just left off the slide into this address? So that's! That's all. It's.

A

It's done in a kernel module in linux called netfilter, which is controlled by.

A

Things that we all call ip tables rules and that's how it works in the most.

A

Common implementation, the calling side.

A

Sees the virtual address: yes, the the destination side believes its own.

A

Address it's it's kind of weird. Oh, let's see, if I get to that, I can't even.

A

Remember, what's on my slides, so let's let's see hold that question. I did not.

A

Get to it, okay, so how does it do round-robin? So basically, what happens if.

A

There are more than one of these endpoints, then it will put in rules such.

A

That the linux kernel netfilter will pick one of the endpoints at random. I.

A

Sort of I sort of have a temptation to actually show you some of those rules.

A

That's a really dangerous thing to do is like live demos in the middle of a does.

A

This thing work if it's working, I think I'm a little suspicious of the.

A

Network right now, which network am I on, I am on that's the.

A

Good one, okay connection to my google cloud: shell has gone well, it's let's.

A

Carry on with the slides and come back to this and see if it worked at the end.

A

Said it did something, did my other thing? Do anything? No okay, nevermind, um okay,.

A

A

Yeah, so for the for the for the benefit of the recording you.

A

Making the point that, if we had, if we had a lot of services and a lot of.

A

Endpoints then this guy would be writing out. Thousands and thousands of ip tables.

A

Rules- and that is correct, and that is why they're replacing this with a.

A

Different implementation, but as of right now, the most common way that people run.

A

This, it really does write out all of those ip tables rules.

A

Okay but yeah yeah, more coil, so service or pod on no aid, for example, asking for.

B

Service, that's my next slide. So yes, as you've just outlined.

A

I was glossing over the fact that we've got packets if we have a network.

A

Conversation we're gonna have packets, we got packets that start here and packets.

A

The end up here and how do they get there? Well, they have to get there.

A

Somehow is, is the the simple answer we and there are like dozens of different.

A

Ways to do this, so I'm gonna talk about the kind of the two big choices and then.

A

Talk about the way that every other choice is covered and if I still didn't.

A

Answer your question then, come back to me yeah. So this this is the question.

A

This is your question right, yeah yeah. How do we, how do we do that? What is what.

A

Is this magical cloudy smokey stuff in the middle? What am I smoking right so.

A

A

The the to the make that the big divide in how you do this comes down to.

A

This question: what it, basically, what is the relationship you have with your.

A

Network people- and this is this- is my you know, I wake up sweating and I this.

A

Is my right, this used to be my relationship with my network people when.

A

I work for a big company, so if you have a great relationship with your network.

A

People or you built the thing yourself out of iron or you rented the whole.

A

Thing from a cloud provider that lets you do this and what you do is you.

A

Program, your reuters, to send the packets- that's it right now it does require that.

A

That you can set up those it does require. You can set up extra ip.

A

Addresses pearpod some some corporates have a ip address base. That's so.

A

Congested they couldn't actually do that. You couldn't set aside a thousand extra.

A

Addresses just to just to kind of do do kubernetes now these are real. These are.

A

These the ones beside the blue squares or pod ips and the ones beside the black squares are node ips. So the these are the.

A

Machines there are no services here, the service the pod mapping is done before.

A

It leaves the machine that was that picture right.

A

Well, let me answer it right. Let me answer it so so, if you are, if you have.

A

The rights to program, your reuters, because you bought the network guy a lot.

A

Of beer, or something like that, this is not coming out very. It's not coming out.

A

Very good contrast, but the typical way you do this is you set aside a range of.

A

These ips on each host and then you just literally, have a list of routes in the.

A

Route in the rooter saying well, if it's in, if it's in this range like like 256,.

A

Addresses go to this machine turn next. 256 addresses go to that machine. Next.

A

256 addresses go to that machine. That is how a lot of people do pod to pod.

A

Networking the packets actually go on the same network as every other packet.

A

There's there's I mean this is kind of the. There is no such thing as container.

A

Networking because we just put them on the same network as the hosts, the one.

A

The one kind of tricky thing we did is we program the rooters to get to get to.

A

The right machine, so we so we know if the address is in this range, it must be.

A

This machine and kubernetes has a certain set of components built into it.

A

That actually automate some of this, like if you're, adding and subtracting nodes.

A

Is that is that helping yeah helping good question the back.

A

To what extent would ipv6 make this easier.

A

So ipv6 takes away the problem that you run out of ipv4 addresses because.

A

There's a lot of them! Well, there's no nats on this picture.

A

Well, yeah, you couldn't, you know, pv6 is there's enough bits to do this in the.

A

Over the whole planet, so so yeah there yeah generally speaking, you you you.

A

Nobody has enough ipv4 address space to do this planet wide, so they they do end.

A

Up nothing but there's no net on this picture. This is just very, very simple.

A

Just routing rules, so yes ipv6, I mean you yeah, you you just.

A

Do exactly the same thing, except that the numbers are longer and they're in.

A

Hex and they have more colons in them and that's that's the that's the main.

A

Difference in in ipv6 kubernetes right now does not support ipv6 very well and.

A

I'll I can extend that answer later after more beer. So almost everyone is.

A

Running kubernetes, you know, I said like I'm going to talk about the most common way that people do this and every single thing I say, has exceptions. So that's.

A

Another one, okay, so so this is, if you have a great relationship with your.

A

Network people and you have the ip address space, then this is an excellent.

A

Way to go, the other way to go is what's called an overlay network.

A

So what happens in an overlay network? Is you take the packet on its way out of.

A

The machine you hide it inside another packet, you, it's called encapsulation.

A

These things are capsules. I spent minutes looking for a clipart.

A

I spent I spent minutes cutting and pasting so when you, when you using an.

A

Overlay network, you encapsulate the packets or things like flannel things.

A

Like we've net our overlay networks, they do this overlay networks mean you never.

A

Have to talk to your network guy because you're actually running another networking software on top of the real network and hope they don't catch you if.

A

You know if they are actually serious network people, because, though, don't like.

A

People running their own networks on top of the real network, but yeah so so.

A

That's overlay networks, there's a little bit of overhead as we do the wrapping up.

A

And the unwrapping there are certain benefits like these. Things could be like.

A

One could be in your laptop and one could be in your data center. You know it doesn't matter. We don't have to be able to program all the routers in between as.

A

Long as this machine can reach this machine, the overlay will take care of.

A

The rest, we have a question well yeah. It's.

A

Effectively, what that's, what vpn to do that that the the you know you need you.

A

Need the automation to assign ip addresses to containers, to be able to.

A

Create and delete these things 100 times a second. You know you need a. You need a.

A

You need a a certain set of tooling, over and above what something like open, vpn.

A

Would do straight out of the box, but yeah, it's basically the same technology.

A

All of this stuff is implemented in the linux kernel. You know what what we we.

A

Make careers out of exposing it in easy-to-use ways: okay, overlay networks.

A

Right now, so I said there are two main ways that people do this and then.

A

There's another 50 ways that people do it. The somebody told me this is the this.

A

Is the initials of spanish, like the equivalent of the cia.

C

Which I find yeah, I find I find that very, very cool. I'm gonna yeah makes me.

A

It makes me the james bond of james bond of packet encapsulation. How am I doing.

A

For time, by the way I'm nearly finished am I am I okay, five minutes. Okay, let's.

A

Let's go yeah see an iso. Cni is a is a api for.

A

Writing plugins, so that kubernetes does not need to know how your network works.

A

So if you have a network technology that is special in some way, all you need to.

A

Do is write. The plug-in couplet calls the plug-in it sends it very simple.

A

Commands, like add and del and the plug-in wires up the pod to the network.

A

That's the purpose of meaning of cni. That is, that, okay, in the next part of my.

A

Talk is about exposing services outside of the cluster and I'm not going to get.

A

Into all the detail in five minutes, let's, let's spend most of the time on.

A

This one, because everything kind of builds on this one.

A

What is a node port? An old port, is the sort of the the default kind of service.

A

You get, if you don't say you want any other kind of service. If you want to.

A

Expose a service outside the cluster kubernetes, if you, if you get a node port.

A

It will arrange things, so your service has a funky port number, like thirty.

A

Thousand and twenty one, you know some some high port number that it picks to.

A

Try not to clash with anything else, and you can call it on any machine in your.

A

Cluster, that's that's the meaning of node port. It's it's on every node! It.

A

Should be like nodes port or my kubernetes has a problem with plurals, so.

A

So if, if you hit a node, where the service is not running, it will bounce it.

A

Across it's all done with iptables rules, in fact, by default it will bounce it.

A

Across, even if you are running one on this node randomly the the default.

A

Implementation is optimized for fairness, so it will just go bouncing across your.

A

Nodes willy-nilly, that's no port. Okay, host port is exactly.

A

The same thing, except it doesn't bounce across machines. It's implemented well,.

A

Yeah, so host port in the beginning is implemented by saying p docker.

A

Whereas node port is implemented in coop proxy oops and that this is what the.

A

Documentation says: do not use host port well, yeah, it maps it on whatever.

A

Whatever node your pod is running on so yeah you'd have to know which pod it was.

A

Running on, or you have to do it for a daemon set which is running on every part of every node or something something you have to do loose like.

A

Labels or something like that to tie it down to them yeah, that's sort of follow.

A

Those instructions hi, let's go on now I'll, come I'll, come back.

A

I'll come load balancer. You can request that your.

A

Service is a load balancer and really what that means is you you've got and.

A

You've got another piece of kit somewhere, that is a load balancer. So if.

A

You're using amazon, that's gonna be an eob, you know could be a. It could be a.

A

Box or like an f5 big ip device could be it. This is kubernetes is going to.

A

Program, your load balancer to do the right thing and.

A

The right thing is going to be to come into the nodes, and then it actually uses.

A

A node port to bounce across to the machine it's really running on so the.

A

Important thing to remember is load balancer. There has to be a specific bit.

A

Of kit, so if you're running it google cloud platform, it's going to be a google load bouncer. If you're running at aws, it's going to be an e lb. This is a layer.

A

4 for those of you are listening in college and finally, ingress, which is.

A

Kind of the same thing but layer 7, so it's completely different and again.

A

Specific implementations so there's an amazon one called an alb or you can.

A

Write net nginx, in this case it's going to run inside your cluster if you're.

A

Running like nginx, and in in that case, you would expose it with a hospital.

A

Where was the question? Yes, this is where it's absolutely necessary because of the.

A

Because we know where the inquest controller is running, so let me just.

A

Show you the config for an ingress controller. I just want to highlight so.

A

It's all about http, you can run. You can route different paths to different.

A

Services, it's it's totally layer, 7! It's! It's all based around receiving the.

A

Request parsing the request forwarding it on which is done in something like.

A

Nginx, which is a reverse proxy, so it's really a completely different animal to.

A

A load balancer, which is layer 4, which is like tcp only but the kind of.

A

Conceptually the same thing: ok, that's it yeah.

A

Yeah yeah, so if you could have some kind of more complicated arrangement.

A

Where you heard like an elb coming into your engine x, or something like that,.

A

Yeah I mean you, couldn't you can string these things together? However, you like.

A

This is if what I'm trying to say is if you, if you write a yeah mall.

A

Specification like that and your cluster is set up to run nginx.

A

Then you'll get something like that. So yeah you can you can layer more things.

A

On top of that, you can put things in front of each other. You can chain things.

A

Together and so on, but that's um these things are abstractions and they can be.

A

Implemented in different ways: yes, the last slide. This is the last the previous.

A

Slide is the other previous slide.

A

Yet so the external client will know nothing at all about food. The external.

A

Client will see a presentation of an address of the cluster.

A

The external client will be will be in the if this is.

A

Nginx, it will be talking to a real machine.

A

A

So the the there yeah there's a whole series of things going on so that.

A

Document would go into the master. The the document is read by the controller.

A

Which then, as you said, right so an actual nginx, that's what I'm saying.

A

There is there's a specific implementation which is nginx or you.

A

Could you could just not do that you could use the specific implementation for alb if, if you're on amazon or you could use the one from google, you know.

A

The the details are abstracted, ingress is a it turns out, is a terrible.

A

Abstraction, because the the reality is, these things are have a lot of overrides.

A

And specific rules that are only work in one implementation, but you know don't.

A

Get everything right first go! So that's what it is.

A

It is what it is. Well, that's why I'm saying they pretty.

A

Much all have their own ingress definition, even even though this format.

A

Is standardized they all right? Yeah.

A

Yeah conceptually, this is a way to request an.

A

Http reverse proxy, without saying exactly how you're going to do it and.

A

That's yeah, that's kind of the point, that's kind of the principle of kubernetes. You know, like I'm gonna request that some work gets done and I'm.

A

Not gonna say exactly how it happens. That's the whole. That's the whole point.

A

Oh yeah, we're hiring.

A