Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Troubleshooting On-Premise Kubernetes Network: Underlay, Overlay and Pod - Tomofumi Hayashi, RedHat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Troubleshooting On-Premise Kubernetes Network: Underlay, Overlay and Pod - Tomofumi Hayashi, Red Hat

Networking is hard. Kubernetes networking can be even harder, especially in on-premise envirnment. Once you install Kubernetes on-premise, you need to manage both network: underlay and overlay. So in case of network issue, operator needs to identify where the error comes from. In addition to container developer, they want to tapping certain pod traffic to somewhere to check the network, too. Based on the last KubeCon presentation by Minhan and Rohit, this talk will presents more practical approach to solve network troubleshooting for on-premise Kubernetes environment, including how to check overlay and underlay network information and how to tapping pod network traffic independent overlay network.

To learn more: https://sched.co/GrWr

A

Good afternoon, everyone thank you for coming. Let's start the my presentation, my I talked about the troubleshooting on Pam's kubernetes network under ray Oberoi and part I'm at Omaha Maharshi from Red Hat.

A

Today's agenda is the following: so the first time I'm introducing why cognitive network is difficult and then they go to the kubernetes and that it's network setting up two troubleshooting now the first step to introduce the kubernetes and then goes to the troubleshooting in the troubleshooting section, I will go to the introduce how to identify the content, interface and then, of course, I'd be terrible and then the last things is the packet capturing as the packet capture, I'm, sharing the ademma stuff, and then this cure code is about the throttle URL. So they are.

A

If you capturing doing stuff in the smartphone, you can see that. So, if you feeling the acutest network, it's easy. Please raise your hand if.

A

Not please raise your hand, okay, comrade I'm I feel so so I'm. Thinking about why kubernetes network is difficult. There are the several of Mississippi stuff, but I directed sharing. There are three items: one is the many decision at design and the co-existed multi-layered Network and then packet modification. The first things is once we are thinking about hey, let's install the equipment netis in my data center, my server at that time. We need to consider a lot of things for design.

A

I mean the first time we need to setting up the host network and then goes to the container networked nowadays. Container network is the Oberoi and then a non-motorized exists. We need to choose that IP address network policy and bandwidth. If we need and then of course the when we setting up the kubernetes cluster, we can choose the row, the branches are network policy or ingress is the option. So we, if we thinking about we, need to know the brands of services at that time.

A

We need to identify the commercial stuff or whatnot or open source, or is there any policy or stuff? So there are little bit complicated at the design phases and then, of course, the amount and the kubernetes network is multi-layered network I mean that not only the host network, as I mentioned before continental Network, it exists and then inside the container network the packet is returning everywhere, so the for example, the software for them praying, for example, the open messages if it exists. There's this rewrite in a pocket I'd be tables at the hot side.

A

Is writing the early writing in a packet by the kubernetes component, as well as the continent Network component and then, of course, the inside the pot? There is I pee table rules, so this is very complicated and then the equipment, a test network resource, is also the IP tables and the IPPS services rotor burns and manuals.

A

There is a lots of concerns so when we're thinking about the one container host running the pot at that time, when the process is the pocketing into the Ethernet 0 and then after that is tough, IP tables right there or something first and then goes the bridges. If the this bridget is the open visit, it may be that this rewrite a pocket again and after that IP tables at the host side. We do same things rewriting in a pocket, so there every time we are in a packet, we do.

A

We do not and identifying a tracking. Then there is a very tough that's the another reason so and then the the first step of the on-premise Cabana. This is install kubernetes so and then, of course, that when I try to the equivalent, as especially in the new container network stuff at that time and goes into trouble, so I drag the sharing idea. The how the kubernetes and the network is setting up in the each step of the queue bottom, so in the simplified via install, is the something like that.

A

So the first time we install the operating systems and also well what stuff then the we did a cube, Ludman in it in the master and akhirin join at the node and then, after the finishing with copying the siena problem and then, of course, the SE ni config file, then kubernetes get into writing at the first step.

A

When we did be a cubed in it and the kid on join at that time, the in the cube, control get node, shows the not already state and then the port at that time and the so the system component right. The acute api server key master at CD is the running set on the other side. Cordy, and s is in the pending state. They are not all pot is in ready, but on the other side, IP table set output shows the additional IP tables will be shown by the cube component.

A

So at the time the kilmer drop mask right past. Writing, adding a several rules, so summarize this stuff, the cube atom after the cubed up in it and giant at that time. The cube must register all nodes by the join, but this is not ready and could keep Adam also setting up the all thought, but the cube, DNS or core DN a-- stuff is still in the pending. This is a Kirsty just waiting the continent network readiness and then the kubernetes component also adding the initial reputable source.

A

So at that time we can verify that host network is ok, but the continent network is not OK, I mean they're, not ready. We are still waiting so once we copying the scene approving and the config file as well. At that time, cube control get not without goes into the ready, yeah right, then Cardenas is called running and then sometimes the discuss is the front case, but the today up new IP address row is already also added, so so they have.

A

This means the year after their Sienna, probably and the configure file copy into the adult everything not only the host network. Also, the continent network is ready, but of course they can go to their third case and then I feel and I think that yeah there is a two common failure case for the step two today up I think the one is the Sienna programming and the config is afraid. I mean the our Sienna config file is the embodied at that time. The is a one purchase and another stuff is the up.

A

Cni config is buried to the assign. I programs successfully lead the news files, but this is not much to the our expectations, our kubernetes expectations at that time via the it's also, the failed so I'd like to show the how we identify, which cases is your case.

A

So the post case, the cubic control get node is saying ready so that we could not identify idea, which case is today, but we can see the port keeper control, get pot and then the in case of the case, a Toria. If CNI configuration is embodied at that time, we can see in the container, creating state in the queue pot and also Dia fared, create pod. Sandbox reason in the describe this is means the once they are seen. A problem is, and the configuration file is copying.

A

The not kubernetes written actually drq brett is trying to create the pot. Then they trying to the invoking the siena program is a configuration file and then, if configuration file is embarrassed or are you only copying the configure file, but not the CNR praveen at that time? This is the explicitly failure case, so we could clear it does not ranching the pot.

A

So, that's why the we see the afraid to create Todd sandbox State, but on the other side, if CNN configuration is the Buried case at that time, of course, cube read successfully launched the pot, but then that's why the Cuba DNS, showing the running state, but the cube control described part shows warning unhealthy state. This is because the readiness probe federal case, so the some pot had the redness proof star in it.

A

The pot llamo file, readiness, prove means the sending in the rest api to somewhere to validate the network status and, of course, the container health check. So if you write the wrong see analytic network configuration at that time, maybe we can create the pot, but this network is not go outside or maybe the equivalent a server or something like that. So at that time maybe we goes through the these unhealthy Venus pool.

A

Then, even though there this is the running, so that's the so we can identify the two cases in case of the the copying CNI and then maybe this can be helpful to identify the first step to introducing the Cuban artists. Them goes to the troubleshooting. The troubleshooting is the very wide topic and then, of course, I could not introduce everything. India thirty minutes or maybe do more than one day, he'll I, don't know, and, of course, that I I don't know everything about the kubernetes in the future as well.

A

So the I direct with introducing the how to identify the continent of areas and then IP packets, iptables on a packet capture very simple stuff, so the Linux Network as the first step is the troubleshooting is IP address shock mess. We can check the IP address and then the its link status and the day stuff, but how about the in the container, especially via we, can see the cni zero interface, but I, don't know they ask which types of the interface is the CNN zero.

A

So the first step is IP command with the option shows the interface type so that this D options rotation is important so that just after the IP commands IP D the address, or maybe the of course there are other commands over the IP, also supporting the the option, so their IP D rings show at that time. Where's the the option: we can see the a bridge keywords: there is not existing the India without D, with the option we can see, they are have the this interface type.

A

So at the time the CNN vero shows the bridge. So this is the Rennicks bridge, so the in case of the akan tenor. We frequently see in the these.

A

These write the B ETH, so this means the a BT. It means the virtual isn't a device which is the point-to-point virtual connection which the Linux kernel provides, but the after the at Merck. It's every time, I'm. Thinking of what I of three means so I'm investigate about that. Let's close up the output, then the here is mapping, so that's the introduce each step. The first step is this number. This number is the eye of index.

A

This is the interface identifier of this namespace over in X corner and then goes to the next is the interface name, just a name. Okay, the eye of three means the this is the opposite side interface index over the virtual Ethernet, so the this bth seven we have a taste-off is goes to I of three index, the interface so the from these informations.

A

So the left side, the diziness is the I offended six and a B th and with the I of three and there they are in another namespace or maybe the sometimes in the same namespace. We can have the I of index three and then end with at math I of six thing. So I I try to find it this stuff and then yep I can find that so we can checking the automatic stuff and then the of course the RFT's ID number we can see which interface is goes to which namespace and then.

A

But there is the another question in case of the container like the nginx or a simple small type. Cleanup container, we don't have the IP command right, so IP commando is the not so handy and, of course, the some container introducing the full version of the IP commands at that time. The d is not exist. How to do that? Linux have the year NS enter commands which do with the t and the and option we can using the hot side program at the accountant ur.

A

So so the disk, the commands execution, is in the cube, node 1, and then this output is the Saudia. This, the this PID is comes from the NZXT container and then the I add the IP the address documents in here so the between the and the IP, we add the dot mark, and then we can see their diesel food at that time.

A

This IP command is comets from hostile site, not their continent site and then this innocent that requires the sudo command. Sorry, you need to have the root privileges as well that the using this stuff, even though the container is a simple and then they don't have the IP commands. We can invoking IP commands from the accountant side and then next is the how we identifying the PID that's the also program, maybe sometimes they become just the PS command to identify in some cases.

A

But if, let's imagine that the Bay Area's nginx container is running there at the same host, of course it's very hard, you know. So that's so there that's the way to identify in the interface the first time. Let's do the equipment control get taught with Jason Aoyama. At that time we can see the continent hiding in its output container ID, maybe or something right, yeah, dr. Cohn, thrush Russ, something along the figure, or maybe the CL I orchestras in case of the cryo case. They stopped and then using this figure in the target.

A

Cube node with the doctor, inspect for doctor out, run time, walk ride, control, inspect and the the stuff were cry. One time we can get the PID from this output, that's the one to identify the PID of the pot, and then we do enter any center like that. We can get the app interface output and then iptables is so where for the app as I mentioned before, iptables is not only inside the account and host.

A

Also the iptables is in the pot, so the first things is who, as the iptables, that's the maybe the first step. We are thinking about almost everything like you know. The queue Brett adds the queue post-wedding Commerc muskrat, killed, Matt drop firewall and the queue proxy here before starting forward each the chain is existing each chain hub the several rules and then CNI also adding the several loose and the we racketeer.

A

We Bosnia weep and weep MPC Network process, a controller stuff for each chain and then, if you using the sto, it's also adding that the several chains for so at that time, maybe the training or something. And then, of course, if you using the another stuff, another component, also adding an IP êtes-vous. It's good for container networking from a user perspective, but it's a very hard to network a transmitter or kubernetes administrator.

A

So the I'd like to see there are some of the hint to the backing that this stuff is especially in the acute proxy and accurate, so they create the hierarchical structure in the IP tables chain. So in case of the not pro routing our output or chain, they created the top of the are these stuff in the cube services.

A

In top of that, and then this chain is the connect to the cubes SBC services items for each services and then cube SEP stuff is continued to that and then cube set, adding the cube mark masquerade stuff, and then this is goes to the right hand, side that was rotting and then some cases to be the they they changing the destination as a mat. And then in that case they. This is the tool chain to the just, adding the cube, post-wedding and then goes to the masquerade.

A

This is very simplified than the crowding output and then the on the other side. The filter input filter for the filter output. There is the fraud, so it's easy, but the Saudia pro routing output is a little bit challenging. So it's pretty care about that and then, regarding the IP table stuff, the last cubicle at the EU, the a Google may hunt and rush. It that's great talk about the IP tables, so black holes in the warm house and has done it troubleshoot in the magic yeah.

A

They take the magic, okay, good and of commence networking so that, if you are checking that these stuff and them there, they introducing the these IP tables talk mainly so there I recommend, if you interested in the packet capture, so the user is in the troubleshooting. The the fundamental stuff is the capture in the packet capturing from there lots of information so where we can, can we capture the packet looking this stuff?

A

There is three stuff one is the container host, isn't it zero and then the bridges and, if either of the pot, but regarding the port side, how do we capture without a Wireshark in the container? So the yeah, of course, I understood the various container images. That's not including the Wireshark and now, of course, they we could not, could not ask the the kubernetes use of hey. Let's see is that is container for it via production.

A

It contained why you sure it's it's funny, so how to do that and then also I'm running and I'm, creating there some tools to the a Cuban lettuce, pod Network, topping this cause a Coco top Coco top taking the argument over the ADIZ part. One is the APOD name, and then next is the bit extra ID and the destination node and the Mira type I will introduce our explain this next right.

A

So, let's imagine the following environment: I mean the acute master and the cube node one and then cube master installed the wire circuit. At that time, we direct to trying the capturing the this.

A

Isn't it your pocket, not literally then, after rewriting the pocket by the other stuff Wendy, we invoke evacuating a coked-up command, the pod name be extra ID and the destination node and the middle type then Coco top creating the mirror interface at the hakeem another one, and then this also using the Linux kissing mirrored functionality.

A

Internet's pocket is the mirrored by the DS interface and then this interface is the be excellent. Interface of the DS goes to the underlay network, with the be extra encapsulation and then this cocoa tab is also creates the extra in the queue master site. So once after creating the ADIZ stuff, we just capturing the pocket at the cube master and we can capturing the pocket and the mirror type is showing the which direction we capturing of the integral.

A

If we choose the ingress at that time, the only the out from outside to the port traffic is captured to the this mirror interface and the ingress is outside via we only capturing the from talk to outside both is the adjuster ingress progress.

A

Okay, that's the demo. I have this is ripe them also cube. Let's see, cube control get all the first. There is the two node and one master yep that stuff. So there is the inside my laptop I ranching, the three virtual machine kumusta in the node and the kiln or two, and then there is the isn't devices, and then this is also connecting with the yet birth, be our Dell devices at the host. Let's go back here, cube to get both I'm running the nginx. Here so and then IP show IP address show there.

A

So currently there is no Miller interface. Yet, okay, that's the.

A

Creates the mira interface.

A

So the coppertop taking the argument, appalled, ng, x, VX, Randi and the destination node- is a cube master, so this command creates the Yama file. So we need to add the pipe to add the cube control to this input. Then this creates the cocoa type pot. Then yep. This is already running state and then, after this stuff IP address, show commands seeing mirror interface in here and then, of course, we do.

A

We do the option at that time we can see the Bieksa interface and the exid is a senior and then let's go to the node aside. This is the port. Engines is running soap, use.

A

And in the center.

A

So with the innocent documents, we can see not only there if other devices also the Mira interface is newly added.

A

Okay, that's capturing the Wireshark cube, control, get hot or wide to seeing the IP address, and then, let's see in the X, then we can capturing the HTTP output at the cube master. Of course, this again we can see that so that's the first step, then the tool removing that is interface, just the delete, then the boss, interface mera interface at the cube master and then the Mira interface and the pot is the mobile IP, the think so, oh maybe maybe we don't have the ID okay.

A

There is no Mira interface as well as angelic side, that's the first stuff and then sometimes the we also drive not to install the wire shark in the Cuban master right. So at that time, I also have their way to do. That is.

A

Best so the instead of the destination node, we could add the some certain IP address for this nation I P, to create the mera interface. So let's, let's try that then get port. So at that time the cocoa pop only creates the mirror interface at the port site, because the the opposite side is unknown for kubernetes. But so this time this IP address is this virtual, pretty stuff. So.

A

So I need to create the VXR interface by hand for now, then, why yeah sure? Let's let's the deal Wireshark, then of course here is the be excellent, Aero interface here then starting a capture then also they are in the cocoa type commands I also did choose the ingress as a mirror type. That's the stuff. Let's that get the occur at that time.

A

At that time, okay, let's filter the HTTP who were easy to the back. At that time we can only see that the get command there is no output. This is because I choosing the minute type into ingress. So there, if you be choosing the egress at this stuff, then we could not see the get command instead of the cat command. We can see the output of the HTTP.

A

That's the demo then go back to this rice yep literally. That's all that's about it. So in this presentation, I'm showing that the kubernetes network is the difficult, the reason is the switched off and then goes to the kubernetes and network setup to identify your first step to troubleshooting and then in the troubleshooting section. I will introduce the how to identify, contain either face I mean the the option after the IP and the IP tables a little bit and then packet capture as a demo.

A

That's about it so I'm showing the actual code of my presentation again so that place look into it if you're, interesting and then Coco top is already in the github. So please use it if you're interested in that's about it. Thank you for your attending.