solo.io Traffic in ambient mesh, 21 Mar 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Traffic in ambient mesh - Part 3: ztunnel, eBPF configuration, and waypoint proxies

Description

In this video we'll talk about configuration on the ztunnels, explain the eBPF redirection mechanism and how to deploy waypoint proxies.

A

In the previous video,, we talked about the configuration that istio cni plugin does on the node and how it sets up the.

A

Iptables rules and ip sets., we talked about the interfaces on the node and how they connect to.

A

The interfaces on the ztunnel running on the same node. in this video we'll explain how the.

A

Ztunnel gets configured., we've mentioned the istio in and istio out interface and that they are connected to p istio in and p istio out, interfaces on the ztunnel pod through the geneve tunnels.

A

So if we look at the routes on the node first,, we run ip route show and then let's find the ztunnel.

A

That's running on that same node and then what we can do is we can shell inside that ztunnel and.

A

Then we can run the ip route show command again.. The configuration on both the node and ztunnel is.

A

Shown in this diagram- and we can see that istio out interface on the mode has the ip 127.1 and.

A

It's connected to the p istio out interface on the ztunnel, with the ip 127.2, and then similarly for.

A

The istio in and p istio in interface. now note that these ip addresses are set by the cni plugin.. We can.

A

Also, look at the iptables rules on the ztunnel., we'll notice, a couple of ports that are being.

A

Used,, for example, port 15008 is the inbound port using h-bone. So all traffic, that's entering the.

A

Node and consequently entering the ztunnel pod will be routed to this port.. The rule is applied.

A

For the incoming traffic that arrives on p istio in interface and has a destination port of 15008.

A

The rule is saying to use t proxy target. t proxy allows transparent interception and redirection.

A

Of the traffic in such a way that the original source, ip and port are preserved, and that allows.

A

The destination to see the original client's ip address and ports, regardless of how many hops it.

A

Goes through. now the on ip and on port parameters are just specifying to redirect the traffic to the.

A

Local port 15001 and to use the loopback address. the t proxy mark parameter is specifying the mark.

A

To be set on the packet. note that the mark is used in other rules as well, and this mark is.

A

Used in the rule list to identify the traffic that needs to be redirected to the ztunnel.

A

Similarly, we have the rules for the outbound traffic that gets redirected to port 15001 and.

A

The port 15006, which is for plain tcp traffic, so traffic that comes outside of the ambient mesh.

A

Now this diagram captures these ports more clearly.. Note that there's also port 15053,, which is used.

A

For dns traffic and the rules for that port only get created if we had the istio meta, dns capture.

A

Variable set to true. now that we've seen the rules on the node side and the rules on the ztunnel.

A

Side,, let's put everything together and talk through how a request sent from a sleep pod on.

A

Node 1 to the http bin pod on node 2 flows through the different components.. Now the request from the.

A

Sleep pod is captured by the rules and iptables configuration on the node., because the pod is part.

A

Of the ambient mesh,, its ip address was added to the ip set on the node and all the packets.

A

Will get marked with the mark 100. the rules on the node specify that any packets marked with.

A

100 are to be directed to the destination 127.2 through the istio out interface.. The rules on the ztunnel.

A

Proxy will transparently proxy the packets from p istio out to the ztunnel outbound port 15001.

A

The ztunnel will process the packets and will send them to the destination ip, which is http bin,.

A

And that gets captured on the dedicated interface that was created for the http bin ip on node b.

A

The rules for inbound traffic will ensure that the packets get routed to the istio in.

A

Interface, and then a tunnel between istio in and p istio in makes the packets land on the ztunnel pod.

A

The iptables configuration will capture the packets from the p istio in, and based on the marks,.

A

It will direct the traffic to port 15008.. The proxy will process the packets and finally send.

A

Them to the destination pod. in the first video, we mentioned how there are two mechanisms.

A

To configure the traffic routing., one was the iptables, and the second one is using the ebpf.

A

Using ebpf,, we can extend the kkernel's capabilities and run sandbox applications without modifying or.

A

Recompiling the kernel.. This means instead of configuring, iptables, route, tables, and.

A

Tunnels, we could write an ebpf program that does exactly what we want and then we could attach it.

A

To a kernel defined event. note that there isn't necessarily anything wrong with iptables,. However,.

A

It does add additional complexity and slowness due to the number of links created,, there's more.

A

Complex kernel, forwarding, rules, and so on. from that standpoint, ebpf is less complex,.

A

More performant, and easier to manage., so how does it work? the traffic control subsystem in.

A

The kernel has a hook that allows us to attach an ebpf program to it.. The hook is called tc, standing.

A

For traffic control, and it's used to intercept and modify the packets that are being sent to.

A

Or from the specific network interface., so instead of configuring firewall rules,, we can write ebpf.

A

Programs that will execute when the packets are being sent to or from a specific network interface.

A

The istio cni installs, four ebpf programs that hook onto the ingress and egress points of.

A

The traffic control subsystem., the app inbound and outbound ebpf programs are attached to the.

A

Ingress and egress points of the workflow pod on the host side, and then on the ztunnel side. The.

A

Programs are attached on the ingress traffic control on the host side and on the pod side.

A

As shown on the image, the ebpf programs basically do the same thing as the iptables.

A

Rules, but they do it in a more efficient way,, which is easier to read and manage.

A

The last thing we'll talk about is the waypoint proxies.. So far, we've only talked about ztunnels.

A

That handle l4 traffic,, however, istio also supports l7 traffic, and in order to handle any l7 concerns,.

A

We need to deploy a waypoint proxy. now before we deploy a waypoint proxy,. Let's look at the new.

A

Proxy config workload command. using this command,, we can check the proxy configuration of a ztunnel.

A

Now the command will list all the pods in the cluster and it tells us two things. first,. It tells.

A

Us whether the pods are part of the ambient mesh or not., we can see that in the protocol column. in.

A

Any pods using the h-bone protocol,, those are part of the ambient mesh. second,. It tells us whether the.

A

Pod is using a waypoint, proxy or not.. We can see that in the l7 support column. note that none of.

A

The pods have waypoint proxies, deployed, so the values in that column are set to false.

A

If we wanted even more information,, we can use the json output format.. Now note that, in addition to.

A

The workload configuration, there's also the authorization policy configuration that will.

A

Show up if there are any l4 authorization policies defined.. So, let's see how to deploy.

A

A waypoint proxy. now the waypoint proxy is an instance of an envoy proxy, and in order.

A

To deploy it we'll have to install the gateway api, crd's first,. So let's do that.

A

And now that we have the gateway api crd's,, we can take a look at the waypoint command. note that the.

A

Waypoint proxies are deployed per service account and they can live and run on any node regardless.

A

Of where the actual workload is running;, they don't have to be co-located with the workloads.

A

So let's generate a yaml first, so we can see how it looks. Like, and I'll use the http bin service.

A

Account as an example., the configuration uses, the gateway class name of istio waypoint, and.

A

It specifies a listener called "mesh" that listens on port 15008. now to deploy the.

A

Waypoint proxy,, we can use the apply command.. Now, let's look at the pod when it starts up,.

A

And now, if we look at the configuration of the ztunnel again,, we'll see the l7 support column.

A

Is set to true and then, if we look at the json configuration,, we'll notice that the waypoint.

A

Address,, the ip, will actually be included in the waypoint addresses field for the http bin workload.

A

Now, as for the waypoint proxy and how it gets configured,, it tries to exactly match the ip.

A

And the port of the destination workload, then it enforces any l7 authentication rules and finally,.

A

Forwards: the request to the destination. now one important thing to note, with the waypoint proxies.

A

Is that they're only part of the request flow on the server side and not on the client side,.

A

So there are strictly reverse proxies for the l7 traffic. for example,. If we have two workloads,.

A

Both with waypoint proxies deployed, and we make a request from workload a to workload.

A

B,, the request will skip the client's side, waypoint proxy, the proxy on workload, a side and then it's.

A

Going to end up on the server side, waypoint, proxy and then finally, land on the workload b.

A

In these series of videos,, we went into a lot of details and looked into different components.

A

Such as istio cni, and how they work together with the platform and how the configuration is set, up.

A

Note that at this time, ambient is part of istio's main branch and will.

A

Be available as part of the regular releases, starting with version 1.18.

A

However, ambient is still in the experimental phase and it's not recommended for production use.. If you.

A

Enjoyed these videos,, please consider subscribing to our channel, and if you have any questions.

A

Please leave a comment: below. thanks for watching and I'll see you in the next video.