Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Provider AWS Update and Roadmap - Nick Turner, Ayberk Yilmaz, Yang Yang, Amazon & Nicole Han

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Cloud Provider AWS Update and Roadmap - Nick Turner, Ayberk Yilmaz, Yang Yang, Amazon & Nicole Han, VMWare

We will give an update on the status of the kubernetes/cloud-provider-aws and kubernetes-sigs/aws-load-balancer-controller repositories, and the roadmap for the future. We loosely divide the talk into 4 topics: 1. the migration from in-tree to out-of-tree (external) cloud-provider, 2. the ECR credential provider, 3. the future of the AWS cloud provider, including the work on the V2 provider, and 4. the current state and future of AWS load balancer controller. We welcome discussion and feedback during the talk.

A

Hello, everybody and welcome to the cloud provider aws update and roadmap at kubecon europe 2021.

A

We'll start off going through the agenda that we have for today, uh so first I'll be talking through the background of the cloud provider, extraction, migration um and the status of the cloud controller manager, then nicole will be talking about the cloud provider v2 effort.

A

uh After that, a burke will touch on the credential provider and finally, yang yang will uh give us a little bit of information about the aws load, balancer controller and the roadmap there.

A

So let's get started um okay, so um so to touch on the background of the cloud provider extraction effort. This is something that's been going on for quite some time in the community.

A

uh The original implementation in kubernetes uh had a go interface called the cloud provider interface that was built into kubernetes, along with the each cloud provider's implementation.

A

So the initial versions of every cloud provider was actually compiled into all to each binary, but the community quickly realized that this was already causing and just going to continue to cause significant bloat as more cloud providers were added um and the the maintenance effort was just going to be too great.

A

um So the the effort was started to extract all of the cloud provider specific code. um So that's like each cloud, provider's, sdk and uh any logic that is specific to a cloud provider would be broken out into separate repositories um and compiled into separate binaries, not using the main build system. So this really, this began a multi-year effort to extract all the code and that's still going on today.

A

um So I just wanted to quickly go over each of the components that had cloud provider code or have cloud provider code and the extraction effort related to those.

A

So first uh the the cube controller manager. There was some cloud provider related code in four of the controllers.

A

That would become the cloud controller manager that includes the node, the node lifecycle, the service and the route controllers, and we'll talk about those in a little bit more detail a little bit later, and then there was also the the volume uh control loops and those um are being replaced by the csi driver effort.

A

So for aws that would be the ebs cs. Csi driver or the uh fsx csi driver and another component is cubelet. Cubelet has a couple areas of quad provider code, one of which is the docker image credential provider. So this was some code that was built into cubelet and four common container registries. Like ecr, there was a package there that would provide the the docker image credentials when cubelet was doing a pull image and that is being replaced by a external binary that cubelet will exec it's very similar logic.

A

It's just outside of cubelet cubelet execs it during pull image and gets the credentials.

A

A

uh Additionally, um the other area of code in cubelet was fetching node addresses, which is being replaced in the cloud controller manager and the node controller.

A

A couple other areas that are worth mentioning is the api server had the capability to do ssh tunnels, but this was so. This was related to cloud provider code because the api server had some flags that it took uh that enabled ssh tunnels only on gcp, and so that effort is being replaced by the network, proxy effort and, finally, the cube api server initially had a persistent volume mutating web hook and persistent volume, labeling mutating web hook and that is being replaced by or that was already replaced by the persistent volume controller.

A

So all the cloud code was removed from.

A

That um so I wanted to give a timeline from aws's standpoint, but this is these are estimates and they're. This is only. This is only going to be true if everything goes perfectly, so um it's relatively unlikely to look exactly like this, but this gives an idea of what like the order that in which things have to happen, um and what the earliest possible dates for some of these would be so you can see for the features. The first is the aha migration framework.

A

This is something that is, uh it was actually it just was merged in 121 in alpha state, and um this allows aj clusters to migrate from just running the cube controller manager to running both the cube controller manager and the cloud controller manager using a migration lock.

A

um So the plan for this is to go beta in 122, then we have the credential provider framework. This was alpha in 120 and along with the aws ecr credential provider, that was alpha recently. Actually, I think it was kind of in between 120 and 121.

A

So, ideally, both of those along with the adobe slide controller manager would be beta in 122, which would allow all of them to go ga and 123.

A

And what all this means is that the earliest, according to you know, like aws uh cloud provider, timeline the earliest possible date for the entry code. Removal is 124, but that will probably be later just because of other dependencies.

A

So, in terms of the aws cloud controller manager, specifically, the upstream features, are no longer being accepted. Pull requests are limited to bug fixes. So, if you are opening a feature pull request, you should open it on the cloud provider aws repository where we've duplicated the v1 provider code there we currently have alpha, builds for the cloud, the cloud controller manager and we're working on beta right now, and um if you do want to go check it out, um there's some uh a couple different ways: you can install it on a cluster.

A

uh If you are, if you use cops, you can um configure it in the cops config, the cops cluster config to uh use the external uh cloud provider and there's also a helm chart provided in the repository, as well as just some yaml files that you can use to install it.

A

So what are the responsibilities of the aws cloud controller manager?

A

For us? It's the node, the node lifecycle, the route and the service controllers.

A

The node controller is responsible for updating node status and node addresses the node lifecycle controller acts on node events, so you might have a node deletion where you actually need to go and delete the node object that would fall under the node lifecycle controller.

A

Then the service controller simply manages cloud load, balancers for load balance, service services and currently it it handles elbs, and it has some nlb code, but we're talking about moving the lb code to the aws load, balancer controller, which yang yang, will talk about later and then the route controller manages a cloud route table. If you have nodes that are using one, if your nodes are set up with one sider, uh one pod set or per node.

B

A

With that, I I'm sorry with that I'll hand it off over to nicole to talk about cloud provider, v2.

B

So, um hey I'm nicole from vmware. I will talk about some updates on the current aws cloud provider. V2 implementation. We will continue to support the existing aws cloud provider in out of trade mode.

B

This new version will address many new issues and gaps in the current v1 provider implementation. um So, for example, we are looking into allowing node names that announce the private dns of the ec2 instance.

B

For now, aws cloud provider forces the node name to be the same as aws private, dns name. For example. This leads to the node being removed if a different node's name is used, so it should be allowed to use any name as the node's name, more descriptive. Nodes names are generally helpful and useful.

B

Currently load balancer can only be created with a machine generating them, giving load balancers easy to rate names, while greatly enhance the user experience of kubernetes on aw aws. We also consider adding something to the service annotations and allow users to directly manage this.

B

So a new v2 implementation will try to address these limitations. Also, we need this since the v1 legacy implementation is going to be removed soon. V2 implementation is in alpha state and after the launch, it will be supported for new clusters.

B

So, to develop cloud provider implementation, we need to implement a few common interfaces like instances load balancer in zones so, for example, the instances and zones interface methods will be called from the node and lifecycle controllers. The load, balancer interface methods will be called from the service controller for instances. We have an initial implementation of instances, v2 interface um instances. V2 is an implementation for instances and should only be implemented by external cloud providers.

B

Implementing instances with you is behaviorally identical to instances, but is optimized to significantly reduce api course to the cloud provider when registering and sticking nodes.

B

For example, we have implemented in the interface methods like whether to check whether the instance is exists or is shut down, where we use the node's name or provider id files to find the node in the cloud provider.

B

As I mentioned before, we are looking into allowing node names that are not the private dns of the ec2 instance. In the near future, we will support node's naming policy other than private dns names.

B

Implementation of this interface will disable course to zones interface. So zones will be uh is deprecated in favor of retrieving zones or region information from instances v2. This interface will not be called if instances v2 is enabled.

B

The existing dom's interface was mainly created for the class to interface, with the local metadata server to fetch a nodes, dom and region with external cloud providers. It makes more sense for the zone or region logic to be coupled to the instances interface after kubernetes version, 1.20, song and region information should be added. In the instance metadata type we are using as part of the new instances, v2 interface in the cloud provider.

B

So in most cases the zone, origin information is contained in the gate, gets instance api core of a cloud provider. This change will reduce the extra core for zone or region in the cloud provider so for load balancer. We have an initial pass of the advanced interface that is currently working in progress. There are some key issues.

B

All we need to address so like first right now, load balancer can only be created with the machine generated names, making it very hard to distinguish in aws if multiple advances have been created within your accounts account also, the default load, balancer name for the cloud provider from the cloud provider package, which is currently in use, have been deprecated in this replacement.

B

Giving load, balancers iterate names well greatly enhance the user experience. It would be nice if we could pick a template to use for naming, for example, a combination of cluster name and service name, mines, work well and prefixing, the loot, bencher name with some meaningful name like the environments, the app stack, etc will be very much helpful, and we also consider adding something to the service annotations and allow users to manage this.

B

um Another thing we are looking into is: we should build more expressive apis for the balancer configuration options right now. It's an explosion of annotations that are really hard to comprehend.

B

So up I'll pass to book to talk about credential provider.

C

Nicole, my name is iberic and I am a soft manager here at aws working on eks and I am going to talk about the production provider extraction.

A

C

It provides some context and then briefly explain what it actually means for the cluster operators.

C

So far, we have been mostly talking about the responsibilities of the control plane, but, as nick mentioned, it also includes some cloud-specific code, I'll be talking only one of those specific cases, workflow that runs specific code, and that is patching image credentials for non-public registries.

C

uh Whenever cuba needs an image from a non-public registry, it somehow has to fetch the credentials, so it can actually get that image and currently shubha does this itself and, as you can imagine, each registry has different apis. Different expectations and cubelet has specific code for each different registry to face those credentials, and that is actually what is actually not what we want at this point because we're extracting class specific code.

C

So somehow we have to get rid of this responsibility and in order to remove this responsibility from the tablet, a new cap was proposed and this step basically proposes a new plugin-based approach where giblet basically talks to these standard binary audience to effect sequences for itself.

C

So how it happens is basically these plugins are nothing but external binary running along with your cube on your nodes and whenever cublet needs credentials to fetch an image, it will basically communicate via startup output and then ask this binary to fetch the credentials for yourself. They use welfare json, it's version and serialized, but again everything is happening through sound input output. I strongly.

B

Suggest reading cap.

C

If you like to learn more, it's very relative written and it explains how this all connects and next like this, so okay, it's not going to fetch these credentials for us anymore, but what does it mean for us? Do we have to do anything?

C

The answer is yes, it means cluster operators will have to configure a cubelet, so it knows what plugin to use for what image. Obviously initially cublet will have no idea about for what image musical what kind of binary.

C

So in order to achieve this, there are two new flags you have to pass the cubelet, so the first one here used on the screen is basically the location of the binaries and I mean plugins, and this is where this is a folder where your binaries are located, and this is where the cube is going to be looking for these binaries and the second one is a pretty important one.

C

uh It is the config file you pass to cuba and this conflict file will calculate which binary to use for what kind of images it is basically a list of regular expression, cloud provider, name pairs and provider name here is the name of the mindlead and cubelet. Basically, whenever there's a match with particular explanation, you'll use the given binary professional images. That being said, it has to be in a very specific format, and you can read the cap to learn about this format.

C

For specific aws users. We already have the ecr plugin, implemented and ready. You can see it. I mean on our repository.

C

We are not shipping it with the cloud provider yet, uh but we are planning to ship it in the future, and we also have more detailed examples and install introductions insert instructions on the documentation and we also have a more important. We also have an example config you can just use. So basically, your pcr uh logic just keeps out of the box, and with that, I'm going to give it to you to talk about the controller.

A

You're muted hanging.

D

So I'm young from amazon, ets team. So today I will talk about amazon adventure controller.

D

uh It's a controller to help manage gymnastics advancers for kubernetes cancer, which satisfies kubernetes, inwards resources by provisioning application advances and satisfying kubernetes services. Resources by provisioning network advisors for me is known as aws alb invest controller. We did a rewrite and rebranded to be elapsed from the banner controller in october last.

D

Year so I'll talk about supporting country so currently with ingress resources and application. Advancer we support insert mode and ip mode, for instance, mode the node will be registered targets.

D

Traffic will reach the node code for your service and pro proxy to your containers and for ip modes traffic will reach your containers directly where, but you are seeing, I must be using amazon's vpc and we also support a feature called ingress groups which allows users to specify a set of ingresses to be supported by a single application advancer which can help customers to saving cost for services resources with networking advances with.

D

Currently we only support ipmo, which is targeting the content across directly, and we provide a compatible api with the entry service and ob support along with these two features. We also support a new theory called target group binding, which exposes applications with a existing target group like aob, technical or an arbitrage.

D

So this allows users to bring their own advancer and tag groups and just net the kubernetes manage the targets inside hyperboost. So customers can use external tools like confirmation, cdk or terraform, to manage, maintain an advanced infrastructure.

D

So what's coming next, so we will release a new version shortly. The new version will include support for lb instance mode, which will be similar to what we have in the entry controller after this feature is released.

D

We recommend to using adapting the bezel controllers to manage the energy for your public service, which is more flexible, and we will support new nlp features more up to date, and it also addresses some technical implications in the existing cloud provider and what we will also support, static support user to specify state ips for analytics, and we will also add a support called node selectors, for instance, mode which allows you to specify the set of nodes used as the backend for your service.

D

So you can use the labels network to match the nodes. This feature will be supported for both invest and service, and our target binding crd for english. We will add a new crd, called ingress class parameters, so is for ingress class support added from kubernetes 118, where you can specify a ingress class for your invoices, which contains additional settings and, along with the ingresscast, you can specify controller specific settings using ingress class parameters we introduced so inside english parameters.

D

Administrator can set up additional restrictions on the the aob settings, such as whether the aob should be public or not, and also sends like ingress grouping or ip address types. Also for ingress. We will add support for the new path type added from kubernetes 118, and we also offer a feature to simplify the excess of reduction configuration for ingresses, which customer can use a single occasion to configure the aob to redirect all http traffic to https.

D

That's what the new features coming for. It is better controller and hand over to me.

A

Okay, so yeah with that wraps up our presentation on the cloud provider aws.

A

So we have some links here. You can see the cloud fighter, aws repository load, away, load, bouncer, controller repository. um The first link is to the documentation for uh the cloud controller manager and any questions that you have. We would be happy to answer either now or ons. You can always find us on kubernetes slack. We have our slack usernames linked down below.

A

So thank you very much appreciate you attending the talk you.