Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Simplified Experience Of Building Cluster API Provider In... - Sahithi Ayloo & Arun Krishnakumar

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Simplified Experience Of Building Cluster API Provider In Multitenant Cloud - Sahithi Ayloo & Arun Krishnakumar, VMware

Speakers: Arun M. Krishnakumar, Sahithi Ayloo
Cluster life cycle management is a challenging task and Cluster API promises to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. With the growing adoption of CAPI in recent times, are you looking into writing your own Cluster API provider for your infrastructure (or) cloud with a unique set of requirements and constraints? We got you covered. This talk will walk you through our journey as a Cluster API provider and all the lessons learned the hard way.

- Building blocks to implement Cluster API provider, and bare essentials like CSI & CPI.
- What are the common patterns around developing and debugging workflows?
- How to enable multi-version API support via webhooks?
- How to address common problems like multi-tenancy, and user quota management in a strong multitenant cloud environment with Enterprise customers?
- How to leverage CAPI in building Kubernetes as a Service layer on your clouds.

Come and learn from the maintainers of a Cluster API infrastructure provider - "our journey around moving from handcrafted Kubernetes life cycle management to Cluster API based life cycle management in a multitenant cloud".

A

Good afternoon, everyone Welcome to our talk. My name is sahity. Ailu here is my colleague Arun Krishna Kumar. Both of us are Engineers from VMware. Today we are going yeah. Today we are going to share our experience um on building cluster API infrastructure provider for a multi-tenant cloud platform.

A

We will also be talking about few of the challenges that we have faced along the way Lessons Learned and discoveries made around few of the problems that we have identified, that one could see in their environments and also um era design patterns around cluster API usage in multi-tenant Cloud platform.

A

Lastly, we'll be covering on how we have actually built kubernetes as a service layer with the underlying technology of cluster API, with that, let's get started um so agenda for us for the first half of the talk I will be covering on cluster API internals and will also be um giving you some resource resources on how to get started with the implementation for the second half Arun, we'll be covering on design patterns around cluster API usage in the context of multi-tenant cloud environment and, lastly, we'll be covering on kubernetes as a service layer on multi-tenon cloud.

A

Okay, before getting into the details of cluster API I would like all of us to have a common understanding on what multi-tenant cloud is.

A

So Cloud basically delivers infrastructure as a service to its tenants in terms of compute storage networking, while providing strict isolation to its tenants and also the security. So then, who are the tenants here? Tenants could be an end individual user, but in this particular case, in our Cloud platform, This multi-tenant Cloud platform, which is VMware Cloud director, uh that's our product's name. So the tenant is an organization.

A

It's an Enterprise level company with group of users and there can exist multiple organizations within this Cloud platform and these users can request for kubernetes clusters, and this is the solution that we have built at Large, that is a kubernetes as a service engine on top of multi-tenant Cloud platform with the underlying technology of cluster API.

A

Okay. So what is cluster API I'm gonna quickly, Breeze through this Slide? The cluster API is a kubernetes project to bring declarative, kubernetes style apis for cluster creation, configuration and management. So the idea here is that so end users would run these commands, which are Cube cuttle traditional familiar commands to create the cluster on an existing cluster. So this existing cluster with cluster API components installed, is what is called management cluster and the children clusters that are created as a result are called workload clusters.

A

So that's the difference between management and workload, cluster, so management cluster is cluster with core copy components installed, and this is where API definitions of workload clusters are also defined and workload clusters are intended to host the modern workloads.

A

Okay, so that's the end user's point of view on what clustered API is now, let's see the developer's point of view on what cluster API is so in simpler terms, cluster cluster API is comprised of one main component, that is core copy provider and three plugable components: infrastructure provider, bootstrap provider and control plane provider.

A

So all these providers have their own set of responsibilities to adhere uh to meet and a few orchestration rules to adhere which are dictated by the core Cappy provider.

A

So at high level, the infrastructure provider responsibilities to create the necessary infrastructure required for the cluster on the chosen Cloud environment, and this is what we are interested in for this talk and bootstrap providers. Responsibility is to generate a script that can convert any machine into a kubernetes, either control, plane or worker node control, plane providers. Responsibility is to manage control, plane, nodes and deal with upgrades Etc, okay, so the so then.

A

The idea here is that user would come in apply a cluster manifest file on this management cluster and the resultant workload clusters get created on the chosen infrastructure provider. So for this talk, uh so this capacity is what we have built. Cluster API provided for VMware Cloud, director and vcd is the acronym that stands for our multi-tenant Cloud platform. That is VMware Cloud director here is the sample copy, manifest file um so cluster. So basically it's a hierarchical structure of API objects and the cluster is at the root.

A

It's the root element of the other objects and it holds owner refs to a few other uh objects associated with other other providers. Note these uh vcd cluster and vcd machine template custom resources.

A

These are associated with our infrastructure provider, that's cap, BCD and you will be actually replacing those with uh whatever the infrastructure uh CRS that you'll be coming up for your infrastructure provider, so the sample copy managers say basically says: I want a kubernetes cluster with one control plane and one worker, one worker node, with so-and-so settings on so on so plus cloud provider.

A

Let's go a bit deeper now to see how cluster API works. What enables smooth interplay of core Cappy and all the other providers that we have just talked about number one: hierarchy of API objects and number two cluster API contract, so um hierarchy of API objects. The diagram that you heard seeing here is the pictorial representation of what we have just seen in the previous slide.

A

The cluster API manifest file and everything. That's in blue here are custom resources associated with core copy and other providers, and that are in green are associated with the infrastructure provider, which is of our interest, and when you apply that yaml file, this is how the resultant API object. Hierarchy is gonna, look like, and all of these resources are being watched by their Associated controllers to bring so they are basically doing continuous reconciliation attempts to bring the their current state to the desired state. So similarly we have these.

A

Oh sorry, so we have these infra cluster and infra machine objects that are created as a result, and these are also be uh are supposed to be watched by the respective controllers, like infra, cluster and infra machine controllers, and this is what uh we are supposed to build as part of the cluster infrastructure provider. So these crds and their Associated controllers at the minimum is what would make the infrastructure provider okay. So now that we have a fair understanding of what hierarchy of API object is going to look like, let's understand, cluster API contract.

A

So, as I've mentioned before, all these controllers from the providers are responsible uh to do certain things and all of these need to adhere to certain orchestration rules dictated by the main component. That is core copy, and these controllers are supposed to interact with each other by variables called well-known fields and let's take an example of how cluster controller and how, in infra cluster controller interact with each other in the sequence diagram.

A

So both of these controllers are watching um watching their Associated CR custom resources here. So the cluster controller is the first one to act here. It sets the owner ref on the infra cluster.

A

So basically it says I kind of own you that an infra cluster controller.

A

This job from that point onwards, is to create the basic infrastructure for the cluster creation to proceed further, like it can create a load balancer, a networking setup, that's unique to your own cloud environments, and it also need to ensure that control plan endpoint is either generator or specified by the user. So once the control plane endpoint is generated, it sets itself to as ready and then cluster controller would consume, that control, plane, endpoint and marks itself ready and it generates the cube con Cube configs secret.

A

So that end users can access begin to access that cluster. Okay, so information controller is mainly responsible for creating nodes. So in the previous slide, what we have seen is infra cluster controller. That was the one that created the basic infrastructure necessary for all of these controllers to proceed further and information controller main job is to create the necessary infrastructure and, at the same time, bootstrap controller generates the necessary script to convert these machines into kubernetes nodes.

A

So bootstrap controller generates the bootstrap script and stores it in a well-known fail in a data secret that is to be consumed by Machine controller Machine controller kind of copies it to an another field which is to be read by information, controller and information controller at this point Provisions the necessary infrastructure using that bootstrap secret.

A

Basically, it takes out the cloud in its kit from that secret and runs it to convert the machine into a kubernetes node, either control plane or worker node node, and if it's a control, plane node, you would also see another controller in the picture that is kcp controller.

A

Okay, so once that is all done, information controller marks itself ready and then Machine controller also marks itself ready and it waits for the node to join the cluster.

A

So the bottom line here is that again, the crd is associated with these infra cluster machine and Associated controllers would what make up the infrastructure provider, and this is what we need to implement as cluster API infrastructure provider. Okay. So now that we have a fair understanding on how internals work, let's get started with the implementation and implementation should become relatively easy and it should all the understanding on the cluster API internals should also help you to debug and troubleshoot.

A

When things do not go as expected, we have used Cube, Builder command, Cube Builder to actually um create the project layout and the scaffolding queue. Builder is a framework that generates um apis kubernetes apis via custom resource definitions, and it will also generate lot of boilerplate code for you. You can just jump in and implement the business logic.

A

Okay. So now, let's assume that we have built the infrastructure provider for your for your own cloud environments now so now, how do we get these infrastructure provider implemented on the clock? Kubernetes cluster? So basically we need to set up the management cluster and we can achieve that by cluster cuttle tool that helps setting up the management cluster and generating cluster manifest files.

A

So this is the command that you need to run for your infrastructure. Vcd is our platform, so basically, this command pulls the content from your GitHub repo and installs those components on the management cluster.

A

So what this also means that you will have to update or edit cluster cuttle code to include your infrastructure provider as list in as part of the huge list of other providers that cluster cutter currently supports, and these are cluster cutter generate commands to generate the sample. Cluster API manifests, which you can run on the management cluster, to create workload clusters and next. Okay. So now we have this management cluster, fully ready with all the components installed. User would come in and create the workload cluster.

A

Now are these workload clusters ready not yet so ready as in ready to host modern applications? Not yet so we need cni to enable container communication CPI to set provider id on the nodes. Note that CPI is kind of a mandatory requirement um from core copy, so it expects cloud provider input interface to be installed on your workload clusters.

A

So so we need to have this cni and CPI installed on the workload cluster to be called fully ready and CSI is to enable state stateful deployments for persistent volumes, and we use CRS cluster resources definitions for installing these components.

A

So if you're planning to uh build cluster API infrastructure provider, you should also plan to implement cloud provider interface for your Cloud environment, okay, so admission controllers and multiversion API.

A

So now that we have the basic implementation of cluster API infrastructure provider, you can make it more robust by implementing these admission controllers, defaulting and validating web hooks.

A

um Basically, they let you write custom code to either set some default values on the resources and validate before the data is persisted in the hcd database. A next multi-version API support. It's a big Topic in itself for this talk, I'm just going to go over the need and the few resources to get started.

A

um So at some point, you'll have to think about bumping up API version and what this means is. It becomes a necessity on the infrastructure provider to be backward compatible with older, API versions again. So when user uh requests for a older API version, kubernetes API server is supposed to return the object in that API version.

A

However, your stored version could be much ahead and Cube API server needs to do the necessary conversions between the desired version and the stored version, and these conversions need to go into conversion web hooks so that Cube API server can actually call them uh to do the necessary conversions.

A

So you can create the scaffolding for these web hooks, also using Cube Builder, and we have used the same so few few of the lessons learned. So the docker provider is an excellent starting point to read through and modify the code, which is what we have actually used uh in the beginning to familiarize ourselves with the infrastructure provider implementation, it's very simple and bootstrap controller. It generates a cloud in its script in ginger template.

A

um We had to do some tinkering to adjust it to our needs and load. Balancers are kind of a first class component of the infrastructure and I have been already talked about CPI and uh do remember to um oh: where is it um yeah for this yeah? Here?

A

uh Sorry, sorry, this uh thing needs to go in here, so we need do. We need to remember to set the cloud provider as external for the cubelet configuration um yeah and.

A

So, lastly, on this page, so these two labels take do take note of these labels. These need to be set on your crds. This is an important step. This basically tells cluster a core component core copy component to use which API version of your infrastructure provider, and it becomes even more important to set these two Fields uh when you have multi version API support in your infrastructure ready.

A

So we actually hit this issue where we had this multiversion API support ready and we some we kind of uh forgot to set these labels and our newer API version resources are somehow getting reset to the older content. So we had no clue why, thanks to Cluster cluster API folks who helped us debug and it's a simple change, but this is very important thing that you need to remember.

A

Okay, lastly, uh Auto scaling kind of comes for free with the cluster API. All you have to do is download this and run this command with cluster API as a cloud provider and set some annotations on the machine deployment object. So there are more references to um here and your clusters are Auto scalable.

A

So with that I'll hand it over to Arun.

A

B

Hi come on thanks, sahiti, so uh yeah, let's uh move from here. So as you see there is a there are lots of gotchas uh the cluster API documentation is one, but, as she mentioned, the labels and so on. Are uh it's tough to find them in the documentation. The documentation is like a 350 page book if you print it out and the labels are mentioned in some particular cases, and it is tough to debug these as well, but recluster API Community is very supportive and we made use of their health quite a bit.

B

So now, let's reiterate on revisit how the VMware, how the cloud provider multi-tenancy looks like how a public I mean how in general, any multi tendency looks like and how VMware Cloud director fits into that model. So uh in VMware Cloud director we follow the principle of the Google private Cloud equivalent, though the other mechanism is also possible, you can have a set of organizations, organ and or two and so on. So does this right so you, for example, Suppose. There are two organizations argon and R2.

B

The cloud is partitioned into these organizations in terms of compute, network storage and so on. So each of them is like a set of virtual data centers from the cloud they are carved out and each org has their own set of data centers and their own set of resources. Now the IDP is also carved out in the sense that uh Oregon users, they do not really know the existence of R2, for example, and org one administrators also do not know the identity of how to there is an Uber cloud provider who is at the top?

B

Who can potentially see everything, but the identity I mean the IDP is unique across all of the orgs. Now we, this is a very rich multi-tenant system. Now we wanted to figure out what our goals were for VMware Cloud director and what cluster API brings the uh brings forward, and how do we marry the two pretty much so on our goals are that tenant users or organization users?

B

They should be able to create cluster API in a self-service manner, so a self-service manner is started because we will talk about that uh in the in a little bit. Self-Service is very important for us. The second part is that to bring out the features of our Cloud platform, VMware Cloud, director into kubernetes clusters, so we have very strong user isolation and quota systems and roles and rights, whereas kubernetes has its own our back system.

B

The quota system of kubernetes is has a lot to be desired, I mean it has one authentication mechanism and it has a separate auth Z mechanism and the two don't talk to each other, pretty much so I mean and um yeah. We are left wanting there, so we wanted to be able to actually represent our user here in the kubernetes cluster. So the third part is the same thing about representation. So we want to ensure that our tenant users can be represented using their own idps in the kubernetes Clusters.

B

So they must be able to authenticate. They must be able to create a cluster, but within the cluster they must be able to show their identity. And the fourth thing is we wanted to administer policies on the user from the cloud side, but this should also flow into kubernetes operations. So if a user has access to create only say 10 VMS, we want to ensure that the nodes, the recubernetes nodes when they are Auto scaling, they shouldn't scale Beyond 10.

B

They should stop at 10, and that should be a policy which is enforced by the cloud and that should automatically be used by the Clusters so on. Then we actually came up with a set of questions. Now. um uh How do we actually first of all satisfy the network requirement, so one hidden thing of cluster API- or it is potentially evident now- is that the management and workload clusters all need to talk to each other.

B

All of the time there needs to be a network connectivity, whereas you saw that the orgs were disjoint with respect to network in the other case. So how do you actually uh have a network connectivity between the management cluster and our workload cluster? Maybe you cannot start off with one Uber management cluster.

B

The second thing is: who creates the management cluster and manages it so potentially there must be somebody who does the life cycle of the management cluster and keeps it secure if the management clusters, for example, handles a thousand workload clusters right, there's also a skew of versions between the management and workload clusters. So at some point, if you want to upgrade the workload cluster, you will also have to upgrade the management cluster. So there is some amount of management task and the administrator of this cluster needs to be kubernetes savvy.

B

So that is on the cluster management side. On the user management side, how do the users create workload clusters in a self-service manner, so the user needs to know that there is a management cluster and they need to go and ask for some access to it and do it. So it's not very self-service. Basically, there has to be somebody who is on the management cluster side. Who is helping them out? How do we enforce the tenant boundaries on the user side?

B

Basically, how do we represent the user, and so that is the next two things and fundamentally, how do we audit the user actions on the cloud side? The user may go and do some kubernetes actions, but they have to be audited with respect to their own user ID. So to solve the user aspect, we made the user a first class citizen inside kubernetes, so what the user would do is they would be able to ask for a token from the cloud and get a refresh token and embed it as a secret in their token.

B

So what I am talking about from now on is sort of patterns in multitarian clusters which exist in other providers and other systems, but they are not really documented or there is no clear acceptance of those as a pattern. So, basically we are talking about what we did and these seem to be the common patterns which are being used nowadays. So one of them is a representation of the user using their secrets, I mean in AWS. It is mounting the secrets and so on.

B

ah So that way, you can actually enforce the policies of the cloud director on the particular user, so the user would just embed their secret into the cluster which they create and they use the tokens in secrets. So we had to build a refresh token methodology in order to get this so because we didn't want to expose the user's credentials directly. ah That is like a token which can be revoked pretty much so. The other thing is that the network boundary we have seen that there can be multiple organizations.

B

uh As a result, there can be multiple management. I mean there needs to be a one management cluster per organization, and uh there has to be uh workload clusters connected to that particular org itself. So this wall is pretty much a network boundary and since uh management and workload clusters need to talk to each other, they have to be within the same networking space which is the same tenant. The user would essentially use their access token and do it.

B

However, this also needs a namespace level multi-tenancy, ah for which we take communication test yourself, and the reason is in the next slide. So, as you can see uh like sahity mentioned, the tenant can apply a kubernetes yaml to the management cluster and get a workload cluster. However, they should not be able to ah view the other clusters, so their access should not flow into this. But how do you manage that? Because the pretty much the tenant needs some accessory?

B

So here we are so, and the third part is that the management cluster should not have a long time access to the Clusters credentials. So we solved that using namespaces. So what happens is that each tenant gets their own namespace. Each tenant user gets their own namespace and they are able to access, only objects within that namespace and they can create workload clusters within that namespace. ah However, this still does not yeah. The other part is that they need to have short uh expiry.

B

I mean not refresh tokens with short expiry times and I'll come to that next, when we are doing a self-service, kubernetes or if you are doing this in the complete schedule. So this namespace pretty much gives exactly this tenant user access to uh on cluster objects and vcd cluster objects and so on a subset of objects. So now let us quickly look at how the workflow would be like.

B

So you have Suppose, there is a Alice management user and a tenant, Alice user, and this user wants to actually go and um ah Crea create a workload cluster. So how do they first of all discover that there is a wireless management cluster? So there is no Discovery and somehow we have to get to know of it from By Word of Mouth, but once this particular tenant gets the cluster uh gets to know of the cluster, then they actually figure out who's.

B

The owner of the cluster is the management cluster admin and they ask for a management cluster access. So this is a human operation which happens. This management cluster now creates a namespace for this user and gives them a particular Cube config, which is the way to access this particular cluster and now the tenant. So, as you can see, the management cluster admins work is done. They can just go out of the picture. Now. The Alice management cluster is available, the namespace is available, the user is still there and the cubeconfig is still there.

B

Now they want to create a cluster, so they will create a cluster with the vcd token embedded as a secret dbcd. Is our cloud provider refresh token this particular token, and they get a workload cluster created using that they can monitor the lifetime of this creation and they can see it getting created now again, so seeing it getting created is a bit of a tricky thing because they don't have access to capacity logs.

B

Okay, so it is not I mean we have to make the logs multi-tenant in a particular Way by which they can actually see. If there is an issue with the creation and then they can go ahead and create it now once that is created ah so as in they should not be able to see any other users logs. So that is where the logs also need to be multiplied at that particular point once the cluster is created, they want they want to actually get the cube config of that cluster.

B

The admin Cube coffee- and that is the means by which they own the cluster right, so they would issue a getcubeconfig, it's a small script which they can run, which would essentially uh take the secrets from the cluster and you get back this particular workload. Cluster and things are done there later when they want to update or upgrade this particular cluster. They again need to come to this particular management cluster and make those changes and their operations are satisfied.

B

So this is the whole set of operations and, as you can see, there are many things in this life cycle and it's not very self-service based right. You actually go and talk to a person and do many things. There are other things with the management cluster as well, which we need to talk about in a distributed system and that's the blast, radius and the security aspects of of it. So you have a management cluster which is handling multiple workload clusters, and you have multiple workload clusters where the users have actually stored their secrets, their tokens.

B

Now what happens if there is a network partition on the management cluster side? Now the vertical clusters cannot be managed at that point. You cannot upgrade or update them, so they are sort of waiting for the management cluster to come back up or the network system to be there or what happens if the management cluster is compromised in some way or becomes evil now all of the tokens are compromised, I mean even if they have short-lived tokens.

B

They cannot use the management cluster anymore in order to upgrade their clusters, so they are in a sort of a limbo State wherein they can. We can always revoke the tokens that is fine, but then you can't go and do anything to your cluster. You can just let the current workload, Runs workloads run even Auto. Scaling will not happen at that particular point. So ah in order to actually solve this issue, we actually came up with something I mean we actually began to use something called a self-managing cluster.

B

ah So this is an old concept which has existed from the beginning of cluster API. However, it has not used as a common pattern nowadays, however, this has helped us quite a bit, so you have the same system wherein you have the management cluster. You have capacity binaries and you have the workload cluster records, the crds, which I think I was mentioning in infra, cluster controller and so on, and uh the user can actually apply a resize and upgrade commands on this management cluster and things work.

B

So what we do is we and things operate on the virtual cluster. What we do is we actually install Cappy and capacity on the workload cluster and we run the cluster Kettle move command, so the move command can be namespace, so you can exactly move those records of this cluster into the workload cluster. All of the labeled objects which, as we mentioned those, would potentially move once these are moved. As you can see, this link between the management, cluster and the workload cluster is lost, and you can actually apply all of your comments on this.

B

So what we have done is, after the cluster has been made self-managing. So this sort of a cluster is called a self managing cluster. Once it becomes self-managing, you can just Auto scale it by just using its own record, so you can do a cube kit. You can change the number of nodes, for example worker nodes from three to five, and you can apply the yaml on this particular workload cluster. It will scale itself up.

B

Likewise, you can see it can scale down as well and you can also do a upgrade operation, so you can actually let it upgrade itself from one version to another. It's completely self. Managing the only caveat is for deletion for deletion. We need to take the help of another cluster and we have in a way around that- and we began to use this on self-service, kubernetes cluster. So now ah we want to have one SAS layer by which you can create a cluster in a networking space where you can't have access.

B

So there is a networking space somewhere which has 192.168 dot, something, for example, in 10 in Pepsi or 10 in group. You have a 192.168 network here now there is a particular kubernetes service layer and a Pepsi user. He cannot access the network of tenant Pepsi directly. He is on his laptop, but, however, he can access a API surface and they want to I mean he or she, and this user wants to be able to run a particular command by which they can actually create a cluster here.

B

So the way we do that is, we make use of the bootstrap cluster mechanism. So basically we create a VM on that. So by using the cloud providers, API I mean by using the cloud API we create a VM on that particular tenant and we actually create a small bootstrap cluster in that and then we create a workload cluster based on that and we move the objects we make it. Self-Managing I mean we install the Cappy capacity and we make it self-managing, and then we destroy the bootstrap cluster.

B

At this point, the Pepsi user can actually go and manage their workload. Clusters on I mean manage their cluster on their own, so there is no intermediate layer. There is no management cluster. There is no overhead of another person coming and administrating one cluster and there is no scaling of requests. So what happens if one management cluster is a handling? Ten thousand other workload clusters: how do you actually ensure that it scales up and works? You don't have any of those questions.

B

You have each Pepsi user I mean each user accessing their own kubernetes cluster and the kis kubernetes as a service layer at this point is just a very thin wrapper.

B

Ultimately came up with the agenda completed the I mean we have finished the implementation of that and we are going to release that as cse40. uh There are I mean other users can also come and create their own self-managing clusters. Now it's a extremely distributed as in its uh embarrassingly parallel right. So there is no Central Authority, which is actually trying to manage multiple clusters, but each cluster is managing itself. All of the requests are directly flowing to that and shout outs yeah thanks a lot.

B

Cappy Community for the help and the giant swarm and some of uh some other partners actually give us some requests and make us ah make capacity better and Implement features better. So there are some references here and a QR code, but if there are any questions uh we can take, so these are the common patterns. The reason why we have mentioned it is when you go to the community and ask for it, there is no fixed set saying that hey. This is the way to do it.

B

It is like some people have done it this way. Some people have done it that way, because- and so this presentation is a very commonly used pattern to do.

B

Any questions or comments.

B

Otherwise, we are sort of at time or Beyond Time.

B