Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2016 (Seattle), 16 Nov 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Imperative to Declarative Infrastructure by For Developers and Operators by Ashwin Raveendran, eBay

Description

Imperative to Declarative Infrastructure - For Developers and Operators - Ashwin Raveendran, eBay

In this talk we present details of eBays journey from a imperatively orchestrated infrastructure to a fully declarative infrastructure model, exposed both to the developers and the operators.

About Ashwin Raveendran
Ashwin Raveendran, is a senior member with the cloud engineering group at eBay. Over his tenure of 5 plus years at eBay, he has led the design and development of multiple generations of cloud computing efforts starting with the creation of a large scale homegrown cloud platform and cloud services. He was also responsible for the operationalization of Openstack at eBay including the architecture for the Software Defined Networking among others. He is currently leading the initiative in the creation of a cloud native ecosystem at eBay with Kubernetes as the main backing technology.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So so I think, let's get started. My name is Ashwin Raveendran I'm, a part of the ebay cloud, engineering group and I'm here to talk about how we manage our Cuban Ruiz clusters declaratively. So.

A

Before, we jump in, let's understand, like know why eBay chose to use given a DS and what are the some of the you know, main reasons why we stand behind. You know the Cuban. It is a cluster management solution. So, first and foremost, like you know, we, like the simplicity aspect of cuban at ease. There are a bunch of these primitives which cuban it is brings in the idea of like the parts and replicas sets. These are like pretty awesome representations.

A

The abstractions which a developer can easily understand so think of these as patterns for distributed computing, which you know if more, the people who know the more is your newer solutions can be built on top of these things, so that simplicity of, like understanding how your application is laid out, is the number one thing which cuban Indies brings to the eBay ecosystem. For us.

A

The second part is like flexibility, so while it is very simple, it also allows you know flexibility in that a developer can take his application, runtime decompose it into one or many units, and you know put it as part of a part, and then you run these parts like in a distributed manner. So though it is very simple, it's also extremely flexible and obviously leg.

A

You know something like Cuban, it is, is very efficient because it has a dynamic scheduler in which powers like how the workloads get portioned onto these different nodes and for operators like us who are operating these Cuban Andes clusters at large scale.

A

It is really really flexible, so we have bent and adapted Cuban, at least to match most of the operational challenges we have, and this talk like we will cover some of the challenges and how we have done that and the the last reason and I think the kind of the foremost reasons is the open source community cube has like an extremely awesome community, like we have had like nothing but good things to say about them, so Jimbo straighten to the declarative models like what we are basing on for writing our abstractions.

A

So this starts with understanding a little bit about a little bit about how cube itself is like you know, are designed to be so. This is roughly like. You know how we see the Cuban. It is machinery, so you have these API server, which is in the center of like you know the universe for Humanities, so think of that as a distributed state store which allows like, in this global view or the global store for all your API objects.

A

So whatever your objects are beter like a pod or, like you know, any other specs which Cuba needs supports, ends up in this API server like that is what this diagram shows like a user here is actually declaring a spec. A spec is like nothing, but, like you know, all the cube objects are specs. You have a part spec, you are a replica spec. You have like in a scaling expect all these fixed gets declared on any on the API server.

A

So API server has the liquor storage, which is a distributed state store and in the default implementation it's an H CD back store.

A

There are a bunch of controllers which are nothing but regular application code which understands basic HTTP primitives. They talk to the API server and they can actually pull out the objects which the user has represented into the API server. In addition to the regular crud operations, which are rest, api is normally support. The cube api server also support an additional one, which is called a watch feature.

A

So one of these controllers can let the api server knows about its intention of wanting to know about updates to certain events which are happening on the api server and api server would push these events or updates back to the controllers so, and these controllers are also just regular application binaries, which you deploy just as regular applications cube ecosystem. So you can have this complex, modeling done and the controller's written all using the Cuban IDs framework itself so to understand how it works.

A

Like you know, let's look at this picture, so this actually explains how we see an eBay developer goes on with this job of, like you know, writing an application and getting it deployed onto a Cuban in this cluster. So, like everything like any other project, you obviously start your journey by writing some code into github. So that is like the the first step here you create like a project in github and then you go and create a booth back a bill.

A

Spec encodes, the information about which repository you have your code and where do you want these artifacts to be built and published into it? Normally is like a darker repository or any other repository, which holds the binary data for the application, but the build spec is a spec which embodies that information, and you have what is called a build controller which is constantly watching your the bill spec. So a developer comes- and you say, is that a this is like the four repos I want, my my code.

A

These are in the four different repos and I want you to actually build. If there is a change in the repo, it's just nothing like nothing, different, it's just Jenkins and others liking it. They do the same thing, but we can abstract that idea of building into this bill. Spec and the controllers can actually spin off like a Jenkins to do the actual job, but for developer, it's just like a single place where it declares this bill spec. So the controller like in this case the build controller they keep.

A

If it has, the smarts can actually like kick off. Another builds another spec in this case the deployment spec to take a compile and build binary and deploy it into the Kuban. It is cluster. So the thing which embodies that is the deployment spec. This is a stock given resub, jecht and obviously there's a deployment controller which comes as part of the cube machinery which watches on the deployment objects. The deployment basically says a you have a set of deployed parts, and this is a new version of the containers which I want to deploy.

A

On top of that, the deployments controller. It looks at this information and it actually like powers. It actually creates like a bunch of replication, specs or, like other additional objects, to deploy your application, so that again is watched by the replication controller, which indeed, in turn, goes and creates a pot spec which is watched by the scheduler. The scheduler sees, like you, know, there's a new part which is come up and these are all objects by the way.

A

All these specs are just specs in the API server, and all these controllers are watching the API server and they see additional information getting created in the case of the part spec, when the replication controller creates a part spec. There is no information about the the actual node, where the part has to be run. That information is filled in by the scheduler, which is also another controller.

A

So it puts it figures like where it has to be running and it creates an entry into the pot spec and finally, like there are the the node, the controllers which are running on the nodes, this cubelet and the queue proxy they watch on the part spec, and they realize that they have to be running this particular load, and then they really they run it. They download the binaries from the cube, the repos, the docker, repos and a part runs on the node.

A

So this is how cuban artists deals with you, know, specs and like how do you, how you create your own controllers to deal with various in e scenarios? So we have taken the same approach and like in a we are wanting to use the same approach with kubernetes takes to manage your applications to manage these clusters. Themself.

A

To understand that part, a little bit I have to just tell you like how you know the things are laid out at eBay from an infrastructure side, so we have a bunch of regions where we operate our data centers in they are spirit. You know, for geographical reasons like if they're spread you closer to the customers and stuff like that and within the each within each regions, you have one or many available reasons.

A

These are fault unit fall domain units where you recommend applications to be in different availability zones for them to be highly available in light of any global failures or like in local failures. Sorry, so each of these availability zones are powered by one or many providers. So in our case we have OpenStack.

A

We have legacy like a bunch of like IAS primitives, which are powering all this availability zones, so our cluster deployment leverages the underlying infrastructure for that, so we have like availability zones like like I mentioned earlier, and it in each of these availability zones. We have this network segments which are created for placing the cluster zone. So in this example, we are talking about.

A

Like you know, in one availability zone it has like two two zones like we have a prods on and a def zone there, segmented through some kind of Sdn or other network segmentation mechanism. Most most of the times. We use software itself to segment those. So the one example to think about that is like if you are in Amazon. This is the V PC constructs which you're talking about so each car is in a single V PC.

A

In our case, this is like a we prominently, use OpenStack in our cloud, so this will be a set of routers and the network segments which OpenStack exposes each of our clusters like have the full set of those things.

A

And at some point in time like we haven't done it yet we will have all the the Federation API server which connects all these clusters into one single unit. So that is like you know how you want to go so to managing, to manage all these different things. We have built a component which we finally call it cube master. The test master so test is like the internal code name, which we have for our the personal management solution and master, obviously because it's managing a bunch of these clusters all across the globe.

A

So it has a bunch of responsibilities like you know, and each of them are sufficiently compact in its own merits and we are going to talk about some of these in the later sessions. So, let's talk about, like you know as an operator, what are the major areas where we have to deal with, starting with networking, which is, you know, probably predominantly one of the complex aspects of like our infrastructure, so the challenges we face in this area are like the following like in. Do we have multiple networking models?

A

We have a flat layer, 3 model which we dominantly, like you know, run our production, and you know the the real business critical workloads on. We also have, like you know, a variety of overlay based Sdn solutions which are there which runs like you know, not so critical application, no workloads like the dev workloads and other experimentation workloads.

A

We have this legacy data centers XZ infrastructure, so we, whatever solution we come up with, has to operate and interoperate with that, and we also have this requirement that every single networking layer which we come up with has to share the single global private IP space. So the last one, the global private IP space itself causes you know some issues for managing the Kuban IDs. The first thing is like in a Cuban, this assumes a private, a suitor block per node.

A

If you look at the default implementation of humanities, you say, oh and again it has to have a network IP block per host. Well, in our case, we have, as I mentioned earlier, we have like one large IP space for everyone and then that is not going to work. In our case, because we easily will run out of the IPS which we have and to prove and Cuban is such a flexible framework, we were able to actually abstract out the idea of, like IP blocks per host, two IP blocks per whatever.

A

We are calling in an arbitrary network scope, so we created this new object, called Network scope and we assign IP addressing to that scopes and all the nodes in our systems get attached to this network scope. So that is a how we have planned this particular problem of not having to have IP blocks per host. We also wrote an eye pam, an IP allocation management system on top of that, so our IP allocation management system also leverages the cube machinery, and it works somewhat like this, so the cluster administrator creates these network scopes.

A

These network scopes are basically objects which are living in the API server space. Just like any other cube of you know, objects, themself are and each of the nodes which approach into the the clustered polls get associated with this network scopes and our I Pam actually is also another controller. A default Cuban DS controller, which runs in the Cuban. It is a cluster.

A

What it does is like when scheduler assigns, like a part to a particular node, our ipam controller kicks in and figures out, like you know where that node is going to land in and it allow a locates an IP based on the location pool which is tied to the node. So this helps us, like you know, not have any requirements to pre-allocate IP addresses to a particular node. We can have a large pool of IP addresses, however, we want and then we can assign the IPS from that block to a particular node of our choice.

A

So that is like the networking challenge. Just like partly the you know, quite a bit of like more complexities. There I don't want to go into all the details, but we can talk about getting up to the talk. The other set of challenges we have are the cluster management challenges itself. How do we set up like these clusters? As I mentioned earlier? We have many available reasons. Many network zones, many data centers like- and we have many many many clusters.

A

We are setting up, so it's a chore to like actually manage all the state across like all these clusters, because, as the SU would know, each of these clusters would come up with its own cluster management components and that's own version of like in software which, as which it has to run, there's a lot of bits which are moving. There are add-ons, there's a configuration state and on the top of it, do you have like the Federated state which is coming up. So all these things have to be managed.

A

Obviously this is a space where a lot of tools exist. You have chefs puppet ansible, but none of them, like you know it gives us like the the declarative primitives to operate them like cuba. Nettie's does so we decided to, like you know, build as part of our test master. We decided to build an object model which roughly looks somewhat like this, so we have an object which is called cluster, which represents the the the needs of a cluster. For example, what is the the domain which the cluster exposes?

A

So we numerically named our clusters, so we would have like in a cluster number 100. So then the hundred dot. You know the Tesla Rio. That is like our internal core name. That will be the name which we would want to create for the clusters and each clusters would have certain storage class requirements, certeyn resource, like in capabilities. So all those information about that gets encoded into the cluster, not just that it also like tells the operator can declare which provider it needs to work with or what is the region it has to work with.

A

So you specify all these details into the cluster object and there's a second set of objects which are called compute node pools. So these are, if you would like the flavors in Amazon or, like you know, flavors in OpenStack, where you say oh I need to like get a bare-metal servers which are of this size. It's got like you know, sixty cores or, like you know, finder GB of RAM and there's a set of these nodes which I want to be part of this cluster.

A

There could be multiple such things that are part of our cluster, but you represent them using the compute node Falls and obviously it has got corresponding aspects of the the specification of the machines they have. The placement, selectors and other aspects which allows us to get the right machines to add into the cluster. We also have the OS image spec, which determines the operating system, which you want to lay down on this thing. So with all those things, this is lovely like how the the classification flow would look.

A

In our scenario, the adminstrator create a cluster spec across the spec, as I mentioned is not just one. It's a bunch of those objects which I showed it short earlier. So the post, this cuts a spec into a mini cube or which is like something which runs locally on your laptop or it could actually use an existing cluster which is already running, but as you'd notice, both of them have the test master control. A controller running in both places.

A

The orange in that these are the controllers which are watching for the existing extents of this faster specs. So the atmosphere like you know proteins is bought this specs onto the clusters.

A

Let's assume that it's a mini cube which is locally onto your box, so that the the controller component gets a hold of the the expectation which the user has, and it actually knows which provider to contact, because all the information encoded into the cluster and it brings up a cluster based on all this respect, which it has and not just that, once it is provision, it actually moves a controller over to the new cluster, and it also moves all the specs into the new cluster and then, at that point of time the controller will start watching the specs which it is managing in the same cluster where it is running.

A

So this is what is called a self hosting. You know you host all the the components which are required for managing the Cuban is cluster bezel for stood along with, given is itself so like that is like the so we just talked about the cluster set up like complexities. So after that, like that, the number one complexity for us is like the hosts runtime itself. These are the operating system bits which we are talking about.

A

So if you, if you ask things, have progressed over this years, the complexities have been moving more and more into the host which you are dealing with so now, like a host has to deal with a lot on lot of, like you know things, for example the cubelet and the queue proxy two interesting things they both run on the host. You have the networking plugins, which are running on the host. We have to manage the keys and other security aspects.

A

We have local storage, we have queue, isolation, logging, monitoring, Damons, a ton of stuff happens on the horse and its you know. Managing these host images and stuff like stuff are like in really really complex and in also like, when you think about how frequently we have to upgrade each of those components, it becomes like a big effort in itself. So our solution to that is like to use a minimalistic OS like we use fedora atomic fedora, but this is roughly how it happens.

A

We have an open, oh, is repo mirror and we take a snapshot of that. You know so that we can freeze them in time and out of this frozen time, snapshots like we create what are called the oyster Erie post. So os3 repost is like a fedora atomic fedora. You know construct which allows you to package your operating system binary, and you know exactly like all the packages and their respective versions, and it's like signed image. So you know what is in the images you are dealing with.

A

What we do like once you create this, create this image is also to push it into container registry, so the entire operating system bits and the repository for that are available as the container and with this particular image, we actually use a hash e Corp Packer tool to create like a base image for our cloud, which is OpenStack. So you start with the known good set of an operating system.

A

You create a binary which is specific to our cloud and the push that into the OpenStack lands and once the compute nodes, all the new compute nodes which are going to provision will use that particular image for getting the, as we add more nodes to the cluster. That is what we use, but what about the existing ones? The existing applications, the company, its existing operating systems, which you have on the cluster, will actually talk to the OS tree repo, which is also running as a container in our cluster, to get the latest updates.

A

So you just restart them using your deployment construct and then all of them lagina gets up to the latest date. So that flow goes like that an administrator goes and create a deployment spec. A deployment spec will in turn goes and creates and changes the node spec and each node spec has the corresponding node, which it is watching.

A

So when you change the operating system on one of the snowed, the node controller does the right things on the back to contact, though you know the machine and get it patched up with the latest version of the operating system, which we want it to have well, in our case, like the operating system has like most of the binaries pre-baked in, for example, docker is pre-baked in and most of the bits are already pre-baked into that image.

A

Our policy is like to run pretty much everything into that in that mode and every components which you want to run on the top of that runs as regular parts or you know in rare cases, docker containers, so nothing changes like your host operating system.

A

So, in addition to these are like the couple of three areas like in a networking. The cluster management and the host operating system ends. When are the three areas where we have defined models for us to operate these clusters, but we also have application modeling challenges, like you know, eBay application developers are, you know they use a set of known internal models, so when they move to the Cuban it is kind of mode. They also have to deal with the existing internal tools. So we have.

A

We are in the process of writing these internal models to declare the aspects of application. We have a well understood notion internally of what an application means. So we have encoded all those things as specs, so a user would use the cube, CTL command and use create application and you get an application and he declares what all he wants inside that and our controllers, like we'll, go realize that information in all the respective systems, with an ebay to make sure that you know it's all good for the user to go also like in.

A

If we have internal models, we write, we have a state singer which actually is another controller which sings to state from API server, and you know populates our internal data store for all the operational aspects.

A

We also had to write a lot of at Mission Control respects, so there was a leg. You know when you operate a site like eBay. There are lots of controls which we have had to put in the past. All those things like has to translate down, because we are letting users directly deal with Cuban IDs.

A

So all those controls we are using the admission controls, that's a feature which, given it is, has where on can write a pluggable solution which allows us to you like in a control like, for example, we will only allow the image if it is scanned for any vulnerabilities and if it is signed by a QA for running in production. So all those kind of situation we handled through admission control specs.

A

So this is another aspect which you are currently implementing, so the ingress becky's like a spec which Kuban it is has but like if you have looked at it like there is a default implementation for GK. Obviously we don't run. We don't have all those things internally, so we implement our own spec using our own internal tools like so OpenStack, if you're using OpenStack that elbe service will be Elbaz. So we have variety of those will be services.

A

We implement English controller to like have our own DNS, our own L load balancers, and we run all the else as regular parts. But an extension of this actually is like you know, to do the entire thing we don't. We are planning to get to this model where we don't even need any physical Road bounces, but you know, do the whole thing using the regular come already hardware using like you know, Cuban Aries, we're all there for traffic direction and the l7 complexities are all handled as Cupid in these constructs.

A

And if you look at the the cue proxy and other things, there's they, they are like almost there with some minimum changes. I think we are. We are hoping that we can get to this model where we don't have to have physical Hardware, physical load, balancers and other bits in our data center.

A

And yeah, in summary, so for us, like cube, rocks like it's one of the awesome pieces of software out there we can, you know, bend and you know change whatever. We need behind the scenes to make it operational in our situation. The good part is like, even after all those things, the experience which the developer gets is like pristine. It's the same experience. They should get on GCE that they can expect. You know even internally to us, so the API missionary, which it provides.

A

It's just pretty awesome like you know you can write your own specs. You can write your own controllers and you don't even have to use cuban at ease. It does not even have to be cuban at ease. We are using it for many other purposes. So my co-workers, who these like you know, going to talk about how we are using the cube api machinery to build complex. You know policy, languages and policy enforcement engines, but that's nothing to do with cuban. It is like you know it. It exists, it can exist outside of cuban.

A

It is so it's pretty powerful framework in itself. So you know please check it out and we are also like you know seriously, considering like open sourcing. All the tools which you have built like over this next few months will be like actively working on trying to see if we can open source it and see. If we can have a set of end users would love to use the same tools as us, and then you know go from there. So that's pretty much it like in caissons, yep.

A

So even has like a set of like existing plot scale, monitoring solution, but for certain small clusters we also use Prometheus, you know only for the Kuban is there because for the application like there are like you know, you know so many other complexities and scale is like a big issue. So we leverage existing solutions for that.

A

Yeah, so we have an existing interface so for all the predominant line and good flows like you know, which are the eBay flows, there's a good, a pass layer which we have so we will continue to have those kind of layers, but when you have applications which does not fit like in the keynote, as I said, there are lots of applications which does not fit the paradigm of pass.

A

So for those like, we are going to expose like the the cube, CTL command line itself, but not just the standard cube seal, but with souped up with all those additional objects and stuff like that, if you're an operator you get like all the cluster objects, compute node pools and all the other things. So if you're an application developer, you get application and other objects which you yeah. We are writing, or on CLI salsa.

B

A

So the the clusters vary from anywhere from hundred physical servers to you know 200 300, so our plans are to accelerate you like in thousands of servers. We are just in the early stages of that, so our typical sizes are like 5 K to 10 K, that's like size of this clusters, which you are looking to get to, but right now it's like 200 300, something like that.

A

So it's not based on so like each availability zone. We typically love to run a single cube clusters, so our availability zones are, like you know: let's fight 5, to 10k physical servers. So you know, if cube scales and everything scales we would love to have that, and that is that size is determined by like you know how much you wanna invest in one availability zone because more the computing put, it could run more application, but the failure domain is going to be bigger. So we have to be judicious about that doesn't answer.

A

A

Yeah yeah yeah, so if you're the so, if you look at the the new project for called K or PS, cops from Cuba notice, they also like started doing that. You know the similar approaches of doing you know creating classes based on a declared respect, it's no different than the cube parts or anything else right. You declare a part and cube realizes the part for you. In our case, you declare in a cluster and the node poles, and it actually goes and creates these. You know bits behind the scenes.

A

So right now like there are like bunch of clusters, so that is what Federation is going to solve right you and all this versus into Federation, and then you specify like what you know envy you want to run it as a label or some other way or like you can.

A

Even we are thinking about right writing like the admission controllers like which default these values, for example, if you know that Custer is earmarked for you know one particular purpose, then every object which gets created will automatically get annotated with that value, and then you know it takes out of it eventually, like anybody want to run all these different environments in a single cluster, but I don't think we are there yet. So that was like a ways to go, but that is the goal. Ultimate goal is to get there.

B

A

So if you delete the cluster object, it just gets really did it goes to the decom for flow like you know, but a cluster itself is not significant. It alone. I mean enough right. So there are other bits which makes up the cluster, for example the the different type of compute node pools, which makes up the Dominions. So you could have like, let's say, a GPU compute node cluster, so another one which is like earmark for storage. You know these are all compute node pools and then you have like a large pool of like general.

A

You know computing cost, you know pools, so they are all part of like the same cluster and they can independently be flexible and flex the out. So, for example, I want to like say, add another five or fifty nodes into that. You know you change the node pool spec to say that you expect 50 more and that will drive the back in opposition to create more compute nodes, run all the software's which are on in needed for running the cubelet and other things and then join it to the cluster.

A

You know this is only for operators, those are operating models like you know, and it's a back protector.

A

So Federation, so all the Cuban ADIZ clusters come with its own control plane. If you would so all the HCD and all the master components they're all in each cluster has its own copy of that, but obviously like. If you have tens of clusters, like user, would have to know like a lot of details, for example, application developer, say said: run this application like in the three regions right.

A

You could actually go and look up all the clusters independently and then go deploy your application in each one of them or, ideally, you would go to a single place and say we run this application. So that is where the Federation is coming in.

A

All your addition is a Cuban. It is feature so like with 1.5. It's going to be like in almost full fledge.

A

Okay cool: let's take this! No more kitchens.

A

The operating system you mean.

B

A

Yeah, so it's a time ago, it's like right now, there's no hard dependency on any of those things, so that you can write a new controller which you know understand and different operating systems upgrade procedures. The specs does not change that much. It's just like.how. So that's the beauty of it right. The spec. If you are designing it properly like, then you don't have your the only thing. You are changing your the controllers themselves, not the spec, so you can yeah. Okay,.

A

So we we have like a as the insulation which you use internally and then we typically like knit into favor the flat layer, 3 Network,.

B

A

Namespaces like so, we ought to create namespace. So when you log in based on your record inches like the authorization token, which you have, we figure out like the the your association to some groups and you create namespaces automatically for you. You can also create your own as many number of namespaces here. So.

A

Well, in its maturity curve right, so you know we need to be comfortable at some point in time to run both because you don't want the developer, taking up all the space, and you know preemption and other things are not yet there. So once the preemption and other features are there, we would want to, like you know, run everything in one single cluster. That's a go.

B

A