Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Continuous Integration at Scale on Kubernetes [B] - Karthik Gajjala, eBay

Description

Continuous Integration at Scale on Kubernetes [B] - Karthik Gajjala, eBay

eBay has a large community of developers working on several thousand applications at any time. To improve developer productivity, we offer Continuous Integration As A Service (CIAAS). This system provides capability to build and test several thousand applications concurrently. This talk will walk the users through our journey of building this system on top of Kubernetes, the challenges
we faced, optimizations we deployed and the scale and reliability we achieved at scale of tens of thousands of builds a day. We plan to continue our journey to leverage public clouds and we want to share our thoughts and initial plans.

About Karthik Gajjala
Karthik Gajjala is a Director of Cloud Engineering at eBay responsible for eBay’s private Cloud that includes Infrastructure As A Service and Platform As A Service. He has been a technologist for close to two decades working in Startups and large enterprises. At eBay, his organization built and operates one of the largest installations of OpenStack Cloud on the planet that powers all of eBay. His organization provides platform services to all of eBay’s developers to improve their productivity through automation, control and transparency.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

My name is Kartika jolla I work in the cloud engineering team at eBay. Our team is responsible for all of infrastructure, service and platform as a service at eBay.

B

So I my name is Joe Kahn I come from Shanghai China, so I'm. Currently a cloud engineer at eBay and recently working.

C

On the next generation continue lies a platform between based on kubernetes at eBay.

A

Today, we're going to walk you through our journey and building a continuous integration platform on top of kubernetes, we'll cover, just in a nutshell, an introduction to our infrastructure and our applications. What they do at eBay- and you know, walk you through our journey in building this platform. Shogun will go more into the details. The technology details will cover what we've learned. What worked, what did not work, how we fix things and then like we will look into going further with this powerful hope.

A

All of you should know. Ebay we've been around for some time and hope. Most of you had an experience either buying or selling on eBay we're a global company we're one of the largest marketplaces online. We have 168 million buyers around the world record early. We do roughly twenty two billion dollars in gross merchandise volume. There's the total amount of transactions that happen in the marketplace about half of it comes through mobile devices and more than half is international. So we have quite a bit of customers across the globe.

A

Eighty-Eight percent of our sales are actually fixed price and more than 80 percent are new items that are what are sold on eBay at any given point in time. You're looking at more than a billion items for sale on eBay.

A

So a business of this size requires a very large infrastructure as well. We started our journey in building a private cloud about six years ago. We have a private cloud that does everything with containers, virtual machines, bare metals. All aspects of the data center are fairly automated. We manage a fleet of about 80,000 bare metals, more than 200,000 virtual machines and roughly five petabytes of managed storage today that our cloud platform manages. So what you see on ebay.com is a composition of about 4,000 different applications that are serving more than 100 billion requests a day.

A

Our internal cloud platform hosts more than 95 percent of the traffic that comes into ebay.com.

A

The applications that are written on ebay are very diverse in nature. We have a very good choice of programming languages, our development teams use which our you know traditionally in Java, but a lot more, a node and Scala and more recently with Co. These development teams are continuously changing their code, releasing their changes to production and the continuous integration is a very critical piece of health. So the platform that we're going to talk more in detail about that today does roughly 10,000 bills a day over 300,000 a month.

A

So few years ago, the continuous integration journey at eBay has been that there's a self-service way to get virtual machines and every team would set up their own Jenkins instance. On that virtual machine. They would find their own way to get the right set of plugins configurations and all that right that only went on for some time. Then we've realized that things are getting out of control and a few years ago we started a journey where we built a missus based continuous integration platform.

A

We have two separate missus clusters for masters and slaves and we have a path layer that provisions a Jenkins instance on these missiles and users could get that self service. We deployed a shared file system, so things could be persisted the configurations to the file system.

A

About a year ago, we made a strategic shift towards kubernetes. That's when we started the journey of rewriting the system to run on on Cuban Edison, thanks to Carlos for his plug in with mrs. plankin. No, he said night here, I attended your talk this morning very good, so we build.

A

We run today, our continuous innovation completely on Cuban. It is with Jenkins, we run two separate pools of nodes for masters and the Builder pods.

A

We have separate images for both Masters and the slaves we use service. We use ingress, we use traffic as an implementation for English controller inside Cuban IDs. We use persistent volume claims the construct of Cuban. It is to persist the configurations for masters that itself is backed by self.

A

It's a wonderful managed to storage at eBay. We have petabytes of it that forms the back end for the PVCs. It's been pretty highly scalable and resilient for us I. Let Shogun go more into the technical details of the architecture.

C

Thank you, okay, so, in the next few minutes, I will show you our C as a service platform in deep on how we build our C a service platform based on kubernetes.

C

Let's take a look at the architecture first, but before that, before go deep into architecture, please be noticed that the big blue rectangle is no kubernetes cluster. The light blue boxes are kubernetes part. The red line in the diagram shows C as a service platform control plan, while the blue line in the right represents the data plan.

C

Okay, so, let's start from a control plane. If a user want a CI, he or she can easily post this third party resource CI config specification, either through pass platform at eBay or against the kuba API server directly. To tell kubernetes cluster that hey I want a CI, and here is my definition. There is a CI controller running in the cluster which watches the CI config object and a convert. The user requirement use a definition for CI Nuria CR instance.

C

Here's their works, CI controller do so first away it will create in a processing volume which will be used as a Jenkins configuration storage and then see our controller will spin up on a Jenkins master part mount with this volume and we leverage replication controller to provide high availability for the Jenkins instance before beside that see. Our controller also create a service for this Jenkins master, as well as the ingress object which wrote the request to this service, so the blue line.

C

When there's a build happens, the Builder request actually first touch the ingress endpoint of C as a service platform, actually there's no ingress controller provided by kubernetes, but we find that traffic ingress controller is amazing, which can meet our requirement of area traffic. Ingress controller will help to wrote the Builder request to specific service, and eventually the dink is master. Pha zinc is master. We leverage the Jenkins kubernetes plugin.

C

Thank you so much and talked to a cube api server just being up jenkins sleep pod to do the actual build. This is the whole architecture of our system.

C

Before going on, let's think about a question so why we need a third party resource, CI config, with a controller loop to spin up a Jenkins master.

C

Some people who are familiar with the kubernetes will see that that is amazing, because that is model driven, but is model driven. The real reason we use it. I would see no model. Children is only a methodology, but not a solution. We should always think from a user requirement perspective. So, let's, let's think about it, what a user really want in most of the cases, what a user want is a CI on a build while not a heavy Jenkins right. So a CI is nothing but a process from source code to manifest.

C

It's not not equal to Jenkins. Jenkins is only one and will build angels.

C

So, let's look at the CI config model in detail, how we model the process from sales code to manifest. We use third party resource in criminalists. We reduced a new kind. We call it CI config. The most important part here is the specification.

C

If a user are using git as their source repository, he can. He need to specify the key to UI in the South. Section strategy means what kind of agent he you use. If you're using Jenkins Jenkins, you need to specify the Jenkins master image with a volume size. There can be more and more other build engines besides Jenkins in builders section.

C

Currently, we support two major bills: standard build and generic build as Karthik just to mention several Misco. They are four major tax tax in eBay that is Java, node.js, Scala and Python as a platform service. What we provided to our user is the most popular cases that is Java and an OGS which covers almost 8 percent of the cases at eBay. We call it standard, build user does not need to provide their own builder image. They just claim their stack, for example, even for others such as Python, the scholar and some very specific build requirements.

C

We dedicate the manageability to our user so that they can their builder image and clay. You see as a service configuration so from CI controller side. After user post CI config into kuba kuba API server. The controller will do the real provisioning of the CI instance, which is PVC pod surveys, the ingress adjusted mission.

C

We also support hibernation. Why? Because Pern our experience in our journey, we found that over 60% of the CIS they are not actively used some CIS the create from the first day, but no use actually because of different reasons. So as a platform, we provide a platform we want to make a high utilization of our computer knows what we do is hibernation by specifying hibernate flag in the CI config specification.

C

Ci controller will tear down the mat support while live persisting Wally, because there is data in position, volume and also service can be deleted, but I will leave ingress. What's the reason, I will talk more about hibernation in England following slots.

C

This is what the build really really happens. Well, there's a builder request: initialized, either by github or a human being, jinkies master will use kubernetes puppy to spin up Jenkins live pod. To do the build jenkins. Kubernetes plugin is nothing but a plugin which managed the Jenkins live part man, the lifecycle on it. The plug-in will compose a Jenkins select part specification. According to the part, template configuration and opposed to API server, then there's a jinkins live pot up and a tooth build.

C

There are a bunch of specification details you can configure in the plug-in configuration actually, but here I just want to list three most important ones. We think here the first one is darker image: it's the Builder image. We recommend our user to compose all the build dependencies, such as libraries and packages into a same builder image. A builder image should be self-contained which can be used in CI or shared by different development. Environment 10, the Toleration.

C

If you want your builder to tolerated, Tunis specify the nose volume and such as a secret, for example, the user might want to push their build manifest to protected repository, which needs credential, I'm sure that they don't want to share the credential with you Ray. They just need to store into kubernetes secret and mounted secret into their build upon hast Pathmark. So why hast Pathmark I just mentioned that it is a best practice to compose all the Tao of the Builder dependencies into darker builder image.

C

But if your project depends on too many dependencies, your builder image can be very huge, maybe 10 gigabytes. So it will take a long time for kubernetes to bring up your sleeve pod because it need to pull image.

C

So some people would say that dependency can be downloaded during build a time or one time such as maven, but it still needs time. How could we solve this problem? We mount a specific host pass from our kubernetes node to the city builder, as the Maven cache, which means on this node. Only the first builder needs to download the maven dependencies and it can be reused in the proceed build in this way. We save a lot of time.

C

Okay, ingress actually without ingress, there's a lot of ways to expose your Jenkins master service to to the user. But, for example, we use a load, balancer whip or node port, but we have met the following two major problems when we design C as a service platform, the first one, every CI instance needs a durable access point for a user.

C

If we use a little balancer, that would be a huge waste of a whip, as you know that the load balancer is really expensive and there's only one part, Jenkins master part in each whip. It's a huge waste right. By leveraging ingress, we can use the air server routing so that we only need a one whip for the ingress and put the second one an ebay is. It is not allowed to use wildcard si or SSA or certificate.

C

It's not secure enough, and it's really hard to issue in a specific SSL certificate for each CI in short term they by enabling ingress. Actually we only need a one certificate for the ingress. End point here is example: CIA s. Corp ebay.com is our ingress and point for C as a service platform.

C

Slash CIA is the CI instance that is air seven past. So when user visit CIA, the requester will go to load, balancer first and then reach traffic and ingress and ingress controller here, TRS, just a terminating an ingress controller will do our similar routine.

C

According to the mapping CIA with a specific service we created for the Jenkins master and then Justin wrote the request to that Jenkins master in cluster using cluster IP.

C

Hibernation as I mentioned, that around 60% of the CIS are not actively used, for example, not used in recent 30 days. So what do we do? Is hibernation.

C

The CI controller will delete the service and a part to release the resource, but leave PV and ingress there. It's very easy to understand that why we live a peewee because there's a Jenkins master data configuration in the precision volume, but why we lived ingress, hibernator CI does not mean with delete a CI. Maybe user want to reuse it after 30 days, so we configure the ingress the else ever routine from the existing service to a reactive service.

C

That means, if the user revisited their CI, the request will be handled by the reactive service, the reactive service, doing a very simple job. It just changed the hibernate flag in the CI config specification and then see our controller will wake up this CI by recreating the pinkest master Potter in seconds and want to the existing volume and a recreate service and configure the ingress rotating from re creation service to that new service which point to the real Jenkins master pod.

C

So in this way, in several seconds, user can get their CI pack without any more click.

C

Ok, this is all about the details of our system, say as a service platform and next I will hand over to Karthik for the learnings and roadmap in the future. Thanks.

A

Evan so few things you know just a reiteration of what we covered in detail, I think the CI configures an abstraction. Is it's a very helpful thing for us. If users want to use something other than Jenkins, they can use the same abstraction from a platform perspective, we run two separate node pools for both masters and builders on the builders is where we use the host Pat mount to leverage the maven caches, so the belts happened very quickly. What other things that we take very seriously at a base developer productivity.

A

So we want to make sure that the pills are occurring as fast as they can and whatever tuning that we need to do to make them faster, be investing to that. The pod templates is a very powerful concept. I think we saw much more detail today in cloud base session about how to use power templates. We've also built an internal monitoring system for this. We roughly have about 2,000 Jenkins masters at any given time on our platform and we've noticed some issues.

A

Sometimes traffic does not catch the creation of new CIS, so we continuously monitor all our CIS to make sure that they are available and if they are not, we look into why they are not.

A

Last thing is, you know: we've been using the persistent volumes for CIS and the volume attachment attachment has been pretty. We went to some issues in terms of reliability and we've introduced fixes into the kubernetes layer as well as under the cube, and it is cloud layer to make that much more robust.

A

Some of the things that we are actively looking into right now is today we use all of our CI platforms on one Cuban. It is cluster in one availability zone in our private cloud. We are finding ways to see if we can span across multiple clusters so based on where the resources are available, we can actually schedule the workloads across multiple clusters.

A

Some of the challenges that we've noticed with pvcs is something that I keep bothering us, so we are looking into maybe if object stole, is a good semantics that we can use for persistence replying with Swift wave, and we have a huge Swift in our private cloud today, few petabytes of surfed, so we're looking at not the other thing is the declarative pipelines. We don't use much of Jenkins pipelines, features that we saw today, but we're looking more into it.

A

Then one of the things that we are also looking to use these leveraged public clouds- we've done a prototype this year to see if we can do out bells on public clouds seamlessly that's an interesting area for us to, but into next year right. It's all. We have and we are actively hiding if you are interested to work at a platform of our scale. You know please reach out to me. My email address is right there. Thank you.