GitHub Universe 2015 - Deploy, 4 Feb 2016

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: CI in the World of MicroServices - GitHub Universe 2015

Description

Surya Gaddipati explores some strategies for setting up smooth, modern and scalable CI using software like Kubernetes, Docker, Mesos, and DotCi, to enable smooth development in the brave new world of microservices.

About GitHub Universe:
Great software is more than code. GitHub Universe serves as a showcase for how people work together to solve the hard problems of developing software.

For more information on GitHub Universe, check the website:
http://githubuniverse.com

A

I'm going to be talking about CI for micro-services it's a couple of years ago. This is how Groupon dot-com looked. I was a web dawa web dev on Groupon dot-com, and we grew pretty rapidly. We went from like a smallish to be on Rails application to.

A

Can you hear me right.

A

I'm just going to hold it like this, so we went from like a smallish Ruby on Rails application to a monolithic Ruby on Rails application and Orsi I set up looked like this. Just a single Jenkins master and our tests were divided among like a bunch of slaves, so nothing special.

A

So fast forward a couple of years- and this is how our stack looks now- we have obviously gone service-oriented, so we use all kinds of tools and technology stacks, which is great for individual teams. But it's hard for someone who is like maintaining CI.

A

So right now we have around 400 jobs and we run like 18 thousand bills per week, so we were able to scale or see our infrastructure without actually scaling.

A

So we were able to scale our CI without actually scaling our team or infrastructure.

A

So my name is Surya Gaddy party and I work I, build release tools at Groupon and.

A

So I'm hoping to talk about how we were able to scale our CI infrastructure without actually scaling the team I'm, hoping that this talk would also touch on some of the aspects of github. Github is not just pull requests and code hosting github comes with, like amazing set of API send tools, so we will see how we can use those in CI.

A

So traditionally, CI has always been set up as an afterthought like people, either teams don't have the experience, the motivation or the time to set up CI properly. They throw a jenkins on some server and call it done it works initially, but after a while, as the project grows, it tends to become slow and flaky, and that tends to have like a strange effect on the morale of the team. So I have personally known people who have like quit their jobs because of bad CI. So.

A

So yeah bad bad CI, bad tools lead to bad culture and bad culture leads to bad tools. So it's like a vicious cycle. So I'm mentioning this because to stress the importance of us setting up a good CI experience.

A

So this is how CI has looked. Traditionally, there would be like a CI sarà, Jenkins, typically, which would pull SEM and schedule jobs on these like carefully managed slaves that are like configured for a particular build.

A

So that leads to poor derricks perience people are not sure how to go about setting up their CI. They often depend on like teams like admin teams which manage Jenkins. So this leads to like lower motivation, / P for people to set up their CI- and this is this- becomes all the more important in SOA world where people are constantly creating and destroying destroying jobs. So this talk is about self hosting CI if hosted C eyes like Travis CI works for you great, but this talk is going to about CI, which you are hosting yourself.

A

So through my experience over the last couple of years, managing CI I was able to come up with things required for perfect CI setup. I used those as like a mission statement for our CI setup.

A

So these are the three things it should be easy to use. Scalable and secure like pretty generic stuff right, like all the web applications need that. So, let's see what it means in the context of a CI, so ease of use, creating a job should be a push-button process. For example, if you must have seen Travis CI Travis CI is was an inspiration for me personally, so creating a job should be push-button as as soon as the dev push pushes a button.

A

Ci system should take care of creating SSH keys, allocating the slaves configuring the job.

A

So, second step in ease-of-use, it's clean execution environments.

A

Often what happens is multiple bills share a single CI server and they tend to flake out the bills. So the goal here should be to give the illusion that each build is running on a brand-new machine. That's not shared with anything else. The.

A

Third step in ease of use is we use ghe at Groupon, so it should be tightly integrated with github. By that I mean that it should. When you create a job, it should create the deploy keys. It should have access to the private repos, it should have security and it should be configured to according to the permissions of the job.

A

So, as I was saying initially CI like when the project starts, CI just works fine, but as the project grows, the CI time tends to grow linearly. As the tests are added so deaf should we allow to paralyze that it has pretty easily it shouldn't, be an arduous process.

A

I'll just ignore that so job can version control, job configurations, so typically job configurations have been managed in a CI system, so I want CI CI configuration to get the same advantages of a version control code, for example. If someone's experimenting with with job configuration, he shouldn't he should be able to do it on their own private band branch without breaking the bill for everyone else, and the last thing I think github.

A

Most people think that devs don't care about beautiful and intuitive UI eyes and get her was a great example of a code hosting system who's. For me, the initial attraction was the clean, clean UI. So CI system should be clean and beautiful.

A

So the leading UI system, CI systems now like Jenkins, the UI for Jenkins, was designed like a decade ago like in 2000s, and it looks and feels like it was designed in mm.

A

So metrics CI system is typically used for all sorts of things we use it for like building deployments, so dev should have easy access to the CI metrics like how long that bells are taking. What parts of bells are taking longer, how long their deployments are taking and it should be accessible to the dev. So they can like optimize and paralyze their bells.

A

And the last thing ci should be extensible. Obviously so Jenkins was a great example of an extensible CI system. So that's my pic of CI servers because it did them I go off now.

A

So that's a pic of my CI servers. It's extremely extensible. It becomes all the more important for a CI system because, as I said, people use CI to do all kinds of stuff. They like run cron jobs. There are now build systems deploy systems, so it's extremely important for CI system to be extensible.

A

Ok, so in the first slide, I mentioned how CI was traditionally set up. Each slave was carefully managed and tailored to a specific built-in devops world. There is a phase for this. They call it pets versus cattle pets. Are the machines you carefully manage the demand, constant attention and cattle is something that each unit is replaceable. So if one of the cows is sick, you can replace it so yeah I like that analogy, a lot- and it was an excuse for me to put my pictures of cats.

A

So those are my three cats by the way.

A

So this thing is specific to github authorization. This ties into security. We we run lots of jobs that require PCI and Sox compliance.

A

The CI system should obviously be secure and it should tightly integrate with github authorization you have already defined, who has access to what and that should translate over to github I mean CI system, for example, if you have configured who has access to push code that should translate to who has access to configure your build, and the last thing is github. Private repo support, so github provides like amazing tools for a ROM private repos like it lets you create your own deploy keys, so CI system should be responsible for managing that.

A

So those are my high-level wishlist items. So, as I said, I used these as a mission statement for managing our CI system at Groupon.

A

So, let's look at, we have looked at all the problems and what we are aiming for. Let's look at the solutions that are out there and how we can like put them together to create a great CI experience. So this is a tool called dot CI. That's it's something we created at Groupon, it's open source. It's at Groupon, slash, dot, CI in a nutshell. What it does is basically takes Jenkins and turns it into Travis CI, it's more than that. But in a nutshell, that's what it does.

A

So what dot CI does it acts as a glue between three major systems that we use for CI, darker, github and Jenkins I'm, going to talk more about how its glues, all of them together.

A

So one of the first things from my slide was a push button, job creation. So this is the UI for dots to create a new job. So when you create a new job, what it does is goes and sets up the github.

A

Stores your OAuth keys and if your repos private, creates deploy, keys and stores them in its own internal database.

A

So no manual environment setup, usually the process at the places I used to work was you would contact us as admin and they would set up a CI machine for you. That was a tedious process and that would often discourage does from creating CI bills. We used to do a custom solution for this, but docker came onto the scene and it was like a match made in heaven for CI. So.

A

Yeah I'm going to talk about docker for the next half an hour. No just rank good, so docker is a declarative syntax for defining your servers, and it obviously goes well with CI context and docker. Compose ena is another great tool. It lets you define so CI environment is usually not just one build server, it's usually build machine and the corresponding services, like like my sequel for Ruby, on Rails or radius or yeah, one of those so docker compose lets. You declare your CI dependencies in a nice and declarative format.

A

So clean environments on demand, what by this I mean see us each building on a CI server, should assume that it's getting a brand-new and clean environment. It shouldn't have shared state, shared ports or stuff left over from previous builds, and this is a really important step for avoiding flaky bills.

A

So Jenkins ha comes with many many many solutions for this, and one of this is doctored plugin. So basically it lets. You define point your Jenkins instance to your darker setup. We usually point it to a darker swarm installation, but you can just point it to like a regular darker installation.

A

So what this does is when a bill comes in it spins up a brand new darker connection, docker container for your bill.

A

So this is another configuration screen for darker plugin. As you can see, you can like define an image that you want to use as a Jenkins slave, so you can like customize your Jenkins slaves to add, like your company specific policies, and it also comes with options to like specify volumes which is used which you can use to like define caches for your build.

A

So that was just one example. There are other examples you can use. Kubernetes, measles and Amazon comes with a brand new container service and Jenkins has plugins too, like all of these services. So if you don't want to like manage your own slave farm, you can like just point it to one of these instances and have your bells run in the cloud.

A

So version control job configurations. As I said, people should be free to experiment with their bill in their own private branch, without breaking the bill for everyone else. So dart CI lets you define a file called out CI ml, much like Travis llamo, and it has all. It basically defines your build configuration in Yama file, which you can check in to your repo and when a bill comes in, it checks out the corresponding dot CI Emma for the particular shop, so people can experiment with their own private branches.

A

So easy parallelization. This is another important things to have for your CI.

A

So this is an example for from dot CI paralyzing. Your test is as simple as specifying three keys in the run section. So when a bill comes in in this example, it would paralyze it into three different groups: test cubes and integration.

A

So we have looked at all the solutions to these individual problems and, let's see I want to present like a small example of how we can like bring all these individual tools together to make good CI experience. So this is an example.

A

From Groupon, this is a ruby gem that we use internally, and we want to test it against against four versions of Ruby and we want to test them in parallel. Obviously so so in this example, each of the environment that we want to build, the gem against is defined in one of these target files. So all the versions that we want to test against are different in these darker files.

A

So this is an example of one of those darker files. So this is an example of Ruby 1/9.

A

So the second step as I was saying you need dr. Campos file to define your test groups, so this is an example for the darker compose file in that repo. Here you can see all the bills have been defined in four different sections: for example, a Niners point towards the docket file for Ruby 1/9 and the build means just build the darker file, so we have defined four different sections and the last piece of puzzle is the dot C IML file, which brings all these things together. So.

A

Here you can see, we have defined four keys and each of these key corresponds to the darker compose file, so ruby 190 in the docker compose file.

A

You can also override the command in the docket file. Here we are doing something special for Ruby to two bills, so this is the end result. This is how it looks when a bill comes in, it was paralyzed for four different versions of Ruby yeah.

A

Okay, the arrows are missing in this slide because lost in the conversion. So in a nutshell, when a bill comes in, it comes to github I mean when a dev checks in he makes a commit to github github fires, a web hook to Jenkins Jenkins spins up these five different docker containers runs the bills reports back the status to github yeah. As I said, the arrows are missing, so use your own imagination.

A

So it reports back the committe status. I was surprised at how, like so many people didn't know about commit status. I was not going to put this in my slide and I realized that so many people have no idea that this even exists.

A

So metrics um this is a slide from dot. C I'd lets you visualize your dog your bill times for these various branches. You can define your own custom metrics. This is an example of how your code coverage was throughout your bills and you can like filter it by lines, packages and files.

A

So github authorization. You can turn you can go to Jing. Once you have installed dot SIA, you can go to the configuration screen of dot, CI and just point it to github other ization. What this would do is map map the permissions you have defined in your github repo to the to the bill. For example, if you have commits to commit access to the build, you would get configured access to the build.

A

So yeah this is the final step. I want my CI system to be extensible. What this means is. You should be able to extend individual parts of your CI experience, and this is how you you can script your dot, C IML here you can see that you might want to run integration tests for if you have defined a step called integration.

A

This dot C IML file is basically a groovy template that get parsed, so you can create for loops and if statements so here you can see you want to notify your HipChat room only if the branch was mastered. So you can do all sorts of cool stuff here. So.

A

Dot SIA is extensible, so Jenkins in itself is extensible, so you can extend Jenkins by writing. Plugins and dot. Ci is a plugin for Jenkins, but you can write plugin for the plug-in so without actually having to contribute code back to dots. Here you can define your own own plugins so here this is an example of adding a new notifier. So if you are using slack or Basecamp or whatever it, you can write your own custom extension and that this automatically becomes a key in the notification section of dot CI.

A

You can also create your own.

A

Own plugins, so this is an example of plugin called code reviewer. What this does is if there is a pull request, it goes through the change set from the payload and.

A

Looks at who committed the most to those files and assigns a code reviewer automatically.

A

You can also do stuff like make code comments in the pull request. If you can hook into the cobertura coverage and get coverage information and make code comments to the github pull request and because you already are authorized to github you, you can use the same worth keys to make comments. So you don't have to setup a special service to do that.

A

This is another example of a plugin that integrates with github deployment, API I, don't know if you have seen the deployments API from github. It's I think they released it like couple of months ago, but maybe it was last year. So what this does is listens to deployment requests and executes builds in a predefined, docker container.

A

So, as I was saying you come, you can define your own custom metrics. It just goes to the metrics section of the UI, so people do all kinds of stuff with CI, so you can create a metric for how often your deploy is happening.

A

So let me summarize what I have been saying for like last 20 minutes, the key takeaways is use docker for CI. It's great Jenkins has like many plugins to to make that easy to use. Yeah just use docker so take advantage of github API, so soget hooks post bunch of information in payloads, for example, chain sets the pusher information, the emails of the pushers, so don't just drop the payload information make the best use out of it.

A

Yeah use commit statuses, so if you're using github flow, I think you should use github flow. Commit statuses are great for telling you, if you can merge your pull request, use, deploy keys. This is another thing from github api. It hub provides an option to create deploy keys for individual repos.

A

So if you have like a PCI compliant repo, for example, you can your CI system can create our deploy, key and use that for checking out old use authorizations API from github, don't duplicate the information you that you have defined and github in your Jenkins use the deployment API it's it's a new API from github that that you can use to track your deployments last thing. This is thing really underused. Feature of github is pull requests comments. Useful requests comments. You can do all sorts of things for pull requests.

A

You can hook up your CI to make comments about the pull request, lines that are missing code coverage or not all failing like a code standard.

A

Yeah so use Jenkins cloud. Api's Jenkins has like many plugins that you can simply point to any of the any of the cloud infrastructures like you can point it to kubernetes group cloud from Google or Amazon's container service or digital ocean service. So if you don't want to maintain your own own slaves, you can simply point it to one of the clouds so yeah. That's the end of the talk, I'm planning to do a demo of some of the stuff I've been talking about.

A

My name is Surya Gaddy party and that's the URL of the dot CI plugin for Jenkins, and my boss asked me to put that on the slide. We're hiring.

A

So I'm hoping to do a quick demo of the dot CI plugin.

A

So here I've configured my dot CI plugin I've created an application in github and pointed it to.

A

Pointed it to my Jenkins instance, you simply have to like it's just a two-step process. You create an application in github and simply enter the keys here and and setting up here is a push-button process. So once you click the button, it.

A

Sets up the gate, hogs and the corresponding deploy key.

A

Yeah, that's all you had to do now. If you like, make a change, let's make a change to one of these files.

A

So that should fire agate hook so the gate hook came in and what Jenkins is doing right now is spinning up a darker instance locally, so the meter those so if I do docker PS. So here you can see.

A

Jenkins spun up our darker slave on demand.

A

So it spun up a docker slave and it's running the build on the docker container instance that just spun up so here it's using the deploy key that it added to the database I think the network flicked out. But you get the idea.

A

Yeah, that was their demo. Thank you.