Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Using Containers for Continuous Integration and Continuous Delivery [I] - Carlos Sanchez, CloudBees

Description

Using Containers for Continuous Integration and Continuous Delivery [I] - Carlos Sanchez, CloudBees

Building and testing is a great use case for containers, both due to the dynamic and isolation aspects, but it increases complexity when scaling to multiple nodes and clusters.

Jenkins is an example of an application that can take advantage of Kubernetes technology to run Continuous Integration and Continuous Delivery workloads. Jenkins and Kubernetes can be integrated to transparently use on demand containers to run build agents and jobs, and isolate job execution. It also supports CI/CD-as-code using Jenkins Pipelines and automated deployments to Kubernetes clusters. The presentation will allow a better understanding of how to use Jenkins on Kubernetes for container based, totally dynamic, large scale CI and CD.

About Carlos Sanchez
Carlos Sanchez specializes in software automation, from build tools to Continuous Delivery. He has spoken at several conferences around the world, including ApacheCON, JavaOne, Fosdem,... Involved in Open Source for more than ten years, he is the author of the Jenkins Kubernetes plugin and a member of the Apache Software Foundation amongst other open source groups, contributing to several projects, such as Jenkins, Apache Maven, or Puppet. He works at CloudBees scaling the Jenkins platform.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

My name is Carlos Sanchez and.

B

A

Talk to you about continuous integration, continuous delivery, all those cool things that you can do with containers. So a little bit about me I'm, an engineer at cloud ways: I work, a scaling, the Jenkins platform, I wrote the well I started the Jenkins kubernetes plugging over three years ago, more or less, when kubernetes was still not very popular and I am a big open source contributor at different foundations, big time Apache, Apache maven and the crease foundation, and now docker images puppet.

A

You name it and I'm also part of the Google development expert program, with the work I'm doing with with Vern Edison, and so so before we start. Ok, who is using kubernetes I'm, not gonna, ask who knows what we're nervous, but okay, keep your hands up, who is using it in production.

B

A

This is still sleeping at night.

A

Ok, who is using Jenkins Jesus, who is using Thank You Smith cover nervous I, can just stop and leave.

B

A

It's perfect: this is the perfect audience. So if I don't talk about something, you expected me to talk, please grab me later, I'll be around the whole week, I'll be at the booth booth starters. So if you find you need to talk to somebody about something, especially kubernetes and jenkins, you can you can find me around. That's where I'm here, I also have a cold since yesterday. So this is not my normal voice.

A

I think I cannot hear it myself very clearly bad all right, so I'm gonna talk a little bit about what we did and how we learn from this experience and covering a little bit of what we do at Clovis and also the the open sources stuff so I'll be very clear about well I hope to be very clear about: what's commercial, what's not open source, what's open source, so our main goal, a few years ago when I joined Club, it was to scale Jenkins, so traditional application and using containers.

A

So one of the reasons is like well everything.

B

A

That came to to me or to us asking well I. We want to run CI city in containers right well, because why not so that's like docker? They hold a hammer for everything. So basically we wanted to run isolated, Jenkins masters and agents and enforce like memories. If you limit so to make sure that people don't have used the system and run it at big scale.

A

So obviously this is not trivial that that's how they move. Containers are run in someplace in the Caribbean until some years ago, and sometimes is what we do resembles that, but so there's two ways to scale Jenkins you can have more agents per master or you can have more masters with their own agents.

A

So the solution about having more agents- that's there's plenty of plugins that you can use to dynamically scale, create agents on demand. It started years ago with the ec2 plugging your Dookie, just our VMs. Obviously, today with containers, you can just start containers, which is a lot faster. If that works for you and the problems is like masters still a single point of failure in Jenkins, you can have different versions of plugins different configurations.

A

So it's not really always useful to have one big master for everybody, especially if you have multiple teams working on different things like mobiles, you have a screen whatever there's a limit of how many ages you can attach, but I mean that's. That limit goes every time it goes higher and higher, and the other solution is that you can have more masters where you can have different teams difference of organizations using the same using different masters. So it's like a charting model where you can have yeah this.

A

These guys that develop job applications have their master with their own plugins and the wrong agents, and these guys that develop something else have have a different master. The purlins is like you lose like single sign-on or you lose the centralized operational configuration of your class. So what we build back in in the last two years, I was close with the incus Enterprise that had the best of both worlds. Using one thing that is called the Operations Center, where you can manage multiple masters and then you have dynamic creation in its master teaching.

A

Whatever one of these plugins that you can, they you can use, find in the open source community. So the first implementation for historical reasons, only was using masses and we use marathon terraform Parker. All these cool technologies and just recently was announced that we are launching this on kubernetes, so Jenkins and kubernetes. We can run both. We think the Masters and it in some kubernetes and the interesting part is especially for the masters.

A

Is the storage, how you handle the storage of the Masters, because they just use filesystem location to store files and output logs and configurations alone and agents. Typically, the meat is storage. So storage is one of the typical problems that people face using coronaries or distributed cluster container, and we use persistent volume claims, so you can plug the wrong implementation or you're persistent volumes using whatever back-end you want. So that's cool thing.

A

The problem side and I just had a dis quote from from one of the engineers like good Jenkins, has like the worst cases and you for Network basic storage, because it has a lot of rights and the small files and many files and so on. So when people try to run Jenkins in like EFS or or an NFS that has does not have a performant server on the backend.

A

They have all these issues, performance issues and also one thing that we had with miso because we built it- and we don't have today on coronaries yet is the multi availability song starts with we CVS, because kubernetes doesn't yet provide the other important power running jenkins at the scale is the networking you had a know that there's like the two ports that jenkins uses the HTTP and the jnlp agent, where the agents connect to so HTTP is typically web UI api's.

A

The NLP is just to connect the agents and we use send in xcode ingress controller, and we use ingress rules like the in coronaries. So basically, we can have multiple clusters running at the same time. This is in different host names and use pathways were raring to go into the operation center or go to into any of the masters you run and for jnlp.

A

So we have the agency that they start dynamically using the kubernetes blocking, and you can also touch agents manually because, let's say you want to have windows box attached a windows agent or do you want to have like a specialized machine? That has something that you cannot contain your eyes or you cannot run on the cloud.

A

So we for that, we expose these ports using node, port and coronaries, so it allows you to plug whatever agent you want and we have what the kubernetes plugin provides his agents with infinite scale for so, whatever definition of infinite you want to use. So it's only limited for the amount of resources you have in your cluster. It's all dynamic, its shading runs in a pub. It has a unique thing to that. Other plugins in Jenkins, don't have I, didn't provision or plugins, which is allowing groups of container spots and I'll, show you demos.

A

Now it also supports Jenkins pipeline who is using Jenkins pipeline today, all right Jesus. This is this. Is my dreams make you find the funders all boring right so far, so you can define your audience with a pipeline support. You can attach persistent workspaces that that's random. Whatever volumes you wanna touch and it's we ought to configure it by the by default.

A

So if you run your master in kubernetes, your agent configuration is already configured automatically to use the same namespace, the same credentials and all that so from now it's demo time- and you can ask me questions if, if you want.

B

You can see.

A

A

Let's, let's make sure I'm the right place.

B

A

Okay, so I'm watching the pods that are running, so this is an instance of operation center. If you use blue ocean that I'm not an ass because I'm sure everybody's using it already, this is like the new UI for Jenkins that doesn't look like the 90s, so.

A

Let's go back to the second I, have two two masters already running, so this is operation center. I can go click here and create a new master, but the interesting bit that we have is I can create what we call now. A team tour.

A

How the cool things? Yes! So what? What is this doing in the backend? This is creating a stateful set, creating the storage for the master. It's creating the making sure that there's one pot running is creating the surface screwing the increase rules, all that is handled by by operation center behind the scenes. So this is still part of plot with Jenkins enterprise and if I go to the logs.

A

So this is the the one I just started: I can click on the logs and I can see everything that has been happening, so the persistent volume claim it's getting bound. The path is created here. There should be yeah, it's creating the service, creating the ingress using the hostname. All that and.

A

Basically, I'm I can see from here what's happening to my master. How is it coming up, and these are all the cool things that kubernetes provides for free? Where I mean if the pot dies, if the null dies? It's all a high availability.

A

Yeah this is also running in the coronaries. Cluster is called CGC and 12 is doing this operation center say if I show you everything that is running in this namespace, because I can also install multiple clusters of this in the in different namespaces I have one stateful set for the operation center, one for one master and two for like three masters in total. This is the one that I just started, tim's store and you see that I get the service. I can get the ingress.

A

We also use roles and service accounts to limit the posture of privileges. So these are all the thing Grace's that we have sorry, maybe at the back, didn't see it. So at some point this is going to come up, but I'll show you in the other one.

A

Well, it already came up so I'm from there on you have your own Jenkins master, where you can install your plugins use it for your theme. So it's one one-click provisioning. Yes,.

A

This is running a kubernetes cluster or maybe a llama, so I used to CBS. We try with CFS with CFS. You have to have a very big star that you committed to have decent performance.

A

So let me switch to the other server. I have another master.

A

So these are automatically configure to use the kubernetes plugin for provisioning and I'll show you what the coroner is. Plugin can do.

A

Some sure and and now the one I'm showing you now, it's a it's a Jenkins open source running on google kubernetes engine. So this has the plugin configure. So it's it's. It will be the same in any of the masters, but I'll show you some examples right. So.

A

If I create a new item and I call it Q, Kong and I create a pipeline.

A

What I can do here is create my own pipeline.

A

Undefined, where I want this this built or this this job to run so you can define a pod template, I can say, name is kyouko and label is Q, come to and then I can say int in in in the nose with a label. Q come run, something SH and let's hostname. So you see the container hostname and sleep for a bit. So we can figure this out.

A

I will not run it. Let me switch context to.

A

Yes, sir, let's watch the pod start coming so there's a node that is has been create a pod that has been created, as this is running so I, just with a very simple syntax. I just got an agent dynamically provision, so one day was the job finishes. The aging is discarded, so there it is. This is the hostname and if I wanted to run this.

A

Multiple times this is basically limited to the to these resources. You have a inter coronaries cluster, there should be more agents coming up more pots and well that's running.

A

Let me edit and create the next pipe pod template pipeline, so what's happening here is that this is by default, creating a container inside the pod. That is basically the agent the remoting agent for Jenkins. So this container by default is an out buying image that just have shell and Java and barely and anything else and I cannot do a lot of useful things with it. So what I can do is I can create my own container image that has all the tools I want. It may have the docker client.

A

They may have Java JavaScript whatever, but that's a pain in the ass, because you have to like basically mix things from different dr. images or I can just compose apart with the tools I want, so you can see, there's more pods coming up and being created well as the jobs keep running.

A

So what I can say is, in this part you're going to create some containers.

A

And let's use a container template.

A

And let's call it moving because I'm sure everybody loves moving here or hated one or the other.

A

And the image is gonna, be maven Alpine and two important things because you know, like containers die, if you don't have a long-running process, what you want to run if this is not like a database or a web server or whatever that it's running what I want to do is City I enabled it's true and I wanna run a command that is cut.

A

So this keeps the container running all the time, and now I can go and say inside the well actually I'll show you like mvn does not exist here, but inside the container maven I can run ambien.

A

Let's sleep for a bit oops.

B

A

So I can run now. I should have changed the name, so it's showing with a different name. Okay I've made that type of somewhere.

B

A

Ambien not fun.

A

Somebody's paying attention I like it I, was going to make some mistakes on purpose so to see if you're worth paying attention, other ones I don't. But you do never know. Oh.

B

A

What I was doing here anyway, so it's MBA, inversion or.

A

Okay, you see that in the first call ambien wasn't did not exist, because this runs in the other container. The jnlp I didn't remoting.

A

Again, I forgot to change the name and now I have a different error. Container maven does not exist, oh yeah, because it's using the old ones, so you can maybe you've gone. Maybe.

A

So what this this looks very simple, but it's really powerful because it allows you to reuse all the images that are already available in the hub, so you don't have to go and create your own images.

A

Okay, now we have the post been created.

A

And this is right, sorry.

A

Both I'll go there in a second son, and this is running on a maven container inside the pot that I just launched okay. Now the gentleman here was asking if this runs cells in the clarity of a plane that you probably all know, which is like a easier, more concise version of pipelines. So by default you.

A

Can use any groovy you want in a pipeline which is really good because you can use any groovy, and this is really bad because you can use any groovy synthesis. You want so there's this idea of the clarity pipelines where you can define the agent in a like Morgan sees concise way. So you can say the same thing. I was saying before I can say: set some label run a grater template that has maven container in there and then run some stages.

A

So if I run this.

A

I'll show you the previous one is exactly the same, so you can have this this very, simpler syntax. If you want because yeah the problem with the groovy one is that it gets complex, pretty pretty easily.

B

A

Another example.

A

So how powerful it is to have multiple to have gone pots with multiple containers on them. So I'm gonna kick this and then I'll explain it as I go so.

A

This is a hard one to let me copy here and show it here.

A

So this is a complex one. This is a selenium template that on a pipeline that will run two tests in parallel.

A

So it's running two maven containers, one for Firefox tests, one for Chrome tests against a selenium hub and another container that selenium hub is has connected to other containers, one with chrome, selenium, chrome and one with selenium Firefox, and so you have five containers, plus the remoting support with six containers, so I'm reusing the pot, the container images that were released by the selenium project now it's interested interesting bits here is that because everything that runs in a pod runs in the same network namespace and these images were designed to run in docker compose, you have to tweak different environment variables.

A

So the pores don't collide. So that's why all this customization like you, you have to change the display port. You have to change the word to look for the selenium hub and but without touching the images you can define this pot. That will run all this for you and in parallel, I'm running a Firefox test with maven.

A

Just passing the selenium browser, Firefox and I have I can run the Chrome browser tests and then there's another option in the kubernetes plugin to get the container locks so at the end, I'm, just printing, the logs firm from both executions. So.

A

You will see I have in parallel the tears running in Chrome and Firefox, and this should have strung.

A

The pots here launching, let's run it again and.

A

I want to go to the previous one, so there's something.

A

So these steps are running against it from the Maven container testing against the selenium hub and then at the end. I can also get the logs from the container. So this is whatever it was printed to the standard output of the of the container.

A

Okay, so the other interesting thing that you can do is also deploy to kubernetes from the from Jenkins.

A

So this is not related to to the kubernetes plugging that will expand the agents, but you could also have both the agents and the deploying to kubernetes cluster that one or not a different one, so imma show you an example.

A

For some reason, this was.

A

Yeah I mean it's: you can get the output if you want to yeah yeah.

A

Yeah yeah, they cube CTL output. Here yeah, it's just showing you that the ports are restarting, the pods are finishing and so on. Let me kill this pose first, to make sure.

A

By default, there.

A

By default, there's going to be a new pod created every time, but you can also set the time that they can stick around. So if you want about to be around for doing other things, Duke- and you can do that- that's typically the confusing part for people that were there if the body sticks around junkies will escape. You know more work in that pod, based on the labels.

B

A

They will get deleted at some point. It's just to make sure they're I'm clean, because it depends on how often they could burn the Jenkins goes through the provisioning phase you may have. You may find both the stick around for longer. One thing we do is to use you you IDs for the labels. So every time every job runs in their own pot and you ensure that it only goes to that one, because the UID is generated in every build and every job run.

A

So I have this croc hunter application that you may know from other people's demos.

B

C

A

C

A

Running the containers inside pots yeah- that was the question. Yes,.

A

That you could mount things and make it available, so you can mount everything you can mount persistent volumes. You can mount files from the host. If you want, you can do all that. Yeah! Yes,.

A

It's just using kubernetes api, it's not talking to the doctor tomorrow.

A

Let me just launch one thing, because this may take a little bit to come up.

A

So I'm launching this this execution- and you have a question here.

A

Okay, so your question is: how do I get this out of the ball into somewhere else?.

A

Okay, you want to use your portable, a docker image, so you can do it the same way as you would do with coronaries. You would mount the docker socket actually I. Think this example. Does it?

A

A

So this example: it's building a go application. It's packaging in that docker container and then is pushing the container either image to the Google container registry.

B

A

Then it's deploying it with kubernetes, so this is a more complex pipeline, so you have again several containers running. Actually this is the branch.

A

Okay, so I have the agent container I, the general P container here: I have a docker container, which is the docker client only with a Google Cloud SDK I just need this to push to the Google registry.

A

I have with the goaline image to build my go long application and then I have the cube CTL image to run cube, CTL command, so I don't have to create one image with all this I just need to create to plug them all together and what I'm doing is mounting the docker socket into the container into the pot right. So this has the the benefit that you can use. Docker commands it has the pran that you can use docker comments, so it's up to you if you wanna use it or not. Yeah, yes,.

A

If you use the docker socket, you are on your own. Basically.

A

A

Now not did you use a darker socket, the plugin will interact with the coronaries api and that's it.

A

So I mean we can talk later after the talk. If you want to talk about how to build docker images in a kubernetes cluster, it's gonna be a long talk.

B

A

Me just finish this example so interesting bits here: building the go application here so inside the container, golang I'm running, make nothing fancy there, but I'm you see, I'm reusing the containers I'm, not creating my own images, then in the container docker that has the docker CLI installed, I'm running the docker build and then the g-cloud docker push. This is creating the container the image I'm, pushing it to the Google registry and in the container cube CTL. What I'm saying is: hey just I.

A

Have some general definitions in github just execute them and install this application? That does this thing whatever it is. So if this run fine, which something II did waiting for next available executors just.

A

Yes, yes, but and that's when there is light yeah I'm in the right place.

A

Of course, the demos never work okay, so this is waiting ah there is it's just patience what's lacking, so this is going to go through that and you know that you can also have approval steps. So what I'm doing here, if type permits there is.

B

A

So I'm asking the are you approving? Do you wanna prove this deployment, and this pipeline is going to wait for one hour they're sitting and waiting for me to click and.

A

Unfortunately, it took a little bit longer to start so in this pipeline. Also what I'm doing if, if I it has a github hook, so I can commit changes and get trigger updates with cube CTL and if I get.

A

Hunter all so there's nothing running here yet.

A

So still going through the Compellent test, so I think this is gonna, make me run out of time. But if you have questions now, yes, if.

A

Well, if your, if any of the containers died in a pause this, what will happen with any coronaries spot is just the container about the ends. It will tell you exited or after or something.

A

You'll get messages in the log, so yeah. One thing we we do is there's a cap that you can set. So this does not spinning things over and over again, let's see okay, so this is the ploy and I got my app build with a container and and I'm missing the part where I was pushing things to eat hub, but you get to get the idea. This is building the image and okay and times up alright. Well, you can. We can talk later a word.

A