Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CI/CD BoF: Delivering Quality Quickly - Brice Fernandes & Marc Carré, Weaveworks (Any Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

CI/CD BoF: Delivering Quality Quickly - Brice Fernandes & Marc Carré, Weaveworks (Any Skill Level)

Join us for talks and conversation on getting code from the editor to the user as fast and reliably as possible.

Topics of discussion will be decided on the day by the attendees.

Contributor to the session who propose a selected topic will receive a prize of their choice out of a selection of great books (including copies of "The DevOps Handbook", "The Phoenix Project", and "Continuous Delivery")

The event will be moderated by Brice Fernandes and Marc Carré from Weaveworks.

About Marc Carré
Software Engineer, Weaveworks

About Brice Fernandes
Brice fell in love with programming while studying physics and never really looked back since. He has a broad technology background that covers everything from embedded C to backendless browser apps using the trendiest javascript frameworks. He taught Game Development and Functional Programming online and founded his own education platform for developers before joining Weaveworks. He now spends his time making microservices and containers so simple even your cat can deploy and monitor them. Come talk to him about programming, creating closed ecological systems and leatherworking.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So cool, so the idea today is: we've got a bunch of contributed topics and we're going to go through and have a vote so we'll start voting on which topic to kind of start talking about we'll rank them in order of kind of preference and we'll spend five minutes on the first one and then after five minutes, I'll get everybody to kind of put their thumbs in the air. If you put your thumb sideways, you don't care whether we continue or stop talking about this.

A

If you put your thumbs up, if you wish to continue talking about this topic, and if you put your thumbs down, you wish to move to the next topic and we'll just keep going down until the session runs out in about thirty five minutes. So that's the idea.

A

So what I'll do first is I'll start listing all the topics and we'll talk through kind of what they are and, if there's any confusion, I will ask the writer the person who actually contributed the topic to start talking about what they meant so that we have a good idea and finally, kind of one thing: that's going to happen is I'm not really going to be contributing a lot to the discussion you guys are.

A

We have mics around the room and the idea is that you're you're going to be contributing you're having discussion about those topics. So what I'd like to what I'd like to set as a ground rule? Is we only talk from experience? So this is like personal experience with whatever the topic is. So if you have heard of a tool, if you've trial the tool that, if you've just heard of something or if you've, just researched a tool, let somebody who's actually tried. It was used it in production.

A

Have the discussion share their experience and otherwise that's basically the session if they see what we're gonna go through so quickly, I'll run down through all the topics and if something isn't clear, I'll ask the person who wrote it down to explain it to us and I've tried to group them into kind of somewhat similar grouping. So there's a couple of a couple of groupings already and so, for example, the first two topics metrics for continuous integration feedback and then the other one is pipeline metrics which to collect so that's kind of metrics.

A

On continuous integration, we have what metrics are quality measured by. So how do you measure quality? We have Jenkins versus CI versus gate, lab or CIC the tool of your choice, why'd that tool how to convince team, how much flexibility to her for the team ease-of-use so who contributed ease-of-use.

A

So when they might guess you could you give us a little bit more about kind of what you mean by ease-of-use like where would you like the discussion to go.

B

About the user interface in this case, okay, so.

A

Is that talking about different solution, which one is easier to use? Yeah? Okay? So what I'm gonna do with the ease of use and I'm gonna bring it to the tools discussion right? This is about like discussing which tools, you've used and comparing different tools. So we have container image building who contributed that topic.

C

Soho CN and see city, sometimes building images is important. Yeah.

A

C

C

A

So is that, like kind of about cuba, Nettie's, okay, so I'm gonna bring that with a CI CD for cuban at ease.

A

We have versioning tagging images getting a fairly straightforward, like a best practice for tagging images, ensuring test reliability and flaky tests. So how do you kind of make sure that your tests, reliable rollbacks, which is quite an interesting question? So how do you once you've deployed it? How do you roll it back? What what you have reproducibility running end-to-end, test with hardware devices?

A

What's the simplest and most effective workflow from get commit to production, so I, guess workflows and pipelines and I'm going to bundle that, with pipeline definitions, declarative versus impressive pipeline I know it's somewhat related on Lieber and.

A

Also, integrating an approval process, because that seems to be about workflows. How do you add an approval process to your workflow so how to do CI, C D, if the whole ecosystem is locked down, that you cannot implement any of the continuous deployment principles? Who wrote that one.

A

That seems like a specific pain point that you felt feels like this. One comes from pain, yeah,.

D

Hi sorry, I'm, Christophe and yeah I have actually worked in the AP world, so I did test automation for a picture corporation where we automated like most of the business processes and what I felt was that everything we learn here and that we learned in university and so on. This does not apply to the ASAP world at least what I learned and you know I was.

D

It was like a whole new world to me so I, you know, I was I, used all these great tools that continuous integration- and you have none of that in that world, and it's also no one knows about that. So that's what I'm, feeling and so I think that we somehow need to bring that into this world because I'm here now at this conference- and it feels to me like this- is like you know back to reality like these are this is the good world and it feels like the bad world.

D

They don't know what's going on, so the question really is, so how can we use the knowledge- and you know, change the all these lockdown places? Ok,.

A

Back yeah, so I'm gonna bring that actually, because that sounds like it's a human problem, not a technical problem. It.

D

Is a human problem, yeah I'm gonna.

A

Bring that with how do you convince your team, yeah cool.

A

So we we can't hear you, can you raise your hands? We can bring a mic to you, yeah. Actually, that's a good question. So it's the question was I'm wondering whether it's coming from a product management or a developer. So could we just have yeah so could we have a hand up if you're, if you consider yourself a developer all right so like the vast majority of the people in this room in developers, who's, who's, kind of project management.

A

To people all right so, like the vast majority of the audience, is development, which is quite interesting, cool, there's one here. This is about one platform: 50 services, five teams, three dev staging prod using kubernetes who, who wrote this one, this? Okay? Excellent? Could you share with you what you want to get after that? It's got home yeah.

E

So that was basically, if you had those requirements, and you wanted to have a CI CD pipeline for kubernetes I'd- be interested to hear what people have put together. There's been successful for them. Okay,.

A

So would it be okay to bundle that, with like tools of choice, would that be appropriate? Yeah.

A

Continuous deployment canary releases I think that's reasonably explicit, like how do you do canary releases when you've got CD? How do you do kind of the automation- and this is one- that's also automate automated pushers to production or just devastating I- think I'm gonna bundle, those two together, I know they're unrelated, but really the question is how do you? What do you automate? How do you do advanced deployment? So there is quite a few questions about advanced deployment strategies.

A

There's a couple about CI, a CI CD on and for service, which is quite interesting. So how do you do CI CD for service and how do DCI CD on service and I? Think that's I'm gonna bundle the two together about service, there's managing feature branches, mono repo is alive. Yes, what are the best practice for a mono repo build once deploy many many strategy with kubernetes I'm gonna bundle that with a bill strategies one. So this is kind of like talking about the bill strategies. How do you advance build strategies?

A

How do you deal with secrets? What's the deal with secrets, what is get ups and get ups versus ownership? So this is this: is our fault I'm, very sorry about the get-ups term, another yet another kind of term coming to the market, but it's all our fault that we've work, so this is kind of what the hell this is. This thing is accessing new environments from tests.

A

I think that's the right one who who proposed that accident and I'm gonna, just guess that you're talking about kind of how do you set up a Cuban, NCS cluster from your test environment? Is that right? Oh I, might yeah, okay, cool so yeah. So how do you deal with access control for the Cuba for your test, Cuba Nettie's environment from your CI CI pipeline? Basically, one big versus many small: how to divide that really I think lives with them on a repo, okay,.

A

So this is a system issue. So what what system do you mean your your CI or CD system.

D

Oh interesting run like that one for the whole company, or should you have like one per team or how do you divide.

A

It yeah absolutely that's, definitely different so like how do you divide a your CI pipelines and your CD pipelines like in your organization.

A

Multi project builds PPC, arm, 64, x86 and test engineering best practices, who's. Okay, is that does that sound on its own, or does that live with one of the other ones? We've talked about so far.

F

Just options for doing building products with multi-out support, okay,.

A

I think I'm gonna keep that on its own and then artifactory repository I guess that's a particular repository like that you're talking about a particular tool who proposed that one.

G

Yes, we are new to the company. Well, at the beginning, we thought a image input array is enough to deliver our product to our customer, but then we got this idea. We need actually use the difficulty reef repertory to do that, I'm wondering if that's a mast or not if it's a master than was the best practice for that. Oh so.

A

You're talking about delivering a product to customer through an artifact repository, yeah, okay, excellent and the final one is chaos, engineering and continuous integration deployment.

A

So how does chaos engineering fit into CI CD, so those are kind of the topics we've got I'll go through them once again, and what I'd like you to do is raise your hand if you think this is an interest, interesting topic to talk about you get as many votes as you want, but I would suggest you restrict yourself to the ones that really interest you so that we can kind of prioritize yeah.

A

If you vote once for all of them, I'd prefer you abstain, it'll make it easier for me to count, but otherwise that's about it. So we'll start with what metrics are quantity? Measured? How do you measure quality and what's important, anybody want to talk about that? Okay, one, two, three, four: five: six, seven, eight nine.

A

Metrics to look at the actual pipeline, how do you monitor your pipeline, so metrics for CI feedback and pipeline metrics switch to collect I, keep them up while I'm counting one two, three, four, five, six, seven, eight nine ten eleven thank you is the mono repo still alive and what are the best practices for the mono repo two three, four, five, six, seven.

A

Managing feature branches.

A

One two: three: four: five: six: seven, eight nine ten, eleven twelve server, listen, CI, CD.

A

This is about advanced deployment strategies like how do you do canary releases? How do you promote between environments? What is that kind of different okie-dokie? It's not actually a surprise. Considering the discussion I've been having 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33 35, this all right, yeah, so I'm gonna call it 40 just about pipeline definitions, declarative versus imperative. How do you set up workflows and pipelines?

A

One two, three, four: five: six.

A

This is the next one is about like practical experience for particular tools. So like Jenkins, kid lab CI ease-of-use, which told you prefer what's a tool for the right use case. Okay, that's quite good as well one one, two, three four.

A

35, so just just below the advanced deployment, how do you convince the team, how much flexibility to offer des like the human side, of how the hell do you make this happen?.

A

16 and to end test with hardware devices.

A

Three, like yeah I, know what you mean like if you're having to do this, you feel the pain really Keely, but it's not very clear what you have to do. Reproducibility. How do you create reproducible builds when you care about one.

A

Versioning tagging images how'd you deal with versioning and I'm gonna yeah one two, three four.

A

Probably pick the biggest one, anybody like keenly wants to vote for a later one. We have about 45 minutes, okay, so who wants to share experience about the advanced deployments? You're? Quite right? Actually, I lost track of time.

A

Anybody wants to talk about just quickly show of hands flaky tests, if it's less than 40 I'm, just going to skip multiple project, builds different architectures roll backs; okay accessing environments from tests, accesing clusters from test environments, artifact repositories, chaos, engineering, okay, andget, ops, okay, I'm, going to call that the third one and secrets okay cool. What's that which one this one right, this one? Oh.

A

Oh yeah, thank you who cares about kind of CIT D on kubernetes in particular, rather than general discussion? Okay, that's about nine people right so who wants to share experiences about the Canary release, advanced deployment patterns we'd be the first to do that. Who does it to start right? In that case, you get to start talking about how you do it.

H

I gave a talk, yes on Tuesday, about how we do deployments at the New York Times through CI CD, and get you can watch a session. I think I cover a lot of what you're looking for here, but specifically for this canary deployment. We started with deploying multiple to we started, deploying our master to ace to our cluster in a low replica count. Alongside with our production and high replicas count, and then through our CDM.

H

We would split the traffic so that it routes the majority to the production cluster and then percentage to the next version, and then people would just track the metrics for that we're thinking about moving to sto so that you can, when you deploy, you can config configure how much traffic you want. Actually want is co2 split for that deployment so that, when you deploy to production, you are determining the split of traffic to your canary, deploy.

A

So what tools are you using to manage that splits and do you autumn ultimate, the scale from so.

H

For that we are playing for is: do we're just going to use the config. So when you run your deployment to the CIC to pipeline, ku still will inject the updated configuration to your cluster and then we'll split the track for you. We're not planning to do like incremental until canary analysis, just a static percentage analysis, so yeah, okay,.

A

Anybody else has done yeah.

F

How would you deal with the canary release that requires a schema change.

H

Yeah, so this is a good question so for things that are doing schema changes, Martin Fowler has a page on implementing your schema changes in a way that you have multiple versions like if you're doing a schema, change, you're gonna create a new column, for example, and keep the existing one, and your code should be updating both, for example, until you're, confident that the new one works and then so at any time you should be able to roll back at least one migrated back one version and then, when you're confident, then you can drop that on the next schema change.

H

A

Who else is actually try to implement some of this stuff in in their pipeline? Do you want to talk to us about what you experienced in what you use.

I

So we were not doing anything on kubernetes or containers, and but we took our VMS and we just partition, 1/10 of them to be our canary cluster in production and what we would do is for a manual process to deploy. We had a tool to help us, but we would apply to the canary cluster and observe and and see if latency or error rates build up on evenly compared to the other clusters. Okay, so.

A

Like it seems like only two people in the room have actually experienced kind of advanced release cycle. We've got some more of that, but you share kind of what your strategy was and how do you implement it there? The microphone is just behind one.

J

But I will probably just repeat what he said about the schema changes. It should always be like the one. It's it's it's in that book. You have the developer handbook as well. Yeah.

A

J

The schema change in the database will be Ford's compatible, so so it will just add things not remove things and then, when you detect that in early stage, so you would detect it in test of staging about when you do the DB migrations, which you of course should have as well so and that works you find you find income in breaking changes in earlier and production. That way and.

A

Just behind you.

K

L

I just wanted to touch on maybe two things that we've seen that they're common misconceptions: oh I get some technical detail as well, if desired, but first thing is a Bluegreen deployment is not a canary deployment. Bluegreen is typically designed to test intended differences between applications or features, because you know you're doing something new and that's what you're looking for a canary deployment you're, not looking for a difference. The difference is a problem, and so the approach is that you want a slightly different.

L

Another thing: that's worth, bearing in mind with Canaries and I've done this many times myself. So that's the the traditional canary I always did was I. Have my production version I spin up a small number of instances of a new version, I route traffic to them, either by sticking them behind a load, balancer and just making sure the numbers match or if I have something where it can explicitly control the percentages like with sto or something else.

L

You set up a specific percentage and then you just wait a while, and then you look at some monitoring dashboard and you pick some set of metrics that you think are relevant and then you make it sort of well. It looks okay and that's fine, but that's a really really basic kind of homebrew canary, like proper canary.

L

If you think about the statistical test, because you're really trying to do a control experiment, you should never compare a small new version against a large existing production version, because they're serving different amounts of traffic and one has been warmed up in the cache and all that kind of stuff.

L

So what you really should do is you should spin up a control which is your production version and the canary, and then you should route the same amount of traffic from the same geographic locales or the same population set to these two and then, if you really do it properly, you should do a statistical analysis to compare both of them and there's plenty of statistical tests.

L

That will do this and then, of course you automate that, because you don't want to do that by hand and I'm not saying everybody should go there or needs to go there, but it's worth bearing in mind the Netflix, for instance. That's what they do for canary analysis and.

A

So what have you got? Experience setting this kind of stuff up well,.

L

The people I know at Netflix do.

A

So unfortunately,.

L

I mean they do it with an internal service that they've just open sourced: okay called Kayenta. So if you search Kayenta on the internet or automated canary analysis, you'll find some stuff.

A

Third time, would you like to continue with this topic? You don't care or you want another topic.

A

Most people want to move on, so the next one is tools of your choice. What tools you use so can we have some good suggestions on tools that you have used and you have personal experience with on kind of the things you you for doing: CI and CD any particular pitfalls, any particular issues with the tools that you use, yeah.

M

We're a big fan of gitlab runners, so if you're still using Jenkins and I haven't used it try it out.

M

It's a it's, a very I like the agent system that has this bullying mechanism and it's it's fits very well with containerized software and mic reserves. So, okay, it's.

H

A

Mean like I'm, just going to repeat the question for the recording, how many teams do you have working on it? Quite.

M

Yeah we are a Research Institute in the Netherlands, but so it's often small teams, but it's growing and growing so over 20 or so I would say in an organization of 3,000 people. Yeah but I, don't know, I, don't know the exact numbers, but it's increasing a lot. Sir. Do we have anybody.

A

From gate lab in the audience actually because I know get labs are here today on the floor. Okay,.

A

Yeah, how did you evaluate the tool.

M

The configuration lives very close to your code so with the gait lab CI Yama foul yeah, easy to set up. We use Jenkins before, but we found it quite tough to. Of course you have this Jenkins fel description as well, but, and it fits closer to two containerize software. In my opinion,.

N

So we started with Jenkins configured with kubernetes plugin and basically we had a dedicated plugin for Jenkins developed inside that was used to configure Jenkins itself provision that, with all the credentials, all the all the plugins and stuff and we were building a containerized Jenkins that we run on top of kubernetes configured so that the built pause would pin up would spawn dynamically on on the job with each job and users were had. Only all they had to do was to was to have a jinkies file alongside a code repository and basically they defined it.

N

Their jobs there and in the end project was scrapped because gitlab CI, with yama configuration, had more appealed more to the general development public than customized. Well, not customized, but what's somewhat customized, we had some custom DSL going on yeah, so.

A

It's interesting, so you found that, like you had a system where that was you dynamically bringing up build pods through, like jenkins instances correct on kubernetes, correct, and I mean like there's a lot of configuration, a lot of effort to get that running. Yes and then you find out some of the kind of the off-the-shelf solution were and if a bit smooth at like the developer experience was smoother developer.

N

Experience was smoother because of llamo files, yeah comparing to the groovy jinkies files, okay, but otherwise the functionality build times was very like similar, and we had, in contrast to to get lab CI. All the configuration for Jenkins was us code in get.

A

Anybody else actually had went the other way around who died. The customized bill system.

F

So we were using git lab and then switch to github, which meant we could no longer used to get lab CI at which points most of the evaluation I did was against Circle CI in Jenkins a few other ones, but having configuration of the jobs as code was key, so that ruled out some tools in the industry. I can agree that so we use the kubernetes plugin with our jenkins and so we're building pods and the configuration is quite complex.

F

It's a easy to abstract it with jenkins libraries, so the developers don't have to be so exposed to the complexity, but it's still not so ideal. That being said, the Jenkins X tool is kind of just coming out and I did some evaluation of that.

F

The other day, I was really impressed with how quick and easy it is and how it gives you everything so for people who are just starting out and the CICE thing I think this is definitely a tool to really consider but be careful, because one of the reasons why we're not going to consider it at all is like you build out these container repositories, but there's no backup structure. There's no high availability, there's no really resiliency to Jenkins X.

F

So you have to do a lot more work to make Jenkins X production ready, but it is a tool to consider and starting off. Okay.

A

Interesting I think there's two points on this. So there's one is the amount of time you spend on. Actually, when you're doing it yourself the amount of engineering time you spend on building it managing it maintaining it. So that's the first one and the second one is configuration is code for your build pipeline. So with anybody has a kind of interesting site and experience on either of those two topics.

A

Yeah you're, about to raise your hand, I saw it like halfway up in the air before you.

H

We also evaluated quite a few services, but eventually we went with drone we're on github. So with that we can't use git lap CI. It does look really enticing, but drone has like a very similar kind of declarative pipeline manner.

H

We just completed Bennett Jenkins because is not the way to go and we productionize our CI system for the entire company. We about 200 repos on there moving towards five hundred fourteen thousand bills a week, and then we are looking to scale that up from ten workers to about twenty soon, but it's pretty satisfactory for our purposes. I know other companies run it in production like Reddit Capital, One Squarespace, that kind of stuff. So it's worked pretty well for us. So far. Okay,.

A

Before we pick up, do you want to continue with this topic? We've got about yep.

F

Just just very small question: did you try.

H

H

Circle, CI and Travis CR kind of restrictive and Drona's open-source project with the enterprise license. So we run it ourselves and it's way cheaper, like ten times at least.

B

So I'm happy that drone was brought into the discussion. We also started using it a bit four to six runners. So it's not a big installation. I think it's it's. It is a great architecture and it has I. The open source approach revealed to me just bear in mind that it's also the hackers approach. So if you, if you are shying away from looking into the source code, then it's sometimes thrown has some rough edges.

B

But if you are into the open source approach and you you can read, go and stuff I think it's it's it's a very nice approach and actually I favor it compared to other solutions like the new Jenkins accent and so on. So it's just my implement.

K

Just a quick pitch for I heard a couple: people say they had trouble, getting the kubernetes plug-in to work and I would just if you could come up and find me after the session I'd like to hear a little bit more about what trouble you had because with OpenShift, we provided Jenkins experience on that and it seems pretty straightforward for me to configure it. So I just wanted to hear more about what issues you had.

A

We had one here and then one here so yeah I think like some of the people who are building the CI pipelines in the conference today. Right so I know the gate lab guys here, nothing, a couple more pipelines are here: oh we.

O

We in Zalando we had three generations of Jenkins. First, it was a few very big Jenkins machines in the data center that didn't scale at all in the build queue sometimes grew up to 5,000 items. Then we had two generations first running Jenkins on AWS provisioned with ansible, then we had another generation of also Jenkins's on AWS I, think Jenkins's means per team. So it's like more than 100 Jenkins in the company and in the second attempt of doing this we had a custom application that was provisioning. Those Jenkins's we're also moving away from that.

O

But this thing is still alive and runs and we ended up building our own CI solution, highly tailored and more or less similar in the concepts to modern sea ice that have just a single llamo configuration. So.

A

How much effort did you put into building your own packet.

O

A lot of effort, so it's a team effort for more than more than a year already so, but we were making talks and other conferences about it. So if you hear about this solando continuous integration, continuous delivery platform, this is us and so far the benefits are that we have strict control of which team can deploy where we have a lot of AWS accounts and kubernetes clusters and the compliance requirements tell us that it should not happen that some someone deploys to some other teams cluster and we also have control over which code gets deployed.

O

We have approval system and we record everything, so we can track back from the deployment to the commit to the author of the comment that got deployed. Actually, so it's a lot of work and we also manage secrets. So in the bills there are no secrets in the build environment or no keys, no certificates, no passwords, no tokens and all the outcoming requests to private resources are intercepted and doesn't it's only she TP, but HTTPS and the authentication header is injected based on. If it's coming from a running, build machine, then it's it's injected.

O

Otherwise, it's it's not so there are many tweaks that we're able to do because it's our own and the usage is growing and people like it at its.

A

Facility and also review opening so state have.

O

You open source state, that's the most popular question and we have it. We have it in the plans, but we did not open source it yet, okay, so we still need to find a way for it, because if you start building it internally for your private use, you inevitably you end up integrating it with other private systems, and there should be a good way of abstracting this and even understanding what you have to abstract and what you can open source. And this is not easy if you are the only one using it. Okay,.

A

So, as a final note like would you say it was worth actually the effort of doing that or should you've gone off the shelf after, like I've explained all this time.

O

Well, you cannot trust me on that. I I would prefer building my own because I'm building it and I'm enjoying it I see there all the happy users, then other other people might often feel the same. So I'm talking to people who actually left our company- and they say they still say this thing was better than what we have now in my new company. Okay,.

A

Thank you very much, that's about time for this session. If you want to move to the next one.