Continuous Delivery Foundation cdCon 2021, 6 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Measuring the Robocat: How Tekton Continuously Delivers Tekton - Christie Wilson & Andrea Frittoli

Description

Measuring the Robocat: How Tekton Continuously Delivers Tekton - Christie Wilson, Google & Andrea Frittoli, IBM

Continuous Delivery (CD) is a set of development processes that support the creation of today’s increasingly powerful (and complicated) software. Whether you’re trying to move to the cloud or break up your monolith into microservices, you need CD automation more than ever. But how do you know over time what’s working and what isn’t? Tekton is a cloud native CD tool with 1102 contributors from 140 companies. It’s important to us to “eat our own dogfood” by using Tekton to continuously deliver Tekton. For this talk, we set out to measure our own CD and see how we stack up through the lens of the DORA metrics and CD best practices. Find out what we’re doing well, and where we have room for improvement! In this talk, you’ll find out how to start measuring your own CD performance, which practices to apply to which kinds of projects and tooling, and how you can leverage Tekton.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

A

Hello, everyone welcome to measuring the robocat.

B

Hey andrea, how are you what's up ben.

A

B

I'm pretty good. Actually, my recovery is 94 today.

A

Oh recovery, what are you talking about.

B

Oh, I'm I'm really into measuring my health lately, and I got this strap on my wrist that measures my heartbeat and my sleep and every morning it says how recovered I am.

A

Oh nice, like a like a fitbit.

B

Yeah exactly like that,.

A

Wow, that's pretty cool, but do you find the data helpful actually.

B

Mostly I mean I do find it interesting to see how it changes over time um and to know where, where to focus my efforts like, should I rest today or should I really go for it? I wish we had something like that for continuous delivery.

A

Oh, like like a fitbit for continuous delivery.

B

Exactly but I don't even know where we'd.

C

A

Well, actually, I have a pretty good idea. Have you heard about the dora matrix.

B

Oh wait, yeah that that sounds really familiar. I wonder what those metrics would look like for the project that we both work on.

A

Oh, that's a really interesting question. hmm Maybe we could take a look at how well decton itself measures up to the dora matrix.

B

That's a great idea, and you know it would be cool if we could throw some best practices in there too, like the basic basic building blocks. That would help us improve those metrics. I'd really like to see how techcon stacks up.

A

I agree totally agree and I mean similar to how you're, using the data to decide when to rest when to exercise.

A

I think, looking at this data will will help us will help us guide on our own cd efforts as well.

B

That is a really good idea that makes total sense. Let's measure the robocat.

B

Actually, maybe we should introduce ourselves first hi, everybody, I'm christy wilson. I lead the tecton team at google and it's probably one of the most exciting, if not the most exciting project. I've ever worked on the first commit that we made in the repo is from august 2018, which makes us very close to techcon's third birthday. I can't believe how fast time has flown and having been responsible for a lot of our decisions early on in the project. I'm really excited to see how well we measure up.

A

And hi I'm andrea fritoli and I work as a software engineer and developer advocates at ibm.

A

My job is to work on open source, which is also my passion, which is great, and it's especially fun working on tecton, so I've been living leading the dog footing, work for tecton since its early stages, it's been marvelous to see how the project and its community have grown and evolved over time.

B

So, today, we're going to show you how we use the doria, metrics and best practices to measure tecton, our very own, robocat. First, we'll start off by explaining a bit about what tecton is and then we'll dive into the measurements themselves. So first, what is tekton? What is this robocat? We're talking about.

B

Tecton itself is a continuous delivery system. You could also call it a ci cd system, it's built on top of kubernetes, using a mechanism for extending kubernetes called custom resources or crds.

B

It also uses the same kind of declarative resource model that kubernetes itself uses, which is often called the kubernetes resource model or krm tecton has been donated to the cdf. That's the continuous delivery foundation, and it's really a group effort and both andrea and I work on this project together.

A

Yeah thanks christy and, as you said, tecton is a continuous delivery system. So there is a bit of inception going on here and yeah. We want to be really clear today. That's what we're going to talk about is the health of the tectum project itself.

A

So one of the things that we've been doing since the beginning is trying to eat our own dog food, so yeah use tecton to build and make tecton itself we're, not all the way there yet. But we have placed a lot of importance on on these, and we've worked on a lot of automation and continuous testing in place from the very beginning.

A

And if you're familiar with the titan project, you probably know that when we say tactum we could mean the overall project of one of the 15 or so project that are actually part of the tecton organization.

B

That's a good point, andrea. So in this talk when we say tekton we're actually talking about seven of those projects, the ones that we collected these metrics for which are pipelines triggers results, the cli dashboard hub and operator, but keep in mind that there's actually more to techton than just that.

B

This is kind of an overview of what all the projects look like and how they relate to each other or all the ones we're talking about. Today we have the dashboard hub and cli, which users will use to interact with our core apis, which are results, pipelines and triggers, and then there's the operator which manages the installations of all these components.

A

Yeah indeed, so um you could roughly group the project, this server project in a few groups and like the dashboard, the hub and cli, are the interfaces. Then we have the core components like blind triggers and results, and finally, the operator that manages them.

D

A

So I guess the question is: with all this different and different kind of projects. um How do we go about measuring their health.

B

So, just like you mentioned at the beginning, that's where the dora metrics come in the door. Metrics are four key metrics that indicate the performance of a software development team, they're based on six years of research from the devops research and assessment, or dora.

B

So what are these metrics? They are deployment, frequency lead time for changes, change, failure, rate and time to restore service.

B

Now something big that changes how these metrics are applied to our project is the types of projects we're talking about. These metrics refer to service deployments and time to restore services, which makes perfect sense if your team is running a service. But what, if you aren't for techton, we do run some services, but ultimately we produce software that is actually run by other people.

B

Let's take a look at the seven tekton projects that we want to measure. What kind of projects are they we can group them broadly into three categories?

B

The hub, which is an interface to the techcon catalog, is the one service that we run available at hub.techton.dev and we need to release to it and keep it up and running, but the rest of our projects are all intended to be run and operated by other people.

B

Five of them are run as services and then there's the cli, which will never be up and running and has no concept of uptime down time, et cetera.

A

Wait wait. You said we don't run um the services, that's not entirely true. I mean we do have a few of this project in our dog, fooling cluster up and running.

D

B

Yeah, that's true, but I mean that's just for us. That's not exactly for our customers.

D

A

Depends what you mean by customers? I mean we have all the contributors that work on tecton. Aren't they our customers too,.

D

Yeah, that's that's a good point.

A

Well, I guess one is what we could do. We could look at this matrix and we could look at our deployments to the folding. Maybe combine both.

B

Words, that's a really good point. We could look at the deployments to our dog food in cluster and maybe another thing that we could do is we could take these metrics and instead of talking about deployments exclusively, we could also talk about releases.

B

So instead of deployment frequency, we could talk about release frequency and instead of time to restore a service. We could talk about the time to release fixes.

A

Yeah, that's a good idea. It sounds interesting indeed um what we could do. Maybe we could look at both releases and deployments and see what we can find out.

B

Sounds good the more data, the better.

A

D

So the first metric.

A

We are looking at is the deployment frequency which we could also call release frequency then.

A

So this is a measurement of how often an organization successfully deploys the production or then, in our cases, releases to end users.

A

The dora folks have measured what this metric look like for the four groups of performance starting from low to medium, high and elite.

A

This means that, although it's just a correlation, you can get a sense of which kind of performer your project is just by looking at how your metrics compare to these groups for deployment frequency, specifically elite performance, deployed multiple times per day, while low performance take one month or more many months to deploy.

A

So, let's take a look at how tax measures are so remember that hub and our dog fooding services are the only actual running services that we deploy. So we're going to take a look at tecton from a couple of different angles.

A

So looking at release frequency, this is an overview at a glance of our seven highlighted projects, and you can see that we started very early with pipeline and then cli and dashboard followed soon after triggers came then, and more recently, we had releases for the other three projects we are discussing about zohab um and operator end results which only had two releases so far.

A

So if we look at the numbers for all these projects- and we compare them against the dora groups- the door buckets- we see that we have a few of the projects that would perform as medium like pipelines triggers cli and dashboard, and the more recent ones results have an operator. They are still in the low bucket.

A

So if we consider um nightly releases as well that takes four of our projects into a higher bucket in the high bucket, because then we have an average day between releases that goes to one.

A

The reason we are a bit hesitant to include 90 releases, though, is that um well, you can install them, uh but we don't have release notes for them, published on github, with warnings and upgrade notes and so forth, and so users would have to discover these releases find out what's inside before they installed them a potential way. We could improve. This, then, would be to to publish our to publish our night releases along with release notes as well.

A

It would have to do a bit more automation, work and there it is no generation, but not too much.

A

So now, instead of looking at the lens of releases, we could also look at our deployments to the full link as we discussed and yeah. These deployments are actually a bit less frequent than our releases. So often what we do we wait a little bit to see is there. Maybe there was a regression or some issue reported that we would do a minor release, and we don't want to get this issue directly into our dog food cluster, which is dear to us.

A

um So at the end we end up pretty much in the same performance category so for pipeline triggers and dashboard which we deployed to the footing. We would be medium and resolved as well. It was only deployed once so. We consider it low performance there.

A

D

You put all together.

A

The different views or lenses that we have seen on deployment the release frequency we end up with most of our project is overall in the medium bucket, except for hub and results, which are most recent ones in the low operator is current to medium because it's building nightly releases. So that increases a little bit its.

B

Performance, nice thanks andrea. That was really interesting. I feel like we found out a lot already about some potential improvements we could make. Let's look at the next of the four dora metrics, which is lead time for changes.

B

This is how long it takes a commit to get into production elite performers get changes into production within the same day and low performers take multiple months so for techton we're going to define production as making releases available to our customers.

B

Even with this metric, there are a few different ways we could look at this protect on the first way we're going to look at. It is the time between releases. So if the commit is merged, it's going to automatically make it into the next release, and you can see that there's actually a pretty wide range of averages here. We go from 35 days for pipeline releases up to currently 78 days for results, but again we have far fewer releases to look at, but all of these fall into the low performance bucket.

B

So that didn't look super great, but a different way. We could look at. It is what's the time between when a change is actually merged and when it makes it into a release.

B

So, instead of just looking at the time between releases, we can look at the time from when something is merged to when it actually makes it into a release, and that looks a little bit better and we're kind of back on track for medium performance in most of our projects, except for the operator, which is a little bit of an outlier and has a lower performance.

B

So that was a little bit better, but it's ignoring all the time that has to pass from when you actually open a pull request and when the merge happens, for example, all the code review and back and forth. That comes with that. So one more way that we could look at this is the time between when somebody opens a pull request and then, when the change actually makes it into a release, and when we look at it through this lens, our metrics become a mix of medium and low performance.

B

So this is how the second dora metric shapes up overall for our techton with us again attempting to kind of average the performances to draw a conclusion. Overall, now, the operator has a significant disadvantage here, because it is downstream of all the other projects, it literally bundles them all together, so its lead time and its performance is probably always going to be less than the other projects. So if we improve the other projects, we'd probably see the operator improve as well.

A

Yeah thanks christy that was really interesting and that's half of the door matrix down so and the two metrics that we just looked at have to do with getting changes in the hands of customers, whether it's a through a release or deployed in an environment that I can use, but the last two have to do with things going wrong, and so the third one that we are looking at now is change failure rate, which is represent the percentage of deployment causing a failure in production which in our case, translates to the percentage of releases which we need a patch fix for.

A

So that is, if the release contained in the bag. That is serious enough that we need to make a new release to fix it.

A

So to measure this one for tekton, we look at the percentage of releases which require the patch fix, and so we have a range of uh percentage. You can see from uh well hub, which is zero, doing really well um up to trigger 61.

A

But if we compare this to the buckets uh from the dora matrix, um we can say that cli and the operator fall a little bit into unknown length, but it's in the middle, so we grant them a medium performance. There.

A

um One thing that is interesting about the way that you're measuring this is that if we had more releases, the metric probably would go down.

A

um For example, if you're doing release every day or counting carbonite releases, then the ratio between releases and batch releases would skyrocket right and if you were releasing that frequently, maybe we wouldn't even need batch releases at all. um Just you know just wait for the next day for the next release and that the fixing is in.

B

Yeah, that's a really interesting point and I think it's very related to the last dora metric, which is the time to restore our service or, in our case, the time to patch a release when there's a problem.

B

Looking at these intervals for elite high, medium and low performers, it feels like it might be a little bit unfair to talk about this with releases, because, if you're talking about deployments, you can automate away a rollback or you can route traffic to working versions instead of a version with an error, and that would count as restoring the service.

B

But you can't really automate identifying and fixing a regression in a release. Even if the resulting patch is a get revert. You're still going to have to involve a person to go in, see that it needs a revert. Do the revert and then decide to make the new release.

B

So it might not be totally fair, but we're going to take a look anyway.

B

So we measured this in tekton by looking at the days between patch releases, so we started initially with a minor release. That's because we don't actually have any major releases yet, but soon so he started with a minor release and then the subsequent patch and then the time between all of the patches that occurred after that.

B

We feel like this is a good representation of the metric for us, because for us, failure and production is a release with a critical bug, and when we have a critical bug, we do a patch.

B

The fact that the gap is so wide here indicates that likely we could do some work to improve our testing and catch these things before they actually make it into releases. So, for example, we could run some of our tests against a full deployment on a nightly basis, or we could run tests in more combinations to catch these to catch these things and not rely on our customers to catch them.

B

For us running, more combinations of test scenarios would normally be pretty expensive, but we're actually looking into a way to make this cheaper, using a tool called kind which is kubernetes and docker and lets us run much more lightweight kubernetes clusters.

B

Okay, well, that was pretty fun. What does it look like when we put them all together? This is an overview of all the metrics. We just looked at across all of our projects and then we kind of tried to average them to draw a conclusion about each project.

B

In general, you can see that the metrics look better when we're talking about releasing and getting changes out, and we don't do as well for the metrics around failures. However, one probably impacts the other, as we've discussed. If we were releasing more, we would probably have a lower change failure rate.

B

Also, the metrics for the hub are kind of interesting. Until very recently we were releasing and deploying manually. So we didn't have that many data points to work with.

B

This exercise has raised some questions for us about how to apply these metrics to techton, since we aren't running any services except for dog fooding, we focus mostly on releases, but in that case, should we be? Should we be comparing our metrics to the performance of projects which do do deployments?

B

Also, techton is an open source project and, as we mentioned at the beginning, we have contributors from more than 150 different companies. So certainly the time to get a change in is going to be very different in an environment like that than it is for proprietary software and, lastly, time to restore, which is a metric we particularly suffered in is tough.

B

When you find a bug in a release, you can't exactly just roll it back the same way you can with the deployment you have to revert the changes, and even in that case, it requires manual intervention with all of that being said, we're going to round up and give ourselves a medium performance rating overall, which feels pretty decent, especially in light of all the questions we raised.

B

So what could we do to get even better? We mentioned that we had nightly releases for several projects, but we were hesitant to call them releases since we don't advertise them or create release notes for them. So if we supported these a bit better and actually numbered our releases and created release, notes that would likely improve all of the metrics.

A

Well, thanks christy, so well, not bad you're doing medium. So, um but besides the dora metrics, we also took a look at how well we are following best practices for continuous delivery.

A

So we'll take a quick look at this. um How what we evaluated! So we don't really want to say that these are the best practices, um even though you could stay tuned for the findings of the cdf best practices, working group as they're, actually looking at defining those.

A

um So this is more about like some more lenses to look at the project room and it's interesting. We see some relationship with what we've seen with dora matrix.

A

um Since discs are kind of a bit harder to measure, but still they should impact the dollar metrics in the long run, you can think of like going back to the example of the fitbits, how your diet will affect your fitness fitness performance, but fitness trackers have a bit of a harder time measuring your diet, what you're eating so um yeah. So what we did? We analyzed some of the cd best practices and we looked like the brain strategy.

A

The test coverage things like the test, mattresses that we are running, how we well we store test results, nightly releases, full releases and finally, continuous deployment. So younger projects like carbon results, are still in the process of ramping up you see a few red crosses there in the table, test, matrixes and test result. Collection.

A

Analysis of flakes are areas where we could improve in general across all the projects, so, for instance, something that we do we started doing recently in pipelines is testing both the stable, as well as the alpha version flavor of the api, so other options that we could do specific in the pipeline project, but also in other projects, are testing, for instance, against different version of kubernetes, and we only use one specific version now or we could test on top of different cloud providers and or even using different container runtimes.

A

We had issues reported with specific container and times before so, but apart from these areas, I mean improving on the matrix testing. I think tecton is in pretty good shape.

B

Yeah, I I agree, I think, overall, we feel pretty good about how techton is doing when we look through the lens of the dora metrics we're a medium performer and we're able to use that to identify that if we increase our release frequency, it's a pretty clear path to getting to high or maybe even elite performance, and by looking at some of the best practices, we could see some clear areas for improvement, specifically around the periodic and matrix testing, which would probably have a positive impact on our weakest metrics around the change failure rate.

B

So, just looking at our projects through these two lenses was really informative and as an open source project in particular, I feel like there are always a million things that need to be done. So it's really helpful to see how these metrics can show us, where it's worth focusing our attention and we're willing to bet that you'd find these lenses just as insightful for your projects. Even if you just started with measuring your deployment or release frequency.

B

That would be a great place to start, especially since it seems to impact all of the other three metrics.

B

So maybe take a look at your project measure the release, frequency and you can find out. How does your project measure up? Are you an elite performer or do you have a ways to go.

B

And that's all that we have for you today thanks very much.

A

B

Thank you, andrea.

A

D

B

A

Hi christy, hello, everyone.

D

Well, so I think we have time for a few questions and it looks like we've got one from adam already. I wish that.

C

um I probably both wrote scripts for this that could more or less be applied to any project, but I don't think they're mine at least is not in like a production-ready sort of state.

A

Yeah, well, we we do have some streets and I think, uh maybe not production really but they're.

A

I think, based on github api a lot, so you can definitely share those um if you want- and um in future um we'd like to we thought about, we could uh start using events cloud events, at least for the services that are running like hub and results that we run them as services. So we could collect data about installing releases from there.

B

Does anybody else have any questions? It looks like we got a few people filling out the polls. um Half of the respondees are using tekton already. So that's pretty cool.

A

D

A

Already started measuring their devops metrics trying to get started, which is quite.

A

D

Go ahead, go ahead.

A

If I were to ask you a question, is um let's say I want to contribute to tecton? Where do I get started.

B

Well, um we have a community repo that has a it should have answers to most or all of the questions you might have about that. It has information about our slack. Our mailing lists our working group meetings um and how to propose projects how to find something to work on um we'd be more than happy to have anyone who's interested in joining, um oh and adam has a link to, but oh shipwright, uh auto draft release notes oh cool yeah. We we have something that auto drafts release notes as well. I don't know.

B

Maybe this goes a bit further. um It seems like maybe there's there's definitely a tool. I think that people will be very happy to have around this. Ours is just still a little bit manual in the I don't know. Everyone has to write, really good release, notes right away in order for the. When are we doing it automatically to work? I think there's always something that we need to tweak a little bit, um or we just embrace that. I guess and just allow them to be a bit flawed.

B

um But thanks for the link.

B

I think we're at time so does anybody have anything else? They'd like to oh wait, one more question: you mentioned cd best practices. Can you mention a few of those? um So there are a few in the slides that are linked a few mentioned at a high level in the slides that we link to there's also a best practices, special interest group as part of the cdf um which you might be interested in, which is trying to kind of nail those best practices down.

B

I think there's a get repo with more information and there might be a birds of a feather about it as well, but I'm not sure if we've missed that already. But here is the information on the on the working group. If you want to join, but at a high level, I think version control is the very first thing. You definitely need definitely testing and then there's kind of more that you can add about testing um like degrees of like unit testing integration, testing system testing, um definitely keeping your source code, always in a releasable state.

B

So you have version control and you always feel like you could release at any time and then from there automate making it possible to easily do the release. um So you might not go as far as continuous deployment, but at least it should be. It should be as simple as hitting a button to actually make the release. Those are some that come to mind for me, andre. I think you had some more that you mentioned in the talk as well.

A

um Well, we had some about yeah testing and kind of coverage, testing matrices, uh different testing, a different combination. In our cases, I guess different combination of kubernetes, for instance, a different uh container, runtimes or different configuration of feature flags, and also keeping a top on your flanks, so in the test, results and tracking test results and making sure that you're on top of lakes and also nightly test periodic tests.

A

So if you have more long-running tests that you cannot fit into the like standard ci to have those running on a regular basis and collecting results and more to them,.

B

And thanks terry for the mention, the birds of the feather for sick best practices is tomorrow. If people want to join question can tecton replace jenkins in the future. um Maybe I think I think our ideal might be more like jenkins interoperates with tecton, so you could use what you liked about jenkins and what you like about tecton and I believe that there's actually a plug-in that lets you use tekton from jenkins.

B

So I think our ideal would be less of a replacement and more of a working together.

A

Definitely I was for interoperability and I think there is a um jenkins contributor meetup on friday or something there is going to be uh talking about the jacob's detective plugin. If you're interested in that.

B

There's a jenkins blog post about techton and oh and jenkins x from jenkins. Oh thanks drop.

B

Great well, that might be it.

B

Well, thanks everybody for coming um hope to see you in person one day, but this is this feels very similar. So and- and now I can go uh I'll- be in my house afterwards, instead of wandering around a conference, for I don't know, if that's better or worse anyway, this is recorded. Isn't it oh okay, I'm done everything's good everything's, great.

A

Thanks everyone.

D

A

Thanks christy bye.