Continuous Delivery Foundation cdCon 2022, 24 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The State of Art in Tackling Flaky Tests - Kohsuke Kawaguchi, Launchable, Inc.

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

The State of Art in Tackling Flaky Tests - Kohsuke Kawaguchi, Launchable, Inc.

Flakiness is a part of the reality for software engineers everywhere, it refuses to go away. This is happening in your team whether or not you know it, it caused frustrations, failed pull requests, added more stress to a hotfix delivery. Come listen to Kohsuke Kawaguchi, the creator of Jenkins to learn how software teams around the world has tackled this problem, from “idol” companies like Google & GitHub, to the “next door neighbor” company just like yours, so that you can tackle this in your team, and see how this is a part of a bigger emerging movement.

A

This is kosuke and I'm from launchable. So today I wanted to talk about the flaky test problem, which I think most of us are familiar with. So in one of the blog posts, google reported that one in seven tests have some level of sickness, even in their. You know, even among google engineers now, if you're a developer, while the quantification is great, this might hardly be a news to you. After all, this is something everybody I think faces in my own experience working on the jenkins project.

A

I certainly had my own share of the fitness program, so let's first set up this conversation by recapping. Why the flaky tests are a problem, so the first, I think, is what I call the crying wolf effect. You know the point of the test is to signal that some problem exists in the code, so when the flakiness happens, meaning the failure happens, even though, when there's no really the all underlying cause, no real underlying cause, then you know it creates a problem the. What is that problem?

A

It's primarily a lost confidence, the system and process right. So this system that's supposed to catch. The problem, while it shows that it's not happening so that results in people slowly like starting to ignore these processes and systems, and if that's left unattended, it can result in loss of confidence to people who are working on those meaning. You know you the devops and years, and aside from this, you know the trust angle, there's also personal toll to the developers.

A

Right I mean how many times you were hoping to march the pull request before you head back home only to discover that the some of the tests failed, and then you know that these are just flaky failures. So what it forces you to do is to re-run the test head back home, maybe attend your family and then later in the night. You're gonna have to come back logging into the system and hope that the tests are fast and now you can mars the uh march.

A

The change and there's also the flip side of the fear is that if you ignore what you think is a flaky test, this could actually be a real problem. In you know, software under test, maybe you're having some rare race conditions or some sort of non-deterministic behavior. That happens really, but it could happen in production. So if you just blindly ignore this flaky test, you could let the bug slip into production and cause a real harm.

A

So you know if everybody has this problem, so what have we done collectively in this industry to tackle this problem? So that's what I wanted to do in this presentation.

A

So the first thing that helps is to know your enemy, so this also comes from the same blog post from google, in which they quantify the correlation between the amount, the memory, a test uses versus the chance of thickness- and you see a very nice linear correlation between those two things and in some sense this is also not the news right. We know instinctively that the larger test, which touches a lot of things, tends to have more flakiness.

A

But you know again, the quantification is great, because we can also quantify how much effort we can spend on it. The other thing that the people looked into is what tends to cause the flakiness, um and then there are two leading causes mentioned by some of the well-respected engineering team that I was able to find. One is time related, so apparently, at the end of the months or beginning of the month or some interesting alignment of stars in the calendar can sometimes lead to the strange behaviors.

A

This is actually something interesting to me, because I never had this kind of thickness in my test, but it could be just me well, he apparently is just me.

A

Another kind I think is is is far more common and certainly is familiar to me, which is a test altering a global state of the program in unexpected way, and so it creates this. You know, reliance decide certain test order or otherwise known as the test interference, so that can create some strange failure when certain things are run in whatever order.

A

The other thing that's well understood and known is that the thickness is not evenly distributed, so our number from google said one in seven tests, but they also report that the only 1.5 percent of the test runs are flaky. Also, that's a huge. You know difference in the number right.

A

So what that means is while many tests, one in seven tests, exhibits some level of thickness, there's um only a small number of tests that fail a lot. So most of these one and seven tests really only fail very rarely so github put the same thing into a much easier graph. They understand so here you know in a given time span. They looked at how many flaky failures happened from a test case, so you can see that whopping 20.

A

I think the 24 of the tests have failed at least once during this time period due to sleekness, which is pretty high, in fact it's higher than one in seven, but you can also see that there's less than one percent of the test that failed more than 100 times during the same time period.

A

So this gives you hope that if you can focus your effort on this small number of very flaky tests, then you can make a big impact without boiling the ocean.

A

So now that on this now that we understood what was done um in I'm sorry now, what was uh now that we understood what are the fakie tests, what we know about flicky test now, let's move on and look into what's being done to tackle this challenge, so it turns out that the there's a lot of things have been attempted by a lot of engineering teams, and many of them have publicly talked about this, so you can go, read them to figure out well and then try to get some inspirations, but in reading those materials I came up, I started to recognize that there is really three: you know: introduce three um intra-related pillars in addressing the flakiness and that's keeping builds green and then measuring flakiness and escalating thickness.

A

Well, I call them that way. So the reason these three things validated is that the fourth and the primary purpose of the delivery pipeline is to make the changes get to the production. So flakiness is kind of like a friction noise in this system, so the first order business therefore becomes. You know making sure that the changes are landing into production smoothly. Despite this heat and friction, so that's the that result. You know that's the keeping build scream and that, if you just ignore this flakiness, then it tends to get worse over time.

A

So if you now need to measure how the overall flakiness that you're, also masking at the same time, by keeping this green and quantifying where the flakings exist and how bad they are, and now that we measure those things you can actually derive the you know attention of the developers to the right parts of the thickness, which is the escalation part.

A

So now that, like you know, we divided the problem into these three spaces, I'm going to quickly look into what's being done in these three separate areas, so, first keeping those green well, not too. Surprisingly, the the most common approach to this problem is just you know, retries, so um at every layer. So this is a the gradle plugin for retry. The similar functionality made and jenkins had its own plugin to you, know tv, try, the tests and so on and so forth.

A

Now, interestingly, some of the more sophisticated dram mechanisms do things like in a running test with a different time, which I guess this speaks to the earlier point that the sum of the common flakiness caused is the time related or move the test in a different virtual machine, and we run it automatically, which is you know, I guess it also makes sense. It's like trying to shake things up and then run things in a cleaner environment.

A

There's also a difference between how transparent, already noisy it is for the developers. You know if you're creating what I call a smaller bitrate loop, for example, in the test runner most often developers will not even see that these bitrates have happened.

A

uh Some of the you know the jenkins plugin that I mentioned well. They actually make you press a button to explicitly kick off at each right. So that's more of a visible approach to um to the thickness.

A

And so the downside, um the the cost of lead rise. Of course, while this is easier for us to take, is that one? It masks? Fakeness and developers now often do not even see that the test flakiness are happening so which necessitates the the measurement which I will talk later and it also delays the feedbacks because often the literary account gets pushed up to whatever um people find it necessary to deal with the most flaky test.

A

So when there's a legitimate failure in let's say one of the long running test cases, so if the retro account is 10 minutes for test integration test that takes three minutes to run, then you know that failure takes whopping 30 minutes just by itself before it can be correctly judged that the test has failed. So there's this, you know a visible real impact to the developer productivity that comes from retrying.

A

um The you know, the the another approach that I think is actually relatively there is just the ignore of leaky tests. um So, for example, some of the engineering teams reported that they you know they don't. Even if the tests are marked as flaky, they don't even run them in primords, where the feedback clip is the most critical and the you know. These teams still run those tests at the post mortgage phase, where the failures are treated more carefully because there's a dedicated team that looks into these things.

A

Now, when I talk about this flaky test problem is the developers they. I often encounter this psychological barrier that the people really hesitate. They mask the thickness. They want to feel that the developer should be held closer to the fire, by which I mean the failures that happen from weakness, which is understandable because they fear that the flakiness will be left unattended, but at the same time I feel it's not necessarily practical, as we saw in aubrey or even engineering teams like you know, google and github, they cannot eliminate the sleekness.

A

So um you know it kind of makes you wonder if it's practical to try to you know, force developers to eliminate them or they have enough capacity. I mean sorry, not the capacity but the space uh or the bandwidth to eliminate those.

A

Another kind of painful aspect is that the these tends to be these things tend to be collective punishment. So often the person who gets penalized by the flaky failure is not the person who has introduced the fakeness so that often creates you know that doesn't really work in terms of the incentive structure.

A

So I think more and more people in east, you know be practical and kind of stop being in ideological, about uh making developers see the flaky test all the time. So, okay, so that's about the keeping bills green. So now that your you know, bills are going smoothly and that tests are running well. Tests are running, green and changes are going intimacy.

A

What can you do to measure the thickness? So it turns out that the most of the you know the really naive approach- and this, I think, is actually the most common you know. Current state of the art is what I call the tribal knowledge.

A

So you know if you work in this team long enough, you kind of know what tests are fakie. So when somebody new sees a bunch of test failures and uh the senior engineer sitting next to them might say well, you know you probably don't need to look into it. Just redone the test, because you know it's it's beam. It's been flaky so step up from. There is what I call the baseline and uh there is surprising uniformity in how people are doing this.

A

It is you know if the same code is tested multiple times and if they result in this different behavior, then they are marked as fakiness.

A

So you know, google etc cetera this way, and you know when launchable computes, the flakiness for customers, we also do it this way, and then you know the interesting thing about these things, as I learned in this in this job is often like this crude.

A

What I almost think of as like a crude mechanism to compute these metrics works surprisingly well, and the incremental gain from doing more sophisticated thing is often you have to be small, but anyway, um the shortcoming of you know. This approach is sometimes the test fail. uh The mass because of some let's say, external dependency is gone like the network is not working, then you know the you stop the test.

A

You fix this problem whatever outside the test environment and you rerun the test so you're, getting the same same tests, run against the same code multiple times and now you're getting the different results. So in this definition it counts as a flaky failure, which is probably not what people typically think of as a test flakiness issue.

A

So the spotify engineering team, like I, did an interesting twist, which kind of makes sense, which is that if these you know tests are fading a must, because of these external reasons, you see this correlation between multiple tests fading all together, whereas the true sleekness, you know, you expect them to show up in more or less a random manner, so they they use the correlation with other failures. They kind of you know fine tune and determine the thickness, which makes sense.

A

Another interesting paper that I saw was from facebook. They use the process known as the bayesian inference um and uh determine so mathematically determine what are the probability that these failures are caused by the thickness. So this is kind of uh I find this like a very facebookish approach. You know they must have hired a phd from uh some graduate school and uh you know if they gave them if they gave them this problem, then it's probably.

A

This is the kind of thing you find so the way they model the world is okay, so that when the test is run, there's a certain probability that the you know, because the code is bad, the test fails um and then, even when the code is good. So when we expect to test it pass well, nonetheless, it could still fail, let's say because of the fakiness.

A

um So the only time you see you know the the other case is the only time you see the pass. So if you model this process like this, like you know, the only thing you observe is past and failure, and you can't really differentiate those two kinds of failures as an observer, then, and then, if you're, given certain sequence of events, so these you know different builds are on the x-axis and tests are running and their results are stacked up.

A

So here three tests are the same tests around three times on this build and they all failed. And here it's like you, know, two failure in the one pass and and so on so forth and from there you can determine what combination of you know. P, f and p b are likely to produce this kind of um uh of observational behavior. So here you know they're like doing this multiple times and plotting out, and you could see um in this case the pf. So this is the chance that the flakiness is happening is about.

A

um I don't know like a 15, maybe 20, something like that, um and then the uh the the probability of real failure is about five percent. So you can say it's probably more likely. This test is more likely fading because of a thickness, and you can kind of quantify that so it's a nice statistical treatment to the problem um so so now is the measuring of thickness. So in this approach by you know, by using some of these approaches, you can see what tests are of the key.

A

So the last step of the peter is you know now that you know which tester flaky, how do you get the developer's attention?

A

So the naive solution, I think, is the that that is just the you know the delete the test, uh basically or well like a create a ticket investigate and then maybe you skip the test or you remove them, uh but you know I I just find people are generally hesitant. They remove the tests, remove the fakie test.

A

I guess it's because um you know people are feeling naked about security and then they um there's a psychological barrier to kind of give up right, but um but at the same time it's you know if you strictly think of this from, like a cost benefit an roi kind of perspective, it might not be worth it to have this test. That creates this lots of noise and have cost to maintain, um and then that might overlay the benefit that you're getting out of this test.

A

So some of more sophisticated engineering teams have done. Let's look at those, so this app is built by the github team. So, as you can see, I mean you can see the links in which they have gone to the they tackle this program. So you see like they came up. Is the impact score. um They said. You know the system can report something like okay, the 2700 bills are affected by this particular kind of flakiness.

A

um That impacted, you know also many developers. You know when this has been seen. The likely cause had to be produced. They even tried to pin down who should start looking into this by adding some kind of bisecting and whatnot. So you know that's and so uh yeah. So this is amazing um and uh another one. This came from, dropbox is in which, when the test is when the test fails, then the test is tentatively marked as noisy. So it's like a in a yellow state.

A

You don't know if this is um the flaky test, or you know a real test. Failure due to the problematic code change, so the difference is normally in most people's environment. This noisy test, this failure that led to the noisy test is going to be seen by developers, but here in this theme this is only seen by the system and this triggers a whole bunch of activity by automated systems.

A

So you know the they. They just automatically run this test multiple times. They also go back in the earlier version of the software, and then you run the same test this and if they fail, um then, like that's a you know, that's a flaky behavior. So by combining these techniques after a while there they'll be able to determine that. Okay, this test is indeed flaky, in which case the test case is put into the quarantine state and they no longer participate in the primords or if the.

A

If this you know, failure is determined to be legitimate, then the change that resulted in this failure. It's going to get backed out, so it failed the you know it failed the test so um that you know they combine techniques like you know, git, bisect and multiple test runs and so on by the system even before humans get involved. So that's you know I think another.

A

I admire this direction in which you're adding more layers and shielding people from getting all sorts of noise.

A

So, as I look at these things, you know the the thing that connect. The question that that I developed is so after all this much effort right, like you know what are devs doing with the escalated fakie test like. Are they actually, you know? Maybe convinced is this all just making people feel better to remove the test. Then you know is that, is it necessary to do all this much work or is that actually creating some kind of behavior change on the ground that I'd be interested in finding out?

A

But so far I hadn't had the opportunity to, and finally, aside from these three pillars that I talked about, there are some noteworthy mentions um that doesn't really fit into any of those.

A

It is, you know the architectural approach so like airbnb is this team in airbnb develops a mobile app, and you know those are notoriously uh hard to test and the tests tend to be flaky, so what they did was they came up with a mobile, app development, mobile, app framework, in which you know I'm going to spare you with the technical details, but they are designed to be testable at the foundational level, so that allows them to make tests very deliberate and fast.

A

So that's you know like that. This is the kind of things that you expect when uh the qa function and the engineering function merges right, so the developers are no longer throwing the stuff over the wall. The the testability gonna give the fundamental change in the way the app is developed in the first place, so you know closer to my home turf. Well, the you know.

A

In java, the ioc framework, like a dependency injection, is in part motivated by this kind of testability, so that it could throw in the mock- or you know, create a test environment in which certain small set of components exist and exercised, so um that this, I think, is the finest example of uh so-called devops.

A

So so you know, as I was reading these approaches. The other thing that started occurring to me is that there's this clear direction that the you know well, the test execution, let's say used to be simple right. You have a view tool, you run the shell command and it runs the test and it spits the result, and that was it.

A

But what I saw across this trend is that there is more and more sophisticated, computation and calculation happen on top of these test failures that lead to a lot more insightful input for the developers that do not waste their time. So it seems to me that this space is kind of dividing into two segments. I mean the two two different areas. The one is, you know things that is local unique to the individual development team.

A

So I call that the build versus spy just because nobody else can solve the problem for you- and here this is like a key building block capability is to be able to run arbitrary test against the arbitrary software under test quickly at the end of the scalable manner. um So this you know sounds easy, but it turns out that a lot of build tools can make this unnecessary hard.

A

So, for example, you need to be able to separate the build step, the act of turning source code into binary from the running the tests, um and then you need to be able to run the tests outside this like a typical delivery type right. So that's another you know pulling out um and then it kind of tends to point to the direction of you having lots of distributed fundamental test runners mentally picture like a container, think you- and it is things like that.

A

So this is you know one direction, one area that that needs to be built out and then this all this additional intelligence that it can be built on top of it. You know like this system, where you trying to get bisect and figuring out what test needs to be rerun and the quarantine all that stuff. That doesn't really seem like it's unique to a different development team and um it seems like the kind of sophisticated thing that not every engineer team should be building, and rather they should be.

A

You know again quote-unquote buying, by which I mean your industry should come up with something that's general purpose, so now that that's in some sense driving my mission that just like when I created jenkins, I saw the opportunity that well, this is not something everybody needs to be built over and over. um So I'm expecting and I'm hoping that there is this.

A

This split emerging in which a portion of it, this low capability of running tests against an arbitrary binary, needs to be exposed from your team, but then how to use that the brain part that drives those is to figure out what test needs to be wrong when with what the software to achieve whatever prophecies that will be reused across the industry.

A

So it's not. um This is the end of my talk and uh you know I left a lot of link in this slide if you're interested in getting the size just drop me a note that slides at launch going to come and I'm happy to send you the uh the you know the live version all right. Thank you very much for listening have a great day.