GitHub GitHub Universe 2020: Developer - DevOps, 10 Dec 2020

Previous Meeting

⏯

youtube image

►

From YouTube: Visual tests on every pull request - GitHub Universe 2020

Description

Presented by Angie Jones, Principal Developer Advocate, Applitools

You’ve created your GitHub repo. You have multiple contributors. You’re reviewing the pull requests. All the signs of a healthy project are there...but what about the tests? Not only do you need tests to ensure your application isn't regressing, but if it’s a frontend application, you'll need visual tests to verify that your application still looks right. In this talk, Angie Jones will discuss AI-powered visual testing and how to integrate it with your build process using GitHub Actions.

For more from GitHub Universe 2020, visit https://githubuniverse.com

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

Hello, everyone and thank you so much for coming to this session today, I'm going to talk about visual testing, but first I want us to start off with a little game of who done.

B

It clearly somebody in this room murdered lord smythe, who at precisely 3 34 this afternoon, was brutally bludgeoned to death. With a blunt instrument, I want each of you to tell me your whereabouts at precisely the time that this dastardly deed took place.

C

I was polishing the brass in the master bedroom. I was buttering his lordship scones below stairs son.

D

Why I was planting my petunias in the potting shed constable arrest.

B

Lady smythe, but how did you know, madam, as any horticulturist will tell you one does not plant petunias until may, is out take her away. It's just a matter of observation. The real question is how observant were.

B

B

Clearly, somebody in this room murdered lord smythe, who at precisely 3 34 this afternoon, was brutally bludgeoned to death. With a blunt instrument, I want each of you to tell me your whereabouts at precisely the time that this dastardly deed took place.

C

I was polishing the brass in the master bedroom. I was buttering his lordship scones below stairs and so.

D

Well, I was planting my petunias in the potting shed constable arrest.

B

A

Smythe, so did you catch those differences or maybe the most important one where the dead man got up and walked away?

A

Probably not right, because we all suffer from what's called unintentional blindness, and this is where we, we don't catch new or unexpected things that come into our viewport, because we're so focused on a specific thing- and this is especially true when we talk about testing our software right, we're so focused on just getting the code out. We might do a little bit of testing to see if the absolute happy path works and we miss a lot of other things that are going on around us. So how do we solve for this?

A

Well, typically, we'll automate our tests, because we figure, if we automate these tests- and we know it's going to take the same steps every single time. It's going to make the same assertions every single time and that way we're safe right, ah not so fast. What? If I told you that by automating your test, you're actually scripting in the unintentional blindness?

A

Why? Yes, because most of the automation tools out there are interrogating and querying the dom for answers. But what about the look and feel of the application? What about the gui that our customers are actually using right, for example, what if some text on the page that we're looking for we in our automated test? We ask: is this text present and it says yes, because it's looking in the dom? But what about? If that text was the same color as the background and therefore couldn't be seen by the users?

A

What, if some element that the customer needed to interact with was overlapped by another element and therefore they couldn't? What? If, when we start changing those viewport sizes, the element or the text or something is bleeding off the edge of the page? None of these things can be caught with the traditional test, automation tools. Now, when I talk about this people start thinking that these are edge cases. But let me tell you once I was trained to uh discover visual bugs I started seeing them everywhere.

A

Let me show you a couple of examples, so this is cinemax. This is the second largest cinema chain in the planet, so not a little small mom-and-pop shop and they had a visual bug on the page that asked you. If you want to store your credit card information so notice there, the two labels are not aligned with the radio button and when we see this, it looks kind of goofy and we might want to laugh and think of this as a lower severity bug.

A

However, let me ask you this question: would you store your credit card information here? Probably not right, I sure, wouldn't why not, because in the back of our heads, we don't trust this page with our credit card information. When we look at this and see this, we think how do they miss this right? They must not test this, which is what kim alluded to in this tweet here and if they didn't catch this part, I know they're, not testing. The back end of this is what we think, but I have a theory.

A

I think they did test it. In fact, I think they automated the test for this, and if they automated the test for this, those labels are present. The radio buttons are present. Technically, this is functional, and this is how this sort of thing gets shipped to production.

A

Here's another example open table, so open table is where you go to make restaurant reservations and I used to live in north carolina home of the barbecue, and I went back to visit.

A

I wanted to catch up with some old friends, so I went to open table to make a nice sized reservation for seven people and they gave me some times to choose from and when I chose the time a model appeared- and I was a bit confused at first because there's nothing centered here but as I got to looking around notice in the upper right corner, there there's two select buttons and on those two select buttons, there's nothing. That's telling me what I'm selecting right first of all like. Why is there two of them?

A

Why are they not aligned and.

A

What am I selecting you know, so I did what any of you would do and I opened chrome, dev tools and began to dig around the dom and when I did so, I realized that hey there are labels present there they're just as far away from those select buttons as possible.

A

So I wanted this barbecue. I went ahead and made the reservation much.

D

Like our automation,.

A

Tools would be able to do because they're interrogating the diamond interacting with the elements via the dom then they're able to complete the test, but as a user not so much right. This is pretty much unusable from a customer standpoint. Our customers aren't necessarily engineers they're, not digging around in chrome devtools. To complete your reservation, I cringe thinking about how much money is lost here now. These aren't just one-offs. This happens to your favorite tech companies as well. Look at this one from amazon. Like I mentioned earlier, changing the viewport size. I promise you.

A

Those mobile wheels is where those visual bugs live. Okay, so right here not only is someone about to buy something, they want to increase the quantity, so they are multiplying the amount of money they're going to spend and they're. Faced with this visual bug, right almost makes it unusable if it's more than two who knows, if they're going to complete this transaction or not, but if we had tests for this, this would pass also look at facebook here.

A

So this is their marketplace where they're going to buy products. Look at the text there! Oh, my goodness, the width is not wide enough and the text is down the edge of the page. You have pictures here where you can barely read the text, but what would our automated tools see? Yes, the text is present and everything would pass, although this makes me not want to spend my money I'll tell you that much and then here's another example, one of my favorite ones. This is instagram, and this is a sponsored ad.

A

This is content that someone paid to have premium. You know placement on instagram here and yet it looks like this again, our automated test. What we ask is this text present. Yes, it's this text present. Yes, nothing indicates that it's actually overlapping each other right.

A

How much money is lost here if I'm buying this ad, I'm no longer going to do it with instagram. In fact, I want a refund on this one right, twitter, look at the tweets overlapping. You see this. What do you do you close the app even google on this pre-order button here right and this one is not so bad, but it is still a point of purchase, which means some high salaries. Google engineer needs to stop innovating new features to come and track this down, not a good use of their time.

A

The great news is, we don't have to live like this. Visual testing can be automated. So what is visual testing exactly well, visual testing is where you know when you, you know you're. Writing your your initial test and you're. Looking at the app as a human being right there, we can take a screenshot of your application, that's in its golden state and we'll consider this the baseline and then for every regression run it's going to take another screenshot and compare those two screenshots to detect.

A

If there are any visual differences, you know the stuff on the screen that our customer is actually seeing. That's what we're going to look for right now, visual testing isn't entirely new. It's been around for a while in the industry, but as an industry we weren't doing this right. We were using pixel to pixel comparison to do this. The problem with pixel pixel comparison is extremely flaky. It was not designed to test software now.

A

Here are some examples. In the first case, this is our baseline image and the second case is the regression run. The regression run failed this gated deployment, so we didn't get a deploy. Why did this fail? Well notice in the second image that google search button is bolding? Well, if you put your mouse over that button, that's the hover effect of that button. So this is automated test. Automated software there's no human involved here, which means it just so happened that the mouse landed right there right.

A

We don't want to write code to make sure the mouse isn't landing in certain places. This is annoying. Here's another example who can see why this one failed yep. If you said that there's no cursor in the second one you're right, the cursor after bad is solid and the second one is not well cursors blink people right. So you don't want your deployment to fail because the the screenshot was captured. You know when the cursor was blinking or something like that.

A

Goofy right, the great news appletools, has come up with a much more sophisticated approach to doing this. It utilizes ai, to mimic the human eye and brain and be able to compare these images just like we would as a human being. So here's another little game, you all don't get to play because you let the dead man get up and walk right.

A

So I ran this through um pixel to pixel comparison, and then I ran it through appliques api, um their ai algorithm, and when I ran it through the pics of the pixel engines, look at this. It flagged everything. It's so sensitive. Every little white space shift gets highlighted right. This is not built for testing software versus when I ran it through applique's api. Look at that, just as we would. It captures the differences okay. So we're going to add this to an existing test. Just so, I can show you how it works in action.

A

So here is the application under test. It's a simple single page, app um eight of my favorite books on testing and the way this works. Is you would type in uh some query here? I type in test, and this shows all of the books that have the word test as part of their title, all right. So the test for this. This is written in javascript with cypress, but everything that I say today applies to any language you're using any test framework, so don't get caught up on that part all right.

A

So here's the test we are querying for the word test. We are expecting these five books to come back and we assert on that by making sure there are five exactly five books returned and then we make sure that the titles match the ones there in our array. Okay, now let's say I wanted to add a little bouncy effect with css. Unlike you all I suck at css right, so I mess up things all the time, but I have tests so I can do whatever I want, because I have tests right.

A

So I just check this in and I know that it'll catch. If I do anything goofy right, okay, so here's the test again look at it and then this is what the application actually looks like with that test catch. This no look at this test. It's green! I love green like the next person, but I want my red when, when it means something you know so this test has passed even though, with a single line of css, I literally flipped this application on its head again.

A

This is why we need the visual testing, so I can revamp that test right.

A

Not only am I getting rid of a lot of code, but I'm going to have more coverage here with the visual testing, so there's three simple commands: the api is called eyes beautiful, so my first command on line three is to open your eyes: ah poetry right, the next command. I say to check the window. This is the magic. This is what's going to take the screenshot. It's going to send that up to the aptitudes cloud that if there's none there already that'll become our baseline.

A

If there is one already it will compare to the baseline and then the final command is closurized. We're done here folks. So let's run that one against that flipped application and notice. Now we get that red failure and we can see here in the aptitudes dashboard what's different, so it highlighted the differences for us beautiful, okay. Now the thing about visual testing in that way is that it's capturing everything. Yes, we've totally eliminated the unintentional blindness there, but sometimes we don't want to capture every single thing.

A

We might want to just scope down to a certain area of the application right, so I can do that in that check window command by just passing in the target. So I want a specific region and then a css selector for that region. That way, it'll only take a picture of that specific region. So here I only get this book right and this is great uh definitely scopes it down. However, we kind of reintroduce a little bit of unintentional blindness when we scope it that way.

A

For example, here I'm typing in agile testing, but none of the books went away because filter is broken. Now and our test would not catch that because we told it find that agile testing book and make sure it's beautiful. Well, it's there and it's beautiful and it kind of misses everything else. That's going around it now, you're still in control, we're utilizing ai, but you need to be the boss here right. So we can enhance our tests.

A

We can mix and match visual assertions with functional assertions, so on line three there, I'm going to slide that assertion back in here and say make sure that it's only one book that comes back and then go ahead and visually verify that book all right. So that gives us the strength and the power that we need. Now. Some people will ask me about dynamic content and say well. Angie.

D

A

Static data, so maybe mine, won't look so great.

B

A

I can't use visual testing to verify it wrong. You can so aptitude has four different algorithms right um to do the comparisons. The first one is strict. That's the default one! We didn't have to specify that one. The next one is exact, which does picks with a pixel. I already told you pixel to pixel sucks you're, not gonna, use pixel, pixel right right, great. The third one is content where it goes colorblind and it only verifies the content on the page and not necessarily the colors, okay and then the last one is layout.

A

This utilizes ai, in a different way, to make sure that the layout of the page is okay right, so not necessarily the content, but it detects a pattern on the page, so, for example, with our books in the baseline image, I have advanced, selenium, followed by cucumber and then in the regression. I have cucumber followed by advanced selenium, because I use the layout mode. This will pass it's just making sure things aren't overlapping or looking goofy there all right, in addition to doing it on a specific browser.

A

This can also work cross-platform.

A

So apple tools has this ultra fast grid, where you can specify what configurations you want to run against and what this does is it takes your javascript, your dom your css, basically the state of your application blast it across all of the configurations that you like to because remember, I told you visual bugs they lie in those um uh viewport changes right, so it'll verify all of that for you in parallel lightning fast to do notice. Here, I'm specifying.

A

I want firefox chrome in ie, 11., there's also support for edge and safari, not only the browser. I'm saying I want iphone, 10 and landscaping portrait mode. I can do iphone 12. I can do galaxy.

D

Whatever I want to do.

A

My test doesn't have to change, so I don't have to go back into all those tests and update them. This is working perfect, and so then I can see how my application looks. I don't have to download this stuff. I don't have to have a device form. All of that is handled in the cloud for me, and this is what it looks like on my dashboard notice.

A

There we have those devices in browsers and configurations also integrates seamlessly with github actions, so you can have these visual tests run on every single pull request and you don't have to worry. You can actually do whatever you want, because you have tests all right. So that's visual testing. In a nutshell, what questions can I.

A