Hyper Cloud CharlestonJS, 27 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Automated Visual Testing - by Gleb Bahmutov at CharlestonJS Meetup

Description

Learn the ends and outs of visual testing from Gleb Bahmutov a distinguished engineer at cypress.io

Cypress.io is an End to End testing framework for all things on the web.

Slides:
https://slides.com/bahmutov/visual-testing-using-cypress

Want some CHSJS SWAG - check out our new t-shirts here:

https://shop.hyper.io/#/collections/charleston-js

A

A

Okay and the meeting has been excellent.

A

Okay, visual testing using cyprus is a topic of passion for me because I love visual testing. I think it's extremely effective and when you combine it with functional end-to-end testing it it could do wonders um you can find me online. I've been working at cypress itself for four years, and I've been using cyprus for a year before that and.

A

Okay, and if you want to argue with me or agree with me, uh you can find me on twitter before we begin. I would like to kind of start like I usually start with a reminder that, despite covet, we have not slowed down greenhouse gas emissions, so we probably will hit 1.5 degrees celsius in the next five years, every degree after that probably will displace one billion. People like the stuff is extremely serious and you know like, fortunately, all. Unfortunately, the answers are surprisingly simple: we have a technology to eliminate 90 of greenhouse gases.

A

We just have to act. You can change your life but, more importantly, you're an organization. There are lots of them. Fighting for survivable planet find the one go to the meeting, attend it virtually do little things and together we can solve it.

A

Okay, slightly back to the topic, I had a long and pretty interesting career where I went from computer vision cc, plus all the way to to browsers, and then I start doing everything with javascript.

A

You can find all my projects on github. I blog a lot and I do argue about technologies. A lot on twitter, all my presentations are on slice.com and these slides are there so they're public. You can take a look at any time.

A

Okay, uh so the cyprus itself is a great company. We grew it to about 50 people, I'm in boston, but majority of people are remote or are in atlanta, that's where it was founded and we work on cyprus to make it the best end-to-end test run for anything that runs in a browser. You have a static site. You should be able to test it.

A

You have every framework that you through you know in that list, including you know, dojo, and you know, google g double key can be tested using cyprus if you never use cyprus before it's as simple as npm install the save dev cyprus.

A

So it's a normal npm installation just like any third-party module and when you install it actually downloads a binary that we built for your operating system. Okay, so we built it one for.

B

A

For mac and for windows, 32 and 64-bit, and everything I'm about to show, you is absolutely open source and under mit license. Okay. So there is no commercial presentation in this in this slide and if it's the first time you install cyprus, you probably want to look at our introduction to cyprus to understand. You know how typical tests is written, how it runs and so on. But this is full end-to-end test right.

A

We use mocha syntax, so we use it and the name of the test and the callback are all commands that you want to run in a real browser that simulate how a user would visit this page. It could be localhost or remote host. Then you will find something using css. Selectors in this case find all elements with class new to do.

A

In this case, it will be only one input box and when.

B

A

Type learn testing followed by.

B

A

Then we type more so we use kind of jquery flowing chaining syntax.

A

If you want to work with element, you just chain commands everything is uh of this global object side that we inject automatically into a browser um after you enter a couple of to-do's in your to-do mvc app, you probably want to get all list items and confirm that there are two of them so load real page enter a couple of to-do's make sure that the application is working and displays that number full end-to-end test, and this is how you know it runs and looks in your browser.

A

So cyprus comes with built-in electron browser, but if it finds any browser installed on your machine, it will use those browsers. You know firefox chrome edge, not safari, though, and then when it runs. This is the browser window, but you see on the right. You see the iframe with actual website under test and it's a real browser. You can interact with notification on the left.

A

You see the test itself so remember we named the test adds to those. So you see the name of a test on the left. Then you know you can use your hooks for visiting, but you can see the commands visit, get selector, then type type again and then get selector and green assertion. So everything from this spec file will be reflected in what we call command log on left.

A

So you know what the task is doing, as you see the application changing on the right, so that was cypress, introduction, right and and for more you can find cyprus documentation. But for this presentation I want to specifically focus on visual testing. Why is it even needed? What are the main principles for visual testing and its implementation?

A

And then we look how to do visual testing yourself a little bit and especially for complicated things like how do you test html5 canvas elements?

A

Okay, so let me ask you a question and you know here's uh amazon.com, and I want to ask you if you can even log in into this page notice, that the sign in and the label for the you know for the email is slightly misplaced.

A

So it actually covers part of the input box and if it covers, but maybe you cannot even focus on that input box or you cannot type, can you sign in but more importantly, would you even try to sign in or would you think that something is wrong, and maybe this is efficient? Oh, not not efficient! um It's like an impersonation attack like a iframe attack right.

A

If you go to amazon and you want to buy this tv, would you actually click the buy button but notice on this page? It's not a buy button. It says, buys now button and underneath it has this lock and secure transaction label with a wink emoji. Would you actually trust this site to sell you this tv, or would you think that something is wrong, so all bugs like this or errors like this?

A

They break user trust and they lead to loss of business and, more importantly, they are very hard to verify using traditional functional sources like I showed like should have you know, length, 2 and so on, because in this case you probably would have to validate that every element on a page has the text you expect you have to hardcode it right and what about everything else?

A

You know things that are not labels, things that are not text things like styles layouts how to actually verify those properties on your page and do it quickly, but we're doing it quickly is very important because manual testing doesn't scale. Imagine if I you know, ask you to go through entire amazon.com to make sure nothing looks wrong.

A

That's absolutely! You know out of a question. It doesn't scale. You cannot expect your developers or even your qa team, to notice all the little details, even small applications.

A

Videos like this sudoku will have a lot of screens and a lot of ui elements that can go wrong. You would have to look at all possible user case scenarios and user stories at every resolution, because the application might actually lay out and show different things at each resolution and the functional tests themselves right. The things that I showed you know find me. The elements check that text find me elements confirm. The number on the page are not very helpful.

A

For example, I can take a to-do nvc application that we had right and I can coming out css file right, so the css would not even load well guess what the application looks completely broken to every user, and yet the functional test still passes because we're still input box, you can still type click enter. It will add a new to the item. It's just everything looks like a mess right, but the functional test still passed because they don't look at the styles.

A

They don't look like a layout and if you try to write functional assertions to confirm css and style and layout properties, it quickly overwhelms you, because you would have to check everything and every possible css property doesn't scale at all. So we need something else and with something else is visual testing. The main idea behind visual testing is that, instead of doing functional, insertions we're going to render the page- or you know it's part like a single element into an image, so basically a text screenshot of a page.

A

So now we have png image and then we will compare that image to a previously rendered and approved image, some an image that some user looked at and said yeah. This looks good, let's keep it as a baseline.

A

So now we will compare the new image to the baseline image.

A

It can look like this. So imagine this is our baseline image. We have a drop down. The open menu took a screenshot well now in a feature branch or in particular new commit.

A

We take an image of the same drop down after we open it, and also we see all this navigation header links below now it could be an accident. Maybe some toggle is gone, but it could be new feature. We actually could have added was headers, but before we decide we have to take an image difference.

A

So we take a delta, we compare with images pixel by pixel, you know red green, blue channel plus alpha, and so we go through each row and each column and we compute, which pixels have changed and majority of pixels have not changed right. Everything is the same except for the bottom part of the drop down, and so usually those visual tips are images with red, pixels kind of marking the spots, the pixels that are different with a new image is different from baseline image, and now you, you know the qa.

A

The developer have to kind of look at this div and say: do they intended to do or did they intend to change their appearance or layout or color? And so you can say no. No! No. This was a mistake. I no no thank you, thank goodness that you caught it. So we can reject the difference. In that case, you know the test fails or you the human can say yeah. This is new feature. We added new navigation menu items, so I accept this.

A

In that case the test passes and also that new image becomes the new baseline image. So from now on we're going to compare new, you know tests for drop down against this new baseline image.

A

So the visual testing replaces this very hard to answer question. Does this page look good to a human eye with a much simpler question? Does this page look the same and this question? Does this page look? The same is very hard for humans to answer right. We would like kind of look through the whole page and, and it just would take forever and we will definitely miss bunch of things. That's why kids love always like find 10 differences between those two pictures, but computers are extremely extremely efficient and fast at image comparison.

A

They literally can just scan the whole array of numbers and say these two numbers are different: easily paralyzable, easily scalable.

A

Okay, for one question that people ask is how many images do I need to actually cover my whole application with visual tests so that we can, you know, catch all those logging problems. We can catch all the buy buttons problem, so my advice is usually like.

A

Take every user workflow, but you have major user stories and when there are visual changes with a nice, you know changes through.

A

We use a story, take a full application, screenshot and of course, if, for example, you have a header and you don't care about the header for the user story, you can limit yourself to just the main area, but take a screenshot every time, something changes, and if you have multiple uh resolutions that you support, make sure you take every user flow again at each support resolution, because your appearance on a mobile and your appearance on desktop can change independently and you don't want to miss the errors, because you never actually took screenshots for mobile resolution.

A

If you have component tests right, both drop-downs could be a component. That shows one item, two items you know subheadings, take the usual tests for each different state of a component, so even in sudoku I have 15 images right so seems like a small application, but the number of images quickly multiplies because there's so many visually different things.

A

So uh this is how sudoku looks when we start again right, but you know the field. For example, in this case we'll have 45 grid cells field, you know and it has the timer zero zero and you know status bar on the right.

A

If we make a move by filling a cell, then it will also highlight our matching numbers right on a field. So six six six. So this is a different visual appearance. So I'll take a screenshot when we are about to finish the game, I probably will take a screenshot just to see just to show how the full field, except one cell, looks, and when we fill the last one, then the game shows us.

A

You sold it in the screen, so I'll take a screenshot here as well to make sure that my game always ends up here and visually. Nothing changes accidentally without me, knowing it and approving it, because the game looks and the layout is different on each resolution. I'll take you know different screens, a tablet and mobile resolution and my component, but status bar or status column on the right it can have toggles.

A

So I will take the same status bar component and I will you know: change the toggle state in my test and I'll take visual screenshot of that component to make sure it doesn't accidentally change. Okay. So this is you know, kind of nice right, just take screenshots compare them.

A

Visual testing presents bunch of challenges, so the first challenge that you'll encounter in a complex application, comparing two images and make sure that pixel, perfect and of the same image is only possible if application is rendering the same data right. Otherwise, pixels will not show the same.

A

So you have to make sure that the application is showing the same stuff. That means, if you have dynamic data, think all your you know, fake dynamically generated user names cannot have that because different name in the input field will show different pixels. uh If you fetch something from the server and you don't control that the data you know from the server the response you'll have different data rendered on the screen: pixel differences.

A

If you have a clock ticking somewhere and the time stamps shown different data, pixel differences so likely cypress provides a lot of mechanism for controlling the network, marking data and controlling the clock. So both things I eat they're easier to do with cyprus.

A

But then, besides, that there is performance question, so imagine you have a larger application and you don't have 15 images, but you have hundreds of images generating them.

A

Comparing them becomes a performance issue because it will slow down your functional tasks. If you have to generate compare generally, compare you also will hit a problem where even generating a consistent image is is a challenge by itself, even with the same data. For example, the same browser will render a complex page differently on mac versus windows versus linux, and everything depends on fonts installed configuration system, libraries, gpu, there's tiny differences in the way browsers work.

A

You know within different modes on ci versus local, locally running tests, so deterministic rendering is a huge problem and just to figure out. What's your workflow for running the tests, if you have a baseline image, where are they stored? Do you commit them to the repository? Do you keep them somewhere else? When you approve a visual div right? How do you do that? Where does the image go? Does it override things right?

A

These are all all questions that you have to figure out in order for your team to start using visual testing.

A

There are lots of options, but you know different ways you can go if you want to try visual testing, so I've written the guide for cyprus that you can check out and many commercial services wrote plugins for cyprus that you can use you know. So you can start writing visual testing right away. The most important number one question you have to answer to yourself immediately is: do you want to do it yourself or do you want to go with third-party commercial service?

A

That's the number one choice, every everything after that is kind of depends on what you picked here. Okay, so let me start with third-party service right: the third-party services, the commercial providers. They don't even take the screenshot on your local machine because that's hard to generate deterministic image across different platforms. Instead, they take the dumb snapshot, write the html css and, like all the styles, and they just send it to their third party cloud.

A

Okay, that's why they will render the image consistently because they use the same browser locked on the same machine. So every time you render from that dumb snapshot themselves, it will give you the same pixels also because they have a cloud of machines. If you send them hundreds of images, they're, all you know displayed and queued up and processed as quickly as possible.

A

Now the baseline images in that case will be stored on the third party image provider right. So you don't have to store anything or update anything you can say yes approve and the image just becomes when your baseline image, you don't have to move any physical files and because this process is usually asynchronous, you don't have to wait for your test to pass. But that means you don't actually know if very visual tests but failed right. So usually you would have said, let's say: github checks.

A

One check will come from functional test and then asynchronously and novice status check will be updated by this third party service.

A

Most of the services become expensive very very quickly. They all give you a free tier. Like you know, the free tier is enough for small projects to try it out for a small project. You'll be enough to actually run it, because you only need like a big screenshot, you don't you don't need that many of them, but I do recommend you started using a third-party service and full disclaimer.

A

You know I have a doctorate degree in image processing.

A

I don't want to do it right, it's it's something I would rather pay and not worry about, and just start using as a tool for my testing, rather than like, create it from scratch. Okay, yeah. Here's! How a typical test looks when, when you use third-party service, so we're going to visit the page right.

A

This is our kind of standard command and then, for example, at some point in inside the test, I'll say sigh that person's snapshot, this command person's snapshot comes from percy, plugin and percy is a very common commonly used third-party image testing service browser browser bottom last year.

A

If you use aptly tools which is extremely popular third-party visual testing service, look at the tutorial, they provide their own custom, cypress command called eyes check window.

A

Where is hapa that they use same thing, just select a page or an element and take a screenshot I've done webinars with each company. You can check that out, see you know how it runs in action, but let's say you have convinced yourself that you can do it. You you decide to go. Do it yourself route? You don't have a pain, you want. You don't need images to be stored anywhere on a third-party. You know cloud provider, so you you want to be safe and sound.

A

Cyprus includes a size, screenshot command that takes a screenshot of a page or a part of a page. So you can take a screenshot of a whole test runner. That's probably not what you want to do right, because you don't care about the command log. You just want to have applications.

A

So a simple size screenshot will just give you like the app screenshot of a whole page or you can select an element using site, get and then say: dot screenshot and that's how you take a screenshot of, for example, a single element, so those screenshots are saved locally as png files and I kind of go for the whole setup. If you want to do it yourself in a blog post, but then you have to manage what files yourself you have to commit them to the repository.

A

If you change your baseline by proving an image, it means like physically moving the image to a folder, and you have to think how you render images so here, for example, on the left. There is mac rendering timer in my game and on the right linux, ci, rendering the same timer to me. They look identical, but notice. The image diffing in the middle shows that the outline of the font characters is different and in red these are pixels that are really different.

A

According to my, you know, image pixel comparison, so you have to think how you're gonna render the same image deterministically on different platforms. Luckily, you can use a docker image even locally and run your tests there, and both will generate very good images as bass lines. That will be consistent because on ci you can run a cypress test in the same docker image, but then again like this is just like: workflow becomes more complicated than necessary.

A

There are plugins for image comparison and even cypress plugins for doing the image comparison yourself right locally, so you have to find it. It's just extra complexity right, so just ask yourself: is it worth it? How much time am I spending doing this rather than actually working on my application?

A

Okay, so for one project, but I'm excited right now about what started last year and they just released version five already. It's called visual regression tracker. So this simplifies image comparison and management.

A

If you're doing everything yourself, okay, so they still take the screenshot using size, screenshot command. That means they haven't solved non-deterministic, rendering problem so rendering would not be consistent across operating systems, but they sold everything else. So they give you a docker compose file and when you start it, it has a docker image for your database that keeps all information about your project keeps all the images there and even gives you a web application with login, where you can review images, approve them and then it like updates it in database.

A

So it takes care of all that you know jazz about storing images, image approval and so on. Here's a typical test for sudoku game.

A

Okay, so my block of tests is called uh sudoku right and my test is called shows the initial game and the first thing I do inside this test. I say vrt, which is visual regression, tracker, start kind of establish connection to my you know, project running locally and you can do it outside the test just once. In this case, I only have one test, so I'm starting inside the test.

A

I want to stop the clock, so cyprus command sided clock freezes, the application clock, so there is no timer running nothing and then, instead of generating a random array, which is what games do I use site fixture to load local json files, which I saved myself before for the initial board and the soul board, and I visit the page and I set both json objects on a window object, and my application knows that when it's running inside cyprus and both arrays are present on a window, object, take them instead of generating a new board randomly.

A

So this is the way I mock my data to remove the source of randomness so that my image is the same okay. So I loaded the page and then I have a couple assertions to make sure the board is where all the grid cells are there. The status timer shows zero, zero, zero and the difficulty shows easy.

A

Okay. So after that I take a screenshot, so I do vr kit track, which is a custom command, but comes from visual regression. Tracker npm package, and I call it sudoku, and this is a wrap around size. Screenshot under the hood okay, so this is what it looks like when it runs notice on the left I have like vrt command started, tracking, sudoku and so on.

A

When I run the first time it just says: okay, I don't have a baseline image. You know for sudoku I'll just store with the image in a database and that's it, and this is the ui for visual regression tracker as, as you can see, I'm running this locally, but you can run it in a cloud right on your companies.

A

You know internal machine and you log in and for this project it says here's an image. It's called sudoku because I use that name. You know it was captured using electron and so on.

A

Okay, if I rerun the test- and I haven't changed anything on the left- you see baseline image on the right. You see current image, they are identical. I haven't changed any styles. I haven't changed any numbers. I haven't changed by layout, so there's a match. The test is green.

A

Now imagine that I changed something so normally it would be changed in application, but in this particular case I'm changing the data. So I'm changing one of the numbers on the initial board and I went to the difficulty drop down and I select medium is sort of easy and I you know, flipped one of the switches.

A

So when I run this after this change, here's what will happen. Cyprus shows in a browser right that their teeth command actually failed right. It says: hey. There are differences in the images that I observed and here's the url. You know you can look at the url and go there and see the actual differences.

A

So if I go back to the you know that url it shows there is one unresolved image, meaning my new image. What I'm comparing is different comparing to baseline- and I can click on this- and this is what it shows right and on the left. It shows bass line on the right. It shows the new image and I can toggle the thing to show the actual div like all the red pixels right and you can see there is a new number, nine change to eight.

A

The difficulty changed to medium right and one of the toggles at the bottom also changed. So it was able to detect all those little things.

A

Now I can click approve. I can say yes, I I wanted this change to happen right. In that case, I can go back to my test, re-run it and immediately it's green again, even though you know both new things are still there. It's just my baseline is the new image all right. So this is how you would approve an image if you really intended to progress to be a change.

A

uh Every time you take visual tests, you still have to fight flake, flake, meaning tesla, sometimes pass sometimes fail, even with visual testing and usually in visual testing. Things are flaky because the animation hasn't changed. So, for example, here I took the image and you see the new image the inside area hasn't expanded, yet so visually it's different image.

A

Maybe the drop down right hasn't rendered yet, and yet I took the image too early. The text hasn't rendered yet, like all the icons are rendered, but none of the text, so this all will lead to flake. So the best advice to fight with flake is to first do a couple of functional assertions.

A

You know the text is where the grid is, where the text is visible and after you you know past voice assertion, then you take an image, and now you compare them pixel by pixel by then you know that animation class, all already uh you know, has been set application finished rendering now you can do image comparison.

A

The html5 canvas presents particular problem. Everything I showed before you can add a functional assertion. But what do you do when you have a canvas? You know complicated chart in this case very simple canvas with just a smiley face that animates how to do image different. In this case and again I have a blog post and open source repository strongly.

A

We cannot even look inside the canvas, it's a single element with just pixels. How do we do visual testing in this case?

A

Well, before we take the image and compare it to the baseline, we can actually hard code a weight. I know there's animation for a smile, I'll record four seconds, and then I take an image and compare it so I'll wait, animation finished now I take the image and it matches the baseline again waiting animation finished.

A

Okay, the same well. If I try to wait for just one second, if I try to speed it up right, then the images are different, because my animation hasn't finished right and those red pixels showing mismatch, especially in ci. I never know when things are done or not.

A

Okay, so before we do this uh next step, I want to explain as something that's a key characteristic of cyprus. Cyprus has what we call retriability in the first test that I've shown we have a command to find all elements with this selector okay, and then we have an assertion. There should be two, so this doesn't immediately check if there are two items or not, and if there are not venus files for tests, because we know that in on the web, nothing happens immediately.

A

So instead, cyprus has this approach where, if there are no items or assertion doesn't pass for whatever reason it will go back to the command like get.

A

Run it again, maybe it will get a different number of elements and then evaluate assertion again, so all commands that are safe to retry. If assertion attached right next to them fail will be retried. So, even if my application is slow in this particular moment, maybe it goes to the server and the server is slow to respond.

A

We don't fail immediately. We retry up to like default command, timeout all right. So in this case we keep retrying, as you can see, in blue on the left and whatever application actually renders us, then the test passes. So we call it restriability.

A

We can do visual retriability for our tests to fight all the flake and not hardcode weights.

A

We need to take a screenshot compared to baseline if it matches all good continue to the next test command, but else if there is a mismatch we can check if we are out of time, if we're out of time, we fail the test else. We go back to take screenshot, comparing again right because taking a screenshot and comparing is non-destructive operation. We can it's almost like querying right. Is it the same? Is the same? No okay?

A

So here's how it looks- and I wrote this little utility called cyprus records to do like, like retries for cyprus easily. So in this case we'll go to the smile page and we'll call recurse, and it only needs two arguments. The first one are cyprus commands that you want to run in this case. Take an image, compare it to the baseline and this will return an object with a match property and we just written match. So this is the predicate. If it returns true, we know recurs finished.

A

If not, it will go back to the first function and try it again. So here's how it looks notice on the left. It keeps retrying taking the image comparing it retrying and when the smile actually looks like the smile has finished, then it finishes the test because it just found an image that looked like the smile we expected.

A

We can factor out all that common code and when it's so easy just say it looks the same and it will rerun visual comparison until it times out or matches the baseline, and we never have to explicitly hard code. The ways we can even automate this inside the browser, so we don't go to the baseline.

A

We know that our animation is still continuing, so we can compare first canvas weight, take the canvas again and do image comparison pixel by pixel in a browser right, and we can automate it where we keep taking the image until the number of different pixels is above a threshold, and then we can log the things.

A

So basically we can ensure the canvas is static. There are no visual changes because the pixels don't match and then compare it to the saved baseline image, which kind of looks like this notice. The number of pixels that are different so keeps going down, and then we compare the image to the baseline.

A

So we kind of wait for application to finish updating and then we can take new image and actual comparing to baseline. This is extremely important for any complex animations and complex apps, like charts. So here I will compare the chart as a loaded.

A

I will add the data set, compare it to a new image and then add one more data set. Compare it then remove data set so do like complex user interactions. With my.

A

Chart with my application and compare at each step to what I already saved, because they were good images, so here's how it looks, notice the complex animations and I never hardcoded any ways. Instead, I keep comparing the images once it matches that particular you know kind of baseline at that particular step. I continue to the next. If, at any point my chart did not look like I expected. The test will fail, because the images never finally reached a consensus.

A

A

Yeah visual testing is simple. Like that super powerful. I never go into details. I just should look like that all this okay, so this is the end of my presentation. If you want to know more about visual testing, I wrote a guide. I have several presentations, so you can find the slides for those. The slides are good because it actually includes a bunch of little videos.

A

I have blog posts and I also have a few videos on my youtube channel, because when you do visual testing, it's much much better to just look at someone actually showing it in action when even reading it or looking at a static, screenshot. Okay, thank you very much for listening and slides online. If you have any feedback, let me know I'm glad mata take care.

B

Great thanks club all right thanks everyone for tuning in and uh hope uh you got a lot out of it. This was pretty awesome just want to let you know that june 3rd next thursday, travis nestlin will be giving a talk on incremental static, bundling and next js and versailles, and then two weeks from now casey wilson will be given a talk on ai and javascript with tensorflow.js.

B

So thanks again for coming thanks again gleb and uh check your inbox, I got um a little gift for you. So have a great day, and all you guys have a great afternoon thanks.

A