Visual Regression Testing with Protractor

Share on facebook
Share on google
Share on twitter
Share on linkedin

The last couple of months I have been working a lot with DevOps and automation along with Angular development. One of the core principles in Continuous Delivery is having your back covered with automatic testing. Without this automatization in place, you are forced to manually regression test the whole app before every release, which will not make Continuous Delivery practical.

Protractor is Angular’s built-in end-to-end testing tool, which is enabling you to make a robot interact with your app automatically. I have a love/hate relationship with end-to-end tests as they on one side provide the most realistic test of whether the site works, but on the other site are the flakiest and most expensive to maintain when doing testing in an Angular app. Also, by default, you have to manually code every assertion that the end-to-end test should do, this is fragile and hard to maintain. There is still a need for doing exploratory testing sometimes before a release, to ensure that the code works acceptable according to acceptance criteria, but you don’t want to keep exploratory testing old features that haven’t changed, that you have already tested in the past. I like to use visual regression testing to indicate the areas in the code, that has changed, so you know where to focus your exploratory testing. Visual regression testing means taking screenshots of your app used for detecting changes by differentiating screenshots of every run with the baseline pictures. This gives awareness about what has changed in the application, making it easier to know where to grasp the visual changes in a pull request and helps QA by information where to focus the exploratory testing effort.

Setting up Visual Regression testing in an Angular app using blue-harvest

blue-harvest is a npm library that allows you to do visual regression testing and gives you a set of action helpers for writing tests easier. With blue-harvest you can take screenshots in your app using browser.takeScreenShot() and compare them against the baseline Screenshots called goldens.

We are now going through how to use blue-harvest for visual regression testing. I have created a complete demo on my Github.

Getting started with blue-harvest

Start with cloning my Angular-demo-with-best-practices which will give you a  TODO app as a foundation to work on.

To get started with blue-harvest you simply install it inside you Angular app by opening the terminal at the root of the project:
npm i blue-harvest

We are gonna make a test that asserts the right amount of todo items, takes a screenshot and compares it with the golden screenshot for that test:

This test navigating to the root path, getting the number of todos and is asserting that there is two todo’s only. Hereafter, it will take a screenshot and compare that against the goldenScreenshot. If any differences would occur in the compareScreenshot, it will create a diff image with the differences marked with pink.

To run this we need to adjust a couple of things in our protractor.conf.js:

Notice here that we are setting:

SELENIUM_PROMISE_MANAGER: false to enable async/await instead of using conventional then callbacks. This allows us to write async tests more like synchronous. Also, we set defaultTimeoutInterval: 999999 to set the timeout to some high number, so it’s not gonna bother us while running tests.

Creating goldens

To create the goldens, which act as a baseline for the screenshots to compare against, we create a new protractor conf for this:

This will take the configuration from protractor.conf.js and set the node environment UPDATE_GOLDENS before running to make blue-harvest create goldens when it runs the method compareScreenshot .

To run an end-to-end test with this config, we are gonna set up the npm scripts in our package.json:

In this post, we will start serving the app with the start script and then running an e2e script.
First, we need to update the webdriver to be able to run scripts with protractor:

npm run webdriver-update

Then we are gonna create the goldens with:

npm run e2e:update-goldens

This should give you a screen like this:

Notice how this has created a folder called goldens with a goldens called todos.png. Which is the screenshot that will be used to compare against then running the npm script e2e:no-serve.

Do a simple regression testing of the app

Lastly, we can run e2e:no-serve to do an actual regression test of  our app:

npm run e2e:no-serve

This should give us a green test like the picture above because the screenshot is matching the golden.

Other alternatives for visual regression testing

I have worked with another alternative for visual regression testing called Screenster. Even though Screenster is much faster to create tests because of the record feature, it has no support for working with feature branches, meaning it has a separate set of goldens for the feature being pull requested. The difference is also who should create and maintain the tests: QA or the developers. The benefit of Screenster is that QA can record tests, which unlike other test recorders I have seen, actually generates useful code. With Protractor tests, it is only the developers who will create the end to end tests.

Conclusion

In this post, we discussed the use cases of visual regression testing to gain awareness of the visual changes in the application by taking screenshots and compare them against a baseline (goldens). We saw how to implement this in a TODO Angular app where we created goldens and used them as a baseline for visual regression testing.

Do you want to become an Angular architect? Check out Angular Architect Accelerator.

Related Posts and Comments

Announcement: Angular Testing Workshop

I’m announcing a new live workshop: Angular Testing Workshop. It will be half-day workshops over three days, 100% online, where we will learn all the industry best practices with Angular testing so you can apply them in your daily work – taking straight out of my experience with doing Angular testing for big projects. The

Read More »

The Most Common Cypress Mistakes

Cypress has become the preferred way of doing UI testing of Angular apps by many Angular experts. It offers great improvements over Selenium-based testing tools by making the testing experience more like a real user using the built-in retry mechanism of assertions and commands (eg. click on the element), a user-friendly GUI which makes it

Read More »

The Ten Commandments of Angular Development

As a consultant, I normally work with companies between 3-12 months at a time and then I am off to the next gig. Most often, I am hired as a “hands-on” coach, were I am called in for an important and urgent project to make stuff happen within a very tight deadline. This requires that

Read More »

The Complete Guide to NgRx Testing (2020)

The main secret behind having better test coverage in an Angular app is to reduce the friction it takes to write tests and enforce test coverage by setting a test coverage threshold to be checked on every commit. NgRx and reactive testing is an area where many people get confused because it seems hard to write

Read More »