The last couple of months I have been working a lot with DevOps and automation along with Angular development. One of the core principles in Continuous Delivery is having your back covered with automatic testing. Without this automatization in place, you are forced to manually regression test the whole app before every release, which will not make Continuous Delivery practical.

Protractor is Angular’s built-in end-to-end testing tool, which is enabling you to make a robot interact with your app automatically. I have a love/hate relationship with end-to-end tests as they on one side provide the most realistic test of whether the site works, but on the other site are the flakiest and most expensive to maintain when doing testing in an Angular app. Also, by default, you have to manually code every assertion that the end-to-end test should do, this is fragile and hard to maintain. There is still a need for doing exploratory testing sometimes before a release, to ensure that the code works acceptable according to acceptance criteria, but you don’t want to keep exploratory testing old features that haven’t changed, that you have already tested in the past. I like to use visual regression testing to indicate the areas in the code, that has changed, so you know where to focus your exploratory testing. Visual regression testing means taking screenshots of your app used for detecting changes by differentiating screenshots of every run with the baseline pictures. This gives awareness about what has changed in the application, making it easier to know where to grasp the visual changes in a pull request and helps QA by information where to focus the exploratory testing effort.

Setting up Visual Regression testing in an Angular app using blue-harvest

blue-harvest is a npm library that allows you to do visual regression testing and gives you a set of action helpers for writing tests easier. With blue-harvest you can take screenshots in your app using browser.takeScreenShot() and compare them against the baseline Screenshots called goldens.

We are now going through how to use blue-harvest for visual regression testing. I have created a complete demo on my Github.

Getting started with blue-harvest

Start with cloning my Angular-demo-with-best-practices which will give you a  TODO app as a foundation to work on.

To get started with blue-harvest you simply install it inside you Angular app by opening the terminal at the root of the project:
npm i blue-harvest

We are gonna make a test that asserts the right amount of todo items, takes a screenshot and compares it with the golden screenshot for that test:

This test navigating to the root path, getting the number of todos and is asserting that there is two todo’s only. Hereafter, it will take a screenshot and compare that against the goldenScreenshot. If any differences would occur in the compareScreenshot, it will create a diff image with the differences marked with pink.

To run this we need to adjust a couple of things in our protractor.conf.js:

Notice here that we are setting:

SELENIUM_PROMISE_MANAGER: false to enable async/await instead of using conventional then callbacks. This allows us to write async tests more like synchronous. Also, we set defaultTimeoutInterval: 999999 to set the timeout to some high number, so it’s not gonna bother us while running tests.

Creating goldens

To create the goldens, which act as a baseline for the screenshots to compare against, we create a new protractor conf for this:

This will take the configuration from protractor.conf.js and set the node environment UPDATE_GOLDENS before running to make blue-harvest create goldens when it runs the method compareScreenshot .

To run an end-to-end test with this config, we are gonna set up the npm scripts in our package.json:

In this post, we will start serving the app with the start script and then running an e2e script.
First, we need to update the webdriver to be able to run scripts with protractor:

npm run webdriver-update

Then we are gonna create the goldens with:

npm run e2e:update-goldens

This should give you a screen like this:

Notice how this has created a folder called goldens with a goldens called todos.png. Which is the screenshot that will be used to compare against then running the npm script e2e:no-serve.

Do a simple regression testing of the app

Lastly, we can run e2e:no-serve to do an actual regression test of  our app:

npm run e2e:no-serve

This should give us a green test like the picture above because the screenshot is matching the golden.

Other alternatives for visual regression testing

I have worked with another alternative for visual regression testing called Screenster. Even though Screenster is much faster to create tests because of the record feature, it has no support for working with feature branches, meaning it has a separate set of goldens for the feature being pull requested. The difference is also who should create and maintain the tests: QA or the developers. The benefit of Screenster is that QA can record tests, which unlike other test recorders I have seen, actually generates useful code. With Protractor tests, it is only the developers who will create the end to end tests.

Conclusion

In this post, we discussed the use cases of visual regression testing to gain awareness of the visual changes in the application by taking screenshots and compare them against a baseline (goldens). We saw how to implement this in a TODO Angular app where we created goldens and used them as a baseline for visual regression testing.