You are currently offline, so some features of this site may not work fully.

Your browser doesn’t support some of the latest CSS features used to build this site.

Read more about my usage of new CSS features.

An overhaul of my initial post on automating visual regression testing, showing new methods and implementations.

What is this regression testing malarkey then?

Regression testing is a method of software testing that checks functionality that already exists against new changes that have been made to a code base, to catch (you guessed it) regressions!

Visual regression testing looks at the user interface of your website and shows any breakages when the front end has been changed, usually when making HTML or CSS updates in your code base.

This guide focuses on how to automate and take the pain out of the boring, laborious process of manual testing — so you can focus on your day to day work building websites, the bit you enjoy!


How to implement visual regression testing

There are many software options when it comes to implementing this kind of testing, to name but a few:

I’ve used all of these frameworks in projects and favour BackstopJS over the others for its ease of setup and comprehensive reporting functionality.

This guide will be using Backstop to write the tests and compare screenshots and Node with Gulp to run the tests in our workflow.

Prerequisites

Along with Node and Gulp installed, you will need PhantomJS and CasperJS installed globally on your machine:

npm install phantomjs casperjs -g

Installation

Next install BackstopJS and save it to your package.json in your development dependencies:

npm install backstopjs --save-dev

Generating a configuration file

Backstop requires you to execute commands from its own gulpfile.js/package.json — within the node_modules folder. You can use the built-in npm explore to do this, for example:

npm explore backstopjs -- npm run genConfig

Running the genConfig task produces a JSON file similar to this one. Modify with your project’s details, set some configuration for your first scenario and you’re ready to go.

"scenarios": [
  {
    "label": "Styleguide",
    "url": "http://localhost:3000/styleguide",
    "selectors": [
    "body",
    ".site-header",
    ".article",
    ".site-footer"
    ],
    "misMatchThreshold": 1
  }
]

Here I’ve changed the label that I see when the report runs, the URL that I’m testing against and added selectors for the DOM elements that I want to screenshot from that page. For now we can leave in the default break points that the tests screenshot against, but see the BackstopJS documentation for more configurable options.

Generating base screenshots

The following command creates reference screenshots as your baseline — an approved set of passing tests. You won’t need to do this very often — usually when you first set up the tests and create new ones.

npm explore backstopjs -- npm run reference
Reference screenshots of a pullquote from my style guide

Making changes and retesting

So now you’re at the point where you can comfortably refactor and update your CSS code, and re-run the tests to see if any regressions have inadvertently occurred. So go and make some changes to your CSS, I’ll wait…

Testing and comparing changes

Now run the tests again with the following command:

npm explore backstopjs -- npm run test

This will open up a server that shows you screenshots of which tests have passed and if any have failed. Adjusting the misMatchThreshold can be tricky, but play around and get it right for your project.

A failing test after making some changes to my CSS
A passing test after fixing the CSS

Where to use this testing

The best place in a project to implement visual regression testing is on a style guide or static component library for a website.

A lot of us work with content managed websites, and dynamic content from a CMS will cause “breakages” when running regression tests — they aren’t smart enough to tell that it is only the content that has changed. So my advice — test with static content.

Who should be using the test suite?

Short answer…everyone! Visual regression testing can help:

  • A specialist front-end developer — giving sanity checks that you know changes haven’t cascaded down to unknown areas of a website
  • A designer/developer of any other discipline — giving comfort and peace of mind that they haven’t inadvertently changed an existing component
  • A QA engineer — automating this kind of regression test will save time and allow them to carry on with more robust manual testing
  • A DevOps engineer — setting up these tests to run automatically on a build can highlight errors before deploying new code to live

Workflow optimisations

Now, typing out those commands is still a little painful every time and as developers, we’re nothing but lazy efficient when it comes to making repetitive tasks as easy as possible.

I add the commands to my project’s package.json in the scripts object, like so:

scripts: {
  "bless": "npm explore backstopjs -- npm run bless",
  "openReport": "npm explore backstopjs -- npm run openReport",
  "reference": "npm explore backstopjs -- npm run reference",
  "visualTest": "npm explore backstopjs -- npm run test"
}

This allows me to enter npm run visualTest in my terminal, for example, to run the regression tests. It also makes it easier to hook into your workflow, either on your CI build or to enforce running these tests before pushing code to a git repository.


TL;DR

Automated visual regression testing is a powerful tool, fairly straightforward to set up and get using right away.

It will definitely give you peace of mind that yourself and other developers aren’t breaking a website when building new features and it is a no-brainer to reducing the amount of regressions when many people are working on an existing code base.

It gives me confidence in refactoring legacy code, comfort that a component I’ve built is more robust when making changes to the rest of the website and also lets developers who aren’t as acquainted with a project (or even front-end development) confidence to make changes without negatively affecting the UI of a site. Implementing this in just one of my team’s projects has already captured errors and sped up our development process.

Demonstrating to your team and stakeholders that you have a website fully tested can also be a quick win to improve their confidence in the project, in refactoring and continuously maintaining and improving your code base.