In this blog post, I will cover why I went from using Protractor to start using Cypress for end-to-end testing. We will look at the flaws in Protractor and how Cypress is fixing them as well as looking into the 7 steps to Cypress e2e testing success. Let’s get started!

If you have been working with Angular for a while, I am sure you have been working with Protractor. You might even have read some of my blog posts about how you “overcome” the common problems with Protractor, as you can find here. This post will leave all this behind and focus on a brighter future.

WARNING! Bright future ahead.

The worst problems with Protractor

Here is my totally opinionated list of the biggest problems with Protractor:

1. It doesn’t test like a real user: it doesn’t retry commands and assertions without you having to implement and maintain custom libraries for this.
2. It will fail when you have a transparent overlay even if you are waiting for an underlying element to be visible
3. It is hard to trace errors when it fails
4. It is hard to trace the executed commands without a lot of console logs

Let’s go through each of them and see why this is.

1) It doesn’t test like a real user

Have you ever heard about a flaky user? Me neither! That is because a user understands the asynchronous nature of web applications and knows to wait and retry for a reasonable amount of time before he reports a problem. An automatic end-to-end testing tool should also test like this or they will always be flaky. If it eg. loads a new page it should wait a reasonable amount for elements to show and being updated. Protractor supports a wait command but with this, you need to be explicit about what you are waiting for every time instead of simply having the wait built into the commands. Of course, you can also create your own helpers for this as I have done in the past but then you need to maintain this as well.

2) It will fail when you have a transparent overlay

This relates to point 1, but especially with a transparent overlay Protractor will have problems as Protractor will try to click on a covered element even if the built-in wait function is waiting for an covered element to appear. Here you need to explicitly tell Protractor to wait for the blocking overlay eg. the spinner overlay to disappear. Wuhu, imagine how much fun that is to maintain!

3) It is hard to trace errors when it fails

Have you ever seen Protractor fail and being confused from the error message why it failed? Especially when it has failed on the CI will this really cause problems for you. You might mitigate this a bit with recording screenshots/videoes but even with this, the error message itself can leave you clueless. Also, you need to implement this by your self as well.

4) It is hard to trace the executed commands without a lot of console logs

Have you ever found yourself writing a lot of console log’s just to get some kind of vision on what is going on when the end-to-end tests are running? Not fun I can tell you and you ideally want one for every command to have the complete traceability of your end-to-end tests when running it on the CI. Have fun doing this for every line.

You can do some workarounds like this, to avoid doing all the console logs yourself, but you are still limited to only getting the browser events and not every single command with the desired information in the console.

How Cypress overcomes these problems

Let’s consider how Cypress overcomes these problems.

Problem 1: It doesn’t test like a real user

Cypress is built so it acts like a real user. What does this mean?

• It will retry commands for x amount of time (timeout) until it works
• It will retry assertions for x amount of time (timeout) until it is passed

All of this is built into Cypress so each command and assertion will retry for the specified amount of time. This will make sure that your tests are working as stable as if a real user were testing.

Problem 2: It fails with a transparent overlay

Because Cypress retries commands and assertions this problem will not occur unless the timeout is set too low.

Problem 3: It is hard to trace errors when it fails

Cypress contains meaningful error messages that contain information about which element failed. Also, Cypress automatically will take and save a screenshot when it fails. This can be used by the CI to investigate why an E2E test failed. Also, it supports video recording (if being run headlessly) which makes it even easier to trace why an E2E test has failed.

Problem 4: It is hard to trace the executed commands

Cypress got its own test running, which contains a log and a snapshot for each action taken. This makes it very easy to trace what has been run in the test and helps fix problems with a broken test.

The number 1 killer of end-to-end tests

The number one killer of end-2-end tests is fragile tests. When tests become fragile the team is not taking them seriously anymore and will soon have the tests disabled on the CI to merge pull requests. I normally recommend teams to start out with simple but stable end-to-end tests before making them more complicated as this will make them harder to maintain.

That is why Cypress will be a big upgrade as it might make your team actually using the end-to-end tests in a productive way instead of just writing them because they are “best practice”.

The 7 step approach to success with Cypress

Let’s consider how simple it can be to get started with Cypress. You might be familiar with my blog post about how to get started with Protractor on your team. Of course, the same steps can be used here as I have never seen anything that has worked better at introducing stable and useful end-to-end tests fast. To do a fast recap:

1. Create a testing strategy (how much of each test type? how does testing fit in the delivery process?)
2. Define top 5 use cases (if nothing else was working, which use cases would you keep alive?)
3. Create a simple smoke test
4. Getting the test data (test users and data associated with the test users)
5. Create the seed script to prepare the test data to run automatic end-to-end tests
6. Implement the top 5 use cases as end-to-end tests
7. Running the e2e tests on the CI

Implementing an end-to-end test suite for a todo-app example

Yes I know, another todo app. Guess what, it still represents the most common Angular scenarios better than anything else I have seen, so I will still use that.

Step 1: Creating a testing strategy

In short, my recommended approach here is:

• Use the testing pyramid to split the tests between approximately: 80 % of the tests should be unit tests, 15 % is integration tests and 5 % is UI tests. Note, even with this automation in place, there will be a need for manual exploratory testing.
• Having a super-efficient, pull oriented delivery process where developers are QA actually work together and share the QA work instead of relying on only handovers. Sometimes the developer will be confident doing the manual exploratory testing (eg. for technical changes with simple regression scope) and sometimes he would want help with the QA either in the form of the QA telling him what to test or delegate the complete regression test to the QA. Doing collaboration instead of handovers will remove bottlenecks as the developer is able to help the QA with the testing work and vice versa.
• For my recommended process approaches, check my continuous delivery post, where I recommend creating feature environments for every pull request and do the testing there before it is shipped to production as soon as the pull request is merged to master. That way, your team can easily do 5+ releases a day and even with more confidence than if you did a traditional “bulk-stage” of many features, because the delta is smaller in every release, making mistakes easier to spot.

Step 2: Defining top 5 use cases

Now, for people who are new to automatic end-to-end testing, only doing top 5 use cases might not seem as enough. Here is the thing: when you have tried actually maintaining end-to-end test in real life and seen how much time they can take to maintain vs. the value they provide you would understand why focusing on just running top 5 use cases that you have a 100 % confidence in, will be the desired approach. Does that mean you can NEVER have more than top 5 use cases covered by end-to-end tests? Of course not, if you find that you easily code maintain more with 0 % flake and 100 % confidence then you can just expand to more use cases, but I have rarely seen this been the case for companies.

“But, I can’t decide the top 5 use cases, they are all important!”

Sometimes I get this objection, and then I ask them: “If nothing else worked, and you could only keep 5 use cases alive, which would you choose?”. Every product owner should by his job definition be able to answer that question.

Defining the top 5 use cases for the TODO app

For the TODO app this will be:

1. See todo list
2. Create todo
3. Mark as complete
4. Delete todo
5. Update todo

If nothing else works the app will still be usable. This is why we focus on the top 5 use cases.

Step 3: Create a smoke test

Yaay! Now we come to some actual code! I bet some of you might have wondered if I was just a talking suit!

Let’s look at how we can start using Cypress to test our Angular app.

I recommend you to use Nx Schematics for integrating Cypress with Angular CLI. If you are already having an NX monorepo then this will be easy for you:

You can add Nx to an existing Angular CLI project using:

ng add @nrwl/workspace

Or you can just generate a new workspace from scratch and copy your app over:

npm init nx-workspace myworkspace

Now, this should give us a new project, that can run Cypress with Angular CLI. You can also just clone my demo repo here.

When you have this setup, you should have an e2e test project beside your app project:

In short these folders mean:

• Fixures: This is for containing test data as json files
• Integration: The E2E tests/spec files
• Plugins: Here we can hook Cyprerss plugins
• Support: For page objects and Cypress commands

The Cypress documentation is ridiculously good so I recommend you just read that for the Cypress fundamentals.

Now we have that in place, let’s start building our first smoke test.

Create the page objects

One architectural note with end-to-end tests. We don’t want our spec files to depend directly on the testing framework. For that reason, we, as a rule of thumb, use page objects to abstract away the Cypress dependencies from our actual test specification using page objects. This also has the benefit of making the spec code more clean as well as ensuring separation of concerns.

We create a new file for the page object:

Here we have a page object with a method for checking that each todo list item has the specified todo title.

Things to note here:

1. I use static here because we don’t want to bother by instantiating an instance every time we use this page object.
2. This contains the Cypress dependencies
3. The name of the method conveys the intent of the method and not some technical jargon

Create the test

For creating the simple smoke test we first make sure to control the server by intercepting the call to the server.

This is using cy.server and then make sure requesting the todo-list endpoint will return the fake todolist.

The actual test is just calling the page object. Note, that this spec file doesn’t contain any Cypress dependencies to keep it framework agnostic and higher level of abstraction.

When running it you should see Cypress open the dashboard and execute the test like this:

Some tips for creating Cypress tests:

• Always use an e2e-* id or data-e2e attribute for selecting elements. Why? More stable than classes/elements selection and tell the developers that this is used in e2e tests. This is only used for selecting elements in the e2e tests.
• Reset the state before each test run. Why? Because by ensuring the same initial conditions it is easier to maintain the tests.
• Keep the tests deterministic. That means, to know the path of test execution at compile time and don’t use dynamic/conditional logic to determine the test path. Why? Doing this complicates the test runs, error tracking, stability, and maintenance when you need to support all these different paths. This is also why Cypress recommends you not to do conditional testing and why they don’t support conditional error handling.
• Keep it simple. Why? Maintaining E2E tests can become a hard job quickly because it is interacting with a real DOM. Keeping stuff simple will give you the best value for money when doing the smoke test. You can always build on top later if you can manage it.

Step 4: Gather the test data

Now, let’s say we wanted to control the test data. There are two ways to go around this:

1. Use Cypress server and request interception to control the returned data
2. Create a seed endpoint to seed the test user BEFORE each test run. Note: always clean up BEFORE and not after the test.

Having this in place will make the test more stable and deterministic because there is no state from the previous test runs that can affect future test runs.

Seed data for the todo app

For a todo app when we consider the different use cases the normal test data requirements would be:

1. A test user (username and password)
2. A todo list belonging to the test users

Step 5: Create the seed script

To ensure that we keep our tests deterministic creating a BE endpoint to reset the e2e test data is beneficial. This simply means that before the e2e tests are run, it will call the endpoint for resetting e2e tests user and then run the test with a reset state every time.

Note, that as an alternative to this, we can also intercept the requests, as we did in the smoke test. The downside of this is that we are not testing against the real server which might be what we want. For third-party services, which we don’t control, it can be a good idea to stub them out if they are causing fragility in the dev environment.

Step 6: Implement end-to-end tests for the top 5 use cases

Now, we expand our test suite to the top 5 use cases.

I will upload a video series soon, which I will post here, to show you how to actually do this.

Otherwise, it will follow the principle of creating the smoke test to make this happen for the top 5 use cases.

If you find it very easy to do these E2E tests and you feel you can handle more, feel free to expand your suite to more than just top 5 use cases. But don’t lie to yourself, only do this if you actually have managed to run the top 5 use cases with easy maintenance and big value for money.

Step 7: Running the end-to-end tests on the CI

Now, the last step to actually get ANY benefit from all this work we need to have this automated as part of the delivery pipeline.

“If it can’t be automated, don’t bother” 

Christian Lüdemann

One common problem for running the e2e tests on the CI pipeline is that it takes “too long” time to execute.

Here is what I normally advise my clients:

1. Run the smoke test on every pull request build
2. Run the top 5 use case regression tests every day at noon and at midnight (if it takes longer than 15 minutes)

The first is because we still want fast feedback to see if the site actually is running. All the unit tests can pass but if there, eg. is a problem with bootstrap, dependency injection or network problems, then the site will not even load. Just the simple smoke test should catch that. Remember: The basis of your automated tests suite should always be unit tests because they are the fastest and easiest to maintain (good value for money).

The second is because running the full regression test suite can take quite a long time (15-25 minutes). Running this at noon and at midnight (against the dev environment reflecting the master branch) will give you fast enough feedback to respond to “red builds” than if you were only to run this on midnight or, urgh… manually.

Also, an important note on faster E2E test runs. I have seen that it can normally take around 10 minutes to build and serve the app before the e2e tests can be run. This is not necessary if you already have your app for the pull request deployed as a feature site, as you can just disable the serve (in Angular.json) and set the base URL to the feature site like:

npm run e2e -- todo-app-e2e --base-url *feature-site-url*

This should cut the e2e execution time in half.

My final takeaway is this: if you can run your whole e2e suite in under 15 minutes, then just run it on every pull request check-in, as it is here automatic test verification provides the most value. Otherwise, run the full suite at noon and midnight.

Conclusion

Awesome, you did it!

We saw in this post why I started to prefer Cypress over Protractor and we went through 7 steps to introducing Cypress on your team. These steps were: Design the testing strategy, define the top 5 use cases, create the smoke test, define the test data, implement the top 5 use cases and run the E2E tests on the CI pipeline.
I will upload a video series with Cypress at a later point to show a more realistic setup with how to use Cypress with a system containing login and a database.