Effective UI/E2E Testing

with ScenariooJS and Protractor

Dominic Langenegger, Michael Werner, Rolf Bruderer

github.com/scenarioo/scenarioo-js

What is it about?

• E2E/UI-Testing and Documentation of Applications
• Structuring and Designing Test Suites
• Using Protractor and ScenariooJS
• Practical Hints and Examples from real Projects

UI/E2E-Tests? WHY?

My Architect told me to only do Unit Tests!

Coder's Ropes and Slidelocks

No Time to Write Tests?

Source: http://change-challenge.blogspot.ch

Test Pyramid

End-to-End Tests

• Registration of new User
  1. Open Application
  2. Click Registration Link
  3. Fill out Form
  4. Hit Registration Button
  5. Check if user in DB

Documentation - Famous Words

Working Software over comprehensive Documentation

Documentation is in the Code

Code IS the Documentation

Tests are Documentation

Scenarioo - How it works ...

ScenariooJS Architecture Overview

Scenarioo Viewer

Jump through User Scenarios like a Kangaroo ...
demo.scenarioo.org

For all Stakeholders

E2E - What are my Ends?

Choose Ends wisely ...

• What have the developers in your team under their control?

• If the test is broken, the developer must be able to fix it!

E2E Test Setup

Subsystem not under your control: fake it!

Fake what you don't control!

• DB: well defined test data set
  
• Other Systems: Stub / Fake / Mock / Test Fixture


• Define it in your source repository in a well maintainable way

Control your Ends

Document Expectations about other Systems

{
    "stringValue": "Our UI can display generic string values!",
    "dateValue": "1978-06-13"    
}

A Specification by Example of what our system can consume

Test Data becomes difficult to maintain

• Engineer it!
  • Design and structure your test data well
  • Only as little test data as needed
  • Write Generators to define maintainable test data
  • NOT use snapshots from testing systems

Good E2E Test Scenarios

• Test System Functionalities
• Concrete Usage Scenarios
• Full E2E Usage Process
• From User's point of view
• Written in Business / User Language
• NOT test Components in isolation!

An Example Scenario

• Order Virtual Machine on System
  • Choose System where you want to Create it
  • Open Machine Creation Wizard
  • Enter Configuration Data
  • Check Order Confirmation and Price
  • Submit Order
  • Expect that the correct Order has been sent

Use Cases and Scenarios

Structure Scenarios by Use Case

• Use Case

  • A goal the user wants to achieve
  • Usually an action = verb and subject(s)
  • Not too fine granular!
    • Boss Test
    • Coffee Break Test

• Scenario = Test Case = One flow through the use case

Good Use Cases Examples

• Open Account
• Create Task
• Edit Person
• Virtual Machine - Add Disk
• Virtual Machine - View

Bad Use Case Examples

• Login
• Select Birthdate for Person
• Validate Date Fields
• Autocomplete Addresses
• Browse User Navigation

Group Usecases by Subject

• User
  • Create Account
  • View Profile
  • Edit Profile
• Task
  • Create
  • Edit
  • Assign
  • Mark Done

Scenarios per Use Case

A hand full of scenarios per use case:

• Most Important Happy Scenario
  • Only with minimal "required properties"
• Most Important Alternative Happy Scenarios
  • at least one with all optional fields set
• Special Validation Scenario
  • first enter wrong
  • check for validation messages
  • then correct it
• Error Scenarios?
  • e.g. when the backend blocks

Good Scenario Examples

Use Case "Task - Create"

• Task in Project
• Sub-Task
• Task with all Options
• Task with Validation
• Fail on Project Closed

Bad Scenario Examples

Use Case "Player - Find"

• Find by exact Name
• Find by Name with Special Characters
• Find by similar Name entered
• Find by partial Name & Nationality
• Find by partial Name & Birthdate
• Find by partial Name and Birthdate only matches Birthyear
• Find by Name and Birthdate, also if month and day in birthdate exchanged

Balance well!

• Avoid E2E-Testing Creep

• 1 hand full of scenarios per use case

• focus on high-level business functionality

Balance well!

• DON'T test all details of complex logic in E2E tests

  • Extract that logic into a Component / Service / Class / Function

  • Extensively test that Logic in Unit Tests

Exercise & Discuss

Think about your Application ...

• What are your Subjects & Use Cases?
• Define Scenarios for one Use Case
• What should be tested as E2E Test?
• Discuss in Small Groups

A simple test

describe('Example Use Case', function() {
  it('Example Scenario', function() {
    browser.get('/index.html');
    element(by.css('li#item_one')).click();
    expect(element(by.id('selected')).getText()).toEqual('one');
  });
});

So far so good, let's look at a bigger example.

A not so simple test

describe('Example Use Case', function() {
  it('Example Scenario', function() {
    browser.get('/index.html');
    var firstname = by.id('#firstname');
    element(firstname).sendKeys('Stephen');

    var lastname = by.id('lastname');
    element(lastname).sendKeys('Strange');
    element(by.id('#submit')).click();

    expect(element(by.id('#message'))).getText()
      .toEqual('Registration successful!');
  });
});

Cool, but it's starting to get a bit messy...

Soon problems arise

• What page am I on?
• What distinct user actions are possible?
• What if elements reappear on other pages?
• Are these tests maintainable?

The solution? Page objects!

Page Objects ...

• ... provide an interface driven by tested functionality
• ... hide the complexity of UI interactions
• ... enable reusable logic for reoccurring components
• ... make nicer code :-)

Example page object

var RegistrationPage = {
  var firstnameTextbox = element(by.id('#firstname'));
  var lastnameTextbox = element(by.id('#lastname'));
  var registerButton = element(by.id('#submit'));

  this.register = function(firstname, lastname) {
    firstnameTextbox.clear();
    firstnameTextbox.setText(firstname);
    lastnameTextbox.clear();
    lastnameTextbox.setText(lastname);

    registerButton.click();
  };
}

Reusable components

var TextBox = function(element) {
  this.element = element;

  this.set = function(value) {
    element.clear();
    element.sendKeys(value);
  }

  this.clear = function(value) {
    element.clear();
  }
}

Reusable components

var RegistrationPage = {
  var firstnameTextbox = new TextBox(element(by.id('#firstname')));
  var lastnameTextbox = new TextBox(element(by.id('#lastname')));
  var registerButton = element(by.id('#submit'));

  this.register = function(firstname, lastname) {
    firstnameTextbox.set(firstname);
    lastnameTextbox.set(lastname);
    registerButton.click();
  };
}

Let's revisit our example

describe('Example Use Case', function() {
  it('Example Scenario', function() {
    registrationPage.open()
        .register('Stephen', 'Strange');

    confirmationPage.expectRegistrationSuccessful();
  });
});

The intention of the code becomes visible.

Summary

• Start early with defining page objects and components
• Interface of page objects is functional
  • Not technical
  • Not driven by UI structure
• Keep tests short
• Don't test too much stuff

Bringing in Scenarioo

For installation and configuration see: https://github.com/scenarioo/scenarioo-js/blob/develop/README.md

Protractor configuration

onPrepare: function onPrepare() {
    global.scenarioo = require('scenarioo-js');
    scenarioo.setupJasmineReporter(jasmine, {
      targetDirectory: './scenariooReports',
      recordLastStepForStatus: {
         failed: true,
         success: true
      },
      ...
  }

https://github.com/scenarioo/scenarioo-js#configuration

And with some slight changes...

describe('Example Use Case', function() {

  afterEach(scenarioo.saveLastStep);

  it('Example Scenario', function() {
    registrationPage.open()
        .register('Stephen', 'Strange');

    confirmationPage.expectRegistrationSuccessful();
  });
});

Documentation activated!

• One use case per describe
• One scenario per it
• Step saved
  • at end of each scenario
  • on expectations failing
  • on each call to scenarioo.saveStep(...)

Defining scenarioo steps

var RegistrationPage = {
  var firstnameText = new TextBox(element(by.id('#firstname')));
  var lastnameText = new TextBox(element(by.id('#lastname')));
  var registerButton = element(by.id('#submit'));

  this.open = function() {
    browser.get('/registration.html');
    scenarioo.saveStep('Registration page opened');
  };
  this.register = function(firstname, lastname) {
    firstnameTextbox.set(firstname);
    lastnameTextbox.set(lastname);
    scenarioo.saveStep('Registration data entered');
    registerButton.click();
  };
}

Define steps inside of page objects!

But wait, there's more

• Scenarioo Fluent DSL
• Cleaner syntax
• Additional information can be defined easily

Example with fluent DSL

useCase('Register User')
  .labels('mobile')
  .describe(function() {

    scenario('Register new user')
      .labels('happy')
      .it(function() {
        registrationPage.open()
            .register('Stephen', 'Strange');

        confirmationPage.expectRegistrationSuccessful();
  });
});

Time to dig into Real Code

• Subjects, Use Cases, Scenarios
• Tests & Page Objects
• Json for Test Data
• Steps inside Page Objects
  • Labels for Steps (for special topics)
• Intercept Backend Calls for Verification
• Test Setup:
  • Scenarioo
  • Well defined Screen Size
  • Animations Turned Off
• Test End:
  • Fail Test on Console Errors

How to Keep Tests Green?

Feature Branches and Pull Requests

• Failed tests on feature Branches are ok!

• Failed tests on master / develop disturb others!

• Let CI Build System verify your Pull Requests

  • automatically run all tests!
  • no merge when tests fail!
  • use it also for Code Reviews!

• Short Demo

Definition of Done?

 # Definition of Done

 * Feature is covered by tests

Any better ideas?

Definition of Done

Our Definition of Done in our Markdown Documentation:

# Definition of Done

* Feature is documented in Scenarioo

• Do Story Reviews in Scenarioo
• This includes allready that there must be a test covering it

Protractor & Promises & Asynchronous Event Queue

What is wrong?

if (element.isPresent()) {
  element.click();
}

element.isPresent().then(function(present) {
    if (present) {
        element.click();
    }
};

Tooling Pittfalls

• Browser & Protractor & Webdriver must be compatible
  • At least one of them is installed globally
  • Rolf's recommendations:
    • Install Protractor and Webdriver also globally
    • Let developer control when to update
    • Causes less dependency problems for developers
  • Other Option: use a versioned docker image to run tests

Internet Explorer Testing

• Selenium for Internet Explorer is most unstable version
• IE testing is very hard!
  • writing working web pages for IE is allready hard
  • writing stable web tests for IE is even harder
  • you will fight with problems of the IE tooling
  • these are problems that are no "real bugs"!
• Is it worth to go that extra mile?
  • we recommend to focus on functional web tests with most stable tooling
  • prefer to use Firefox or Chrome for automated testing
  • cover IE by using it for additional manual tests

More Topics (if Time)

• Debugging Protractor Tests
• Mocking & Proxy Engine for Backend
• Other Questions?

Scenarioo in Other Environments

https://github.com/scenarioo/scenarioo-java https://github.com/scenarioo/scenarioo-cs

• do not depend on any Testing Toolkit
• integration with Selenium up to the user
• more flexibile and additional data can be written

Scenarioo-JS versus Scenarioo-Java/C♯

• Advantages of ScenariooJS
  • Integrates easy with Protractor and Jasmine 2
  • Comes with comfortable Fluent DSL
  • Automatic Step on Error and Success
  • Protractor is easy to write tests
  • JSON is handy for passing data to page objects
• Disadvantages of ScenariooJS
  • No support for additional App-specific detail data yet
  • Difficult to integrate with other testing toolkits