Praxis / Doc /

The Automator a.k.a. Praxoid

The Automator (aka Praxoid) is basically a customized version of Marvin-JS. It relies on several other technologies, including Mimik, Selenium WebDriver for Node and Mocha, to run the automated Praxis test suite.

The tests themselves are initially written in plain English (given/when/then) and parsed by Yadda, a BDD framework inspired by Cucumber. For example, a login.feature test might look like this:

Given I have admin credentials
When I enter a username and password and click Log In
Then I am logged in to the site and I see the main screen

Mimik can then be used to generate stub code for the tests. 

mimik generate tests/features/pod/login.feature

Given(/I have admin credentials/, function(){
    // test code goes here
});

When(/I enter a username and password and click Log In/, function(){
    // test code goes here
});

Then(/I am logged in to the site and I see the main screen/, function(){
    // test code goes here
});

Marvin wraps things up with some convenience methods to further abstract test code from the plain-language test logic, and creates some beautiful reports.

Multiple Browser Challenge

Unlike most web applications, Praxis requires testing real-time interactions between multiple clients (browsers). It's not unique (there are chat/collaborative applications out there), but the need to verify test action results in multiple browsers simultaneously makes automated testing more complicated.

Marvin is a very basic test runner/reporter. It relies on a single session object throughout the code. This object contains a reference to the selenium webdriver instance. This session also keeps track of the time that the session was launched, which is used to name the report results/screenshot output folder. This was a problem because our tests need to be able to instantiate multiple concurrent drivers. To work around this, Automator overrides several methods in Marvin's classes (mostly in Page) to accept a driver object, which overrides the default session.getDriver().

Also, tests in Marvin are typically executed against one browser configuration at a time. The config.json file contains a browsers array. Each browser configuration is tested individually.

In order to test Praxis's real-time collaborative features on a single machine, it's necessary to be able to launch multiple browser platforms concurrently. To do this, Automator adds a factory utility that allows the test developer to write a small amount of test logic to launch additional browsers with specific features. An automatorBrowsers array was added to the config.json file containing the different possible browser configurations. Thus, the test developer can determine what browser platform is currently under test (one of the browsers in the config.json browsers array) and, when a second browser is necessary, select a configuration from the automatorBrowsers array.

Test Execution Order

By default, Mocha executes tests in alphabetical order (feature, then scenario). It's possible to specify order by passing the feature files in the Mocha command line in the desired order, but since features are typically (see Multiple-file Features below) defined in a single file, it does not sort scenarios.

To work around this, Automator allows specifying a testOrder array in config.json. Each array element is either a string (just the feature name) or an object (with a (feature) title attribute and scenarios array attribute).

Multiple-file Features

By default, Marvin/Mocha expect a 1-1 relation between Feature and file. This is ok until the number of scenarios and steps within a feature start to grow. Automator adds the ability to spread a feature's scenarios across multiple files. Simply include the same "Feature: " spec at the top of each file. The Automator will merge them automatically into a single feature.

Shared Data

Test data is stored in JSON files under tests/data. Occasionally, data created during tests needs to be used in subsequent tests during the same session. For example, when creating folders in the inject library that are later used to store injects. In this case, the objects are named such that future tests can select them without needing to hard-code them. For example, folders are named "Test Folder " with the session.launchDate appended. Injects are named session.launchDate with the inject type appended.

Be sure to assign session data variables within the correct scope. For example, several tests re-use inject creation steps. Prior to execution of certain steps, the inject data needs to retrievable. The data can be set in the session object in the step (Given/When/Then) the first time it is needed.

Webdriver Promises Gotcha

The javascript version of the webdriver returns promises everywhere, so you need to:

select.findElements(webdriver.By.tagName, 'option').then(function(opts){
    require('selenium-webdriver').promise.map(opts, function(opt){
        // do something
    });
});

Intermittent Stale Elements

A frequent problem in testing dynamic content is the StaleElementReferenceError. Tests may run fine sometimes, and other times, you get this error. In a nutshell, you cannot be sure that references to a page element that appear later in your code will actually be valid, because the code may execute before the asynchronous code that fetches the element has returned. Webdriver provides the ability to pause test execution until specified DOM elements are available, but some of the promisified element methods don't seem to behave as expected.

Following solutions on SO (e.g., http://stackoverflow.com/questions/16882860/selenium-webdriver-js-explicit-wait), several attempts were made using driver.wait() with WebElement.isDisplayed(), but the stale element errors would still come up regularly (1 occurrence every 3 or 4 runs). (It seems like element.isDisplayed() may be the source of the problem.)

When driver.wait() was used with webdriver.until.elementLocated(), stale element errors disappeared completely. Unfortunately, it only worked for newly added elements. Checking for elements that become visible (using element.isDisplayed() or until.elementIsVisible()) or checking when element has text does not work. The ONLY solution that works in all situations is an explicit wait using driver.sleep() before methods that check content or element visibility.

Step Uniqueness

Yadda keeps all steps in a single store, keyed on the feature step text (e.g., "{when|given|then} I do xxxx"). This a PitA because a) you get frequent duplicate warnings if you have a consistent writing style, and b) if you try to take a advantage of that to create reusable step library, you cosntantly have to check back to make sure you have the exact same feature step text to ensure it gets re-used.

To minimize duplicate warnings, try to situational phrasing (e.g., "when I click the Create button in the New Inject dialog). At the same time, for steps you may want to re-use, try to keep the step text generic enough so that it will apply across all the re-use contexts. Don't re-use steps if you need to alter the business logic. If you need to alter the data, use the session object for that (i.e, in a unique step, set the value in a session attribute, then retrieve it in the reusable step).

Setting Up

Test framework, test scripts, and results are all stored in /tests.

The /tests/config.json file contains some settings that can be edited, including browsers under test, tags (annotations you can specify to disable/enable which feature tests to run), and file locations.

Folder structure is as follows:

See also

WebDriverJS-specific (used by Praxoid)

Other