View on GitHub

Preceptor

Test runner and aggregator

Download this project as a .zip file Download this project as a tar.gz file

Preceptor Logo

Preceptor is a test-runner and test-aggregator that runs multiple tests and testing frameworks in parallel, sequential, or a combination there of, aggregating all of the test-results and coverage-reports.

Build Status npm version

NPM

API-Documentation

Coverage Report

Quick Look

Main-Features:

Features through plugins:

Table of Contents

Installation

Install this module with the following command:

npm install preceptor

To add the module to your package.json dependencies:

npm install --save preceptor

To add the module to your package.json dev-dependencies:

npm install --save-dev preceptor

What is Preceptor?

Today, there are a lot of testing frameworks out there, many of which are optimized for testing specific areas. A couple of these frameworks are:

All of these testing-frameworks produce their own test results - often in different output formats. Additional testing tools, like Selenium, create even more results, generating the results for tests that were run in multiple browsers. CI systems help here a bit since they can usually handle multiple results, but all the results are usually plugged into one root-node, making it hard to organize and analyse the test-results. Some engineers spend a good amount of time aggregating all of these results in a reasonable manner, creating often one-off hacks to merge all the results. Others just manually parse through the results, determining if any kind of problems were encountered during the most recent test-run, loosing the ability to have a one-look green/red summary.

Code-coverage metrics pose a similar problem: code-coverage reports are created independently and in isolation of tests that run for the same system in different testing frameworks or in separate tests (unit, component, integration, functional, acceptance, ...). It is challenging to have a complete overview of all coverage metrics as there is never only one coverage-report for all the tests. What is usually the case is that only some tests have coverage reports, possibly having only a subset of the metrics that could have been collected - most testing systems, for example, are not collecting client-side coverage-reports when running integration tests with Selenium; these are valuable coverage results.

In addition, all of these tools need to be coordinated, running sequentially, in parallel, or even a combination thereof. However, many testing tools do not have this capability build-in, resulting in third-party developers create their own solutions which are quite often not much more than a hack, having also quite possibly inconsistent interfaces as they are usually not created by the same people. All of this coordination can easily turn the code-base into an unmaintainable mess, quite possibly triggering false-positives due to flaky testing infrastructure and code.

Preceptor tries to solve all of these problems by using a consistent way of configuring the tools, managing the workflow within a single configuration file, and aggregating the test-results and code-coverage reports in one place. In addition, it adds a flexible and extensible plugin infrastructure to the test-runner, introducing even more features like a Selenium / WebDriver integration, injecting client/server code in setup/teardown methods of different testing frameworks, keeping the test-code free of glue-code.

Preceptor comes with the following features out of the box:

As mentioned, plugins add even more features, including:

Getting Started

Preceptor comes with a command-line tool that takes a configuration file that defines what sequence the task should run, what data needs to be collected, and how some tasks affect others.

Let's have a look at some examples that uses Mocha as sole testing tool. Preceptor can be used to mix multiple testing frameworks and independent tests at the same time, however, I will only use Mocha for the next examples.

To get started, create the following file with the name config.js:

module.exports = {
    "configuration": {
        "reportManager": {
            "reporter": [
                { "type": "Spec" },
                { "type": "List", "progress": false }
            ]
        }
    },

    "tasks": [
        { // Task
            "type": "mocha",
            "title": "Everything", // Just giving the task a title
            "configuration": {
                "paths": [__dirname + "/mocha/test1.js"]
            }
        }
    ]
};

This configuration adds a Mocha task and two reports:

As SUT, add the following lib.js file to the same folder:

console.log('Mostly Harmless');

module.exports = {

  answerToLifeTheUniverseAndEverything: function () {
    return 42;
  },

  whatToDo: function (phrase) {
    return (phrase == "don't panic");
  },

  allThereIs: function () {
    return 6 * 9;
  },

  worstFirstMillionYears: function () {
    return 10 + 10 + 10;
  },

  message: function () {
    console.log("So Long, and Thanks for All the Fish.");
  },

  startInfiniteImprobabilityDriver: function () {
    throw new Error("infinitely improbable");
  }
};

The tests live in a sub-folder with the name mocha. For now, let's add a couple of simple tests that give you a glimpse on how test-results will look.

Copy the following content into mocha/test1.js:

var assert = require('assert');
var lib = require('../lib');

it('should know the answer to life, the universe, and everything', function () {
    assert.equal(lib.answerToLifeTheUniverseAndEverything(), 42);
});

describe('The End', function () {

    it('should print something', function () {
        lib.message();
    });
});

Preceptor adds a command-line tool that can be executed by calling preceptor in the shell (when installed globally). Run Preceptor with the above created configuration file:

preceptor config.js

The result should look something like this (colors are not shown):

  ✓ should know the answer to life, the universe, and everything

  The End
    ✓ should print something

You can see that the output that was created in lib.js isn't visible anywhere. By default, Preceptor will hide any output created by tests, the SUT, and the testing-framework, displaying only the output of loaded reporters. This behavior can be changed by modifying values in the task-options for each individual tasks. For more information on how to do this, please refer to the API documentation or see below.

Now, to show some of the features, let's add two more test files:

mocha/test2.js

var assert = require('assert');
var lib = require('../lib');

it('should find all there is', function () {
    assert.equal(lib.allThereIs(), 42);
});

it('should calculate the worst first million years for marvin', function () {
    assert.equal(lib.worstFirstMillionYears(), 30);
});

mocha/test3.js

var assert = require('assert');
var lib = require('../lib');

it('should not panic', function () {
    assert.ok(lib.whatToDo("don't panic"));
});

it('should panic', function () {
    assert.ok(!lib.whatToDo("do not panic"));
});

Let's run them all sequentially. Simply add them to the task-list:

module.exports = {
    "configuration": {
        "reportManager": {
            "reporter": [
                { "type": "Spec" },
                { "type": "List", "progress": false }
            ]
        }
    },

    "tasks": [
        {
            "type": "mocha",
            "title": "Everything",
            "configuration": {
                "paths": [__dirname + "/mocha/test1.js"]
            }
        },
        {
            "type": "mocha",
            "title": "Calculations",
            "configuration": {
                "paths": [__dirname + "/mocha/test2.js"]
            }
        },
        {
            "type": "mocha",
            "title": "Phrasing",
            "configuration": {
                "paths": [__dirname + "/mocha/test3.js"]
            }
        }
    ]
};

Run Preceptor:

  ✓ should know the answer to life, the universe, and everything

  The End
    ✓ should print something
  1) should find all there is
  ✓ should calculate the worst first million years for marvin
  ✓ should not panic
  ✓ should panic

  1) Root should find all there is
     54 == 42
AssertionError: 54 == 42
    at Context.<anonymous> (.../mocha/test2.js:5:9)
    ... // Abbreviated

This example shows a failed test, displaying some details and the stack-trace that is abbreviated here in the output for our examples. Let's keep this failing test around for now.

Great! However, now, the tests appear to run all in the same top-level test-suite - we cannot easily distinguish which tests were run in which task. Let's change this. Add a suite:true to each task, turning each task into a virtual test-suite that creates a new level of test-suites in the test-results using the title given.

module.exports = {
    "configuration": {
        "reportManager": {
            "reporter": [
                { "type": "Spec" },
                { "type": "List", "progress": false }
            ]
        }
    },

    "tasks": [
        {
            "type": "mocha",
            "title": "Everything",
            "suite": true,
            "configuration": {
                "paths": [__dirname + "/mocha/test1.js"]
            }
        },
        {
            "type": "mocha",
            "title": "Calculations",
            "suite": true,
            "configuration": {
                "paths": [__dirname + "/mocha/test2.js"]
            }
        },
        {
            "type": "mocha",
            "title": "Phrasing",
            "suite": true,
            "configuration": {
                "paths": [__dirname + "/mocha/test3.js"]
            }
        }
    ]
};

The output for this configuration is:

  Everything
    ✓ should know the answer to life, the universe, and everything

    The End
      ✓ should print something

  Calculations
    1) should find all there is
    ✓ should calculate the worst first million years for marvin

  Phrasing
    ✓ should not panic
    ✓ should panic

  1) Calculations should find all there is
     54 == 42
AssertionError: 54 == 42
    at Context.<anonymous> (.../mocha/test2.js:5:9)
    ... // Abbreviated

Great! That looks already a lot better.

But now, I want to run the first tests in parallel, and the last test should run when the first two tests are both completed. Simply wrap the first two tasks in an array literal. This will group the tasks together.

module.exports = {
    "configuration": {
        "reportManager": {
            "reporter": [
                { "type": "Spec" },
                { "type": "List", "progress": false }
            ]
        }
    },

    "tasks": [
        [{ // <-----
            "type": "mocha",
            "title": "Everything",
            "suite": true,
            "configuration": {
                "paths": [__dirname + "/mocha/test1.js"]
            }
        },
        {
            "type": "mocha",
            "title": "Calculations",
            "suite": true,
            "configuration": {
                "paths": [__dirname + "/mocha/test2.js"]
            }
        }], // <-----
        {
            "type": "mocha",
            "title": "Phrasing",
            "suite": true,
            "configuration": {
                "paths": [__dirname + "/mocha/test3.js"]
            }
        }
    ]
};

Let's run it again:

  Everything

  Calculations
    ✓ should know the answer to life, the universe, and everything

    The End
      ✓ should print something
    1) should find all there is
    ✓ should calculate the worst first million years for marvin

  Phrasing
    ✓ should not panic
    ✓ should panic

  1) Calculations should find all there is
     54 == 42
AssertionError: 54 == 42
    at Context.<anonymous> (.../mocha/test2.js:5:9)
    ... // Abbreviated

You can see that the tests are running in parallel now. However, the Spec reporter prints the test-results as soon as it receives them from the test-clients, turning the output into a mess. This is the default behavior of most of the console reporters. We can change this behavior by setting the progress property to false. When this flag is set to false, the reporter will organize all test-results in a tree, printing them in an ordered manner just when Preceptor completes all tests.

To make it a little bit more interesting, let's add also the Duration reporter; it will print the total duration. The value of progress for the Duration reporter is by default false, so no need to set this explicitly.

module.exports = {
    "configuration": {
        "reportManager": {
            "reporter": [
                { "type": "Spec", "progress": false },
                { "type": "List", "progress": false },
                { "type": "Duration" }
            ]
        }
    },

    // ...
};

The output is now:

  Everything
    ✓ should know the answer to life, the universe, and everything

    The End
      ✓ should print something

  Calculations
    1) should find all there is
    ✓ should calculate the worst first million years for marvin

  Phrasing
    ✓ should not panic
    ✓ should panic

  1) Calculations should find all there is
     54 == 42
AssertionError: 54 == 42
    at Context.<anonymous> (.../mocha/test2.js:5:9)
    ... // Abbreviated

Time: 531 milliseconds

The test-results are printed correctly now. Also, the duration is printed just as we wanted it.

With Preceptor, you can add as many reporters as you want, all of which are run at the same time - no matter if they are console or file reporters.

Let's add some file reporters: JUnit and TAP

module.exports = {
    "configuration": {
        "reportManager": {
            "reporter": [
                { "type": "Spec", "progress": false },
                { "type": "List", "progress": false },
                { "type": "Duration" },
                { "type": "Junit", path: "junit.xml" },
                { "type": "Tap", path: "tap.txt" }
            ]
        }
    },

    // ...
};

When running this configuration, two files will be created in the current working directory called junit.xml and tap.txt, holding JUnit and TAP results respectively.

Now, I want to collect code-coverage reports from all the tests that were run. For that, add a coverage section to the global Preceptor configuration, activating it with active:true, and may be supply additional configuration options.

module.exports = {
    "configuration": {
        "reportManager": {
            "reporter": [
                { "type": "Spec", "progress": false },
                { "type": "List", "progress": false },
                { "type": "Duration" },
                { "type": "Junit", path: "junit.xml" },
                { "type": "Tap", path: "tap.txt" }
            ]
        },
        "coverage": {
            "active": true, // Activate coverage

            "path": "./coverage" // This is the default path

            // Add here any other coverage options.
            // See below for more information
        }
    },

  // ...
};

The output is now:

Everything
  ✓ should know the answer to life, the universe, and everything

  The End
    ✓ should print something

Calculations
  1) should find all there is
  ✓ should calculate the worst first million years for marvin

Phrasing
  ✓ should not panic
  ✓ should panic

1) Calculations should find all there is
   54 == 42
AssertionError: 54 == 42
  at Context.<anonymous> (.../mocha/test2.js:5:9)
  ... // Abbreviated

Time: 572 milliseconds


=============================== Coverage summary ===============================
Statements   : 96.3% ( 26/27 )
Branches     : 100% ( 0/0 )
Functions    : 92.31% ( 12/13 )
Lines        : 96.3% ( 26/27 )
================================================================================

By default, the coverage reporter creates three reports:

This concludes the introduction of Preceptor. Please read the general overview that follows, or refer to the API documentation that is included with this module.

Usage

Command-Line usage

Preceptor is started through the command-line that can be found in the bin folder. You can run the application with

preceptor [options] [config-file]

The config-file is by default rule-book.js or rule-book.json when left-off, where by the first one has priority over the second.

The command-line tool exposes a couple of flags and parameters:

--config j          Inline JSON configuration or configuration overwrites
--profile p         Profile of configuration
--subprofile p      Sub-profile of configuration
--version           Print version
--help              This help

For profiles, see the profile section below.

The config options adds the possibility to create or overwrite configuration directly from the console. The inline configuration-options are applied after the profile selection. Objects are merged if a configuration file was selected, and arrays will be appended to already available lists.

Testing Lifecycle

To understand how Preceptor works, we have to understand first what the testing lifecycle is. Tests are run usually through a common set of states that can be defined as the testing lifecycle.

This include:

Each individual state will be triggered in Preceptor so that plugins can react on these state changes. This includes the reporter, the client, the client decorators, the tasks, and Preceptor itself.

Preceptor breaks them (and some additional states) into 4 groups:

Let's describe them in more detail by describing their function and what information in conveyed.

Administrative states

Administrative states will be triggered for events that change the collection and run behavior of Preceptor itself.

State: Start

The start state will be triggered just when Preceptor starts to run.

Parameters:

State: Stop

The stop state will be triggered just when Preceptor completes all tests.

Parameters:

State: Complete

The complete state will be triggered when Preceptor completed all the export jobs.

Parameters:

Suite management

These states handle any changes to test-suites.

State: Suite Start

The suiteStart state will be triggered when a new test-suite starts. Please be aware that it is possible to have test-suites within other test-suites.

Parameters:

State: Suite End

The suiteEnd state will be triggered when a test-suite completed.

Parameters:

Test management

Test management states are states that transmit test and test-result information.

State: Test Start

The testStart state will be triggered when a new test starts.

Parameters:

State: Test Passed

The testPassed state will be triggered when a test has passed.

Parameters:

State: Test Failed

The testFailed state will be triggered when a test has failed. A failure is an unexpected test-result, often triggered by assertions.

Parameters:

State: Test Error

The testError state will be triggered when a test had an error. An error is an unexpected exception that was not triggered by an assertion.

Parameters:

State: Test Undefined

The testUndefined state will be triggered when a test hasn't been defined. This is triggered when the testing-framework recognizes a test, but could not find the test-implementation. Not all testing frameworks support this state.

Parameters:

State: Test Skipped

The testSkipped state will be triggered when a test has been skipped. Tests are usually skipped when for example sub-systems that are dependencies for test are not available.

Parameters:

State: Test Incomplete

The testIncomplete state will be triggered when a test is available, but incomplete. This happens when tests are written, but are not completed yet.

Parameters:

Configuration

The configuration file has three top-level properties:

Profiles

The configuration file supports multiple layers of profiles:

As an example, let's use the following abbreviated configuration:

config.js

module.exports = {

    "configuration": {
        "reportManager": { "reporter": [ { "type": "Spec" } ] }
    },

    "tasks": [
        {
            "type": "mocha",
            "configuration": {
                "paths": [__dirname + "/mocha/test1.js"]
            }
        }
    ]
};
Global Profile

Global profiles can be used to toggle between full-configurations:

module.exports = {

    "profile1": {
        "configuration": {
            "reportManager": { "reporter": [ { "type": "Spec" } ] }
        },

        "tasks": [
            {
                "type": "mocha",
                "configuration": {
                    "paths": [__dirname + "/mocha/test1.js"]
                }
            }
        ]
    },

    "profile2": {
        "configuration": {
            "reportManager": { "reporter": [ { "type": "Dot" } ] }
        },

        "tasks": [
            {
                "type": "mocha",
                "configuration": {
                    "paths": [__dirname + "/mocha/test2.js"]
                }
            }
        ]
    }
};

To select the second profile, call Preceptor as follows:

preceptor --profile profile2 config.js

The first profile can be selected by supplying profile1 instead, or whatever name you give the corresponding profiles. Be sure to always supply a profile when it is available since the default behavior is to not look for a profile.

Tasks Profile

Task profiles are very similar to global profiles, but are there for toggling task-lists. This can be very useful, when global configurations are shared between multiple profiles. Task profiles are also called sub-profiles since they toggle configuration below the global profile selection.

An example looks like this:

module.exports = {

    "configuration": {
        "reportManager": { "reporter": [ { "type": "Spec" } ] }
    },

    "tasks": {
        "sub-profile1": [
            {
                "type": "mocha",
                "configuration": {
                    "paths": [__dirname + "/mocha/test1.js"]
                }
            }
        ],
        "sub-profile2": [
            {
                "type": "mocha",
                "configuration": {
                    "paths": [__dirname + "/mocha/test2.js"]
                }
            }
        ]
    }
};

To select the first sub-profile, call Preceptor as follows:

preceptor --subprofile sub-profile1 config.js

Be sure to always supply a sub-profile when it is available since the default behavior is to not look for a sub-profile.

Combine Profiles

It is also possible to combine both profile methods.

module.exports = {

    "ci": {
        "configuration": {
            "reportManager": { "reporter": [ { "type": "Dot" } ] }
        },

        "tasks": {
            "acceptance": [
                {
                    "type": "mocha",
                    "configuration": {
                        "paths": [__dirname + "/build/mocha/test1.js"]
                    }
                }
            ],
            "integration": [
                {
                    "type": "mocha",
                    "configuration": {
                        "paths": [__dirname + "/build/mocha/test2.js"]
                    }
                }
            ]
        }
    },
    "dev": {
        "configuration": {
            "reportManager": { "reporter": [ { "type": "Spec" } ] }
        },

        "tasks": {
            "acceptance": [
                {
                    "type": "mocha",
                    "configuration": {
                        "paths": [__dirname + "/../mocha/test1.js"]
                    }
                }
            ],
            "integration": [
                {
                    "type": "mocha",
                    "configuration": {
                        "paths": [__dirname + "/../mocha/test2.js"]
                    }
                }
            ]
        }
    }
};

With this example, you could run the acceptance tests from your local machine with:

preceptor --profile dev --subprofile acceptance config.js

Global configuration

Preceptors task-independent behavior is configured in this section. It has the following properties:

Reporting

The report manager describes what reporters should be used and what it should listen for to receive testing lifecycle events from unsupported clients. See the preceptor-reporter project documentation for more information on the reportManager property.

Coverage

Coverage report collection can be configured in this section. It has the following options:

This feature uses Istanbul for collecting and merging coverage reports. See the Istanbul project website for more information.

Additional Reports Preceptor adds the file report to the list of reports to be able to disable the export of the JSON data. The file value will not be given to Istanbul since it is not available there.

Shared configuration

Any value that is assigned to the 'shared' object in the configuration root will be assigned as default for all task options. Task options then can overwrite these values. This gives a developer the opportunity to set own default values for properties, overwriting the default values that were given by the system.

Plugins

The Preceptor system supports the loading of custom plugins by installing the modules through NPM, and by adding the module name to this list. Preceptor then tries to load each of these modules by executing the loader function on the exported module interface, giving it the instance of Preceptor itself. The plugin can then register itself to Preceptor. See the preceptor-webdriver project for an example.

Tasks

The following tasks for testing-frameworks are supported by default:

Preceptor implements also some general purpose tasks:

The following sections describe the configuration and usage of these tasks

General Configuration

Tasks have a common set of configuration options that can be set on the root of the task. Any custom configuration option, that will be listed for each task below, should be set on the configuration object instead of the root (see "Object and List Configuration"). The reason for this separation is flexibility for new features in future version of Preceptor without breaking any task or custom task that might add similar named options.

Value Configuration
Object and List Configuration
Flag Configuration

Cucumber Task

The type-value for this task is cucumber.

Task options:

Mocha Task

The type-value for this task is mocha.

Task options:

Kobold Task

The type-value for this task is kobold.

Task options:

The storage configuration has also the following values by default:

Loader Task

This task imports already available test-report files, including JUnit and TAP files. The type-value for this task is loader.

Task options:

Example:

    {
        "type": "loader",
        "title": "JUnit import",
        "suite": true, // Wrap it in a suite using the title from above

        "configuration": { // Loader-task specific configuration

            "format": "junit", // Use "junit" as the format (is default)
            "path": "junit-*.xml", // Glob to select import files

            "configuration": {
                // Custom JUnit loader configuration
                "topLevel": false
            }
        }
    }

Node Task

The type-value for this task is node.

Task options:

Shell Task

The type-value for this task is shell.

Task options:

Group Task

The type-value for this task is group.

Task options:

The group Task Decorator uses this task to add the array-literal feature to the configuration.

Task Decorators

A task decorator is a plugin that can modify task-options before they are applied. This makes the configuration very flexible as the whole configuration scheme can be changed with these plugins.

The following task decorators are build-in:

Client Decorators

Client decorators are "tasks" that will be run in the client process by the client-runner. The decorators attach themselves to test-hooks that are triggered during the testing lifecycle (see above).

There are two types of hooks:

There are four activity hooks:

Out of the box, Preceptor supports the following client decorators:

The preceptor-webdriver plugin adds the WebDriver client-decorator for injecting Selenium setup/tear-down code into the client.

Client decorators can be added to every task that supports the testing lifecycle events (generally all testing frameworks), and it is added through the decorators list in the task-options.

Configuration

Client decorators has the following common options:

The general client decoration configuration namespace is reserved for future changes to all client decorators. Client decorator specific configuration should be described in configuration.

Client Runner

Clients are generally run in its own process. Each of these processes communicates with Preceptor, making it also possible to instruct the client to run client decorators in its process. For this to work, each testing-framework needs a client-runner. This is abstracted away, but is required when writing a plugin. The following client-runners are available:

Please see the above files in the source-code for details on how to implement these. Much of the common behavior and the communication is implemented in the client.js file.

Plugin Naming

The plugin module naming should follow the Grunt plugin naming convention. Plugins contributed by the community should have "contrib" in the module name (example: preceptor-contrib-hello-world). Plugins that are supported by the Preceptor team will be named without the "contrib" keyword (example: preceptor-webdriver).

API-Documentation

Generate the documentation with following command:

npm run docs

The documentation will be generated in the docs folder of the module root.

Tests

Run the tests with the following command:

npm run test

The code-coverage will be written to the coverage folder in the module root.

Third-party libraries

The following third-party libraries are used by this module:

Dependencies

Dev-Dependencies

License

The MIT License

Copyright 2014 Yahoo Inc.