Preceptor-Reporter
Reporting library for the preceptor test-runner and aggregator.
Table of Contents
- Usage
- Manager
- Plugins
- API-Documentation
- Tests
- Third-party libraries
- License
Usage
This module is bundled with Preceptor by default and does not need to be included in the plugins
list of the global configuration section in the Preceptor configuration file.
However, you need to define the reporters that should be attached to the Preceptor test-events. You can add multiple reporters at the same time, all of which receive the same events from the Preceptor clients. Here is an example adding the Junit
, the List
and the Spec
reporter plugin to the reporter list:
"configuration": // ... "reportManager": "listener": "reporter": "type": "Spec" "type": "List" "progress": false "type": "Junit" "path": __dirname + "/test-results.xml" // ...
Configuration
The following configuration options are available for this module:
reporter
- List of reporters to activate (see above example)listener
- List of listeners to activate
Manager
The Manager object is the main access-point to the reporting plugin. It gives access to the plugin management methods, to the defined test-framework hooks, the defined messengers, and all abstract objects for creating your own reporting plugins.
Plugins
Reporter
Reporter plugins are the core of this library and represent the plugins that can export events.
The following reporter are available:
Dot
Duration
Event
JenkinsSauceLabs
Json
Junit
LineSummary
List
Plain
Preceptor
Spec
Summary
Tap
TeamCity
Plugin management methods
Here is a short summary of the methods available to access reporter plugins and instances thereof (see API documentation for more information):
getActiveReporter
- Gets a list of active reportergetReporterList
- Gets a dictionary of all registered reporterhasReporter
- Checks if a specific reporter is registeredgetReporter
- Gets a specific reporter from the list of registered reporter. It will return the constructor of the reporter.registerReporter
- Registers a new reporter by giving it an identifier and the reporter constructorregisterReporterRange
- Registers a list of new reporter with a dictionary of identifier and reporter constructoraddReporter
- Creates a new instance of a specific reporter and adds it to the active-reporter listaddReporterRange
- Creates a list of reporter and adds them all to the active-reporter list
Common configuration
All reporter have a common set of configuration options:
type
- Type of reporterpath
- Filesystem path to where the test-results should be exported. If none is given, then no filesystem output will be done.configuration
- Custom configuration options for a specific reportercolor
- Flag that determines if the output should be in color. (default: true)output
- Flag that determines if there should be an output to the console. (default: undefined - leaving it to the reporter to decide)progress
- Flag that determines if there should be a constant update when action states are changing. (default: undefined - leaving it to the reporter to decide)
Generally, there are two modes a reporter can be in (and both at the same time):
- file-output
- screen-output
Some of the reporters might want to prescribe how it should behave (like showing progress), but it can always be overwritten by the user.
Dot
This reporter prints a character for each test. The character will tell what the result of the test has been.
.
- Test has passedF
- Test has failedE
- Test had errorU
- Test is undefinedS
- Test was skippedI
- Test is incomplete
Duration
This reporter prints the total duration of all tests when all test had been finished.
Event
Every test-event is emitted by this reporter, and could pick-up by any listener that is attached to the object.
For every test-event, the event reporter will trigger three individual event types, including sub-types:
message
- Generic message, giving the area-type, the message-type, and the message-data as an array of items- * ```admin``` - Administration events, grouping messages of ```start```, ```stop```, ```complete``` * ```item``` - Item events, grouping messages of ```itemData``` and ```itemMessage``` * ```suite``` - Suite events, grouping messages of ```suiteStart``` and ```suiteEnd``` * ```test``` - Test events, grouping messages of ```testStart```, ```testPassed```, ```testError```, ```testFailed```, ```testUndefined```, ```testSkipped```, and ```testIncomplete```
- * ```start``` - Start of Preceptor * ```stop``` - End of Preceptor * ```complete``` - Completion of all processing after ```stop``` * ```itemData``` - Custom test-event data, giving the id of the test-action and the data as JSON string * ```itemMessage``` - Custom test-event message, giving the id of the test-action and the message * ```suiteStart``` - Start of a test-suite, giving the id of the test-action, the parent-id, and the suite-name * ```suiteEnd``` - End of a test-suite, giving the id of the test-action * ```testStart``` - Start of a test, giving the id of the test-action, the parent-id, and the test-name * ```testPassed``` - Successful completion of test, giving the id of the test-action * ```testFailed``` - Failed test, giving the id of the test-action, the message, and the reason of failing * ```testError``` - Test with unexpected error, giving the id of the test-action, the message, and the reason of the error * ```testUndefined``` - Test-case that hasn't been defined yet, giving the id of the test-action * ```testSkipped``` - Skipped test-case, giving the id of the test-action and the reason of skipping * ```testIncomplete``` - Test that is still WIP, giving the id of the test-action
JenkinsSauceLabs
The reporter prints the job-name for a Selenium session, binding the job-name to the Selenium test-run.
The session-id is determined by taking several steps in a row until a session-id is found. The following steps are taken in order:
sessionId
value in the test-rootitemData
propertysessionId
value of theJenkinsSauceLabs
reporter configurationSELENIUM_SESSION_ID
in the environment variables
The job-name is determined by taking the following steps:
jobName
value in the test-rootitemData
propertyjobName
value of theJenkinsSauceLabs
reporter configurationAPP_NAME
andBUILD_NUMBER
in the environment variables, combining in the format: "APP_NAME(#BUILD_NUMBER)"
Json
This reporter exports all test-data at once.
Junit
The JUnit reporter exports all test-data as an XML format.
LineSummary
With this reporter, the test results are summarized in one line when all the tests are completed.
List
The reporter lists-up all the issues which were reported during the test-runs.
Plain
This reporter prints out all test events in a plain text-format. This reporter can be used for debugging purposes when creating a custom reporter, learning about all the test-events that are triggered.
Preceptor
The Preceptor reporter prints out one-to-one test-events so that a parent Preceptor instance could pick-up all the events, integrating it into the parent test-result data.
Format
The general Preceptor format is as follows:
#|#<messageType> <data>#|#\n
The protocol sends out the version identifier for this format before any other message is send out. Version - Version of data-format
#|#version 1#|#\n
Here is a full list of all the messages sent out. Please see the API documentation for more information.
ItemData - Custom data for test-action
#|#itemData [<id>, <json>]#|#\n
ItemMessage - Custom message for test-action
#|#itemMessage [<id>, <message>]#|#\n
SuiteStart - Start of a suite
#|#suiteStart [<id>, <parentId>, <suiteName>]#|#\n
SuiteEnd - End of a suite
#|#suiteEnd [<id>]#|#\n
TestStart - Start of a test
#|#testStart [<id>, <parentId>, <testName>]#|#\n
TestPassed - End of test, marking test as passed
#|#testPassed [<id>]#|#\n
TestFailed - End of test, marking test as failed
#|#testFailed [<id>, <message>, <reason>]#|#\n
TestError - End of test, marking it as having an error
#|#testError [<id>, <message>, <reason>]#|#\n
TestUndefined - End of test, marking it as undefined
#|#testUndefined [<id>]#|#\n
TestSkipped - End of test. marking it as skipped
#|#testSkipped [<id>, <reason>]#|#\n
TestIncomplete - End of test, marking it as incomplete
#|#testIncomplete [<id>]#|#\n
Spec
The Spec reporter will print all test-events in an ordered and consistent format. This reporter has a very similar output format as the "spec"-reporter from Mocha.
Summary
With this reporter, the test results are summarized with multiple lines when all the tests are completed.
Tap
The TAP reporter exports the test-events in the TAP format.
TeamCity
The TeamCity reporter exports the test-events in the TeamCity format. All tools understanding this format should be able to pick-up the test results reported by this reporter. This includes the IntelliJ IDE.
Custom Plugin
You can create your own reporter plugin by using the exposed AbstractReporter
object that is a property on the manager object.
var ReportManager = ;var manager = ; var CustomReporter = ReportManagerAbstractReporter; manager;manager;
There are a couple of terms that might need clarification:
- action - There are three action-types available: the root-element of all test-results, any test-suite, and any tests.
itemMessage
- These are custom messages that can be attached to any test-action.itemData
- This is an object that could be used to save custom data for any test-action.
start
and end
is called once before and after all the test-results respectively.
In general, all actions have a start and end. For tests, however, there is a multitude of end methods, each defining a different end-result. The start methods take an ID that is given by the reporting instance and should be used again when the action is ended, finishing up the pending action. Should any action be still pending after all tests are completed, then the reporting manager will throw an exception since a testing-framework might possibly have been interrupted and some of the data might be missing.
The getOutput
method is called when all the collected data needs to be retrieved at once. This could be for saving the results to a file, or when printing the results at the end of the tests when progress
is set to false.
All test-results and actions are saved in the test-container object.
thisname
The following items are saved for each action-type and can be accessed the same way as describe above:
id
- Identifier of actionstartTime
- Start of action as an interger timestamppending
- Flag if action is pending or nottype
- Type of action. Possible values are: 'root', 'suite', 'test'name
- Name of actionlevel
- Numeric depth-level starting with 0parentId
- Parent action
See the above mentioned plugins for examples.
Listener
Lister plugins are the opposite of Reporter plugins - these Listener plugins import data.
The following listener are available:
Preceptor
TeamCity
Plugin management methods
Here is a short summary of the methods available to access listener plugins and instances thereof (see API documentation for more information):
getActiveListener
- Gets a list of active listenergetListenerList
- Gets a dictionary of all registered listenerhasListener
- Checks if a specific listener is registeredgetListener
- Gets a specific listener from the list of registered listener. It will return the constructor of the listener.registerListener
- Registers a new listener by giving it an identifier and the listener constructorregisterListenerRange
- Registers a list of new listener with a dictionary of identifier and listener constructoraddListener
- Creates a new instance of a specific listener and adds it to the active-listener listaddListenerRange
- Creates a list of listener and adds them all to the active-listener list
Common configuration
All listener have a common set of configuration options:
type
- Type of listenerconfiguration
- Custom configuration options for a specific reporterplaceholder
- List of placeholder given to theparse
method when parsing the input (see section about custom listener plugin).
Preceptor
The Preceptor listener can pick any information printed by a Preceptor process, integrating the processes result into its own results. See the reporter section above for more information about the data-format.
TeamCity
The TeamCity listener can listen for any messages that were meant for TeamCity receivers, for example the IntelliJ IDE. Any testing-framework that can export test-results to IntelliJ (and their like) can also be used to send these test information a Preceptor instance.
Configuration
This listener has the following configuration options:
parentId
{string} - Identifier of parent action
Custom Plugin
You can create your own listener plugin by using the exposed AbstractListener
object that is a property on the manager object.
var ReportManager = ;var manager = ; var CustomListener = ReportManagerAbstractListener; manager;manager;
The parse method will be called with text that needs to be parsed. This is usually output of a process. The placeholder might have key-value pairs of text that needs to be replaced within the messages.
See the above mentioned plugins for examples.
Loader
Loader plugins are objects that import common test and coverage reports, and they can be used independently by other projects since they are exposed by the getLoaders
method on the Manager object constructor.
The following loaders are available:
junit
- Object that imports JUnit xml test-report filestap
- Object that imports TAP test-report filesistanbul
- Object that imports Istanbul JSON coverage files
Common configuration
All loaders have a common set of configuration options:
type
- Type of loader (currently not used)configuration
- Custom configuration options for a specific loaderpath
- Glob path to files that should be imported
JUnit
The JUnit loader has the following configuration options:
topLevel
- Flag that indicates if thetestsuites
level should be imported. (default: false)
TAP
The TAP loader currently does not have custom configuration options.
Istanbul
mapping
- List of mapping objects that hav propertyfrom
andto
.from
is a regular expression (as string), andto
is a string.
Example
var junit =
Custom Plugin
You can create your own loader plugin by using the exposed AbstractLoader
object that is a property on the manager object.
var ReportManager = ; var CustomLoader = ReportManagerAbstractLoader;
See the above mentioned plugins for examples.
Messenger
Messenger plugins are objects that create low-level messages for external systems, and they can be used independently by other projects since they are exposed by the getMessengers
method on the Manager object constructor.
The following messengers are available:
JenkinsSauceLabs
- Object that prints Sauce-Labs job information to the Jenkins build log, letting the Jenkins SauceLabs plugin know about the job-name.Preceptor
- Object that prints Preceptor event messages to the standard output.TeamCity
- Object that prints Team-City event messages to the standard output. Most of the messenges can also be understood by the IntelliJ IDE.
It is important to note that these messengers have no consistent interface as they represent low-level objects. However, messengers trigger always an event on the messenger itself with the event-name message
that is the exact message that is created for the external system.
Common configuration
All messenger have a common set of configuration options:
type
- Type of messengerconfiguration
- Custom configuration options for a specific messengeroutput
- Flag that determines if the events are printed to the console. (default: false)
Example
var messengers = Preceptor;var teamCityMessenger = ; teamCityMessenger; teamCityMessenger;teamCityMessenger; teamCityMessenger;teamCityMessenger; teamCityMessenger;
Custom Plugin
You can create your own messenger plugin by using the exposed AbstractMessenger
object that is a property on the manager object.
var ReportManager = ; var CustomMessenger = ReportManagerAbstractMessenger;
See the above mentioned plugins for examples.
Hooks
Hooks are custom reporters for different testing frameworks, giving Preceptor access to different stages of the test lifecycle.
The following hooks are available:
cucumber
- Cucumber hooksmocha
- Mocha hooks
API-Documentation
Generate the documentation with following command:
npm run docs
The documentation will be generated in the docs
folder of the module root.
Tests
Run the tests with the following command:
npm run test
The code-coverage will be written to the coverage
folder in the module root.
Third-party libraries
The following third-party libraries are used by this module:
Dependencies
- glob: https://github.com/isaacs/node-glob
- preceptor-core: https://github.com/yahoo/preceptor-core
- sax: https://github.com/isaacs/sax-js
- promise: https://github.com/then/promise
- underscore: http://underscorejs.org
- xmlbuilder: http://github.com/oozcitak/xmlbuilder-js
Dev-Dependencies
- chai: http://chaijs.com
- codeclimate-test-reporter: https://github.com/codeclimate/javascript-test-reporter
- coveralls: https://github.com/cainus/node-coveralls
- istanbul: https://github.com/gotwarlost/istanbul
- mocha: https://github.com/visionmedia/mocha
- cucumber: http://github.com/cucumber/cucumber-js
- sinon: http://sinonjs.org
- sinon-chai: https://github.com/domenic/sinon-chai
- yuidocjs: https://github.com/yui/yuidoc
License
The MIT License
Copyright 2014-2015 Yahoo Inc.