TAP

This plugin adds support for TAP test result files in Jenkins.

It lets you configure an Ant-style pattern to find your TAP files. The plugin scans the matching files and creates test result views in Jenkins.

The TAP plugin depends on tap4j, a TAP implementation for Java, and the Jenkins JUnit plugin.

The plugin looks for TAP files like this:

1..2
ok 1 - Yahoo!
not ok 2 - org.tap4j.Error...

When a TAP stream is found, the plugin delegates parsing to tap4j. The parsed results are then processed and displayed in Jenkins using graphs and custom pages.

The easiest way to learn TAP is by looking at examples. The following stream shows two test cases, with comments explaining each line.

1..2 # The test plan: there are two test cases, numbered from 1 to 2.
ok 1 - Yahoo! # The first test passed successfully.
not ok 2 - org.tap4j.Error... # The second test failed and includes an exception message.

The plugin cannot process the default output generated by prove, because it contains additional information besides TAP and causes the tap4j parser to fail.

The recommended approach is to use the Perl module TAP::Harness::Archive.

For example, if your tests are inside the t/ directory, you can archive the TAP results with:

prove t/ --archive output

The result files are stored under output/t/. You can then configure the plugin with a pattern such as:

t/**/*.t

TAP supports attachments using YAMLish. If you are familiar with YAML, this format should look familiar.

The following example includes an attachment:

1..2
  ok 1
  not ok 2 - br.eti.kinoshita.selenium.TestListVeterinarians#testGoogle
  ---
extensions:
  Files:
    my_message.txt:
      File-Title: my_message.txt
      File-Description: Sample message
      File-Size: 31
      File-Name: message.txt
      File-Content: TuNvIGNvbnRhdmFtIGNvbSBtaW5oYSBhc3T6Y2lhIQ==
      File-Type: image/png
  ...

The Files entry contains the attached files. The content must be Base64 encoded.

Subtests allow you to group multiple TAP streams into a single stream. This works similarly to test suites in JUnit or TestNG.

Indentation is important when using TAP subtests.

Note that subtests and YAMLish are not part of the official TAP 13 specification.

  1..3
  ok 1 - First test
  1..2
  ok 1 - This is a subtest
  ok 2 - So is this
  1..2
  ok 1 - This is a subtest
  ok 2 - So is this
  ok 2 - An example subtest
  ok 3 - Third test

1. Install the Jenkins TAP plugin using the Plugin Manager or by manually copying the .hpi (or .jpi) file.
2. Enable TAP publishing in your job configuration and configure the file pattern and other settings.
3. Run your build and view the test results.

1. English (American)
2. Portuguese (Brazil) Work in Progress
3. Spanish (Thanks to César Fernandes de Almeida) Work in Progress

1. If the file type of the TAP report is considered as binary by the Jenkins webserver then the TAP plugin does not consider this file for inclusion in TAP reports (see #15813 for further details). To make sure your TAP report is considered for inclusion use e.g. the file name suffix .tap ( so instead of a file named report use report.tap).

For commercial support, please get contact us via @tupilabs.com at BlueSky.

1. This plug-in is going to be part of "Make your tests speak TAP" presentation in JCertif by Bruno P. Kinoshita, in September 2011. In this presentation will also be presented the tap4j project, how to enable TAP in JUnit and TestNG, integrate Perl and Java tests and an Eclipse TAP editor.
2. Gábor Szabó (2009), Test Reporting system: Smolder wish-list http://szabgab.com/blog/2009/07/test-reporting-system-smolder-wish-list.html.
3. Gábor Szabó (2009), Reporting Test Results http://szabgab.com/blog/2009/04/reporting-test-results.html.
4. tap4j - The TAP implementation for Java.
5. Test Anything Protocol (official webpage).
6. Performance tests with phantomjs and yslow (uses the plug-in for plotting the TAP results)
7. TAP Plugin for Matlab

JCertif - Make your tests speak TAP - Speaker: Bruno P. Kinoshita
September 2011 - Brazzaville, Congo

Moved to CHANGES.md.

1. Update Jenkins API
2. Update tap4j to match the latest protocol specification (13 and 14)
3. Remove deprecated code
4. Simplify code (it's old!)

1. Add configurations like validate number of tests with test plan, a TODO causes a test to fail
2. Diagnostics image gallery Done! Fixed as a new plug-in
3. Diagnostics exception code formatting
4. Add a link to open the file in the Build workspace (think about remote and local issues) Done!

The idea for this plugin started after tap4j was created. After learning about Smolder, it became clear that Jenkins could be used as a replacement by adding TAP support and a UI for displaying test results.

After discussions on the Jenkins developer mailing list, Max, Nick, Bruno (tap4j), and Cesar (tap4j) started working together on the initial design.

The first version of the plugin was released in July 2011.

The graph code was adapted from the TestNG plugin (thanks to its development team). The diagnostics support (YAMLish) was implemented using Jelly, Java, and CSS.

You can use a heredoc to write a TAP file with Shell, and use it with the plug-in. This is useful for testing.

#!/bin/bash

for x in {1..100}; do

cat > $x.tap <<EOF
ok 1
not ok 2
ok 3 # SKIP
not ok 4
ok 5
ok 6
ok 7
EOF

done ;