Results Format of Tests Written Using Polyspace Test API

You can write C/C++ unit tests using the Polyspace® Test™ xUnit API. When you build these tests, you create a test runner executable that can be used to run the tests later. The test runner executable prints test results (test pass/fail information) on the console. You can optionally pipe the console output to a file for further processing.

This topic shows the various formats in which the test results can be displayed.

Prerequisites

This topic describes test authoring using the Polyspace Test xUnit API. To compile these tests, you are required to know some file paths in advance. For your convenience, you can define environment variables to stand for the file paths, or otherwise include the file paths in your build. For more information, see Set Up C/C++ Testing and Code Profiling Using Self-Managed Builds.

How to Change Results Format

You can change the display format of test results in one of the following ways:

For example, suppose the test runner executable is named testrunner.exe in Windows® (or ./testrunner in Linux®). By default, the executable displays test results in a human-readable tabular format. To display results in another format, for example, Test Anything Protocol (TAP), enter:

testrunner.exe -format tap

To see all formats available, enter:

testrunner.exe -h

The test results can be displayed using the following formats.

FormatOptionConfiguration MacroMore Details
Tabular (default)-format table or -format t 
#define PST_OUTPUT PST_OUTPUT_TABLE
Tabular Format
Brief-format brief or -format b 
#define PST_OUTPUT PST_OUTPUT_BRIEF
Brief Format
Test Anything Protocol (TAP)-format tap or -format a 
#define PST_OUTPUT PST_OUTPUT_TAP
Test Anything Protocol (TAP) Format
Machine-readable-format mrf or -format m 
#define PST_OUTPUT PST_OUTPUT_MRF
Machine-Readable Format

Within each format, you can choose to further tweak the result display. For example:

  • You can print the test file and line number in the format file:line using the option -format dots or file(line) using the option -format parens. By default, a colon separator is used. The format file(line) is more suitable for a parser that also parses other messages in that format, for example, on IDE-s such as Microsoft® Visual Studio®.

  • You can choose to log assessments that pass using the option -format passed. By default, you only see assessments for failed tests.

To specify multiple option values for -format, enter a comma-separated list. For example, to specify that results must be displayed in tabular format with file and line information in the form file(line), enter:

testrunner.exe -format table,parens

Supported Formats

The following sections show the various formats in which test results can be displayed.

Example Files

To see the test results format described in this topic:

  1. Copy the source files in the folder polyspaceroot\polyspace\examples\doc_pstest\test_results_format to a writable location. Here, polyspaceroot is the Polyspace installation folder, for example, C:\Program Files\Polyspace\R2026a.

  2. Build and run the tests as usual.

    For example, the file test_all_pass.c contains tests for functions defined in the file example.c. To build and run these tests with a GCC compiler, enter the following at the command line:

    gcc example.c test_all_pass.c <PSTUNIT_SOURCE> -I <PSTUNIT_INCLUDE> -o testrunner
    For more information on the file paths <PSTUNIT_SOURCE> and <PSTUNIT_INCLUDE>, see Set Up C/C++ Testing and Code Profiling Using Self-Managed Builds.

    To run the tests, in Windows for example, enter:

    testrunner.exe -format table

The folder contains two test files:

  • test_all_pass.c contains two test suites, suiteGlobalSumTestsAllPass and suiteStatusTestsAllPass. Each suite contains two tests, both passing.

  • test_some_fail.c contains two test suites, suiteGlobalSumTestsSomeFail and suiteStatusTestsSomeFail. Each suite contains a failing test.

    By default, the test results show the details of the failing test such as:

    • Location of failed assertion in the format fileName:lineNumber or fileName(lineNumber) (if you use -format parens).

    • Argument of failed assertion along with values involved. For example, if the assertion involves an equality such as:

      PST_VERIFY_EQ_INT(gSum, expected_gSum);
      the equality expression is quoted along with the left and right hand sides of the equality, for example:
      Expression '(pst_long_long_t)(gSum) == (pst_long_long_t)(expected_gSum)' evaluated to false
lhs="4294967295" rhs="2147483648"

    • Any custom message that you print for failing tests.

Tabular Format

By default, the test runner executable displays results in tabular format. You can also explicitly request a tabular format by using the option -format table or -format t with the test runner executable (or by setting the configuration macro PST_OUTPUT to PST_OUTPUT_TABLE). The tabular format is the most complete yet human-readable of the results display formats.

In the previous example, the results of test execution in tabular format looks like this:

  • All tests pass:

    | Running tests   |            |
|-----------------|------------|
| Running         | suite      | suiteGlobalSumTestsAllPass | test_all_pass.c
|-----------------|------------|
| Running         | test       | suiteGlobalSumTestsAllPass/globalSumTestOne
|            PASS | test       | suiteGlobalSumTestsAllPass/globalSumTestOne
| Running         | test       | suiteGlobalSumTestsAllPass/globalSumTestTwo
|            PASS | test       | suiteGlobalSumTestsAllPass/globalSumTestTwo
|-----------------|------------|
|            PASS | suite      | suiteGlobalSumTestsAllPass
|-----------------|------------|
| Running         | suite      | suiteStatusTestsAllPass | test_all_pass.c
|-----------------|------------|
| Running         | test       | suiteStatusTestsAllPass/statusTestOne
|            PASS | test       | suiteStatusTestsAllPass/statusTestOne
| Running         | test       | suiteStatusTestsAllPass/statusTestTwo
|            PASS | test       | suiteStatusTestsAllPass/statusTestTwo
|-----------------|------------|
|            PASS | suite      | suiteStatusTestsAllPass

|        |  Total | Passed | Failed | Incomplete
|--------|--------|--------|--------|------------
| Suites |      2 |      2 |      0 |          0
|  Tests |      4 |      4 |      0 |          0

  • One of the tests fails in both test suites (globalSumTestTwo in suite suiteGlobalSumTestsSomeFail and statusTestTwo in suite suiteStatusTestsSomeFail):

    | Running tests   |            |
|-----------------|------------|
| Running         | suite      | suiteGlobalSumTestsSomeFail | test_some_fail.c
|-----------------|------------|
| Running         | test       | suiteGlobalSumTestsSomeFail/globalSumTestOne
|            PASS | test       | suiteGlobalSumTestsSomeFail/globalSumTestOne
| Running         | test       | suiteGlobalSumTestsSomeFail/globalSumTestTwo
|     VERIFY FAIL |            |
test_some_fail.c:42: Verify failed
Saturating sum did not return expected value.
Expression '(pst_long_long_t)(gSum) == (pst_long_long_t)(expected_gSum)' evaluated to false
lhs="4294967295" rhs="2147483648"
|            FAIL | test       | suiteGlobalSumTestsSomeFail/globalSumTestTwo
|-----------------|------------|
|            FAIL | suite      | suiteGlobalSumTestsSomeFail
|-----------------|------------|
| Running         | suite      | suiteStatusTestsSomeFail | test_some_fail.c
|-----------------|------------|
| Running         | test       | suiteStatusTestsSomeFail/statusTestOne
|            PASS | test       | suiteStatusTestsSomeFail/statusTestOne
| Running         | test       | suiteStatusTestsSomeFail/statusTestTwo
|     VERIFY FAIL |            |
test_some_fail.c:72: Verify failed
Saturating sum did not return expected value.
Expression '(pst_long_long_t)(status2) == (pst_long_long_t)(expected_status)' 
  evaluated to false
lhs="1" rhs="0"
|            FAIL | test       | suiteStatusTestsSomeFail/statusTestTwo
|-----------------|------------|
|            FAIL | suite      | suiteStatusTestsSomeFail

|        |  Total | Passed | Failed | Incomplete
|--------|--------|--------|--------|------------
| Suites |      2 |      0 |      2 |          0
|  Tests |      4 |      2 |      2 |          0

Brief Format

To see a brief summary of the test results, use the option -format brief or -format b with the test runner executable (or set the configuration macro PST_OUTPUT to PST_OUTPUT_BRIEF). In this format, you only see details of failing tests and a summary of test execution.

In the previous example, the results of test execution in brief format looks like this:

  • All tests pass:

    # Running tests

# Executed suites: 2. Pass: 2. Fail: 0. Incomplete: 0
# Executed tests: 4. Pass: 4. Fail: 0. Incomplete: 0

  • One of the tests fails in both test suites (globalSumTestTwo in suite suiteGlobalSumTestsSomeFail and statusTestTwo in suite suiteStatusTestsSomeFail):

    test_some_fail.c:42: Verify failed
Saturating sum did not return expected value.
Expression '(pst_long_long_t)(gSum) == (pst_long_long_t)(expected_gSum)' evaluated to false
lhs="4294967295" rhs="2147483648"
# Done test suiteGlobalSumTestsSomeFail/globalSumTestTwo: FAIL
# Done suite suiteGlobalSumTestsSomeFail: FAIL

# Failure(s) in test suiteStatusTestsSomeFail/statusTestTwo:
test_some_fail.c:72: Verify failed
Saturating sum did not return expected value.
Expression '(pst_long_long_t)(status2) == (pst_long_long_t)(expected_status)' 
  evaluated to false
lhs="1" rhs="0"
# Done test suiteStatusTestsSomeFail/statusTestTwo: FAIL
# Done suite suiteStatusTestsSomeFail: FAIL

# Executed suites: 2. Pass: 0. Fail: 2. Incomplete: 0
# Executed tests: 4. Pass: 2. Fail: 2. Incomplete: 0

Test Anything Protocol (TAP) Format

To see test results in the Test Anything Protocol or TAP format, use the option -format tap or -format a with the test runner executable (or set the configuration macro PST_OUTPUT to PST_OUTPUT_TAP). This format allows you to use TAP consumers to parse the test results and show them in a format of your choice. For more information on the TAP format, see Test Anything Protocol.

In the previous example, the results of test execution in TAP format looks like this:

  • All tests pass:

    ok 1 suiteGlobalSumTestsAllPass.globalSumTestOne
ok 2 suiteGlobalSumTestsAllPass.globalSumTestTwo
ok 3 suiteStatusTestsAllPass.statusTestOne
ok 4 suiteStatusTestsAllPass.statusTestTwo
1..4

  • One of the tests fails in both test suites (globalSumTestTwo in suite suiteGlobalSumTestsSomeFail and statusTestTwo in suite suiteStatusTestsSomeFail):

    ok 1 suiteGlobalSumTestsSomeFail.globalSumTestOne
not ok 2 suiteGlobalSumTestsSomeFail.globalSumTestTwo
# test_some_fail.c: 42: verify fail '(pst_long_long_t)(gSum) == (pst_long_long_t)(expected_gSum)' 'lhs="4294967295" rhs="2147483648"' Saturating sum did not return expected value.
ok 3 suiteStatusTestsSomeFail.statusTestOne
not ok 4 suiteStatusTestsSomeFail.statusTestTwo
# test_some_fail.c: 72: verify fail '(pst_long_long_t)(status2) == (pst_long_long_t)(expected_status)' 'lhs="1" rhs="0"' Saturating sum did not return expected value.
1..4

In this format, each line starts with the test status: ok (passing test case) or not ok (failing test). The test status is followed by an identifying number and the name of the test that failed in the format suiteName.testName. After each failing test starting with not ok, the next line starts with a # and contains details of the failing test.

Machine-Readable Format

To see test results in a machine-readable format, use the option -format mrf or -format m with the test runner executable (or set the configuration macro PST_OUTPUT to PST_OUTPUT_MRF). Like the TAP format, this format allows you to parse the test results and show them in a format of your choice. The format is useful when you run several tools together and need to extract Polyspace Test results from messages printed by other tools.

The results in this format consist of several parts, each beginning with a ##[ and ending with a ]. For more details on the format, see Machine-Readable Results for Tests Written Using Polyspace Test API.

HTML Format

You can generate test results in the .pstestr format and open the results in the Polyspace Platform user interface or generate an HTML report from the results. To generate results in HTML format via a .pstestr file, you have to follow these steps:

  1. Generate results in the machine-readable (.mrf) format using the test runner executable.

  2. Generate a .pstestr file from this intermediate format using the command polyspace-test -convert. You can open this file in the Polyspace Platform user interface. See also Open Polyspace Results in Polyspace Platform User Interface.

  3. Generate an HTML report from this .pstestr file, using the command polyspace-test -report.

For more information, see Define Test Results Format.

See Also

Topics

External Websites