QTapTestLogger Class

Header: #include <QTapTestLogger>
Inherits: QAbstractTestLogger

Detailed Description

\internal \inmodule QtTest

QTapTestLogger implements the Test Anything Protocol v13.

The Test Anything Protocol (TAP) is a simple plain-text interface between testing code and systems for reporting and analyzing test results. Since QtTest doesn't build the table for a data-driven test until the test is about to be run, we don't typically know how many tests we'll run until we've run them, so we put The Plan at the end, rather than the beginning. We summarise the results in comments following The Plan.

YAMLish

The TAP protocol supports inclusion, after a Test Line, of a "diagnostic block" in YAML, containing supporting information. We use this to package other information from the test, in combination with comments to record information we're unable to deliver at the same time as a test line. By convention, TAP producers limit themselves to a restricted subset of YAML, known as YAMLish, for maximal compatibility with TAP consumers.

YAML (see yaml.org for details) supports three data-types: mapping (hash or dictionary), sequence (list, array or set) and scalar (string or number), much as JSON does. It uses indentation to indicate levels of nesting of mappings and sequences within one another. A line starting with a dash (at the same level of indent as its context) introduces a list entry. A line starting with a key (more indented than its context, aside from any earlier keys at its level) followed by a colon introduces a mapping entry; the key may be either a simple word or a quoted string (within which numeric escapes may be used to indicate unicode characters). The value associated with a given key, or the entry in a list, can appar after the key-colon or dasy either on the same line or on a succession of subsequent lines at higher indent. Thus

 - first list item
 -
   second: list item is a mapping
   with:
   - two keys
   - the second of which is a list
 - back in the outer list, a third item

In YAMLish, the top-level structure should be a hash. The keys supported for this hash, with the meanings for their values, are:

  • message: free text supplying supporting details
  • severity: how bad is it ?
  • source: source describing the test, as an URL (compare file, line)
  • at: identifies the function (with file and line) performing the test
  • datetime: when the test was run (ISO 8601 or HTTP format)
  • file: source of the test as a local file-name, when appropriate
  • line: line number within the source
  • name: test name
  • extensions: sub-hash in which to store random other stuff
  • actual: what happened (a.k.a. found; contrast expected)
  • expected: what was expected (a.k.a. wanted; contrast actual)
  • display: description of the result, suitable for display
  • dump: a sub-hash of variable values when the result arose
  • error: describes the error
  • backtrace: describes the call-stack of the error

In practice, only at, expected and actual appear to be generally supported.

We can also have several messages produced within a single test, so the simple message / severity pair of top-level keys does not serve us well. We therefore use extensions with a sub-tag messages in which to package a list of messages.

See also QAbstractTestLogger.