Executing Tests

7 Executing Tests

The Jakarta EE 8 CTS uses the JavaTest harness to execute the tests in the test suite. For detailed instructions that explain how to run and use JavaTest, see the JavaTest User’s Guide and Reference.

This chapter includes the following topics:

Jakarta EE 8 CTS Operating Assumptions
Starting JavaTest
Validating Your Test Configuration
Running a Subset of the Tests
Test Reports

Note	The instructions in this chapter assume that you have installed and configured your test environment as described inChapter 4, "Installation," and Chapter 5, "Setup and Configuration," respectively.

7.1 Jakarta EE 8 CTS Operating Assumptions

The following are assumed in this chapter:

Jakarta EE 8 CI is installed and configured as described in this guide.
Detailed configuration will vary from product to product. In this guide, we provide details for configuring the Jakarta EE CI, Eclipse GlassFish 5.1. If you are using another CI, refer to that product’s setup and configuration documentation.
Java SE 8 software is correctly installed and configured on the host machine.
Jakarta EE 8 CTS is installed and configured as described in this guide.
Implementations of the technologies to be tested are properly installed and configured.

7.2 Starting JavaTest

There are two general ways to run Jakarta EE 8 CTS using the JavaTest harness software:

Through the JavaTest GUI
From the command line in your shell environment

Running the JavaTest harness from JavaTest GUI is recommended for initial configuration procedures, for validating your configuration, for selecting tests to run, and for general ease-of-use when running tests and viewing test reports.

Running the JavaTest harness from the command line is useful in headless server configurations, and for running tests in batch mode.

Note	The `build.xml` file in `<TS_HOME>/bin` contains the various Ant targets for the Jakarta EE 8 CTS test suite

7.2.1 To Start JavaTest in GUI Mode

Set the TS_HOME environment variable to the directory in which the Jakarta EE 8 CTS is installed.
Change to the <TS_HOME>/bin directory.
Ensure that the ts.jte file contains information relevant to your setup.
Refer to Chapter 5, "Setup and Configuration," for detailed configuration instructions.
Execute the ant gui target to start the JavaTest GUI:

ant gui

Using the JavaTest GUI to run CTS tests is described later in this guide. For detailed information about using the JavaTest interface, see the JavaTest User’s Guide.

Note

The forward and reverse keywords are available to filter the interop and/or rebuildable tests during a selected test run when running tests in the following directory only:

<TS_HOME>/src/com/sun/ts/tests/interop

Forward tests are interop tests that run from the Vendor Implementation to the Compatible Implementation, as well as rebuildable tests that run only against the Vendor Implementation. Reverse tests (with test names ending in _reverse) are interop tests that run from the Compatible Implementation to the Vendor Implementation, as well as rebuildable tests that run only against the Compatible Implementation.

To set one of these keywords in the Javatest GUI, select the Configure menu item, then select Change Configuration, then select Keywords, and set the appropriate keyword.

When one of these keywords has been set, executing tests in the directories above causes only those tests that match the keyword to be run. This can be useful when trying to debug failures with a particular test configuration. Note, however, for certification all tests in both directions must pass.

7.2.2 To Start JavaTest in Command-Line Mode

Set the TS_HOME environment variable to the directory in which Jakarta EE 8 CTS was installed.
Change to any subdirectory under <TS_HOME>/src/com/sun/ts/tests.
Ensure that the ts.jte file contains information relevant to your setup.
Refer to Chapter 5, "Setup and Configuration," for detailed configuration instructions.
Execute the runclient Ant target to start the JavaTest:

ant runclient

This runs all tests in the current directory and any subdirectories.

Example 7-1 Running the Jakarta EE 8 CTS Signature Tests

To run the Jakarta EE 8 CTS signature tests, enter the following commands:

cd <TS_HOME>/src/com/sun/ts/tests/signaturetest/javaee
ant runclient

Example 7-2 Running a Single Test Directory

To run a single test directory in the forward direction, enter the following commands:

cd <TS_HOME>/src/com/sun/ts/tests/jaxws/api/javax_xml_ws/Dispatch
ant -Dkeywords=forward runclient

Example 7-3 Running a Subset of Test Directories

To run a subset of test directories in the reverse direction, enter the following commands:

cd <TS_HOME>/src/com/sun/ts/tests/jaxws/api
ant -Dkeywords=reverse runclient

7.3 Validating Your Test Configuration

7.3.1 To Validate Your Configuration in GUI Mode

Start the JavaTest GUI and step through the basic configuration steps, if required, as described in Section 5.5.2, "The Configuration Interview."
In the JavaTest GUI tree view, expand the following directories: com, sun, ts, tests, samples.
Highlight the samples directory, right-click, and choose Execute These Tests.
If a work directory has not been specified, you are prompted to specify or create a new one.
From the JavaTest main menu, select File, then select Create Work Directory. The Create Work Directory dialog is displayed.
Locate or enter the name of the directory to which the test harness will write temporary files (for example, /tmp/JTWork), and click Create.
From the JavaTest main menu, select Run Tests, then select Start to run the default tests.
If your configuration information is incomplete, you are prompted to supply the missing parameters.
The JavaTest status bar grows while JavaTest tracks statistics relative to the files done, tests found, and tests done.
Check the results.
Test progress and results are displayed by the JavaTest harness.

7.3.2 To Validate Your Configuration in Command-Line Mode

Go to the <TS_HOME>/src/com/sun/ts/tests/samples directory.
Start the the test run by executing the following command:
```
ant runclient
```
All sample tests will be run, and should pass.
Generate test reports by executing the following commands:
1. Change to the <TS_HOME>/bin directory:
  cd <TS_HOME>/bin
2. Run the report Ant target:
  ant report
  Reports are written to the report directory you specified in <TS_HOME>/bin/ts.jte. If no report directory is specified, reports are written to the /tmp/JTreport directory (Solaris/Linux) or C:\temp\JTreport (Windows).

7.4 Running a Subset of the Tests

7.4.1 To Run a Subset of Tests in GUI Mode

From the JavaTest main menu, select Configure, then select Edit Configuration.
In the Configuration Editor, select Specify Tests to Run? from the option list on the left.
You are asked whether you want to run all or a subset of the test suite.
Click Yes, and then Next to run a subset of tests.
Select the tests you want to run from the displayed test tree, and then click Done.
You can select entire branches of the test tree, or use Ctrl+Click or Shift+Click to select multiple tests or ranges of tests, respectively.
After clicking Done, you are returned to the JavaTest main window.
Select Run Tests, then select Start to run the tests you selected.

7.4.2 To Run a Subset of Tests in Command-Line Mode

Change to the directory containing the tests you want to run.
For example, <TS_HOME>/src/com/sun/ts/tests/samples.
Start the test run by executing the following command:
```
ant runclient
```
The tests in <TS_HOME>/src/com/sun/ts/tests/samples and its subdirectories are run.

7.4.3 To Run a Subset of Tests in Batch Mode Based on Prior Result Status ^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^

You can run certain tests in batch mode based on the test’s prior run status by specifying the priorStatus system property when invoking Ant.

Invoke ant with the priorStatus property.

The accepted values for the priorStatus property are any combination of the following:

fail
pass
error
notRun

For example, you could run all Jakarta EE 8 tests with a status of failed and error by invoking the following commands:

cd <TS_HOME>/src/com/sun/ts/tests/ejb
ant -DpriorStatus="fail,error" runclient

Note that multiple priorStatus values must be separated by commas.

7.5 Using Keywords to Test Required and Optional Technologies

The Jakarta EE CTS includes some tests that may be optional depending on your implementation. For example, certain technologies are now optional for implementations of the full Jakarta EE Platform. There are other technologies which are optional for Web Profile implementations, but may be implemented. If implemented, optional tests must be run and pass. There are two mechanisms in place in the CTS which control whether or not a given set of tests is run - the javaee.level property in the ts.jte file (see Section 7.5.1, "Setting the javaee.level Property") and keywords (see Section 7.5.2, "Using Keywords to Create Groups and Subsets of Tests").

7.5.1 Setting the javaee.level Property

The ts.jte file includes the javaee.level property. This property serves two purposes. First, it is used to determine whether the implementation under test is a Jakarta EE Full profile (full) or Jakarta EE Web profile (web). Either "full" or "web" must be specified in the list values. A setting of "full" instructs the test harness to deploy EAR files. A setting of "web" instructs the test harness to deploy WAR files. The javaee.level property is also used to help determine which APIs in the signature tests are to be tested. The comments that precede the property setting in the ts.jte file provide additional information about setting this property.

The default setting is as follows:

javaee.level=full jaxr jaxrpc javaeedeploy

7.5.2 Using Keywords to Create Groups and Subsets of Tests

Each test in CTS has keywords associated with it. The keywords are used to create groups and subsets of tests. At test execution time, a user can tell the test harness to only run tests with or without certain keywords. This mechanism is used to select or omit testing on selected optional technologies. The "keywords" property can be set to a set of available keywords joined by "&" and/or "|".

To set the keywords system property at runtime, you must either pass it on the command line via -Dkeywords="" or in the JavaTest GUI, by opening the test suite and performing the following steps:

Select View, then select Filters, then select CurrentConfiguration.
Select Configure, then select ChangeConfiguration, then select Keywords.
In the Keywords dialog, select the Select Tests that Match check box, specify the desired keyword in the field, then click Done.
Only tests that have been tagged with that keyword will be enabled in the test tree.

The examples in the sections that follow show how to use keywords to run required technologies in both the Full and Web profile, run/omit running optional sets of tests in CTS, and run the Interoperability and Rebuildable tests in forward and reverse directions.

7.5.2.1 To Use Keywords to Run Required Technologies

Example 7-4 Running Tests for Required Technologies in the Full Profile

cd <TS_HOME>/src/com/sun/ts/tests
ant -Dkeywords=javaee runclient

Only tests that are required by the Full Profile will be run.

Example 7-5 Running Tests for All Required Technologies in the Web Profile

cd <TS_HOME>/src/com/sun/ts/tests
ant -Dkeywords=javaee_web_profile runclient

Only tests that are required by the Web Profile will be run.

Example 7-6 Running All Required Tests Except Connector Tests in the Full Profile

cd <TS_HOME>/src/com/sun/ts/tests
ant -Dkeywords="javaee & !connector" runclient

Example 7-7 Running All EJB Tests in the Full Profile

cd <TS_HOME>/src/com/sun/ts/tests
ant -Dkeywords=ejb runclient

Example 7-8 Running All EJB 3.2 Tests in the Full Profile

cd <TS_HOME>/src/com/sun/ts/tests
ant -Dkeywords=ejb32 runclient

Example 7-9 Running All EJB Tests in the Web Profile

cd <TS_HOME>/src/com/sun/ts/tests
ant -Dkeywords=ejb_web_profile runclient

7.5.2.2 To Use Keywords to Run Optional Technologies With the Full Profile

Keywords can be used to run subsets of tests from areas that are not required by the Jakarta EE 8 platform specification. Table 7-1 lists optional subsets of tests that can be run for the Full Profile and provides the technology-to-keyword mappings for each of the optional areas.

Table 7-1 Keyword to Technology Mappings for Full Profile Optional

Subsets

Technology Keyword

Technology	Keyword
EJB 1.x, CMP, BMP, entity beans	`ejb_1x_optional or` `javaee_optional`
EJB 2.x, CMP, BMP, entity beans	`ejb_2x_optional` or `javaee_optional`
EJBQL	`javaee_optional`
Jakarta Deployment	`javaeedeploy_optional` or `javaee_optional`
JAXR	`javaee_optional`
JAX-RPC	javaee_optional

EJB 1.x, CMP, BMP, entity beans

ejb_1x_optional or javaee_optional

EJB 2.x, CMP, BMP, entity beans

ejb_2x_optional or javaee_optional

EJBQL

javaee_optional

Jakarta Deployment

javaeedeploy_optional or javaee_optional

JAXR

javaee_optional

JAX-RPC

javaee_optional

Example 7-10 Running Tests for All Optional Technologies in the Full Profile

cd <TS_HOME>/src/com/sun/ts/tests
ant -Dkeywords=javaee_optional runclient

Example 7-11 Running the Optional JAXR and JAX-RPC Tests With the Required Full Profile Tests

cd <TS_HOME>/src/com/sun/ts/tests
ant -Dkeywords="javaee | jaxr | jaxrpc" runclient

The optional JAXR and JAX-RPC tests and the tests that are required by the Full Profile will be run.

7.5.2.3 To Use Keywords to Run Optional Subsets of Tests With the Web Profile

Keywords can be used to run subsets of tests from additional areas that are not required by the Jakarta EE 8 Web Profile specification. For example, if your server implements the Jakarta EE 8 Web Profile and the Jakarta Connector Architecture 1.7 technology, set the keywords to javaee_web_profile|connector_web_profile to enable running tests for both areas. The command below shows how to specify these keywords to run the tests in both areas.

ant -Dkeywords="(javaee_web_profile|connector_web_profile) runclient

Table 7-2 lists optional subsets of tests that can be run for the Web Profile and provides the technology-to-keyword mappings for each of the optional areas.

Table 7-2 Keyword to Technology Mappings for Web Profile Optional

Subsets

Technology Keyword

Technology	Keyword
Jakarta Connectors	`connector_web_profile`
Jakarta Authorization (formerly JACC)	`jacc_web_profile`
Jakarta Authentication (formerly JASPIC)	`jaspic_web_profile`
Jakarta Mail (formerly JavaMail)	`javamail_web_profile`
Jakarta Registries (formerly JAXR)	`jaxr_web_profile`
Jakarta XML RPC (formerly JAX-RPC)	`jaxrpc_web_profile`
Jakarta Messaging(formerly JMS)	`jms_web_profile`
XA	`xa_web_profile`

Jakarta Connectors

connector_web_profile

Jakarta Authorization (formerly JACC)

jacc_web_profile

Jakarta Authentication (formerly JASPIC)

jaspic_web_profile

Jakarta Mail (formerly JavaMail)

javamail_web_profile

Jakarta Registries (formerly JAXR)

jaxr_web_profile

Jakarta XML RPC (formerly JAX-RPC)

jaxrpc_web_profile

Jakarta Messaging(formerly JMS)

jms_web_profile

xa_web_profile

To add tests for other technologies, select the appropriate keyword from Table 7-2. This table provides a mapping of keywords to optional technologies (test directories) in the test suite and indicates optional test areas for the Jakarta EE 8 Web Profile.

Example 7-12 Running Tests for All Optional Technologies in the Web Profile

cd <TS_HOME>/src/com/sun/ts/tests
ant -Dkeywords=javaee_web_profile_optional runclient

Example 7-13 Running the Optional Jakarta Authorization and Authentication Tests With All Required Web Profile Tests

cd <TS_HOME>/src/com/sun/ts/tests
ant -Dkeywords="javaee_web_profile | jacc_web_profile | jaspic_web_profile" runclient

7.5.2.4 To Use Keywords to Run Optional Subsets for Jakarta Enterprise Beans Lite

Table 1-1 shows the CTS keywords you can use to test optional Jakarta Enterprise Beans (formerly EJB) Lite components. Components denoted with an asterisk (*) are pruned components; components without an asterisk are not required by EJB Lite.

Table 7-3 CTS Keywords for Optional Jakarta Enterprise Beans Lite Components

Component CTS Keyword

Component	CTS Keyword
Message-Driven Beans	`ejb_mdb_optional`
1x CMP/BMP Entity Beans *	ejb_1x_optional
2x CMP/BMP Entity Beans, Remote/Home Component, Local/Home Component *	`ejb_2x_optional`
3x Remote	`ejb_3x_remote_optional`
JAX-RPC Web Service Endpoint *	`ejb_jaxrpc_optional`
EJB QL *	`ejb_ql_optional`
Persistent Timer Service	`ejb_persistent_timer_optional`
Remote asyncrhonous session bean	`ejb_remote_async_optional`
RMI-IIOP Interoperability	`ejb_rmi_interop_optional`
EJB Embeddable Container	`ejb_embeddable_optional`

Message-Driven Beans

ejb_mdb_optional

1x CMP/BMP Entity Beans *

ejb_1x_optional

2x CMP/BMP Entity Beans, Remote/Home Component, Local/Home Component *

ejb_2x_optional

3x Remote

ejb_3x_remote_optional

JAX-RPC Web Service Endpoint *

ejb_jaxrpc_optional

EJB QL *

ejb_ql_optional

Persistent Timer Service

ejb_persistent_timer_optional

Remote asyncrhonous session bean

ejb_remote_async_optional

RMI-IIOP Interoperability

ejb_rmi_interop_optional

EJB Embeddable Container

ejb_embeddable_optional

Support for the following features has been made optional in this release:

EJB 2.1 and earlier Entity Bean Component Contract for Container-Managed Persistence and Bean-Managed Persistence
Client View of an EJB 2.1 and earlier Entity Bean
EJB QL: Query Language for Container-Managed Persistence Query Methods
Jakarta RESTful Web Services (formerly JAX-RPC) Based Web Service Endpoints
Jakarta XML-RPC (formerly JAX-RPC) Web Service Client View
Jakarta Registries (formerly JAXR) 1.0
Jakarta Deployment 1.2 are also optional

7.5.2.5 To Use Keywords to Run Tests in Selected Vehicles

The following vehicle keywords can be used to select or exclude the vehicles in which tests are run:

connectorservlet_vehicle
ejblitesecuredjsp_vehicle
ejbliteservlet_vehicle
ejbliteservlet2_vehicle
jaspicservlet_vehicle
pmservlet_vehicle
puservlet_vehicle
wsservlet_vehicle
servlet_vehicle
jsp_vehicle
web_vehicle
appclient_vehicle
wsappclient_vehicle
ejb_vehicle
wsejb_vehicle

These vehicles are defined in the <TS_HOME>/src/com/sun/ts/tests/common/vehicle subdirectory structures.

Example 7-14 Running Tests in the Jakarta Enterprise Beans (EJB) Vehicle Only

ant -Dkeywords="ejb_vehicle"  runclient

Example 7-15 Running Tests in Vehicles Other Than the Jakarta Enterprise Beans Vehicle

ant -Dkeywords="!ejb_vehicle"  runclient

7.5.2.6 To Use Keywords to Run Tests in Forward and Reverse Directions

The forward and reverse keywords can be used to filter the interop and/or rebuildable tests during a selected test run when running tests in one of the following directories only:

<TS_HOME>/src/com/sun/ts/tests/jaxws
<TS_HOME>/src/com/sun/ts/tests/jws
<TS_HOME>/src/com/sun/ts/tests/interop

Forward tests are interop tests that run from the Vendor Implementation to the Compatible Implementation, as well as rebuildable tests that run only against the Vendor Implementation. Reverse tests (with test names ending in _reverse) are interop tests that run from the Compatible Implementation to the Vendor Implementation, as well as rebuildable tests that run only against the Compatible Implementation.

To set one of these keywords when running in command-line mode, set the appropriate keyword using the keyword system property.

Example 7-16 Running Tests in the Forward Direction

ant -Dkeywords=forward runclient

Example 7-17 Running Tests in the Reverse Direction

ant -Dkeywords=reverse runclient

To set one of these keywords in the Javatest GUI, select the Configure menu item, then select Change Configuration, then select Keywords, and set the appropriate keyword.

7.6 Running Interop or JWS Reverse Tests _~_~_~_~_~_~_~_~_~_~_~_~_~_~_~~~

If you are running Interop or JWS reverse tests, which run against the Jakarta EE 8 CI, you must start the standalone deployment server in a separate shell on the same host as the CTS harness. The default deployment porting implementation goes through a standalone deployment server with a dedicated classpath. To start the standalone deployment server, change to the <TS_HOME>/bin directory and execute the start.auto.deployment.server Ant task.

7.7 Rebuilding Test Directories

The following directories require rebuilding, which is done by running the configure.datasource.tests Ant target:

com/sun/ts/tests/ejb30/lite/packaging/war/datasource
com/sun/ts/tests/ejb30/assembly/appres
com/sun/ts/tests/ejb30/misc/datasource

When the configure.datasource.tests Ant target is run from any directory, it rebuilds these directories and any required subdirectories.

The com/sun/ts/tests/jms/ee20/resourcedefs directory must also be rebuilt. Run the build.special.webservices.clients Ant target to rebuild the tests in this directory.

The database properties in the CTS bundle are set to Derby database. If any other database is used, the update.metadata.token.values ant target needs to be executed for metadata-complete tests.

The following directories require rebuilding: src\com\sun\ts\tests\appclient\deploy\metadatacomplete\testapp.

This can be done by running the update.metadata.token.values Ant target.

7.8 Test Reports

A set of report files is created for every test run. These report files can be found in the report directory you specify. After a test run is completed, the JavaTest harness writes HTML reports for the test run. You can view these files in the JavaTest ReportBrowser when running in GUI mode, or in the web browser of your choice outside the JavaTest interface.

To see all of the HTML report files, enter the URL of the report.html file. This file is the root file that links to all of the other HTML reports.

The JavaTest harness also creates a summary.txt file in the report directory that you can open in any text editor. The summary.txt file contains a list of all tests that were run, their test results, and their status messages.

Although you can run the Ant report target from any test directory, its support is not guaranteed in the lower level directories. It is recommended that you always run the report target from <TS_HOME>/bin, from which reports are generated containing information about which tests were or were not run.

7.8.1 Creating Test Reports

7.8.1.1 To Create a Test Report in GUI Mode

From the JavaTest main menu, select Report, then select Create Report.
You are prompted to specify a directory to use for your test reports.
Specify the directory you want to use for your reports, and then click OK.
Use the Filter list to specify whether you want to generate reports for the current configuration, all tests, or a custom set of tests.
You are asked whether you want to view report now.
Click Yes to display the new report in the JavaTest ReportBrowser.

7.8.1.2 To Create a Test Report in Command-Line Mode

Specify where you want to create the test report.

To specify the report directory from the command line at runtime, use:
```
ant report -Dreport.dir="report_dir"
```
Reports are written for the last test run to the directory you specify.
To specify the default report directory, set the report.dir property in <TS_HOME>/bin/ts.jte.
For example, report.dir="/home/josephine/reports".
To disable reporting, set the report.dir property to "none", either on the command line or in ts.jte.
For example:
```
ant -Dreport.dir="none"
```

Troubleshooting

Although you can run the report Ant target from any test directory, its support is not guaranteed in the lower level directories. It is recommended that you always run the report target from <TS_HOME)/bin, from which reports are generated containing information about which tests were or were not run.cc

7.8.2 Viewing an Existing Test Report

7.8.2.1 To View an Existing Report in the JavaTest Report Browser

From the JavaTest main menu, select Report, then select Open Report.
You are prompted to specify the directory containing the report you want to open.
Select the report directory you want to open, and then click Open.
The selected report set is opened in the JavaTest Report Browser.

7.8.2.2 To View an Existing Report in a Web Browser

Use the Web browser of your choice to view the report.html file in the report directory you specified from the command line or in ts.jte.

The current report directory is displayed when you run the report target.