Interface TestRunner
TestRunner
The Smart GWT TestRunner is a system for running a suite of Selenium tests on a periodic basis, comparing the results to previous results, and generating email alerts reporting on new test failures or fixes to tests that were previously failing.TestRunner is a key piece of implementing the Continuous Integration methodology, whereby continuous testing is applied so that regressions are caught immediately. This allows a product or application to be kept continuously at a very high level of quality, allowing for more frequent and predictable releases.
Database Setup
Each time TestRunner executes it by default stores results to a SQL database via two SQLDataSources:
- batchRun.ds.xml: stores global information about the run as a whole: an ID for the run, when it started and ended, and optional data to be used in emails generated by the system.
- testResult.ds.xml: stores the result of each individual test, including when it started and ended, and information about errors that occurred, if any
selenium/
directory of your SDK. If you choose to move them elsewhere, simply update the DataSources
location (configured by project.datasources in
server.properties).
These DataSources behave just like other SQLDataSources:
- they are compatible with all the database types that Smart GWT supports
- they will use the default database configured for your project, or you can set
DataSource.dbNamein the .ds.xml file to use a second database instead - you can setup the database connection and generate SQL tables using the
AdminConsole - you can build your own UI for viewing test results, by loading the
batchRunandtestResultDataSources like any other SQLDataSource and binding components such as ListGrids to them. - if you deploy an application that includes these DataSources, third-party tools can access these DataSources via the RESTHandler servlet
If needed, the IDs of these DataSources can be configured via the
server.properties settings
autotest.batchRunDS and autotest.testResultDS.
Note: If you use the default server.properties shipped with the SDK, TestRunner and the SDK web server will share a common SQL database, so that the web server and TestRunner cannot both run at once. This means that you must point TestRunner at the web server of a separate SDK installation - on a separate matchine or in a separate filesystem location.
Adding Test Files
TestRunner currently supports tests written in Selenese, Selenium's HTML-based format for
recording automated tests. The Selenium IDE can be used to record tests and save them in
Selenese format. For more background on the Selenium IDE, Smart GWT's extensions, and
the use of Selenium WebDriver, see the
Automated Testing Overview.
Test files should be saved with the extension .rctest.html. They should all appear under
a common root directory (called testRoot), but any level of nesting is allowed,
and any other files that appear under testRoot will be ignored; only
.rctest.html files will be processed.
The testRoot directory is passed to TestRunner when you execute it. In the
database and in emails, test are identified by their directory path relative to
testRoot.
Adding a test to the test suite is as simple as placing the .rctest.html file somewhere under
the testRoot directory; on the next TestRunner execution, TestRunner will
notice the new test and start reporting results for it (including reporting it as a failure
if it fails in its very first run).
The included test result viewing application (see below) also provides an interface to upload tests if you prefer not to allow direct filesystem access to the machine where TestRunner executes.
Running Test Runner
TestRunner is an ordinary Java class - com.isomorphic.autotest.TestRunner - and can be run from the command line in the usual fashion, or run programmatically from within a Java application using the wrapper class com.isomorphic.autotest.TestRunnerDriver. We also provide convenience scripts to run the TestRunner Java class in the SDK root directory.
Minimally, TestRunner needs to know the base directory of a set of test files. All files saved anywhere under this base directory which end in the extension .rctest.html will be assumed to be Selenese test files and executed.
As is standard for Selenese test files, the first command in the file is typically an "open" command with the URL of the application which should be opened in a browser so that subsequent commands can be run.
The assumption is that the application that will be tested is already deployed by the time TestRunner is run; how to automate building and deployment of applications is outside of the scope of this document, however, the recommendation is that a Revision-Control System (such as SVN, git or CVS) is used, and that every time a developer "checks in" or "commits" changes, the application being tested is built and deployed to a test server, then TestRunner is run. Continuous Build Servers (such as Hudson, Bamboo or CruiseControl) may help automate the step of building from source control and deploying to a test server, then such a Build Server can typically be configured to trigger TestRunner.
TestRunner requires several resources to be in expected default locations from the current
directory unless you provide overrides via the command line or server.properties.
Some of
the required resources are:
- the batch report email template, by default at mailTemplats/batchReport.template
- the selenium test template and batchRun/testResult dataSource XML files, by default in tools/selenium
- the dataSource XML schema files, by default in isomorphic/system/schema
Command-line Examples
The following command-line would run TestRunner, execute all tests under the default testRoot directory "tests", and commit the results:
java com.isomorphic.autotest.TestRunnerThis assumes your classpath environment has been set to include the isomorphic SDK JARs; you may invoke the convenience script test_runner.sh|.bat|.command in the SDK root directory to run the TestRunner Java class without having to set the classpath.
The following command-line would execute TestRunner as above, but run all tests under the directory "foo/bar" relative to the current directory, and email a report of the results:
java com.isomorphic.autotest.TestRunner -tr foo/bar -e user@company.comTo do the same, but only run a particular test, you can use the files option (-f):
java com.isomorphic.autotest.TestRunner -tr foo/bar -c -f test1.rctest.html -e user@company.comNote that when a file is specified, the default is not to commit the results unless requested via the commit option (-c).
Java API
The following Java code would do the same as the last command-line example:
TestRunnerDriver driver = new TestRunnerDriver();
driver.setTestRoot("foo/bar");
driver.setBatchCommit(true);
driver.setFiles(new String[] { "test1.rctest.html" });
driver.setAlertEmail("user@company.com");
driver.run();
TestRunner Configuration
TestRunner supports several more command-line options, or equivalent settings that can be applied via Java. The following table summarizes the command-line options, equivalent Java Setter in the DriverConfiguration interface, and it's behavior (including default behavior).
| Command-line Option | Java Setter | server.properties
Name | Behavior |
|---|---|---|---|
| -b <browser> | setBrowser | N/A | Specifies the browser string passed to Selenium. Default is *firefox
See UsingSelenium |
| -br <branch> | setBranch | autotest.branch | Specifies a branch for the batch, used in the batch run record and email notification. Default is MAIN.Test result comparison occurs per branch. |
| -c/-nc | setBatchCommit | N/A | This pair of argumentless options allows you to force the batch results to be committed (-c) or not committed (-nc). This is useful to override the default of committing (or of not committing if the -f option has been passed). |
| -cs | setCaptureScreenshot | N/A | Configures TestRunner to capture a PNG screenshot of the browser if a Selenium test fails, adding the image to the test result record. |
| -f <files> | setFiles | N/A | Specifies a file or list of files to run. This option can restrict which Selenium scripts under testRoot get run. Relative paths from the testRoot or bare filenames may be provided. When present, this option also disables the default behavior of committing test results. |
| -fr <path> | setFileRoot | N/A | Sets the root directory for all other file system paths. If not set, defaults to current working directory where Java was launched, or the web server root if TestRunner is run in a servlet. |
| -h | N/A | N/A | Lists available command-line options. |
| -hp <port> | setHttpPort | N/A | Sets the web server port Selenium should use to run the tests. Default is 8080 |
| -ht <host/IP> | setHttpTarget | N/A | Sets the target web server Selenium should use to run the tests. Default is localhost |
| -lg <message> | setBatchLog | N/A | Provides a log message to include in the record for this batch run. (No Default) |
| -lp | N/A | N/A | Informs TestRunner that a message or file has been piped to STDIN as the batch log message. |
| -sm | setSaveMessages | N/A | Configures TestRunner to save the client log messages for each test to the associated test record if the test fails. |
| -sr <path> | setServerFileRoot | N/A | Sets the serverFileRoot directory. Default is /. Selenium scripts executing open() commands on the httpTarget server will by use this default path. |
| -t <timestamp> | setTimestamp | N/A | Forces comparison of the batch results to be against the batch run with a timestamp closest to that provided, rather than the most recent batch run. Format is "2012-12-31 23:59:59" in the local time zone. |
| -tr <path> | setTestRoot | autotest.testRoot | Sets the testRoot directory relative to the current directory. By default, its
value is tests, and all Selenium scripts under the testRoot will be executed
by TestRunner. If TestRunner is being run in s servlet container, then this path must use Unix-style path separators (forward slashes) and be absolute (starting with a separator). In the host filesystem, the path will be interpreted relative to the root container directory. |
| -un <userName> | setUserName | autotest.userName | Specifies a user name for the batch run record. (No Default) |
| -vm <mode> | setServerLogMode | N/A | Configures TestRunner to collect the server log messages for each test if it fails. Legal modes are "none", "some", or "all". Default is some. |
| -vo <out> | setServerLogOutputMethod | N/A | Configures the output method that TestRunner will use to report or persist any server log messages; only has an effect for server log modes of "some" or "all". Legal values are "email", "datasource", or "both". Default is email. |
| -x/-nx | setMaximizeBrowser | N/A | Sets whether to maximize the browser for Selenium tests. If not explicitly set via this call, the browser will be maximized if and only if screenshots are being taken. |
Email Notifications
At completion of the batch of tests, TestRunner can automatically send out an email
notification summarizing the results of the test run, including error messages for
any newly failing tests. A velocity template file is used to control its format; see
Velocity Support.
The following velocity variables are available:
- $firstBatchFound. Whether baseline batch was found with which to compare
- $fixed. A list of the test results for tests fixed in this batch run
- $regression. A list of the test results for tests broken in this batch run
- $totalTestFiles. The total number of tests run in this batch run
- $passedTestFiles. The number of tests that passed in this batch run
- $batchStartTime. Timestamp associated with start of this batch run
- $batchLog. Log message, if any was provided for this batch run
The following options govern the Email Notifications:
| Command-line Option | Java Setter | server.properties
Name | Behavior |
|---|---|---|---|
| -cc <recipient> | setCcEmail | autotest.ccEmail | Sets the recipient email address for batch report email. This recipient will always be cc'd a copy of the batch report email, regardless of whether fixes or regressions are present. (No Default) |
| -e <recipient> | setAlertEmail | autotest.alertEmail | Sets the recipient email address for batch report email. If the "repeat email" recipient address has also been set via -re, this address will only be sent "alert email" reports where fixes or regressions are present. Otherwise, it will receive all batch report email. (No Default) |
| -m <mailHost> | setMailHost | autotest.mailHost | Specifies what mail host to use to send mail. If not provided, your mail software will decide what host to use. |
| -ms <subject> | setMailSubject | autotest.mailSubject | Sets subject line base to use when sending the email reporting batch results. Regressions and fixes info will be appended to the provided subject content. |
| -mt <file> | setMailTemplate | autotest.mailTemplate | Specifies what velocity template file to use when generating the batch report email for this batch run. Default is mailTemplates/batchReport.template |
| -ne | setNoEmail | N/A | Disables sending any email for the batch run. If recipient email addresses have not been set through the command line, Java setters, or server.properties, it's not needed. However, it may be useful in suppressing email in cases where they have been set. |
| -re <recipient> | setRepeatEmail | autotest.repeatEmail | Sets the recipient email address for batch report email. If the "alert email" recipient address has also been set via -e, this address will only be sent "repeat email" reports where no fixes or regressions are present. Otherwise, it will receive all batch report email. (No Default) |
| -se <sender> | setSenderEmail | autotest.senderEmail | Sets the sender email address for batch report email. Only needed if there is a problem sending email using the sender address generated by default. |
Note: If you choose not to have any email sent upon completion of a batch run, and decide not to commit the results to the DataSources, the results of each batch run can still be determined by examining the Java console log, which captures the output of each RC test script.
Result Viewer
TestRunner comes with a very very simple application for interactively viewing and searching test results, implemented in SmartClient technology. This application is meant as a starting point for building your own application for interactive viewing of test results, if you prefer to go beyond email notifications.
The source code for this application is just a single testResultViewer.jsp file in the
"selenium"
directory in the SDK; copy it anywhere under webroot in a project that includes
the Smart GWT Server and it will function.
The result viewing application also includes the ability to upload new test files to
testRoot as an alternative to providing testers with direct access to the
filesystem for the machine where TestRunner executes.
Getting Started FAQ
Q: When I run TestRunner, I want to target the SGWT showcase, but TestRunner fails due to
HSQLDB reporting a locked database.
A: By default, TestRunner uses the HSQLDB associated with the SGWT showcase when run from
the SGWT ZIP root directory. Therefore, if samples/showcase/war has been deployed to a
webserver, you must stop it before running TestRunner. One simple alternative is to deploy
the file showcase.war from the SGWT ZIP root, which has a separate copy of the HSQLDB. You
may also simply install another copy of the SGWT ZIP in a different location on the same
machine, or point TestRunner at a different machine using the -ht command-line option.
Q: When I run TestRunner, TestRunner fails reporting that DataSource BatchRun or TestResult
cannot be found.
A: These DataSources must be imported into the default HSQLDB before TestRunner can be used.
Use the "import" option of showcase/tools/adminConsole.jsp under the deployed SGWT showcase
to select and import the BatchRun and TestResult DataSources prior to running TestRunner.