Description

The ability to control which testcases should be included in a specific run is important. The ability to code a single testcase to cover different requirements avoids duplication and deduces maintenance costs. Here are some considerations for subsetting testcases:

1. OS environments like Windows, Linux, etc...
2. External dependencies that may not always be available
3. Code versions for software being tested or dependent software
4. Performance considerations - having tests that can meet various turn-around requirements
5. Failing testcases that need to be deferred waiting for a fix
6. Testcases written before implementation is completed or available to QA organizations

Using

GINT provides mechanisms to mark testcases for different subsets and run tests to include or exclude testcases based on subsets. Subsetting can be done statically (declared) or dynamically (determined by script logic). In continuous integration (CI) and other automated environments, dynamic subsetting can be used to prevent external factors from failing test runs unnecessarily. Other automation can report directly on the failing external conditions and notify the appropriate people.GINT uses the term level. Each testcase has a level value that defaults to 1. The following Parameters and Functions control the interaction with the testcase level and what testcases are run.

• level - defaults to 1. Include testcases that have an integer level less than or equal to this setting.
• includeLevels - defaults to [ true ]. Include testcases that have a non-integer level in the list of values.
• excludeLevels - defaults to [ false ]. Exclude testcases that have a level value in the list of values. Exclude takes precedence over other factors.

Functions

The following functions are available to test scripts:

• getLevel, setLevel
• getIncludeLevels, setIncludeLevels, addIncludeLevels
• getExcludeLevels, setExcludeLevels, addExcludeLevels

Examples

OS environments

Test script dynamically figures out the tests to be included.

...
gint.addIncludeLevels(helper.isWindows() ? 'windows' : 'linux')

gint.add('list',
    [name: 'dir', level: 'windows', cmd: 'dir c:\\.', data: ['Program Files', 'Windows']],
    [name: 'ls',  level: 'linux',   cmd: 'ls /',      data: ['usr', 'bin']],
])
...

External dependency

Test script needs the database configuration provided as a testcase parameter or property in order to run this testcase. The level in this case is a simple boolean - true to include, false to exclude.

...
haveDatabase = databaseConfig.database != null   // if no database is configured, need to defer testcases

gint.add(
    [name: 'getConnection', stopOnFail: true, level: haveDatabase,
        inline: { connection = sqlHelper.getConnection(databaseConfig); true; },
    ]
)
...

Performance

By default, a test run will only run testcases with level less than or equal to 1. Anything higher will be deferred. So identifying long running tests makes it possible for quick turn around times on most tests and allow for nightly or special request testing to cover all tests. Specify a parameter of -Dlevel=9 to include the long running test. The meaning of level numbers is test defined.

...
gint.add([
    [name: 'quick', inline: { shortRunningFunction() } ],  
    [name: 'slow',  inline: { longRunningFunction()  }, level: 9 ],
])
...