10 October 2010

Development Productivity

Integration testing can be time consuming, difficult, and complex. GINT reduces the time and difficulty of writing testcases while helping to manage the complexity.

1. Declarative style for defining testcases - a testcase is a key-value map declared in groovy code, a CSV file, or a database table. Minimal programming skill is required for testcase definitions.
2. Groovy scripting - more complex tests may require some level of scripting and Groovy is a powerful and compact scripting language. Groovy also has a short learning curve.
3. Command line based - the quickest way to run and debug tests. Run last failed, individual testcases, groups of testcases, or all. Easy to provide parameters.
4. Parameters - the ability to easily provide control and testcase specific parameters to minimize duplication and maximize re-use. More ...
5. Automatic comparisons - text or regex based comparisons on result data to determine success or failure of testcase. /wiki/spaces/GINT2/pages/116588607.
6. Subsets - the ability to subset testcases for a test run. Subsets for specific environments (like Windows, Linux, etc...), code versions, or turn around requirements (short verses long running tests). Another example - testcase can be marked as deferred waiting for a fix. This behavior can be controlled statically (declared) or dynamically (determined by script logic).
7. setUp and tearDown support.

Example script

includeTool << org.swift.tools.Gint  // test framework

gint.initialize(this) // required

gint.add([name: 'simple1', inline: { assert 1 == 1 } ])

gint.finalizeTest() // final preparations for running tests

Example run

gant -f simple.gant

 = = = = = = = = = =  simple started at Tue Aug 24 21:26:13 CDT 2010  = = = = = = = = = =

    [start] simple1
   [ending] simple1
 [complete] simple1 - 0.022 secs


     [info] Successful testcases  . . . . . . . . . . : 1    <<< TEST SUCCESSFUL
     [info] Total testcases . . . . . . . . . . . . . : 1
     [info] Elapsed run time  . . . . . . . . . . . . : 1.027 secs

 = = = = = = = = = =  simple completed at Tue Aug 24 21:26:14 CDT 2010  = = = = = = = = = =

BUILD SUCCESSFUL
Total time: 2.51 seconds

Performance

Integration level tests can often take a long time to run serially. GINT provides automatic parallel running of testcases with simple options to manage ordering and dependencies. This makes effective use of multi-core processors to reduce execution time significantly. Reduced execution time means that more tests can be run in CI environments while still providing quick turnaround. GINT capabilities make it relatively easy to exploit this capability and manage complexity.

/wiki/spaces/GINT2/pages/116555886

Serial

[info] Successful testcases  . . . . . . . . . . : 103    <<< TEST SUCCESSFUL
[info] Deferred testcases  . . . . . . . . . . . : 9
[info] Level of run  . . . . . . . . . . . . . . : 1
[info] Deferred levels . . . . . . . . . . . . . : [false, 3, 99]
[info] Total testcases . . . . . . . . . . . . . : 112
[info] Elapsed run time  . . . . . . . . . . . . : 125.19 secs

Parallel

[info] Successful testcases  . . . . . . . . . . : 103    <<< TEST SUCCESSFUL
[info] Deferred testcases  . . . . . . . . . . . : 9
[info] Level of run  . . . . . . . . . . . . . . : 1
[info] Deferred levels . . . . . . . . . . . . . : [false, 3, 99]
[info] Total testcases . . . . . . . . . . . . . : 112
[info] Elapsed run time  . . . . . . . . . . . . : 64.361 secs

Continuous Integration

Continuous integration (CI) based development relies on tests that can be run frequently using automation. GINT makes it easier to include integration level tests in a CI environment. GINT supports CI tools like Bamboo and similar. Specific GINT features make this possible and effective:

• Command line driven.
• JUnit-style XML output. /wiki/spaces/GINT2/pages/116588712.
• Test parameters. /wiki/spaces/GINT2/pages/116588789.
• Test subsets. /wiki/spaces/GINT2/pages/111870064.
• Parallel running of testcases. /wiki/spaces/GINT2/pages/116555886.

Parameters

Integration tests need to cover a wide gambit of environments and conditions. Often, tests are dependent on external factors that may change frequently. One technique to handle these environments and factors is through external parameterization of testcases and test runs. GINT provides extensive support for parameters to minimize duplication and maximize re-use of tests.

Use

1. Control parameters are used to determine how the test is run. For example, subsetting of what testcases should be run.
2. Test specific parameters are used in testcase logic.

Where specified

1. Test script
2. Command line parameters. /wiki/spaces/GINT2/pages/116588789
3. User specific property file. /wiki/spaces/GINT2/pages/116555873
4. Project or current directory based property file. /wiki/spaces/GINT2/pages/116555873
5. Test profiles using multiple user defined property files. /wiki/spaces/GINT2/pages/111935618

Example

...
gint.add([name: 'test6', description: 'Green test', level: 'green', ...])
...

gant -f level.gant -Dclean -DincludeLevels=green,blue -Dlevel=0 

 = = = = = = = = = =  level started at Tue Sep 07 09:20:40 CDT 2010  = = = = = = = = = =
    [start] test6 (Green test)
    [start] test7 (Blue test)
   [ending] test6 (Green test)
 [complete] test6 - 0.017 secs

   [ending] test7 (Blue test)
 [complete] test7 - 0.027 secs
  
     [info] Successful testcases  . . . . . . . . . . : 2    <<< TEST SUCCESSFUL
     [info] Deferred testcases  . . . . . . . . . . . : 5
     [info] Level of run  . . . . . . . . . . . . . . : 0
     [info] Deferred levels . . . . . . . . . . . . . : [false, 1, 99, true]
     [info] Total testcases . . . . . . . . . . . . . : 7
     [info] Elapsed run time  . . . . . . . . . . . . : 0.744 secs
   
 = = = = = = = = = =  level completed at Tue Sep 07 09:20:40 CDT 2010  = = = = = = = = = = 
  
BUILD SUCCESSFUL
Total time: 2.05 seconds

Interactive testing

Manual and semi-automated tests can be included in GINT test scripts. GINT uses instant messaging (IM) to make it possible to interact with testcases requiring manual steps. This can improve test execution productivity and provide standard reporting.

/wiki/spaces/GINT2/pages/111870041

Default content

[name: 'simple', inline: { }, im: true],  // default content, before test is run

Testcase simple from test interactiveexample is requesting a response before running. 
successResponses: [success, s, yes, y, true], failResponses: [fail, f, no, n, false],
waitResponses: [wait], skipResponses: [skip]

Specific content

[name: 'manualTest1', inline: { }, im: [content: "Instructions for running manualTest1"]]

Instructions for running manualTest1    successResponses: [success, s, yes, y, true],
failResponses: [fail, f, no, n, false], waitResponses: [wait], skipResponses: [skip]

Extensions

Open Source

Client Platforms