|Data regression testing|
Test case implementation
The test cases should be provided in terms of a single executable with two operating mode (selected with input parameters, as in the example proposed): a WRITER, to be used to produce the reference data of the current release, and a READER, to be executed against the data for all the available releases.
In order to execute the output comparison with the reference, the oval configuration should only include the READER command (see example).
Input data for the READER, as data output for the WRITER, are stored in a specific local (in the execution directory) folder named ./data .
Reference data storage area
The area /afs/cern.ch/sw/lcg/app/pool/reference_data has been created (reference data storage area) and it is currently fully accessible. It contains for the moment the folders for two previous pool releases: POOL_1_3_0 and POOL_1_3_3 .
In each release folder are hosted sub-folders for the available data regression tests.
Scripts for data export/import from/to the storage area
Two scripts are available for the steering of the regression test process:
It takes the release label as an argument (like POOL_1_3_0) and copies in the storage area the files involved in the test case, which are assumed hosted in the ./data local folder. This script is actually not supposed to be used by the developer for the release under validation, but only to store data for previous releases. The data export for the current release will be executed by the release manager.
It imports from the storage area data from the releases for which the current test label is available, and runs the corresponding test checking the output against the reference. Files are copied from the storage area in the local ./data area, which is cleaned up at each cycle. Output files and logs are kept for each release respectively in the working directory and in a log folder.
#First run the WRITER for the release you want to store:
#Export data using the script (only a previous release! For the current
release only the release manager will do it!) ~/POOL_1_3_0/src/Tests/DataRegression_MyComponent_MyTestCase>../../../ReleaseTools/Scripts/pool_export_data
#The previous command copies all files stored in ~/POOL_1_3_0/src/Tests/DataRegression_MyComponent_MyTestCase/data in /afs/cern.ch/sw/lcg/app/pool/reference_data/POOL_1_3_0/DataRegression_MyComponent_MyTestCase, creating the folders when they don't exist already.
# use the stored data to check the current release: ~/POOL_1_4_0-kappa/src/Tests/DataRegression_MyComponent_MyTestCase>../../../ReleaseTools/Scripts/pool_regression_test
For each release in the storage area where the label DataRegression_MyComponent_MyTestCase
is found, the script will do:
- remove the folder src/Tests/DataRegression_DataSvc_MyComponent_MyTestCase/data with its contents, if any
- copy the files found in src/Tests/DataRegression_MyComponent_MyTestCase/data.
- run the test execution with 'oval run'.
- compare the output produced with the reference output.
When all the available release are scanned, an ending message is sent out. For each release, the output is available in the file test_DataRegression_MyComponent_MyTestCase_RELEASEXXX.log, while the log of the execution+comparison is in log/RELEASEXXX.log.
Download the scripts and run the available examples:
1) checkout pool/Tests/DataRegression_DataSvc_CMSCollDataModel with tag
2) checkout pool/Tests/Libraries/TestsBase with tag Tests-Libraries-TestBase-4-0-1
3) checkout pool/ReleaseTools (head)
4) compile src/Tests
5) cd src/Tests/DataRegression_DataSvc_CMSCollDataModel
6) run the script: ReleaseTools/Scripts/pool_regression_test