assert

ETLUnit 3.9.6

 

Draft in Progress

This document is a draft and is under development.

 

Description

  • Declare a condition that should be true.
  • Throw an error if the test breaks for an unexpected reason.
  • End in failure if the assertion proves false.

Attributes

List of assert() Attributes

assert() for Database

 

 

 

 

 

 

 

- >> REF:

 Database Feature Module - Extract

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

- >> REF:

 Database Feature Module - Execute Common

 

 

 

 

- >> REF:

 Database Feature Module - Configuration

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

default-implementation-id:  - >> TYPE:  string

implementation-defaults- >> TYPE:  object (key-value pairs, comma-separated)

{

<implementation-default-name>:  - >> TYPE:  string

{

server-name- >> TYPE:  string

server-port- >> TYPE:  integer

implementation-properties:  - >> TYPE:  object (key-value pairs, comma-separated)

user-name- >> TYPE:  string

password- >> TYPE:  string or null

}, ... multiple implementation defaults possible

}

default-connection-id- >> TYPE:  string

data-file-null-delimiter- >> TYPE:  string

data-file-column-delimiter- >> TYPE:  string

data-file-row-delimiter- >> TYPE:  string

database-definitions:   - >> TYPE:  object (key-value pairs, comma-separated)

{

<database-name>  - >> TYPE:  string

{

implementation-id:  - >> TYPE:  string

server-name- >> TYPE:  string

server-port- >> TYPE:  integer

implementation-properties- >> TYPE:  object (key-value pairs, comma-separated)

user-name- >> TYPE:  string

password- >> TYPE:  string or null

schema-scripts:    - >> TYPE:  list of strings

}, ...  multiple database definitions possible

}


 

connection-id:  - >> TYPE:  string

mode- >> TYPE:  string 


 

source-table- >> TYPE:  string

target- >> TYPE:  string

target-file- >> TYPE:  string

source-schema- >> TYPE:  string

sql-script- >> TYPE:  string

sql- >> TYPE:  string

source-reference-file-type- >> REF:

 File Feature Module - Reference File Type

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Either:

- >> TYPE:  string

Or:

- >> TYPE:  object (key-value pairs)

{

id- >> TYPE:  string

classifier- >> TYPE:  string

version- >> TYPE:  string

column-list- >> TYPE:  list of unique strings

column-list-mode:  - >>

POSSIBLE VALUES:

include

exclude

}

 



target-reference-file-type- >> REF:

 File Feature Module - Reference File Type

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Either:

- >> TYPE:  string

Or:

- >> TYPE:  object (key-value pairs)

{

id- >> TYPE:  string

classifier- >> TYPE:  string

version- >> TYPE:  string

column-list- >> TYPE:  list of unique strings

column-list-mode:  - >>

POSSIBLE VALUES:

include

exclude

}

 



reference-file-type- >> REF:

 File Feature Module - Reference File Type

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Either:

- >> TYPE:  string

Or:

- >> TYPE:  object (key-value pairs)

{

id- >> TYPE:  string

classifier- >> TYPE:  string

version- >> TYPE:  string

column-list- >> TYPE:  list of unique strings

column-list-mode:  - >>

POSSIBLE VALUES:

include

exclude

}

 



column-list- >> TYPE:   list of unique strings

column-list-mode- >>

POSSIBLE VALUES: 

include

exclude



assertion-mode- >>

POSSIBLE VALUES:

equals  - - default mode

empty

notEmpty

failure-id- >> TYPE:  string

Go to top

assert() for File

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

- >> REF:

 File Feature Module - Configuration

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

reference-file-types:  - >> TYPE:  object (key-value pairs)

{

<reference-file-type-name>:  - >> TYPE:  string

{

default-version:  - >> TYPE:  string

default-remote-version:  - >> TYPE:  string

default-classifier:  - >> TYPE:  string

default-remote-classifier:  - >> TYPE:  string

classifiers:  - >> TYPE:  object (key-value pairs)

{

<classifier-name>:  - >> TYPE:  string

{

default-version:  - >> TYPE:  string

default-remote-version:  - >> TYPE:  string

}, ... multiple classifiers possible

}

}, ... multiple reference file types possible

} 


 

source-file- >> TYPE:  string  - - REQUIRED

sql- >> TYPE:  string

sql-script- >> TYPE:  string

target- >> TYPE:  string

classifier- >> TYPE:  string

context- >> TYPE:  string

producer- >> TYPE:  string

failure-id- >> TYPE:  string

assertion-mode- >>

POSSIBLE VALUES:

equals  - - default mode

exists

notExists

empty

notEmpty

reference-file-type- >> REF:

 File Feature Module - Reference File Type

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Either:

- >> TYPE:  string

Or:

- >> TYPE:  object (key-value pairs)

{

id- >> TYPE:  string

classifier- >> TYPE:  string

version- >> TYPE:  string

column-list- >> TYPE:  list of unique strings

column-list-mode:  - >>

POSSIBLE VALUES:

include

exclude

}

 



base-source-reference-file-type- >> REF:

 File Feature Module - Reference File Type

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Either:

- >> TYPE:  string

Or:

- >> TYPE:  object (key-value pairs)

{

id- >> TYPE:  string

classifier- >> TYPE:  string

version- >> TYPE:  string

column-list- >> TYPE:  list of unique strings

column-list-mode:  - >>

POSSIBLE VALUES:

include

exclude

}

 



source-reference-file-type- >> REF:

 File Feature Module - Reference File Type

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Either:

- >> TYPE:  string

Or:

- >> TYPE:  object (key-value pairs)

{

id- >> TYPE:  string

classifier- >> TYPE:  string

version- >> TYPE:  string

column-list- >> TYPE:  list of unique strings

column-list-mode:  - >>

POSSIBLE VALUES:

include

exclude

}

 



target-reference-file-type- >> REF:

 File Feature Module - Reference File Type

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Either:

- >> TYPE:  string

Or:

- >> TYPE:  object (key-value pairs)

{

id- >> TYPE:  string

classifier- >> TYPE:  string

version- >> TYPE:  string

column-list- >> TYPE:  list of unique strings

column-list-mode:  - >>

POSSIBLE VALUES:

include

exclude

}

 



column-list- >> TYPE:  unique list of strings

column-list-mode- >>

POSSIBLE VALUES:

include

exclude

Go to top

assert() for Data Sets

 

 

 

data-set-name:  - >> TYPE:  string  - - REQUIRED

data-set-id:  - >> TYPE:  string

failure-id:  - >> TYPE:  string

ignore-data-set-properties:  - >> TYPE:  boolean

Go to top

Individual assert() Attributes

assert() for Database Context

[Database Feature Module - Configuration properties not listed here.]

connection-id:

  • Corresponds to id property in the @Database annotation.
  • Any value other than one that corresponds to the id property in the @Database annotation is not allowed.

mode:

  • Corresponds to one member in the list of modes, found in the @Database annotation.
  • Any value other than one that corresponds to one of the modes listed in the @Database annotation is not allowed.

source-table:

  • Database table with "actual" data for the assert operation.

target:

  • Refers to a data file, found in folder data/, a sibling to the ETLUnit test class.

target-file:

  • Not supported.

source-schema:

  • The source table's schema name.

sql-script:

  • Store the SQL script in a file with the extension ".sql" in its name, in folder sql/, a sibling to the ETLUnit test.

sql:

  • In-line SQL script.

source-reference-file-type:

  • Name of the FML file to be applied to the source table.  Does not include extension ".fml."

target-reference-file-type:

  • Name of the FML file to be applied to the target data file.  Does not include extension ".fml."

reference-file-type:

  • Synonym to source-reference-file-type.

column-list:

  • A list of column names to include or exclude.

column-list-mode:

  • Valid values:
    • include
      • Include for consideration the columns enumerated in column-list.
    • exclude
      • Exclude for consideration the columns enumerated in column-list.

assertion-mode:

  • Valid values:
    • equals
      • Byte for byte, the source content matches the target content.
    • empty
      • The source is empty.
    • notEmpty
      • The source is not empty.

failure-id:

  • In case of failure in the assert operation, use this failure id instead of the one that would normally be generated.

Go to top

assert() for File Context

[File Feature Module - Configuration properties not listed here.]

source-file:

  • Required
  • This is the "actual" data in the assert operation.  It may come from one of two places:
    • Output from the execute operation.
    • A file that has been registered with ETLUnit with the register operation.

sql:

  • In-line SQL script.

sql-script:

  • Store the SQL script in a file with the extension ".sql" in its name, in folder sql/, a sibling to the ETLUnit test.

target:

  • Refers to a data file, found in folder data/, a sibling to the ETLUnit test class.

classifier:

  • Value is a string.  Ad-hoc name of a classifier that will prevent, for instance, a registered file from producing an ERR_REGISTER_FILE_NAME_DUPLICATED error.

context:

  • Used like classifier, it sets the assert in a different context than similar asserts.  May prevent a registered file from producing an ERR_REGISTER_FILE_NAME_DUPLICATED error.

producer:

failure-id:

  • In case of failure in the assert operation, use this failure id instead of the one that would normally be generated.

assertion-mode:

  • Valid values:
    • equals - - the default
      • Byte for byte, the source and target files compare favorably.
    • exists
      • The source file exists
    • notExists
      • The source file does not exist
    • empty
      • The source file is empty
    • notEmpty
      • The source file is not empty

reference-file-type:

  • Synonym to source-reference-file-type.

base-source-reference-file-type:

  • Reference file type for the source file in a three-tiered assertion (source-file, sql or sql-script, and target).
  • source-file must have FML for this to work.
  • Example:

 

The variable $file-table-name, below, is internal to ETLUnit.  In this case, it points to the registered file, TEST.

file_register_iv.etlunit

@JoinSuite(name:'file_register')

class file_register_iv

{

@Description(description: 'Copied from OptionsIntegrationTest, in etlunit-file.')

@Test

sqlPolymorphic()

{

register(

file: 'POLYM',

registered-name: 'TEST'

);

assert(

source-file: 'TEST',

target: 'TEST_SQL_POLY',

sql: "SELECT FILE_NAME_DATE, 'POP' || POS_TYPE || 'STAR' AS POP_STAR FROM $file-table-name",

base-source-reference-file-type: 'sqlPolymorphic',

source-reference-file-type: 'test_ii',

target-reference-file-type: 'test_ii'

);

}

}

 

 

source-reference-file-type:

  • Name of the FML file to be applied to the source file.  Does not include extension ".fml."

target-reference-file-type:

  • Name of the FML file to be applied to the target file.  Does not include extension ".fml."

column-list:

  • A list of column names to include or exclude.

column-list-mode:

  • Valid values:
    • include
      • Include for consideration the columns enumerated in column-list.
    • exclude
      • Exclude for consideration the columns enumerated in column-list.

Go to top

assert() for Data Sets Context

data-set-name:

  • Required
  • Matches the name of a text file in a folder 'dataset,' which is a sibling of the ETLUnit test class.
  • The data set text file has file extention '.dataset.'

data-set-id:

  • The data-set-id's value matches the value of the id property in the data set member's properties section.

failure-id:

  • The failure to be generated by ETLUnit if the operation using the data set fails.
  • A failure elsewhere in the test method will not invoke this failure ID.

ignore-data-set-properties:

  • Causes the ETLUnit test to ignore properties included in the data set member's properties section.
  • This does not include the id property in the data set member's properties section.

Go to top

Examples

Here is an example of the assert operation in the Database context that involves some configuration and FML.

Supporting Files

Configuration

 

  • The POM is placed in the root of the project.
pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

...

<dependencies>

<dependency>

<groupId>org.bitbucket.bradleysmithllc.etlunit</groupId>

<artifactId>etlunit-core</artifactId>

<version>${etlunit.project.version}</version>

</dependency>

<dependency>

<groupId>org.bitbucket.bradleysmithllc.etlunit</groupId>

<artifactId>etlunit-h2-database</artifactId>

<version>${etlunit.project.version}</version>

</dependency>

<dependency>

<groupId>org.bitbucket.bradleysmithllc.etlunit</groupId>

<artifactId>etlunit-cli</artifactId>

<version>${etlunit.project.version}</version>

</dependency>

<dependency>

<groupId>org.bitbucket.bradleysmithllc.etlunit</groupId>

<artifactId>etlunit-file</artifactId>

<version>${etlunit.project.version}</version>

</dependency>

</dependencies>

...

</project>

  • etlunit.json may go under folder src/test/resources/config.
etlunit.json

{

  "install-features": ["h2-database", "file"],

  "features": {

    "database": {

      "database-definitions": {

        "edw": {

          "implementation-id": "h2",

          "schema-scripts": [

            "schema.ddl"

          ]

        }

      }

    }

  }

}

  • schema.ddl might be placed under src/main/h2/edw, in this case.  The project in this example is using an h2 database engine, and edw is the name of a declared database (above).
schema.ddl

CREATE TABLE TEST_TABLE_VII

(

  ID INT NOT NULL,

  COL1 VARCHAR(256) NOT NULL,

  COL2 INTEGER NOT NULL,

  COL3 DATETIME NOT NULL,

  COL4 VARCHAR(256) NOT NULL,

  COL5 VARCHAR(256) NOT NULL,

  COL6 VARCHAR(256) NOT NULL,

  COL7 VARCHAR(256) NOT NULL,

  PRIMARY KEY(ID)

);;

  • reference-file-type-catalog.json should be placed in src/main/reference/file/fml folder.
reference-file-type-catalog.json

{

    "reference-file-types":

    {

        "description": "Database types for the edw-pet data schema",

        "packages":

        {

            "edw.public":

            {

                "description": "Tables in the package",

                "types":

                {

                    "TEST_TABLE_VII": {}

                }

            }

        }

    }

}

  • test_table_vii.json in this case is placed in src/main/reference/file/fml/edw/public.  TODO: Fix this link: SEE FML File Placement Options for more info.
test_table_vii.json

{

    "description": "Desc",

    "default-version": "edw.1",

    "versions": ["edw.1"]

}

id_date_mdy_stg.fml is placed in the folder data/, a sibling of the ETLUnit test class.

id_date_mdy_stg.fml

{

    "flat-file" : {

        "format-type" : "delimited",

        "row-delimiter" : "\n",

        "column-delimiter" : "\t",

        "null-token" : null,

        "columns" : [ {

            "id" : "ID",

            "type" : "INTEGER",

            "length" : 10,

            "basic-type" : "integer"

        }, {

            "id" : "COL3",

            "type" : "TIMESTAMP",

            "length" : 8,

            "basic-type" : "string",

            "format": "MMddyyyy"

        } ],

        "orderBy" : [ "ID" ]

    }

}

Place file test_table_vii+edw.1.fml in a location like src/main/reference/file/fml/edw/public.  TODO:Fix this link:  Check out FML File Placement Options for more info.

test_table_vii+edw.1.fml

{

  "flat-file" : {

    "format-type" : "delimited",

    "row-delimiter" : "\n",

    "column-delimiter" : "\t",

    "null-token" : null,

    "columns" : [ {

      "id" : "ID",

      "type" : "INTEGER",

      "length" : 10,

      "basic-type" : "integer"

    }, {

      "id" : "COL1",

      "type" : "VARCHAR",

      "length" : 256,

      "basic-type" : "string",

      "default-value": {"expression": "ID - ${rowData['ID']}"}

    }, {

      "id" : "COL2",

      "type" : "INTEGER",

      "length" : 10,

      "basic-type" : "integer",

      "default-value": {"initial-value": 10000, "increment": -3}

    }, {

      "id" : "COL3",

      "type" : "TIMESTAMP",

      "length" : 23,

      "basic-type" : "string",

      "format": "MMddyyyy",

      "default-value": {"initial-value": 1012000}

    }, {

      "id" : "COL4",

      "type" : "VARCHAR",

      "length" : 256,

      "basic-type" : "string",

      "default-value": {"initial-value": 400}

    }, {

      "id" : "COL5",

      "type" : "VARCHAR",

      "length" : 256,

      "basic-type" : "string",

      "default-value": {"initial-value": 500}

    }, {

      "id" : "COL6",

      "type" : "VARCHAR",

      "length" : 256,

      "basic-type" : "string",

      "default-value": {"initial-value": 600}

    }, {

      "id" : "COL7",

      "type" : "VARCHAR",

      "length" : 256,

      "basic-type" : "string",

      "default-value": {"initial-value": 700}

    } ],

    "primaryKey" : [ "ID" ],

    "orderBy" : [ "ID" ]

  }

}

 

Data

 

ID_AND_DATE_X_MDY.delimited
/*-- ID    COL3  --*/
/*-- INTEGER    VARCHAR  --*/
1    01012014
2    02022014
3    03032014
4    04042014
5    05052014
6    06062014
7    07072014
8    08082014
9    09092014
10    10102014
11    11112014
12    12122014

 

 

Sample Test Code

 

assert_with_columns_ii.etlunit

@Description( description: 'Shows how to ignore columns on the assert')

@Database(id: 'edw')

class assert_with_columns_ii {

@Test

assert_many_against_few_with_implied_src_FML(){

stage(

source: 'ID_AND_DATE_X_MDY',

target-table: 'TEST_TABLE_VII',

reference-file-type: {

id: 'id_date_mdy_stg'

}

);

 

assert(

source-table: 'TEST_TABLE_VII',

target-reference-file-type: {

column-list-mode: 'include',

column-list: ['ID','COL3']

},

target: 'ID_AND_DATE_X_MDY'

);

}

}

 

 

Go to top