Description

Execute one or more processes.  Most likely, these will be the processes being tested.

Attributes

List of execute() Attributes

execute() for Database

Go to top

execute() for Informatica

Go to top

execute() for Java Executable

Go to top

execute() for Runtime Executable

Go to top

Individual execute() Attributes

execute() for Database Context

[Database Feature Module - Configuration properties not listed here.]

connection-id:

• Corresponds to id property in the @Database annotation.
• Any value other than one that corresponds to the id property in the @Database annotation is not allowed.

mode:

• Corresponds to one member in the list of modes, found in the @Database annotation.
• Any value other than one that corresponds to one of the modes listed in the @Database annotation is not allowed.

sql:

• Literal text of an SQL statement to execute.
• The database used by the SQL statement should be declared with the @Database annotation before the test method or test class.

sql-script:

• Name of an SQL script file to execute.
• The file is located in the sql/ folder, a sibling to the ETLUnit test.
• The SQL script file should have an ".sql" file extension.
• The value of sql-script should exclude the .sql file extension.

sql-procedure:

• Name of an SQL stored procedure to run.

parameters:

• A list of strings.
• These are parameters that will be passed to the stored procedure.
• Not valid without sql-procedure.

Go to top

execute() for Informatica Context

[All but the last property below originate in the ETLAgent project.]

domain-defaults:

client-type:

agent:

host-name:

host-port:

informatica-bin-directory:

security-domain:

working-root:

username:

password-encrypted:

• • A string

password:

client-version:

connectivity-host:

connectivity-port:

default-domain:

domains:

• JSON object, key-value pairs. Each includes these:

<domain-name>

[same available properties as domain-defaults above, plus:]

default-repository:

repositories:

• JSON object, key-value pairs.  Each includes these:

<repository-name>:

default-web-services-hub:

default-integration-service:

integration-services:

web-services-hubs:

• JSON object, key-value pairs. Each include these:

<web-service-hub-name>:

host-name:

host-port:

workflow:

folder:

task:

informatica-integration-service:

informatica-domain:

informatica-repository:

context-name:

• TODO: Fix link
• Compare to context-name in the stage operation.

log-name:

context:

source-files:

lookup-files;

run-instance-name:

parameters:

• Unique list of JSON objects, (key-value pairs). Each include these:


<parameter-label>:

scope:

parameter-name:

parameter-value:

informatica-integration-service:

informatica-domain:

informatica-repository:

error-id:

• If an error occurs during workflow execution, the error ID to return in the CLI.

Go to top

execute() for Java Executable Context

java-class:

• Required.
• A string.
• The qualified name of your Java class.
• In other words, the Java class name preceded by any package hierarchy.
• e.g.:
  • java-class: 'MyJavaClass'
  • java-class: 'com.sonicdrivein.etl.edw.ListParameters'

arguments:

• List of strings.
• Arguments to pass to the Java main method.
• e.g.:
  • arguments: ['arg1','arg2']

exit-code:

• An integer.
• The expected exit code from the Java class.
• If execution of the Java class errors out, you may not know about it unless you check the logs.
• But if you provide an exit-code of 0, and execution of the Java class errors out, the test will error out.
• An unmatched exit code returns error 'ERR_JAVA_EXECUTION_ERROR_BAD_RETURN_CODE.'

run-id:

• A string.
• You can manually find the run-id in the test's log.  

system-properties:

• An object (key-value pairs).
• These become system properties that become available to the Java class when it is executed.
• TODO: Make this code accessible? There is sample code in the java-samples module of the etlunit-specific-examples project on Bitbucket.

  Go to top 

Special Instructions

execute() for Java Executable

In order to be able to run a Java class in your ETLUnit project, here are some tips that may help.

• Include the etlunit-execute dependency in your POM.

• The java-executor feature must be configured for your project.
  
  • The default file for defining your project's configuration is etlunit.json, located in folder src/test/resources/config.
  • You may also place the configuration information in another file, (e.g., java.json) so that you can run your Java tests in another profile.  In this case the profile name would be "java."  TODO: Fix link. SEE the ETLUnit Test Profiles page for more information about profiles.
• At a minimum, your profile configuration file must declare java-executor as one of the installed features of your project.

• TODO: Make this code accessible? There is sample code in the java-samples module of the etlunit-specific-examples project on Bitbucket.
• Place your Java source code under folder src/main/java.  You may use packages if you wish.
• ETLUnit will execute a Java class only if it has a main method.
• When you run a Maven compile or package on your project, binaries for Java classes you wrote will be written to target/classes.  Package structure will be maintained.
• Out-of-the-box, ETLUnit will only be able to find your Java classes when the tests are run by Maven from the command line.  Running an ETLUnit test from Maven looks like the following:

mvn test -Detlunit-test=#my_java_test

• If you can recognize it, the test name is preceded by the pound sign ("#"), which is part of the command line syntax to run a test method (contrast this with the syntax to run a test package, a test class, or an operation).  Not all special symbols used in the CLI will work in the context of the Maven test command.  TODO: Fix link. SEE the ETLUnit CLI page for more information on running tests from the ETLUnit command shell.
• That said, you can run Java classes from the command shell if you manually reconfigure the batch file that starts the command line interface ("CLI").
• There are two batch files that are used to start the shell.  They both reside in folder bin/ under your project root. 
  
  1. Batch file "etlunit" will be invoked when your command shell is a Unix-like shell, like BASH. 
  2. Batch file "etlunit.bat" will be invoked when your command shell is the Windows command shell.

# Replace the following line in the script:
  java -classpath target\classes;target\test-classes -Djava.ext.dirs=lib org.bitbucket.bradleysmithllc.etlunit.cli.CliMain
# with this line:
  java -classpath target\classes;target\test-classes -Djava.ext.dirs=lib -Djava.class.path=/c/src/MY_PROJECT/target/classes org.bitbucket.bradleysmithllc.etlunit.cli.CliMain


# Be sure to make the java.class.path property's path reflect the location of your project.

rem Replace the following line in the script:
    java -classpath target\classes;target\test-classes -Djava.ext.dirs=lib org.bitbucket.bradleysmithllc.etlunit.cli.CliMain
rem with this line:
    java -classpath target\classes;target\test-classes -Djava.ext.dirs=lib -Djava.class.path=.;C:\src\MY_PROJECT\target\classes org.bitbucket.bradleysmithllc.etlunit.cli.CliMain

rem Be sure to make the java.class.path property's path reflect the location of your project.

• When your Java class has completed:
  • You can find standard out and error output in a location like this:
    • target/generated-sources/log/<testClass>/<testMethod>/0_execute/
  • You can find html reports in a location like this:
    • target/html-reports/<hash>
  • You can just run the report ("r" or "report" from the CLI) and see the contents of the log files from there.
• Once all the configuration pieces are in place, you will be able to run an ETLUnit test that uses the execute operation in the Java Executor context.
  • Your test will succeed when run using mvn test.  That means that continuous integration batch tests that make use of Maven to run your project's tests will not break.
  • Your test will succeed when run in ETLUnit's CLI.

Go to top

Examples

execute() for Database Example

The following example uses custom FML to do its job.  There are three FML files included.

1. The source data file has only one column.  By providing FML for the single column source, a column list does not have to be provided.
2. The test will fail if FML is not provided for the stage operation's target database file, ADDRESS_TYPE.
3. The target data file in the assert operation uses its own two-column FML description.

The assert operation's source-reference-file-type property narrows the count of columns that will be compared with the target to just two.

TODO: Make this project accessible? This example is from the sql-database-samples module of the etlunit-specific-examples project in Bitbucket.

 CREATE TABLE dbo.ADDRESS_TYPE
(
    ADDRESS_TYPE_ID SMALLINT  NOT NULL ,
    DESCRIPTION VARCHAR(256) NULL ,
    POPULATION_DATE DATETIME NOT NULL
)
;;
ALTER TABLE dbo.ADDRESS_TYPE
    ADD CONSTRAINT XPKADDRESS_TYPE PRIMARY KEY  CLUSTERED (ADDRESS_TYPE_ID ASC)
;;


CREATE TABLE dbo.ADDRESS_TYPE_IV
(
    ADDRESS_TYPE_ID SMALLINT  NOT NULL ,
    DESCRIPTION VARCHAR(256) NULL ,
    POPULATION_DATE DATETIME NOT NULL
)
;;
ALTER TABLE dbo.ADDRESS_TYPE_IV
    ADD CONSTRAINT XPKADDRESS_TYPE_IIV PRIMARY KEY  CLUSTERED (ADDRESS_TYPE_ID ASC)
;;


CREATE PROCEDURE CopyBillingAddressID_TypeToIV @ID smallint, @Type varchar(256) = NULL
  AS
  INSERT INTO address_type_iv
    SELECT * FROM address_type
    WHERE ADDRESS_TYPE_ID = @ID
    AND DESCRIPTION = ISNULL(@Type, DESCRIPTION)
;;

 {
    "flat-file" : {
        "format-type" : "delimited",
        "row-delimiter" : "\n",
        "column-delimiter" : "\t",
        "null-token" : null,
        "columns" : [ {
            "id" : "ADDRESS_TYPE_ID",
            "type" : "SMALLINT",
            "length" : 5,
            "basic-type" : "integer",
            "default-value" : {
                "initial-value" : 2,
                "increment" : 2,
                "modulus" : 32768
            }
        }, {
            "id" : "DESCRIPTION",
            "type" : "VARCHAR",
            "length" : 256,
            "basic-type" : "string",
            "default-value" : {
                "initial-value" : 3,
                "increment" : 3,
                "expression" : "$strings.constrain(\"DESCRIPTION[${defaultValue}]\", 256)"
            }
        }, {
            "id" : "POPULATION_DATE",
            "type" : "TIMESTAMP",
            "length" : 23,
            "basic-type" : "string",
            "default-value" : {
                "initial-value" : 4,
                "increment" : 4,
                "expression" : "$math.add(2000, $math.modulo($defaultValue, 100))$numbers.leftPad($math.add(1, $math.modulo($defaultValue, 12)), 2)$numbers.leftPad($math.add(1, $math.modulo($defaultValue, 28)), 2)"
            },
            "format": "yyyyMMdd"
        } ],
        "primaryKey" : [ "ADDRESS_TYPE_ID" ],
        "orderBy" : [ "ADDRESS_TYPE_ID", "DESCRIPTION", "POPULATION_DATE" ]
    }
}

{
    "flat-file" : {
        "format-type" : "delimited",
        "row-delimiter" : "\n",
        "column-delimiter" : "\t",
        "null-token" : null,
        "columns" : [ {
            "id" : "DESCRIPTION",
            "type" : "VARCHAR",
            "length" : 256,
            "basic-type" : "string",
            "default-value" : {
                "initial-value" : 3,
                "increment" : 3,
                "expression" : "$strings.constrain(\"DESCRIPTION[${defaultValue}]\", 256)"
            }
        } ]
    }
}

{
    "flat-file" : {
        "format-type" : "delimited",
        "row-delimiter" : "\n",
        "column-delimiter" : "\t",
        "null-token" : null,
        "columns" : [ {
            "id" : "ADDRESS_TYPE_ID",
            "type" : "SMALLINT",
            "length" : 5,
            "basic-type" : "integer",
            "default-value" : {
                "initial-value" : 2,
                "increment" : 2,
                "modulus" : 32768
            }
        }, {
            "id" : "DESCRIPTION",
            "type" : "VARCHAR",
            "length" : 256,
            "basic-type" : "string",
            "default-value" : {
                "initial-value" : 3,
                "increment" : 3,
                "expression" : "$strings.constrain(\"DESCRIPTION[${defaultValue}]\", 256)"
            }
        } ],
        "primaryKey" : [ "ADDRESS_TYPE_ID" ],
        "orderBy" : [ "ADDRESS_TYPE_ID" ]
    }
}

Go to top

execute() for Informatica Example

Go to top

execute() for Java Executable Example

TODO: Make this code accessible? SEE the java-samples module of the etlunit-specific-examples project in Bitbucket.

To get this to work using the CLI, follow the instructions above for ETLUnit CLI script modifications.

The following Java executable example has several pieces.  The log assertion annotation is included to give the exercise a mite more meaning.

• Project folder structure for the artifacts used by this example
• Source for Java class SystemInfo
  • Should be placed under src/main/java
• Log expression file fandango_surprise.expr
  • In this case, contains a fragment of the log contents representing an expected log excerpt string.
• Test class java_demo.etlunit

Go to top

execute() for Runtime Executable Example