Starting a New ETLUnit Project

Owned by Michael Bruton (Deactivated)
Last updated: Jun 16, 2015
8 min read

ETLUnit 3.9.6

 

Draft in Progress

This document is a draft and is under development.

 

From Archetype

Prerequisites

Generate ETLUnit Project from Archetype

Types of Projects of Interest

  • Schema project
  • ETLUnit project

Archetype Generation Steps

In a command window, navigate to the folder in which the project should reside.

 

Navigate to new project location
userme@LO-PPBBKK55 /c
$ mkdir src/my_etlunit_proj
userme@LO-PPBBKK55 /c
$ cd src/my_etlunit_proj
userme@LO-PPBBKK55 /c/src/my_etlunit_proj

 

Obtain a list of ETLUnit-related available archetypes from Maven Central.

 

Example of filtered list of available archetypes
$ mvn archetype:generate | grep bradleysmithllc
768: remote -> org.bitbucket.bradleysmithllc.etlunit:etlunit-database-schema-archetype (-)
769: remote -> org.bitbucket.bradleysmithllc.etlunit:etlunit-feature-archetype (-)
770: remote -> org.bitbucket.bradleysmithllc.etlunit:etlunit-project-archetype (-)
771: remote -> org.bitbucket.bradleysmithllc.etlunit:feature-archetype (-)
772: remote -> org.bitbucket.bradleysmithllc.etlunit:project-archetype (-)

 

Press <Ctrl>-C to end the command.

Next, list ALL archetypes from which you can select the one you want.

 

mvn archetype:generate
...
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): 582:
...
111: 3.5.0
112: 3.5.1
113: 3.6.0
114: 3.6.1
115: 3.7.0
116: 3.8.0
117: 3.8.1
118: 3.9.0
119: 3.9.1
120: 3.9.2
121: 3.9.3
122: 3.9.4
123: 3.9.5
124: 3.9.6
Choose a number:

 

After the mvn archetype:generate command, at each of the two prompts above, type in the appropriate answer.  The actual numbers will change as Maven Central adds more archetypes.

 

Below are some sample responses to the prompts that follow, to help project generation.  The colons indicate prompts.

 

Sample answers to POM questions
Define value for property 'groupId': : org.userme.etlunit.samples
Define value for property 'artifactId': : isolate_fml_pkg
Define value for property 'version':  1.0-SNAPSHOT: : 1.0.0
Define value for property 'package':  org.userme.etlunit.samples: :
Confirm properties configuration:
groupId: org.userme.etlunit.samples
artifactId: isolate_fml_pkg
version: 1.0.0
package: org.userme.etlunit.samples
 Y: : Y <Enter>
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: etlunit-project-archetype:3.9.6
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: org.userme.etlunit.samples
[INFO] Parameter: artifactId, Value: isolate_fml_pkg
[INFO] Parameter: version, Value: 1.0.0
[INFO] Parameter: package, Value: org.userme.etlunit.samples
[INFO] Parameter: packageInPathFormat, Value: org/userme/etlunit/samples
[INFO] Parameter: package, Value: org.userme.etlunit.samples
[INFO] Parameter: version, Value: 1.0.0
[INFO] Parameter: groupId, Value: org.userme.etlunit.samples
[INFO] Parameter: artifactId, Value: isolate_fml_pkg
[INFO] project created from Archetype in dir: C:\src\my_etlunit_proj\isolate_fml_pkg
[INFO] ------------------------------------------------------------------------

 

Your project has been generated.  There isn't much there yet.

Go to top

Generated POM

Below is a sample of the Maven generated POM.

 

pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.userme.etlunit.samples</groupId>
    <artifactId>isolate_fml_pkg</artifactId>
    <version>1.0.0</version>
    <packaging>jar</packaging>

    <name>isolate_fml_pkg</name>
    <url>http://org.userme.etlunit.samples</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <etlunit.project.version>3.9.6</etlunit.project.version>
    </properties>

    <repositories>
        <repository>
            <id>distrib-snapshots</id>
            <name>Snapshots</name>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>org.bitbucket.bradleysmithllc.etlunit</groupId>
            <artifactId>etlunit-core</artifactId>
            <version>${etlunit.project.version}</version>
        </dependency>
        <dependency>
            <groupId>org.bitbucket.bradleysmithllc.etlunit</groupId>
            <artifactId>etlunit-cli</artifactId>
            <version>${etlunit.project.version}</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.bitbucket.bradleysmithllc.etlunit</groupId>
                <artifactId>etlunit-maven</artifactId>
                <version>${etlunit.project.version}</version>
                <executions>
                    <execution>
                        <phase>test</phase>
                        <goals>
                            <goal>test</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.4</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/etlunit-lib</outputDirectory>
                            <overWriteReleases>false</overWriteReleases>
                            <overWriteSnapshots>false</overWriteSnapshots>
                            <overWriteIfNewer>true</overWriteIfNewer>
                        </configuration>
                    </execution>
                    <execution>
                        <id>unpack-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>unpack</goal>
                        </goals>
                        <configuration>
                            <artifactItems>
                                <artifactItem>
                                    <groupId>org.bitbucket.bradleysmithllc.etlunit</groupId>
                                    <artifactId>etlunit-cli</artifactId>
                                    <type>tar.gz</type>
                                    <version>${etlunit.project.version}</version>
                                    <overWrite>false</overWrite>
                                    <classifier>bin</classifier>
                                    <outputDirectory>${basedir}</outputDirectory>
                                </artifactItem>
                            </artifactItems>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

 

 

Folder Structure

Following is an example of some basic the folder structures for your new project.

 

 

 

Folders generated by archetype:generate

 

Basic layout before addition of database

Basic layout after adding Informatica

EDW_EXTRACTS matches the name of an Informatica folder.

 

Go to top

Basic Configuration for Database

Example Using SQLServer Database

Create or Make Available the Database Schema

You can put the schema in your project or in another project and then make that project available by including its JAR file as a dependency in your project.

Example

Place a data definition file in a location named for the database.

 

Configure the database folder, the database name and the schema script in etlunit.json.

 

etlunit.json

{

"install-features": ["database", "sql-server-database", "log-assertion"],

"features":

{

"database":

{

"database-definitions":

{

"edw":

{

"implementation-id": "sql-server",

"user-name": "user",

"password": "pwd",

"schema-scripts":

[

"EDW.SQL"

]

}

}

}

}

}

 

  • "sql-server" implementation-id: The implementation-id corresponds to the folder in which the database folders are placed.
  • "edw" database-definition:  Corresponds to the folder bearing the database name.  This name will be used in the @Database annotations in ETLUnit tests.
  • "EDW.SQL" schema-script: Name of the file with the data definitions.

Push to Bitbucket

You can follow steps to setting up a Bitbucket repository for your project on the Bitbucket Create a Repository page.

You may also want to use a graphical user interface like SourceTree.

Go to top