Concept
Etlunit uses a concept of reference file types which are core to how it handles data in almost every context. Some examples of when reference file types are used:
...
When discussing ETL processes - especially testing - the data inputs for the test and the assertions performed on the result of the data transformation are the key features of a testing platform. In etlunit, we take that seriously and the discussion that follows should help to fully understand the process so that your tests are well understood and stable - keys to agile unit testing and continuous integration success.
Determining effective reference file type
Whenever a reference file type is required, etlunit uses the following algorithm to determine which one to use.
- If a reference-file-type is specified, that is used.
- If the operation is operating on a named thing, such as a database table or assertion file, and a reference-file-type exists that has a matching name, then that is used.
- If the thing being acted upon has a generic name, then a reference file type matching that name is used. E.G., a database table named Table, in the schema edw, and connection id db, will first match a reference file type named Table.fml, and secondly db-edw-Table.fml.
- If there are two data sets involved, and the other data set has been identified using this same process, the same file type will be used for this one.
- At this point no file type has been found and may cause an error.
Locating Reference File types
Reference file types are located in the src/main/reference/file/fml/name.fml folder in the project. Any named file type will be located in this folder first. The next step is to search the classpath for reference/file/fml/name.fml.