Home

Welcome to the OrientDB-ETL module. By using this module you can easily move data from and to OrientDB by executing an ETL process. OrientDB ETL is based on the following principles:

• one configuration file in JSON format
• one Extractor is allowed to extract data from a source
• one Loader is allowed to load data to a destintion
• multiple Transformers that transform data in pipeline. They receive something in input, do something, return something as output that will be processed as input by the next component

How ETL works

EXTRACTOR => TRANSFORMERS[] => LOADER

Example of a process that extract from a CSV file, apply some change, lookup if the record has already been created and then store the record as document against OrientDB database:

+-----------+-----------------------+-----------+
| EXTRACTOR | TRANSFORMERS pipeline |  LOADER   |
+-----------+-----------------------+-----------+
|   FILE   ==>  CSV->FIELD->MERGE  ==> OrientDB |
+-----------+-----------------------+-----------+

## Installation Starting from OrientDB v2.0 the ETL module will be distributed in bundle with the official release. If you want to use it, then follow these steps:

• Clone the repository on your computer, by executing:
• git clone https://github.com/orientechnologies/orientdb-etl.git
• Compile the module, by executing:
• mvn clean install
• Copy script/oetl.sh (or .bat under Windows) to $ORIENTDB_HOME/bin
• Copy target/orientdb-etl-2.0-SNAPSHOT.jar to $ORIENTDB_HOME/lib

Usage

$ cd $ORIENTDB_HOME/bin
$ ./oetl.sh config-dbpedia.json

## Available Components

• Block
• Extractors
• Transformers
• Loaders

Examples:

• Import DBPedia

Configuration Syntax

{
  "config": {
    <name>: <value>
  },
  "begin": [
    { <block-name>: { <configuration> } }
  ],
  "extractor" : {
    { <extractor-name>: { <configuration> } }
  },
  "transformers" : [
    { <transformer-name>: { <configuration> } }
  ],
  "loader" : { <loader-name>: { <configuration> } },
  "end": [
   { <block-name>: { <configuration> } }
  ]
}