importing

Importing data

Each "store" has a list of available "data" that has been imported or saved into it.

Data can be imported from different sources (files, databases, etc) into a common format (TDataItem class), and optionally persisted to local disk files in binary format for fast saving and loading.

Each data consists of the following files:

mydata.def

The *.def file is optional. It is only created and used when manually importing data using the Data Manager editor.

A *.def file is just an *.ini text file with the appropriate settings that define where is the source of the data to import and importing parameters, like a server connection name when importing a database.

These settings are manually edited using the Data Manager.

Programatically, *.def files can be created and modified using the TDataDefinition class

mydata.bi

A *.bi file is a binary file containing the information of the data structure. This is the number of "columns", type of each column and other parameters.

The TPersistence class (in BI.Persist unit) allows reading and writing *.bi files at low-level.

mydata.databi

A *.databi file is a binary file containing the "real" data arrays, one array for each column, and optional arrays for the column "missing" (null) values and the "map" (index) of the column array.

The TDataPersistence class (in BI.Persist unit) allows reading and writing *.databi files at low-level.

The "Data Manager" editor dialog, usable at design-time and run-time, allows adding/removing data items and configuring them to import and persist the data.

Importing data can also be done at run-time without the Data Manager editor, for example, importing an XML file:

uses BI.Data.XML;
var X : TBIXML;
X:=TBIXML.Create;

var Books : TDataItem;
Books:= X.Import('books.xml');

A simplified version of the above code automatically detects the file extension to avoid needing to create a TBIXML variable:

uses BI.DataSource, BI.Data.XML;
Books:= TBIFileSource.ImportFile('books.xml');

The same applies to files with *.json, *.csv, *.xls, *.cds etc extensions. Just using the appropriate BI.Data.xxx unit, enables the automatic extension discovery.

Other helper methods in TBIFileSource and importing classes:

TBIFileSource.ImportURL( 'http://acme.com/mydata.json' );

X.Import( Memo1.Lines );  // any TStrings containing xml data

TBIFileSource.Import('c:\myfolder', '*.*', '', True)  // recursive folder import

TBIDB.Import( FDConnection1 );   // database connections

TBIDatasetSource.Import( MyDataSet1 );  // any TDataset-derived

TDataItem.Create( TTypeProvider<TCustomer>.Create( MyCustomers ) ); // ORM mapping

The Data Manager dialog (and all other dialogs) are available for both VCL and Firemonkey frameworks:

Different sources can be configured in the Data Manager, like:

• Text files (CSV, JSON, XML) from a folder or an URL.

• Microsoft Excel spreadsheets

• Database connections (using FireDAC, SQLExpress or other libraries)

• Data that is in another BIWeb server

• Contents from compressed *.zip files

Notes about the different data formats and importing them:

The units and classes related to importing data (BI.Data.xxx) can be used also by code, with more methods and flexibility (like for example importing contents of a TMemo.Lines or TStrings, or importing a live TDataSet instance of any class).

XML, JSON and databases can be imported using a default "engine" or specifying a custom one.

The above example uses the "default" engine (just using the parameterless Create constructor).

For XML, the available engines are:

• XmlDom (the standard xml code included by default in Delphi)

• Omni http://www.kluug.net/omnixml.php

• OXml https://code.google.com/p/omnixml

To specify an "engine", the only change is the import class should be created like:

uses BI.Data.XML.Omni;
X:=TBIXML.CreateEngine(TOmniXML.Create);

Similar to xml, JSON import supports:

• System.JSON (or DBXJSON for older Delphi versions)

• SuperObject https://github.com/hgourvest/superobject

and the same code to use any of them, for example:

uses BI.Data.JSON.SuperObject;
X:=TBIJSON.CreateEngine(TSuperObject.Create);

Importing databases can also be done specifying a different engine.

The available options are:

• FireDAC

• SqlExpress / dbExpress

• SqlDB for Lazarus

• BDE (Borland Database Engine deprecated)

• UniDAC

uses BI.Data.DB.SqlExpr;
X:=TBIDB.CreateEngine(TDBSqlExprEngine.Create);  // <-- use SqlExpress instead of FireDAC

Other "engines" for any kind of data will be implemented in the near future on-demand.