Quandl Scraper

Downloads Quandl Data (Bulks) without a premium account and import it to a PostgreSQL (+ 9.2) database.

Everything gets structured N I C E L Y

Dependencies

• PostgreSQL (9.2 or later) must be installed

Setup

• In class: QuandlScraper/TecEnergyQuandl/Utils/Constants.cs
  • Replace API_KEY with your Quandl api key
  • Replace USER with your Postgres user
  • Replace PASSWORD with your Postgres password

Thats it. The program will handle by itself the creation of database/schemas/tables etc..

Usage

1. Program will download all databases available and insert them into quandl.databases (in a database named quandl in the postgres instance) and then pause

• Here you have to select the databases in postgres by setting import to true. Example:

update quandl.databases set import = true where databasecode = 'WIKI';

2. Program will download all datasets available from WIKI and insert them into quandl.datasets and then pause

• Here you have to select the datasets in postgres by setting import to true. Example:

update quandl.datasets set import = true where databasecode = 'WIKI' and datasetcode = 'AAPL';
update quandl.datasets set import = true where databasecode = 'WIKI' and datasetcode = 'FB';

3. Program will download all the data available from WIKI/AAPL and WIKI/FB, dynamically creates table quandl.wiki and inserts all the data

(Extra) Usage for datatables

1. Program will insert all datatables available and insert them into quandl.datatables (in a database named quandl in the postgres instance) and then pause

• Here you have to select the datatables in postgres by setting import to true. Example:

update quandl.datatables set import = true where name = 'ZACKS/CP';

2. Program will download all the data available from ZACKS/CP, dynamically creates table ZACKS/CP and inserts all the data

Info

The program consists in 3 key tasks (4 if you count datatables fetch). Each one is independient from each other. So you can skip a task (commenting the code line) and the program will still run fine.

1. Fetch Databases available
2. Fetch Datasets available _{(from selected databases downloaded in step 1)}
3. Fetch Data _{(from selected databases downloaded in step 2)}
4. Fetch Datatables (Extra) _{(from selected datatables downloaded)}

If something unexpected happens a file log.txt (inside same directory where the excutable is running) will be created with the information, however, the program will do his best to handle the error and keep working.