From bfdbdb576a118600b5d8670cda183747aed04bf2 Mon Sep 17 00:00:00 2001 From: Andreas Bader Date: Tue, 23 May 2017 17:10:29 +0200 Subject: [PATCH] Fixing & Updating Readme; Adding documentation (Merging Readme branch) --- README.md | 250 +++++++++++++++++++----------------- docs/dev/adding_database.md | 23 ++++ docs/ei/aws.md | 10 ++ docs/ei/digitalocean.md | 10 ++ docs/ei/openstack.md | 46 +++++++ docs/ei/virtualbox.md | 11 ++ docs/ei/vsphere.md | 47 +++++++ docs/tsdb/akumuli.md | 7 + docs/tsdb/axibase.md | 7 + docs/tsdb/blueflood.md | 33 +++++ docs/tsdb/dalmatinerdb.md | 31 +++++ docs/tsdb/databus.md | 7 + docs/tsdb/druid.md | 91 +++++++++++++ docs/tsdb/elasticsearch.md | 7 + docs/tsdb/gnocchi.md | 7 + docs/tsdb/graphite.md | 7 + docs/tsdb/h5serv.md | 73 +++++++++++ docs/tsdb/influxdb.md | 23 ++++ docs/tsdb/kairosdb.md | 35 +++++ docs/tsdb/kdbplus.md | 7 + docs/tsdb/monetdb.md | 7 + docs/tsdb/mysql.md | 13 ++ docs/tsdb/newts.md | 38 ++++++ docs/tsdb/opentsdb.md | 62 +++++++++ docs/tsdb/postgresql.md | 13 ++ docs/tsdb/prometheus.md | 37 ++++++ docs/tsdb/redis.md | 9 ++ docs/tsdb/rhombus.md | 22 ++++ docs/tsdb/riakts.md | 7 + docs/tsdb/scylladb.md | 14 ++ docs/tsdb/seriesly.md | 7 + 31 files changed, 844 insertions(+), 117 deletions(-) create mode 100644 docs/dev/adding_database.md create mode 100644 docs/ei/aws.md create mode 100644 docs/ei/digitalocean.md create mode 100644 docs/ei/openstack.md create mode 100644 docs/ei/virtualbox.md create mode 100644 docs/ei/vsphere.md create mode 100644 docs/tsdb/akumuli.md create mode 100644 docs/tsdb/axibase.md create mode 100644 docs/tsdb/blueflood.md create mode 100644 docs/tsdb/dalmatinerdb.md create mode 100644 docs/tsdb/databus.md create mode 100644 docs/tsdb/druid.md create mode 100644 docs/tsdb/elasticsearch.md create mode 100644 docs/tsdb/gnocchi.md create mode 100644 docs/tsdb/graphite.md create mode 100644 docs/tsdb/h5serv.md create mode 100644 docs/tsdb/influxdb.md create mode 100644 docs/tsdb/kairosdb.md create mode 100644 docs/tsdb/kdbplus.md create mode 100644 docs/tsdb/monetdb.md create mode 100644 docs/tsdb/mysql.md create mode 100644 docs/tsdb/newts.md create mode 100644 docs/tsdb/opentsdb.md create mode 100644 docs/tsdb/postgresql.md create mode 100644 docs/tsdb/prometheus.md create mode 100644 docs/tsdb/redis.md create mode 100644 docs/tsdb/rhombus.md create mode 100644 docs/tsdb/riakts.md create mode 100644 docs/tsdb/scylladb.md create mode 100644 docs/tsdb/seriesly.md diff --git a/README.md b/README.md index 17ebdb3..b3fc0b5 100644 --- a/README.md +++ b/README.md @@ -1,135 +1,151 @@ -# TSDBBench: Overlord -[![Build Status](https://api.travis-ci.org/TSDBBench/Overlord.svg?branch=master)](https://travis-ci.org/TSDBBench/Overlord) - -The Python and Vagrant part of TSDBBench - -The VMs are running on vSphere or Openstack, but you can choose to run TSDBBench.py on your local pc or on a Control VM. - -## Setup Elastic Infrastructure -For your elastic infrastructure you need image(s) as bases for vagrant to derive generator and tsdb vms from. You can choose wheter you want to use vSohere or Openstack, you don't need both. - -1. Create/Copy images: - - All images are available at http://nemarcontrolvm.iaas.uni-stuttgart.de/isos/, you can use them directly from there (no need to download to your local pc!). - - for vSphere: - - Copy http://nemarcontrolvm.iaas.uni-stuttgart.de/isos/debian-8.1.0-amd64-netinst-vsphere-autoinstall.iso to your vSphere datastore - - for Openstack: - - Copy http://nemarcontrolvm.iaas.uni-stuttgart.de/isos/debian-8.1.0-amd64-netinst-openstack-autoinstall.qcow2 to your images in Openstack - - If you still want to create own images (changed preseed file e.g.) (run on your local pc): - - for vSphere: - - cd /path/to/another/folder/ - wget http://cdimage.debian.org/debian-cd/8.2.0/amd64/iso-cd/debian-8.2.0-amd64-netinst.iso - cd /path/to/some/folder/ - git clone https://github.com/baderas/TSDBBench - cd TSDBBench - ./MakeDebianIso.py -t /path/to/tmpfolder -i /path/to/another/folder/debian-8.2.0-amd64-netinst.iso -f /path/to/outputfolder/ -p preseed-vsphere.cfg - - for Openstack - - sudo apt-get install qemu-kvm libvirt-bin - sudo gpasswd -a $USER kvm - sudo gpasswd -a $USER libvirt - sudo reboot (or logout and login) - cd /path/to/another/folder/ - wget http://cdimage.debian.org/debian-cd/8.2.0/amd64/iso-cd/debian-8.2.0-amd64-netinst.iso - cd /path/to/some/folder/ - git clone https://github.com/baderas/TSDBBench - cd TSDBBench - ./MakeDebianIso.py -t /path/to/tmpfolder -i /path/to/another/folder/debian-8.2.0-amd64-netinst.iso -f /path/to/outputfolder/ -p preseed-openstack.cfg - ./MakeDebianQcow2.py -i /path/to/outputfolder/debian-8.2.0-amd64-netinst-openstack-autoinstall.iso -f /path/to/outputfolder/ - upload the resulting file to your Openstack Storage -2. (Only for vSphere) Create templates (run on your local pc): - - cd /path/to/some/folder/ - git clone https://github.com/baderas/TSDBBench - cd TSDBBench - nano vagrant_files/vagrantconf.rb (add everything that says '') - nano vagrant_files/vagrantconf_db.rb (add everything that says '') - nano vagrant_files/vagrantconf_gen.rb (add everything that says '') - ./pySetupVsphere.py -f /path/to/some/folder/TSDBBench/vagrant_files/ - -## Setup -Everything was tested and used on Debian Jessie x64, but should work on Ubuntu. But Ubuntu has different package names for a lot of the packages, you need to find and change them! -### Choice 1: local pc with openstack -1. Install packages: - +# Overlord +[![Build Status](https://api.travis-ci.org/TSDBBench/TSDBBench.svg?branch=master)](https://travis-ci.org/TSDBBench/TSDBBench) + +The Python and Vagrant part of TSDBBench that automatically setups and benchmarks tims series databases (TSDBs). + +The benchmarking is done with VMs that are automatically setup by Vagrant and are running on one of the five supported elastic infrastructures (EIs). + +Different cluster sizes and configurations as well as different workloads can be used to benchmark one or more TSDBs. + +The benchmark is done with [YCSB-TS](https://github.com/TSDBBench/YCSB-TS). + +## Supported databases +* Timeseries databases with a Requirement on NoSQL DBMS: + * [Axibase](docs/tsdb/axibase.md) + * [Blueflood](docs/tsdb/blueflood.md) + * [Databus](docs/tsdb/databus.md) + * [KairosDB](docs/tsdb/kairosdb.md) + * [NewTS](docs/tsdb/newts.md) + * [OpenTSDB](docs/tsdb/opentsdb.md) + * [Rhombus](docs/tsdb/rhombus.md) + * [Graphite](docs/tsdb/graphite.md) +* Timeseries databases with no Requirement on any DBMS: + * [Akumuli](docs/tsdb/akumuli.md) + * [Druid](docs/tsdb/druid.md) + * [InfluxDB](docs/tsdb/influxdb.md) + * [Gnocchi](docs/tsdb/gnocchi.md) + * [Seriesly](docs/tsdb/seriesly.md) + * [Prometheus](docs/tsdb/prometheus.md) +* Column-oriented DBMS + * [MonetDB](docs/tsdb/monetdb.md) + * [Kdb+](docs/tsdb/kdpplus.md) +* Relational databases: + * [MySQL](docs/tsdb/mysql.md) + * [PostgreSQL](docs/tsdb/postgresql.md) +* Other: + * [Elasticsearch](docs/tsdb/elasticsearch.md) + * [h5serv](docs/tsdb/h5serv.md) + +## Unsupported databases (WIP) +* [DalmatinerDB](docs/tsdb/dalmatinerdb.md) +* [ScyllaDB](docs/tsdb/scylladb.md) +* [RiakTS](docs/tsdb/riakts.md) +* [Redis](docs/tsdb/redis.md) + +## Supported Elastic Infrastrctures +* [VMware vSphere](docs/ei/vsphere.md) +* [OpenStack](docs/ei/openstack.md) +* [VirtualBox](docs/ei/virtualbox.md) +* [DigitalOcean](docs/ei/digitalocean.md) +* [Amazon Web Services](docs/ei/aws.md) + +## Initial Setup of the Elastic Infrastrcture +* This must done only once and only for one elastic infrastructure +* All images are available at http://tsdbbench.allweathercomputing.com/bin/, can use them directly from there. +* See the articles for the five supported elastic infrastructures for specific instructions + * [VMware vSphere](docs/ei/vsphere.md) + * [OpenStack](docs/ei/openstack.md) + * [VirtualBox](docs/ei/virtualbox.md) + * [DigitalOcean](docs/ei/digitalocean.md) + * [Amazon Web Services](docs/ei/aws.md) + +## Initial Setup of local pc or Control-VM +* To control TSDBBench, a local pc or a Control-VM (a VM with everything preinstalled) can be used (only vSphere and OpenStack) +1. Local PC: + 1. Install packages: + ```bash sudo apt-get install python-dateutil python-jinja2 python-numpy python-pandas python-flask python-redis python-requests python-six python-tornado python-werkzeug python-markupsafe python-greenlet python-zmq python-yaml python-pip wkhtmltopdf python-magic fabric vagrant zlib1g-dev zlib1g libxml2 libxml2-dev libxslt1.1 libxslt1-dev python-webcolors python-pyvmomi -2. Install pip packages: - + ``` + 2. Install pip packages: + ```bash sudo pip install bokeh python-vagrant pdfkit -3. Install vagrant plugins: - - vagrant plugin install vagrant-vsphere + ``` + 3. Install vagrant plugins: + ```bashvagrant plugin install vagrant-vsphere vagrant plugin install vagrant-openstack-provider -4. Reconfigure locales and make sure that en_US.UTF-8 is generated - + ``` + 4. Reconfigure locales and make sure that en_US.UTF-8 is generated + ```bash sudo dpkg-reconfigure locales -5. Checking out & Prepairing Git Repo - + ``` + 5. Checking out & Prepairing Git Repo + ```bash cd /path/to/some/folder/ git clone https://github.com/baderas/TSDBBench cd TSDBBench - vagrant box add TSDBBench/tsdbbench_dummy.box --provider openstack + vagrant box add --name dummy dummy.box copy hooks/pre-commit .git/hooks/ cd .. -6. Edit Config (change everything that says '') - - cd /path/to/some/folder/TSDBBench + ``` + 6. Edit config for the chosen elastic infrastructure (change everything that says '' for your chosen elastic infrastructure) + ```bash + cd TSDBBench nano vagrant_files/vagrantconf.rb nano vagrant_files/vagrantconf_db.rb nano vagrant_files/vagrantconf_gen.rb - -### Choice 2: Control VM - - Coming soon... + ``` +2. Control-VM + 1. Create Control-VM according to [VMware vSphere](docs/ei/vsphere.md) or [OpenStack](docs/ei/openstack.md) + 2. Login to your Control-VM + 3. Edit config for the chosen elastic infrastructure (change everything that says '' for your chosen elastic infrastructure) + ```bash + cd TSDBBench + nano vagrant_files/vagrantconf.rb + nano vagrant_files/vagrantconf_db.rb + nano vagrant_files/vagrantconf_gen.rb + ``` ## Running a testworkload - - without creating html file: - - cd /path/to/some/folder/TSDBBench - ./TSDBBench.py -t /path/to/tmpfolder -f /path/to/some/folder/TSDBBench/vagrant_files -d mysql1 -l --provider 'vsphere' -w "testworkloada" - - with creating html file: - - cd /path/to/some/folder/TSDBBench - ./TSDBBench.py -t /path/to/tmpfolder -f /path/to/some/folder/TSDBBench/vagrant_files -d mysql1 -l --provider 'vsphere' -w "testworkloada" -m - + - without creation of html file: + ```bash + cd TSDBBench + ./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l``` + - with creation of html file: + ```bash + cd TSDBBench + ./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l -m``` + - with creation of html files and multiple databases: + ```bash + cd TSDBBench + ./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 postgresql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l -m --provider "vsphere"``` + ## Creating html files (when not using -m) - Creating a html file from a ycsb_*.log file: - - cd /path/to/some/folder/TSDBBench - ./ProcessYcsbLog.py -f some_ycsb_logfile.log - + ```bash + cd TSDBBench + ./ProcessYcsbLog.py -f some_ycsb_logfile.log + ``` - Creating a html file from a ycsb_*.ydc file: - - cd /path/to/some/folder/TSDBBench - ./ProcessYcsbLog.py -f some_ycsb_logfile.ydc - + ```bash + cd TSDBBench + ./ProcessYcsbLog.py -f some_ycsb_logfile.ydc + ``` - Creating a combined html file a set of from a ycsb_*.ydc/.log files: - - cd /path/to/some/folder/TSDBBench - ./ProcessYcsbLog.py -f some_ycsb_logfile1.ydc ome_ycsb_logfile2.log ome_ycsb_logfile3.ydc ... - -## Steps to add a new TSDB: -for this example consider your new tsdb would be opentsdb: - 1. Add Vagrantfiles: - 1. Create at least one folder in /path/to/some/folder/TSDBBench/vagrant_files - - e.g. opentsdb1 - 2. Create at least one Vagrantfile in this new folder - - named same as the folder but with _ + number + .vagrant as suffix - - e.g. /path/to/some/folder/TSDBBench/vagrant_files/opentsdb1/opentsdb1_0 - 3. In this Vagrantfile put deployment tasks like installing and configuring - - but nothing where you need to know IPs from other nodes of the cluster (that comes later) - 4. Use basic scripts as much as possible - - see /path/to/some/folder/TSDBBench/vagrant_files/basic - 2. Add the python part - 1. Add a python file to /path/to/some/folder/TSDBBench/databases - - named like the folder + .py (.e.g. opentsdb1.py) - 2. In this file you add: - 1. Deployment tasks that requires IPs from all nodes of the cluster - 2. Checks that the database is running - 3. Some basic db configs (everything that starts with db_) - 3. Look at other database python files, there are comments that explain every field in every file - 3. Testing - - You need to copy the database files to /path/to/some/folder/TSDBBench/vagrant_files/generator/files/databases/ - - You can run either run "hooks/pre-commit" or "git commit -a" (runs the hook as well if setup correctly). - -## Related Projects - * [VPS Benchmarks](http://www.vpsbenchmarks.com/) - measurement of VPS performance + ```bash + cd TSDBBench + ./ProcessYcsbLog.py -f some_ycsb_logfile1.ydc ome_ycsb_logfile2.log ome_ycsb_logfile3.ydc ... + ``` + +## Additional Information +* Everything was tested and used on Debian Jessie x64, but should work on Ubuntu. + * Ubuntu has different package names for a lot of the packages, you need to find and change them +* Logfiles/Benchmark Results are stored compressed as .ydc Files + +## Development Information +* Development specific details on databases and elastic infrastructures can be found on their specific files (see links at the beginning) +* [Adding a New Database](dev/adding_database.md) + +## Related Work +* [Comparison of TSDBs (Andreas Bader)](http://www2.informatik.uni-stuttgart.de/cgi-bin/NCSTRL/NCSTRL_view.pl?id=DIP-3729&mod=0&engl=0&inst=FAK) +* [Survey and Comparison of Open Source TSDBs (Andreas Bader, Oliver Kopp, Michael Falkenthal)](http://www2.informatik.uni-stuttgart.de/cgi-bin/NCSTRL/NCSTRL_view.pl?id=INPROC-2017-06&mod=0&engl=0&inst=IPVS) +* [Ultimate Comparison of TSDBs](https://tsdbbench.github.io/Ultimate-TSDB-Comparison/) diff --git a/docs/dev/adding_database.md b/docs/dev/adding_database.md new file mode 100644 index 0000000..7d8da12 --- /dev/null +++ b/docs/dev/adding_database.md @@ -0,0 +1,23 @@ +# Adding a New Database +* for this example consider your new tsdb would be opentsdb: + 1. Add Vagrantfiles: + 1. Create at least one folder in TSDBBench/vagrant_files + * E.g. opentsdb_cl1_rf1 (cl = cluster size, rf = replication factor) + 2. Create at least one Vagrantfile in this new folder + * Named same as the folder but with _ + number + .vagrant as suffix + * Each file is one VM that is created and provisioned + * E.g. TSDBBench/vagrant_files/opentsdb_cl1_rf1/opentsdb_cl1_rf1_0.vagrant + 3. In this Vagrantfile put deployment tasks like installing and configuring + * But nothing where you need to know IPs from other nodes of the cluster (that comes in a later step) + * Can't be done with Vagrant + * Consider files in basic folder that can be reused + * Eee TSDBBench/vagrant_files/basic/opentsdb.rb for example + 2. Add the python part + 1. Add a python file to TSDBBench/vagrant_files/generator/files/databases/ + * Named like the folder + .py (.e.g. opentsdb_cl1_rf1.py) + 2. In this file add: + 1. Deployment tasks that requires IPs from all nodes of the cluster + 2. Checks that the database is running + 3. Some basic db configs (everything that starts with db_) + 3. Look at other database python files, there are comments that explain every field in every file + * E.g., TSDBBench/vagrant_files/generator/files/databases/opentsdb_cl1_rf1.py diff --git a/docs/ei/aws.md b/docs/ei/aws.md new file mode 100644 index 0000000..20aca95 --- /dev/null +++ b/docs/ei/aws.md @@ -0,0 +1,10 @@ +# Amazon Web Services (AWS) + +## Links +* https://aws.amazon.com + +## Information +* For AWS a existing default debian image is used, so no specific image creation/preparation is required + * As a drawback a lot of things must be installed on each run, which takes more time than using an preinstalled image + +[Back to README.md](../../README.md) diff --git a/docs/ei/digitalocean.md b/docs/ei/digitalocean.md new file mode 100644 index 0000000..b313629 --- /dev/null +++ b/docs/ei/digitalocean.md @@ -0,0 +1,10 @@ +# DigitalOcean + +## Links +* https://www.digitalocean.com/ + +## Information +* For DigitalOcean a existing default debian image is used, so no specific image creation/preparation is required + * As a drawback a lot of things must be installed on each run, which takes more time than using an preinstalled image + +[Back to README.md](../../README.md) diff --git a/docs/ei/openstack.md b/docs/ei/openstack.md new file mode 100644 index 0000000..7a56da8 --- /dev/null +++ b/docs/ei/openstack.md @@ -0,0 +1,46 @@ +# OpenStack + +## Links +* https://www.openstack.org/ + +## Information +* for OpenStack qcow2 images are sued + * the images are created from the autoninstalling iso files + * the iso files for OpenStack have only minor differences to those of vSphere, see the corresponding preseed files. + +## Initial Elastic Infrastructure Setup +1. Copy http://tsdbbench.allweathercomputing.com/bin/debian-8.8.0-amd64-netinst-vsphere-autoinstall.qcow2 to your images in Openstack +2. (optional) If you want a Control-VM, do the following: + 3. Copy http://tsdbbench.allweathercomputing.com/bin/debian-8.8.0-amd64-netinst-vsphere-controlvm-autoinstall.qcow2 to your images in OpenStack + 4. Create a new VM using the Control-VM image + +## Creating Your Own Images (Generator/Database) +* If you want to create your own images, do the following (example on Debian Jessie): + 1. `sudo apt-get install qemu-kvm libvirt-bin` + 2. `sudo gpasswd -a $USER kvm` + 3. `sudo gpasswd -a $USER libvirt` + 4. `sudo reboot` (or logout and login) + 5. `cd /path/to/another/folder/` + 6. `wget http://cdimage.debian.org/debian-cd/8.8.0/amd64/iso-cd/debian-8.8.0-amd64-netinst.iso` + 7. `cd /path/to/some/folder/` + 8. `git clone https://github.com/baderas/TSDBBench` + 9. `cd TSDBBench` + 10. `./MakeDebianIso.py -t /path/to/tmpfolder -i /path/to/another/folder/debian-8.8.0-amd64-netinst.iso -f /path/to/outputfolder/ -p preseed-openstack.cfg` + 11. `./MakeDebianQcow2.py -i /path/to/outputfolder/debian-8.8.0-amd64-netinst-openstack-autoinstall.iso -f /path/to/outputfolder/` + 12. Upload the resulting file to your Openstack Storage + +## Creating your own Control-VM Image +1. `sudo apt-get install qemu-kvm libvirt-bin` +2. `sudo gpasswd -a $USER kvm` +3. `sudo gpasswd -a $USER libvirt` +4. `sudo reboot` (or logout and login) +5. `cd /path/to/another/folder/` +6. `wget http://cdimage.debian.org/debian-cd/8.8.0/amd64/iso-cd/debian-8.8.0-amd64-netinst.iso` +7. `cd /path/to/some/folder/` +8. `git clone https://github.com/baderas/TSDBBench` +9. `cd TSDBBench` +10. `./MakeDebianIso.py -t /path/to/tmpfolder -i /path/to/another/folder/debian-8.8.0-amd64-netinst.iso -f /path/to/outputfolder/ -p preseed-openstack-controlvm.cfg` +11. `./MakeDebianQcow2.py -i /path/to/outputfolder/debian-8.8.0-amd64-netinst-openstack-controlvm-autoinstall.iso -f /path/to/outputfolder/` +12. Upload the resulting file to your Openstack Storage + +[Back to README.md](../../README.md) diff --git a/docs/ei/virtualbox.md b/docs/ei/virtualbox.md new file mode 100644 index 0000000..c5940a7 --- /dev/null +++ b/docs/ei/virtualbox.md @@ -0,0 +1,11 @@ +# Oracle VirtualBox + +## Links +* https://www.virtualbox.org/ + +## Information +* Virtualbox was used as first locally running development plattform +* A specific BaseBox must exist in Vagrant (comparable to templates in vSphere or qcow2 images in OpenStack) + * Steps how to create this BaseBox are missing + +[Back to README.md](../../README.md) diff --git a/docs/ei/vsphere.md b/docs/ei/vsphere.md new file mode 100644 index 0000000..0e95f10 --- /dev/null +++ b/docs/ei/vsphere.md @@ -0,0 +1,47 @@ +# VMWare vSphere + +## Links +* http://www.vmware.com/products/vsphere.html + +## Information +* for vSphere autoinstalling iso files are used to create templates that can be reused + * the whole process is scripted + +## Initial Elastic Infrastructure Setup +1. Copy http://tsdbbench.allweathercomputing.com/bin/debian-8.8.0-amd64-netinst-vsphere-autoinstall.iso to your vSphere datastore +2. `cd /path/to/some/folder/` +3. `git clone https://github.com/baderas/TSDBBench` +4. `cd TSDBBench` +5. `nano vagrant_files/vagrantconf.rb` (add everything that says '' and is vSphere related) +6. `nano vagrant_files/vagrantconf_db.rb` (add everything that says '' and is vSphere related) +7. `nano vagrant_files/vagrantconf_gen.rb` (add everything that says '' and is vSphere related) +8. `nano SetupVsphere.py` (change baseDbVmConfig and baseGenVmConfig) +9. `./SetupVsphere.py -f /path/to/some/folder/TSDBBench/vagrant_files/` +10. (optional) If you want a Control-VM, do the following: + 11. Copy http://tsdbbench.allweathercomputing.com/bin/debian-8.8.0-amd64-netinst-vsphere-controlvm-autoinstall.iso to your vSphere datastore + 12. Create a new VM and use the image to install the Control-VM on it (automated) + +## Creating Your Own Images (Generator/Database) +* If you want to create your own images, do the following: + 1. `cd /path/to/another/folder/` + 2. `wget http://cdimage.debian.org/debian-cd/8.8.0/amd64/iso-cd/debian-8.8.0-amd64-netinst.iso` + 3. `cd /path/to/some/folder/` + 4. `git clone https://github.com/baderas/TSDBBench` + 5. `cd TSDBBench` + 6. `./MakeDebianIso.py -t /path/to/tmpfolder -i /path/to/another/folder/debian-8.8.0-amd64-netinst.iso -f /path/to/outputfolder/ -p preseed-vsphere.cfg` + 7. `nano vagrant_files/vagrantconf.rb` (add everything that says '' and is vSphere related) + 8. `nano vagrant_files/vagrantconf_db.rb` (add everything that says '' and is vSphere related) + 9. `nano vagrant_files/vagrantconf_gen.rb` (add everything that says '' and is vSphere related) + 10. `nano SetupVsphere.py` (change baseDbVmConfig and baseGenVmConfig) + 11. `./SetupVsphere.py -f /path/to/some/folder/TSDBBench/vagrant_files/` + +## Creating your own Control-VM Image +1. `cd /path/to/another/folder/` +2. `wget http://cdimage.debian.org/debian-cd/8.8.0/amd64/iso-cd/debian-8.8.0-amd64-netinst.iso` +3. `cd /path/to/some/folder/` +4. `git clone https://github.com/baderas/TSDBBench` +5. `cd TSDBBench` +6. `./MakeDebianIso.py -t /path/to/tmpfolder -i /path/to/another/folder/debian-8.8.0-amd64-netinst.iso -f /path/to/outputfolder/ -p preseed-vsphere-controlvm.cfg` +7. Import the resulting iso file to your vSphere datastore, create a new VM and use the iso image to install the Control-VM on it + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/akumuli.md b/docs/tsdb/akumuli.md new file mode 100644 index 0000000..f600a49 --- /dev/null +++ b/docs/tsdb/akumuli.md @@ -0,0 +1,7 @@ +# Akumuli + +## Links + +* https://github.com/akumuli/Akumuli + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/axibase.md b/docs/tsdb/axibase.md new file mode 100644 index 0000000..8cf174a --- /dev/null +++ b/docs/tsdb/axibase.md @@ -0,0 +1,7 @@ +# Axibase + +## Links + +* https://axibase.com/ + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/blueflood.md b/docs/tsdb/blueflood.md new file mode 100644 index 0000000..1e6b882 --- /dev/null +++ b/docs/tsdb/blueflood.md @@ -0,0 +1,33 @@ +# blueflood + +## Links + +* http://blueflood.io/ +* https://github.com/rackerlabs/blueflood + +## Information + +* Uses Cassandra +* Does not support TAGs ([source](https://github.com/rackerlabs/blueflood/wiki/FAQ)) +* Does not support SUM ([source](https://github.com/rackerlabs/blueflood/wiki/10minuteguide#send-numeric-metrics)) + * Using MIN instead of SUM in YCSB-TS +* Elastic Search (ES) is optional for searching metrics (not needed for YCSB-TS, only using one known metric) +* Unclear how to combine multiple instances of blueflood + * Zookeeper can be used to coordinate Rollups and shard locking ([Source1](https://github.com/rackerlabs/blueflood/wiki/Deployment-Dependencies), [Source2](https://github.com/rackerlabs/blueflood/blob/master/blueflood-core/src/main/java/com/rackspacecloud/blueflood/service/CoreConfig.java)) + * Unclear if and how load can be distributed + * Multiple cassandra nodes can be used for one shard. If one node fails, a non-failing one takes over. +* Only support for the following granularities: 5, 60, 240, 1440 +* Due to the size of the .jar file (72 MB) and a long buildprocess, the jar file is downloaded in TSDBBench instead of built everytime + +## How to install/build: + +* sudo apt-get install maven openjdk-7-jdk +* git clone https://github.com/rackerlabs/blueflood.git +* cd blueflood +* mvn test integration-test +* mvn package -P all-modules +* The result (if the build process was successful) is in blueflood-all/target/blueflood-all-2.0.0-SNAPSHOT-jar-with-dependencies.jar +* Go on with the [10 Minuten Guide](https://github.com/rackerlabs/blueflood/wiki/10minuteguide) + + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/dalmatinerdb.md b/docs/tsdb/dalmatinerdb.md new file mode 100644 index 0000000..94643bd --- /dev/null +++ b/docs/tsdb/dalmatinerdb.md @@ -0,0 +1,31 @@ +# DalmatinerDB + +## Links + +* https://github.com/dalmatinerdb/dalmatinerdb +* https://docs.dalmatiner.io/ + +## Information + +* Has no requirement on any DBMS +* Uses ZFS as filesystem + * Can use other filesystems, but may be slower then +* DEB/RPM packages are not available, but can be build +* Not (yet) integrated into TSDBBench/YCSB-TS + * Since the bug seems to be fixed ([issue](https://github.com/dalmatinerdb/dalmatinerdb/issues/4)), it may work know + +## How to install/build: + +* https://dalmatiner.io/docs/installation.html#from-sorce +* Datastore: + * sudo apt-get install git rebar erlang + * git clone https://github.com/dalmatinerdb/dalmatinerdb.git + * cd dalmatinerdb + * make deps all rel + * cp -r rel/dalmatinerdb $TARGET_DIRECTORY + * cd $TARGET_DIRECTORY + * cp etc/dalmatinerdb.conf.example etc/dalmatinerdb.conf + * vi etc/dalmatinerdb.conf # check the settings and adjust if needed + * ./bin/ddb start + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/databus.md b/docs/tsdb/databus.md new file mode 100644 index 0000000..0b97480 --- /dev/null +++ b/docs/tsdb/databus.md @@ -0,0 +1,7 @@ +# Databus + +## Links + +* https://github.com/linkedin/databus + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/druid.md b/docs/tsdb/druid.md new file mode 100644 index 0000000..0561b9f --- /dev/null +++ b/docs/tsdb/druid.md @@ -0,0 +1,91 @@ +# Druid + +## Links + +* http://druid.io/docs/latest/ +* http://druid.io/docs/0.8.0 + +## Information + +* "high-performance, column-oriented, distributed data store" +* Real-time Analysis +* Has five node types: Historical, Coordinator, Broker, Realtime, Indexing Service +* Supports multiple Ingestion methods + * Two of them would be suitable for YCSB_TS: + * [Realtime Data Ingestion](http://druid.io/docs/latest/ingestion/realtime-ingestion.html) via REST HTTP + * Requires Realtime Node + * Uses EventReceiverFirehose + * Not very well documented + * Should not be used ([source](https://groups.google.com/forum/#!topic/druid-development/DR89YlMzyKU)) + * Tranquility (Finagle Service) + * Requires Coordinator, Historical, Overlord, Broker, and Middlemanager Node or Indexing Service + * See "Batch Ingestionn Using the Indexing Service" [here](http://druid.io/docs/latest/ingestion/batch-ingestion.html) + * All can be installed on the same node + * [Tranquility](https://github.com/druid-io/tranquility) is the Java library for the client +* Multiple Deep Storages supported: local FS, HDFS, and S3 +* Kafka is not recommended: + * Does not scale very well compared to Tranquility/Finagle ([souce](https://groups.google.com/forum/#!searchin/druid-development/fangjin$20yang$20%22thoughts%22/druid-development/aRMmNHQGdhI/muBGl0Xi_wgJ)) +* Supported query granularities: none (ms when ms was ingested), minute, fifteen_minute, thirty_minute, hour, day, or all (one bucket) + +## Implementation Notes + +* Using tranquility 0.4.2 + * 0.5.0 does not work from Maven +* No inserts into past possible, even with 1 year windowperiod and a fitting SegmentGranularity +* Tagnames have to be deinfed as dimensions before using +* SUM as aggregating function is available in two flavours: LongSum (for 64bit integer) and DoubleSum (for 64bit floating point) + * Using DoubleSum in TSDBBench, changable via flag +* AVG is only available as post-aggregation function + * Uses SUM + Count as preceding functions +* Replication of five is not possible in a five not cluster setup, since 5 historical nodes would be required. + * The idea of replication if to achieve a higher availabily (see https://groups.google.com/forum/#!topic/druid-development/0TBL5-3Z2PI) + +## Steps to setup Druids Cluster Wikipedia Example with Overload (for Tranquility) +* Old: + * `wget http://static.druid.io/artifacts/releases/druid-0.8.0-bin.tar.gz` + * `tar -xvzf druid-0.8.0-bin.tar.gz` + * `sudo apt-get install zookeeperd mysql-server` + * `sudo systemctl start zookeeper.service` + * `cd druid-0.8.0` + * `mysql -uroot -e 'CREATE DATABASE druid DEFAULT CHARACTER SET utf8; GRANT ALL PRIVILEGES ON druid.* TO druid IDENTIFIED BY "diurd";'` + * `nano examples/wikipedia/wikipedia_realtime.spec`: + * "segmentGranularity": "HOUR", -> "segmentGranularity": "FIVE_MINUTE", + * "intermediatePersistPeriod": "PT10m", -> "intermediatePersistPeriod": "PT3m", + * "windowPeriod": "PT10m", -> "windowPeriod": "PT1m", +* execute in parallel (can take some time when running the first time due to extension loading): + * `java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common:config/coordinator:lib/* io.druid.cli.Main server coordinator` + * `java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common:config/historical:lib/* io.druid.cli.Main server historical` + * `java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common:config/broker:lib/* io.druid.cli.Main server broker` + * `java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common:config/overlord:lib/* io.druid.cli.Main server overlord` +* If a "real" Index Service is required: + * `java -Xms64m -Xmx64m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common:config/middlemanager:lib/* io.druid.cli.Main server middleManager` +* New with script (2015-10-13): + * `sudo systemctl start zookeeper.service` (if not running) + * `./run.bash` + +## Other Related Links + +* https://github.com/gianm/druid-monitorama-2015 +* https://github.com/jwang93/test-druid/wiki/MySQL-Setup +* http://curator.apache.org/index.html +* https://github.com/druid-io/tranquility +* https://github.com/mark1900/druid-sandbox/tree/master/kafka-storm-tranquility-druid-topology-test +* https://groups.google.com/forum/#!forum/druid-development +* https://groups.google.com/forum/#!forum/druid-user +* Important on Google Groups: + * https://groups.google.com/forum/#!topic/druid-user/qgfKipXPzeE + * https://groups.google.com/forum/#!topic/druid-development/eIiuSS-fM8I + * https://groups.google.com/forum/#!topic/druid-development/Siwd4gA7Yjg +* Important on Google Groups on Problems with Tranquility: + * https://groups.google.com/forum/#!topic/druid-development/PU6njY0gE5U + * https://groups.google.com/forum/#!topic/druid-user/UT5JNSZqAuk + * https://github.com/druid-io/druid/issues/1448 + * https://groups.google.com/forum/#!topic/druid-user/LKqvur7wWmo + * https://groups.google.com/forum/#!topic/druid-user/1YsRnLPMkhw + * https://groups.google.com/forum/#!topic/druid-user/IKJXd1d51TM +* http://brianoneill.blogspot.de/2015/09/druid-vagrant-up-and-tranquility.html +* https://github.com/Quantiply/druid-vagrant +* https://github.com/boneill42/druid-vagrant +* http://www.datascienceassn.org/sites/default/files/users/user1/DruidDataIngest_0.6.62.pdf + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/elasticsearch.md b/docs/tsdb/elasticsearch.md new file mode 100644 index 0000000..06e5241 --- /dev/null +++ b/docs/tsdb/elasticsearch.md @@ -0,0 +1,7 @@ +# Elasticsearch + +## Links + +* https://www.elastic.co/de/products/elasticsearch + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/gnocchi.md b/docs/tsdb/gnocchi.md new file mode 100644 index 0000000..b5fa9cc --- /dev/null +++ b/docs/tsdb/gnocchi.md @@ -0,0 +1,7 @@ +# Gnocchi + +## Links + +* http://gnocchi.xyz/ + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/graphite.md b/docs/tsdb/graphite.md new file mode 100644 index 0000000..23ce4fe --- /dev/null +++ b/docs/tsdb/graphite.md @@ -0,0 +1,7 @@ +# Graphite + +## Links + +* http://graphite.readthedocs.io/en/latest/ + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/h5serv.md b/docs/tsdb/h5serv.md new file mode 100644 index 0000000..d5fe6e9 --- /dev/null +++ b/docs/tsdb/h5serv.md @@ -0,0 +1,73 @@ +# h5serv (HDF5) + +## Links + +* https://www.hdfgroup.org/HDF5/ + +## Information + +* HDF5 is a file system + * no distribution + * strong consistency + * often used in research +* No TSDB, just a filesystem +* Disadvantages when using as TSDB: + * No append() function + * Only put() that eneds a specific index, requirement for a counter that counts along + * COUNT, AVG, SUM not available + * No indexing functions + * time series as compound data type possible + * h5table also possible for time series, but not yet available in h5serv + * basically the same as compound, but easier to use +* h5serv is a server with REST interface and hdf5 filesystem + * uses hdf5-json + +## Implementation Notes + +* Problems with string with variable length + * see https://github.com/HDFGroup/h5serv/issues/77 + * Fix: using constant length strings +* H5T_STD_U64BE does not work with queries + * see https://github.com/HDFGroup/h5serv/issues/80 + * *_I64BE variants work + * *_*_LE is faster than *_*_BE + + +## How to install/build + +* https://dalmatiner.io/docs/installation.html#from-sorce +* Datastore: + * sudo apt-get install git rebar erlang + * git clone https://github.com/dalmatinerdb/dalmatinerdb.git + * cd dalmatinerdb + * make deps all rel + * cp -r rel/dalmatinerdb $TARGET_DIRECTORY + * cd $TARGET_DIRECTORY + * cp etc/dalmatinerdb.conf.example etc/dalmatinerdb.conf + * vi etc/dalmatinerdb.conf # check the settings and adjust if needed + * ./bin/ddb start + +## Other related links + +* https://www.hdfgroup.org/HDF5/ +* https://www.hdfgroup.org/projects/hdfserver/ +* https://www.hdfgroup.org/HDF5/doc/ +* http://h5serv.readthedocs.org/en/latest/ +* http://h5serv.readthedocs.org/en/latest/Diagram.html +* https://github.com/HDFGroup/h5serv +* https://travis-ci.org/HDFGroup/h5serv/jobs/106399076 +* https://travis-ci.org/HDFGroup/hdf5-json/jobs/99532396 +* https://travis-ci.org/HDFGroup/h5serv/jobs/104515484 +* http://h5serv.readthedocs.org/en/latest/FAQ/index.html +* https://github.com/HDFGroup/hdf5-json +* http://permalink.gmane.org/gmane.comp.programming.hdf/5946 +* https://www.hdfgroup.org/pubs/papers/Big_HDF_FAQs.pdf +* http://stackoverflow.com/questions/15379399/writing-appending-arrays-of-float-to-the-only-dataset-in-hdf5-file-in-c +* http://www.pytables.org/index.html +* https://github.com/pydata/numexpr +* https://code.google.com/archive/p/numexpr/wikis/UsersGuide.wiki#Supported_operators +* http://www.pytables.org/usersguide/condition_syntax.html#condition-syntax +* http://www.pytables.org/usersguide/libref/structured_storage.html#tables.Table.where +* https://code.google.com/archive/p/numexpr/ + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/influxdb.md b/docs/tsdb/influxdb.md new file mode 100644 index 0000000..cec311a --- /dev/null +++ b/docs/tsdb/influxdb.md @@ -0,0 +1,23 @@ +# InfluxDB + +## Links + +* https://www.influxdata.com/products/open-source/ +* https://github.com/influxdb/influxdb-java + +## Information + +* The open source InfluxDB has limited cluster capabilities + +## Implementation Notes + +* A cluster always has 3 RAFT Server + * status can be seen with `/opt/influxdb/influx -execute "SHOW SERVERS"'` +* 0.9.4.2 has problems with MEAN() when running on clusters + * see https://github.com/influxdb/influxdb/issues/4170 + * newer versions don't have this problem +* GROUP BY with more than 100000 buckets can't be used + * see https://github.com/influxdb/influxdb/issues/2702 + * Fix: Using seconds instead of millisceonds, checking if the range does not exceed 100000 seconds + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/kairosdb.md b/docs/tsdb/kairosdb.md new file mode 100644 index 0000000..36ef92f --- /dev/null +++ b/docs/tsdb/kairosdb.md @@ -0,0 +1,35 @@ +# KairosDB + +## Links + +* https://kairosdb.github.io/ +* https://github.com/kairosdb/kairosdb + +## Information + +* 3 possible variants: + * KairosDB with H2 (slow, only for testing) + * KairosDB with HBase (deprecated, does not support seconds(?)) + * KairosDB with Cassandra +* When using READ() only a timerange (not a timestamp) is supported +* Status can be seen with `nodetool status` +* Performance notes: https://code.google.com/p/kairosdb/wiki/GettingStarted#Configuring_Cassandra +* Hardware notes: https://wiki.apache.org/cassandra/CassandraHardware +* Cassandra Cluster setup hints: + * http://docs.datastax.com/en/cassandra/1.2/cassandra/initialize/initializeSingleDS.html + * https://www.digitalocean.com/community/tutorials/how-to-configure-a-multi-node-cluster-with-cassandra-on-a-ubuntu-vps + +## Implementation Notes + +* Cassandra 22X did not work, error: "all host pools marked down. Retry burden pushed out to client" +* Cassandra 21X works with the same config +* When using e.g. SCAN(), a message is thrown: "******************* Type=number" + * See https://github.com/kairosdb/kairosdb-client/issues/43 + + +## Other Related Links + +* http://de.slideshare.net/codyaray/scalable-distributed-stats-infrastructure +* https://prezi.com/ajkjic0jdws3/kairosdb-cassandra-schema/ + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/kdbplus.md b/docs/tsdb/kdbplus.md new file mode 100644 index 0000000..1f9f892 --- /dev/null +++ b/docs/tsdb/kdbplus.md @@ -0,0 +1,7 @@ +# KDB+ + +## Links + +* https://kx.com/discover/ + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/monetdb.md b/docs/tsdb/monetdb.md new file mode 100644 index 0000000..cb2e767 --- /dev/null +++ b/docs/tsdb/monetdb.md @@ -0,0 +1,7 @@ +# MonetDB + +## Links + +* https://www.monetdb.org/Home + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/mysql.md b/docs/tsdb/mysql.md new file mode 100644 index 0000000..53faf4c --- /dev/null +++ b/docs/tsdb/mysql.md @@ -0,0 +1,13 @@ +# MySQL + +## Links + +* https://www.mysql.com + +## Implementation Information + +* Uses 3 tags by default (3 columns) +* Uses an auto-incrementing id primary key +* All time series of TSDBBench functions can be performed using SQL and are implemented + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/newts.md b/docs/tsdb/newts.md new file mode 100644 index 0000000..f1ac7ac --- /dev/null +++ b/docs/tsdb/newts.md @@ -0,0 +1,38 @@ +# NewTS + +## Links + +* http://opennms.github.io/newts/ +* https://github.com/OpenNMS/newts + +## Information + +* Filtering can only done by time or by TAGs when using the API +* When filtering for TAGs, you have the choice between x,y,z exist or one of x,y,z + * but you can't say: a=x and b=y and c=z + * see https://github.com/OpenNMS/newts/blob/master/api/src/main/java/org/opennms/newts/api/search/QueryBuilder.java + * see https://github.com/OpenNMS/newts/blob/master/cassandra/search/src/main/java/org/opennms/newts/cassandra/search/CassandraSearcher.java +* REST API does not support searching/filtering for time -> https://github.com/OpenNMS/newts/wiki/Search +* No COUNT(), SUM() -> https://github.com/OpenNMS/newts/wiki/ReportDefinitions +* Bucket size must be at least 2 ms when using AVG(), MAX(), MIN() +* Unlimited bucket size impossible + * If bucket is too big, "java.lang.IllegalArgumentException: resolution must be a multiple of interval" is thrown + +## Implementation Notes + +* Instead of COUNT() MIN() is used +* Instead of SUM() MAX() is used +* 1 year bucket size is used for "unlimited" +* NewTS daemonizes itself in a ugly way. Only way to control it headlessly is with a systemd service file. +* Changing the replication factor must be done by hand in Cassandra keypsace + * https://cassandra.apache.org/doc/cql3/CQL.html#createKeyspaceStmt + * http://www.opennms.org/wiki/Featurebranch/Newts#Updating_the_replication_factor +* Filtering for Tags does also not work with CassandraSearcher + * disabled + +## Other Related Links + +* https://github.com/OpenNMS/newts/wiki/JavaAPI +* https://github.com/OpenNMS/newts/wiki/UsingJava + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/opentsdb.md b/docs/tsdb/opentsdb.md new file mode 100644 index 0000000..30902e7 --- /dev/null +++ b/docs/tsdb/opentsdb.md @@ -0,0 +1,62 @@ +# OpenTSDB + +## Links + +* http://opentsdb.net + +## Information + +* Requires three components: + * Zookeeper + * HBase + * Java +* Gnuplot is required for graphs +* Metrics must be created explicitly +* Filtering for tags required version >= 2.2 +* Every query requires an aggregating function + * pure SCAN()/READ() is not possible +* COUNT() requires version >= 2.2 +* Milliseconds are accepted but not guaranteed + * see http://opentsdb.net/docs/build/html/user_guide/writing.html#timestamps +* Not sure how tu use more than one OpenTSDB node + * see http://wiki.cvrgrid.org/index.php/OpenTSDB_Cluster_Setup#Installing_OpenTSDB + * Vanish or DNS Round Robin would be a possibility but only for READ() + +## Implementation Notes + +* No direct java API, requires REST/JSON via HTTP +* Zookeeper is started by start-hbase.sh, do not start by manually +* JAVA_HOME must be set correctly +* Gnuplot is required for graphs +* MIN() is used for SCAN()/READ() +* HBase stores by deault in /tmp/hbase-root which is a tmpfs (faster than HDD/SSD) + * changed to use HDD +* /etc/opentsdb/opentsdb.conf says that tsd.storage.hbase.zk_quorum must be a space seperated list, but it must be a comma seperated list +* LZO should be used for better performance + * see http://opentsdb.net/setup-hbase.html + * installation: http://opentsdb.net/setup-hbase.html + * FIX1: https://github.com/twitter/hadoop-lzo/issues/35 + * FIX2: ignore CLASSPATH -> `JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 CFLAGS=-m64 CXXFLAGS=-m64 ant compile-native tar` +* OpenTSDB's webinterface is accessible on http://:4242/ +* Haddop is accessible on http://:50070 +* HBase ist accessible on http://:16010 (16020/16030 on regionservers!) +## Other Related Links + +* http://arcticfoxontheweb.blogspot.de/2015/03/querying-opentsdb-with-java-yes-it-can.html +* http://opentsdb.net/docs/build/html/api_http/query/index.html +* http://opentsdb.net/docs/build/html/api_http/put.html +* https://hbase.apache.org/book.html#quickstart +* http://www.dreamsyssoft.com/blog/blog.php?/archives/5-How-to-use-HBase-Hadoop-Clustered.html +* https://alijehangiri.wordpress.com/2013/05/17/cloud/ +* http://hbase.apache.org/book.html#standalone_dist +* http://hbase.apache.org/book.html#quickstart_fully_distributed +* http://chaalpritam.blogspot.de/2015/01/hadoop-260-multi-node-cluster-setup-on.html +* http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/ +* http://www.bigdata-insider.de/infrastruktur/articles/469385/ +* http://wiki.cvrgrid.org/index.php/OpenTSDB_Cluster_Setup#Installing_OpenTSDB +* Performance/Benchmarking: + * http://www.moredevs.com/opentsdb-and-hbase-rough-performance-test/ + * https://peritusconsulting.github.io/articles/2014-06-02-next-generation-monitoring-using-opentsdb.html + * https://www.mapr.com/blog/loading-time-series-database-100-million-points-second + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/postgresql.md b/docs/tsdb/postgresql.md new file mode 100644 index 0000000..5fe856b --- /dev/null +++ b/docs/tsdb/postgresql.md @@ -0,0 +1,13 @@ +# PostgreSQL + +## Links + +* https://www.postgresql.org/ + +## Implementation Information + +* Uses 3 tags by default (3 columns) +* Uses an auto-incrementing id primary key +* All time series of TSDBBench functions can be performed using SQL and are implemented + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/prometheus.md b/docs/tsdb/prometheus.md new file mode 100644 index 0000000..d6e1546 --- /dev/null +++ b/docs/tsdb/prometheus.md @@ -0,0 +1,37 @@ +# Prometheus + +## Links + +* https://github.com/prometheus/prometheus +* https://github.com/prometheus/pushgateway +* https://github.com/prometheus/client_java +* https://prometheus.io/ + +## Information + +* Uses LevelDB for indexes and uses a custom storage layer when persisiting data on the local filesystem +* "multi-dimensional data model" +* Features a flexible query language + * Recording rules to save results of frequently needed expressions in separate time series +* Indirect pushing of time series data via pushgateway + * Prometheus-Clients available to push data of the push metrics types counter, gauge, histogram or summary + * Supported exposition format is the Protocol buffer format + * No support of own timestamps, time of push is used instead + * HTTP API to push to pushgateway + * Supported exposition formats are plain text format and Protocol buffer format + * Own timestamps can be used when using plain text format +* Two different query formats of ranges + * Using offsets from the currect point of time and a granularity of s, m, h, d, w, y from there to include the desired time series data in a vector + * Can be used by buitlin functions to calculate values as sum, avg, min, max + * Using two timestamps rfc3339- or Unix-timestamps and a step size of ms, s, h, d, w, y to to include the desired time series in a matrix + * No further postprocessing possible with builtin functions +* Prebuilt releases for prometheus and pushgateway are available + +## Implementation Notes + +* As custom timestamps are used in insert steps, the plaintext format was used +* As postprocessing functions are used in the scan steps, the vector result format was used + * Timestamps were used to calculate offset and granularity to include the desired range of time series and overestimate the range as little as possible + + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/redis.md b/docs/tsdb/redis.md new file mode 100644 index 0000000..ecf6dfc --- /dev/null +++ b/docs/tsdb/redis.md @@ -0,0 +1,9 @@ +# Redis + +## Links + +* https://github.com/TSDBBench/Overlord/issues/1 +* https://www.infoq.com/articles/redis-time-series +* https://redis.io/ + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/rhombus.md b/docs/tsdb/rhombus.md new file mode 100644 index 0000000..e82601a --- /dev/null +++ b/docs/tsdb/rhombus.md @@ -0,0 +1,22 @@ +# Rhombus + +## Links + +* https://github.com/Pardot/Rhombus + +## Information + +* Rhombus is a library that uses Cassandra, no seperate running daemon +* Creates a Cassandra Keyspace at init() +* The cassandra layout must be given in a json file + * Amount of TAGs is not dynamic! +* No AVG() or SUM() and therefore no granularity choosable + +## Implementation Notes + +* 3 TAGs are supported by default (same as SQL) +* "ShardingStrategyNone" is used as strategy since it is not expected that tag wide rows overflow +* Nothing must be installed on the server besides Cassandra +* COUNT() instead of AVG()/SUM() + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/riakts.md b/docs/tsdb/riakts.md new file mode 100644 index 0000000..18c75bd --- /dev/null +++ b/docs/tsdb/riakts.md @@ -0,0 +1,7 @@ +# RiakTS + +## Links + +* http://docs.basho.com/riak/ts/1.5.2/ + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/scylladb.md b/docs/tsdb/scylladb.md new file mode 100644 index 0000000..3ad1d36 --- /dev/null +++ b/docs/tsdb/scylladb.md @@ -0,0 +1,14 @@ +# ScyllaDB + +## Links + +* http://www.scylladb.com/ +* https://github.com/scylladb/scylla + +## Information + +* ScyllaDB is a drop-in replacement for Cassandra +* Should be usable with all TSDBs that use Cassandra + * Cassandra-dependant stuff must be checked, e.g. Keyspace creations (Rhombus), setting of replication factors (NewTS), etc. + +[Back to README.md](../../README.md) diff --git a/docs/tsdb/seriesly.md b/docs/tsdb/seriesly.md new file mode 100644 index 0000000..9e33ab4 --- /dev/null +++ b/docs/tsdb/seriesly.md @@ -0,0 +1,7 @@ +# Seriesly + +## Links + +* https://github.com/dustin/seriesly + +[Back to README.md](../../README.md)