Skip to content

Latest commit

 

History

History
190 lines (160 loc) · 6.81 KB

index.md

File metadata and controls

190 lines (160 loc) · 6.81 KB
Raw

Overlord

This is the Python and Vagrant part of TSDBBench

Supported databases

Timeseries databases with a requirement on NoSQL DBMS

Timeseries databases with no requirement on any DBMS

Column-oriented DBMS

Relational databases

Other

Unsupported databases (WIP)

Supported Elastic Infrastructures

Initial Setup of the Elastic Infrastructure

Initial Setup of local pc or Control-VM

  • To control TSDBBench, a local pc or a Control-VM (a VM with everything preinstalled) can be used (only vSphere and OpenStack)
  1. Local PC:

    1. Install packages:

      sudo apt-get install python-dateutil python-jinja2 python-numpy \
 python-pandas python-flask python-redis python-requests python-six \
 python-tornado python-werkzeug python-markupsafe python-greenlet \
 python-zmq python-yaml python-pip wkhtmltopdf python-magic fabric \
 vagrant zlib1g-dev zlib1g libxml2 libxml2-dev libxslt1.1 libxslt1-dev \
 python-webcolors python-pyvmomi

      A version of this package-list adapted for Arch-Linux is available as package-requirements

    2. Install pip packages:

      sudo pip install bokeh python-vagrant pdfkit

      Alternatively use sudo pip install -r control-requirements.txt

    3. Install vagrant plugins:

      vagrant plugin install vagrant-vsphere
vagrant plugin install vagrant-openstack-provider

    4. Reconfigure locales and make sure that en_US.UTF-8 is generated

      sudo dpkg-reconfigure locales

    5. Checking out & Prepairing Git Repo

      cd /path/to/some/folder/
git clone https://github.com/baderas/TSDBBench
cd TSDBBench
vagrant box add --name dummy dummy.box
copy hooks/pre-commit .git/hooks/
cd ..

    6. Edit config for the chosen elastic infrastructure (change everything that says '' for your chosen elastic infrastructure)

      cd TSDBBench
nano vagrant_files/vagrantconf.rb
nano vagrant_files/vagrantconf_db.rb
nano vagrant_files/vagrantconf_gen.rb
  2. Control-VM
    1. Create Control-VM according to VMware vSphere or OpenStack
    2. Login to your Control-VM
    3. Edit config for the chosen elastic infrastructure (change everything that says '' for your chosen elastic infrastructure) 
      cd TSDBBench
nano vagrant_files/vagrantconf.rb
nano vagrant_files/vagrantconf_db.rb
nano vagrant_files/vagrantconf_gen.rb

Running a testworkload

  • without creation of html file: 
    cd TSDBBench
./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l```
  • with creation of html file: 
    cd TSDBBench
./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l -m```
  • with creation of html files and multiple databases: 
    cd TSDBBench
./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 postgresql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l -m --provider "vsphere"```

Creating html files (when not using -m)

  • Creating a html file from a ycsb_*.log file: 
    cd TSDBBench
./ProcessYcsbLog.py -f some_ycsb_logfile.log
  • Creating a html file from a ycsb_*.ydc file: 
    cd TSDBBench
./ProcessYcsbLog.py -f some_ycsb_logfile.ydc
  • Creating a combined html file a set of from a ycsb_*.ydc/.log files: 
    cd TSDBBench
./ProcessYcsbLog.py -f some_ycsb_logfile1.ydc ome_ycsb_logfile2.log ome_ycsb_logfile3.ydc ...

Additional Information

  • Everything was tested and used on Debian Jessie x64, but should work on Ubuntu.
    • Ubuntu has different package names for a lot of the packages, you need to find and change them
  • Logfiles/Benchmark Results are stored compressed as .ydc Files

Development Information

  • Development specific details on databases and elastic infrastructures can be found on their specific files (see links at the beginning)
  • Adding a New Database

Funding

TSDBBench received funding from the Federal Ministry for Economic Affairs and Energy in the context of the project NEMAR.

BMWi