Overlord

This is the Python and Vagrant part of TSDBBench

• Supported databases
  • Timeseries databases with a requirement on NoSQL DBMS
  • Timeseries databases with no requirement on any DBMS
  • Column-oriented DBMS
  • Relational databases
  • Other
• Unsupported databases (WIP)
• Supported Elastic Infrastructures
• Initial Setup of the Elastic Infrastructure
• Initial Setup of local pc or Control-VM
• Running a testworkload
• Creating html files (when not using -m)
• Additional Information
• Development Information
• Funding

Supported databases

Timeseries databases with a requirement on NoSQL DBMS

• Axibase
• Blueflood
• Databus
• KairosDB
• NewTS
• OpenTSDB
• Rhombus
• Graphite

Timeseries databases with no requirement on any DBMS

• Akumuli
• Druid
• InfluxDB
• Gnocchi
• Seriesly
• Prometheus

Column-oriented DBMS

• MonetDB
• Kdb+

Relational databases

• MySQL
• PostgreSQL

Other

• Elasticsearch
• h5serv

Unsupported databases (WIP)

• DalmatinerDB
• ScyllaDB
• RiakTS
• Redis

Supported Elastic Infrastructures

• VMware vSphere
• OpenStack
• VirtualBox
• DigitalOcean
• Amazon Web Services

Initial Setup of the Elastic Infrastructure

• This must done only once and only for one elastic infrastructure
• See the articles for the five supported elastic infrastructures for specific instructions
  • VMware vSphere
  • OpenStack
  • VirtualBox
  • DigitalOcean
  • Amazon Web Services

Initial Setup of local pc or Control-VM

• To control TSDBBench, a local pc or a Control-VM (a VM with everything preinstalled) can be used (only vSphere and OpenStack)
  1. Local PC:
  1. Install packages:

      sudo apt-get install python-dateutil python-jinja2 python-numpy \
       python-pandas python-flask python-redis python-requests python-six \
       python-tornado python-werkzeug python-markupsafe python-greenlet \
       python-zmq python-yaml python-pip wkhtmltopdf python-magic fabric \
       vagrant zlib1g-dev zlib1g libxml2 libxml2-dev libxslt1.1 libxslt1-dev \
       python-webcolors python-pyvmomi
     

     A version of this package-list adapted for Arch-Linux is available as package-requirements

  2. Install pip packages:

      sudo pip install bokeh python-vagrant pdfkit
     

     Alternatively use sudo pip install -r control-requirements.txt

  3. Install vagrant plugins:

      vagrant plugin install vagrant-vsphere
      vagrant plugin install vagrant-openstack-provider
     

  4. Reconfigure locales and make sure that en_US.UTF-8 is generated

      sudo dpkg-reconfigure locales
     

  5. Checking out & Prepairing Git Repo

      cd /path/to/some/folder/
      git clone https://github.com/baderas/TSDBBench
      cd TSDBBench
      vagrant box add --name dummy dummy.box
      copy hooks/pre-commit .git/hooks/
      cd ..
     

  6. Edit config for the chosen elastic infrastructure (change everything that says ‘’ for your chosen elastic infrastructure)

      cd TSDBBench
      nano vagrant_files/vagrantconf.rb
      nano vagrant_files/vagrantconf_db.rb
      nano vagrant_files/vagrantconf_gen.rb
     

     1. Control-VM
  7. Create Control-VM according to VMware vSphere or OpenStack
  8. Login to your Control-VM
  9. Edit config for the chosen elastic infrastructure (change everything that says ‘’ for your chosen elastic infrastructure)

      cd TSDBBench
      nano vagrant_files/vagrantconf.rb
      nano vagrant_files/vagrantconf_db.rb
      nano vagrant_files/vagrantconf_gen.rb
     

Running a testworkload

• without creation of html file: bash cd TSDBBench ./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l
• with creation of html file: bash cd TSDBBench ./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l -m
• with creation of html files and multiple databases: bash cd TSDBBench ./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 postgresql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l -m --provider "vsphere"

Creating html files (when not using -m)

• Creating a html file from a ycsb_*.log file:

   cd TSDBBench
   ./ProcessYcsbLog.py -f some_ycsb_logfile.log
  

• Creating a html file from a ycsb_*.ydc file:

   cd TSDBBench
   ./ProcessYcsbLog.py -f some_ycsb_logfile.ydc
  

• Creating a combined html file a set of from a ycsb_*.ydc/.log files:

   cd TSDBBench
   ./ProcessYcsbLog.py -f some_ycsb_logfile1.ydc ome_ycsb_logfile2.log ome_ycsb_logfile3.ydc ...
  

Additional Information

• Everything was tested and used on Debian Jessie x64, but should work on Ubuntu.
  • Ubuntu has different package names for a lot of the packages, you need to find and change them
• Logfiles/Benchmark Results are stored compressed as .ydc Files

Development Information

• Development specific details on databases and elastic infrastructures can be found on their specific files (see links at the beginning)
• Adding a New Database

Funding

TSDBBench received funding from the Federal Ministry for Economic Affairs and Energy in the context of the project NEMAR.