Overlord

The Python and Vagrant part of TSDBBench

View project on GitHub

Overlord

This is the Python and Vagrant part of TSDBBench

Supported databases

Timeseries databases with a requirement on NoSQL DBMS

Timeseries databases with no requirement on any DBMS

Column-oriented DBMS

Relational databases

Other

Unsupported databases (WIP)

Supported Elastic Infrastructures

Initial Setup of the Elastic Infrastructure

  • This must done only once and only for one elastic infrastructure
  • All images are available at http://tsdbbench.allweathercomputing.com/bin/, can use them directly from there.
  • See the articles for the five supported elastic infrastructures for specific instructions

Initial Setup of local pc or Control-VM

  • To control TSDBBench, a local pc or a Control-VM (a VM with everything preinstalled) can be used (only vSphere and OpenStack)
    1. Local PC:
    1. Install packages:
       sudo apt-get install python-dateutil python-jinja2 python-numpy python-pandas python-flask python-redis python-requests python-six python-tornado python-werkzeug python-markupsafe python-greenlet python-zmq python-yaml python-pip wkhtmltopdf python-magic fabric vagrant zlib1g-dev zlib1g libxml2 libxml2-dev libxslt1.1 libxslt1-dev python-webcolors python-pyvmomi
      
    2. Install pip packages:
       sudo pip install bokeh python-vagrant pdfkit
      
    3. Install vagrant plugins: bashvagrant plugin install vagrant-vsphere vagrant plugin install vagrant-openstack-provider
    4. Reconfigure locales and make sure that en_US.UTF-8 is generated
       sudo dpkg-reconfigure locales
      
    5. Checking out & Prepairing Git Repo
       cd /path/to/some/folder/
       git clone https://github.com/baderas/TSDBBench
       cd TSDBBench
       vagrant box add --name dummy dummy.box
       copy hooks/pre-commit .git/hooks/
       cd ..
      
    6. Edit config for the chosen elastic infrastructure (change everything that says ‘’ for your chosen elastic infrastructure)
       cd TSDBBench
       nano vagrant_files/vagrantconf.rb
       nano vagrant_files/vagrantconf_db.rb
       nano vagrant_files/vagrantconf_gen.rb
      
      1. Control-VM
    7. Create Control-VM according to VMware vSphere or OpenStack
    8. Login to your Control-VM
    9. Edit config for the chosen elastic infrastructure (change everything that says ‘’ for your chosen elastic infrastructure)
       cd TSDBBench
       nano vagrant_files/vagrantconf.rb
       nano vagrant_files/vagrantconf_db.rb
       nano vagrant_files/vagrantconf_gen.rb
      

Running a testworkload

  • without creation of html file: bash cd TSDBBench ./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l
  • with creation of html file: bash cd TSDBBench ./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l -m
  • with creation of html files and multiple databases: bash cd TSDBBench ./TSDBBench.py -t /path/to/some/tmpfolder -f vagrant_files -d mysql_cl1_rf1 postgresql_cl1_rf1 --provider 'vsphere' -w "testworkloada" -l -m --provider "vsphere"

Creating html files (when not using -m)

  • Creating a html file from a ycsb_*.log file:
     cd TSDBBench
     ./ProcessYcsbLog.py -f some_ycsb_logfile.log
    
  • Creating a html file from a ycsb_*.ydc file:
     cd TSDBBench
     ./ProcessYcsbLog.py -f some_ycsb_logfile.ydc
    
  • Creating a combined html file a set of from a ycsb_*.ydc/.log files:
     cd TSDBBench
     ./ProcessYcsbLog.py -f some_ycsb_logfile1.ydc ome_ycsb_logfile2.log ome_ycsb_logfile3.ydc ...
    

Additional Information

  • Everything was tested and used on Debian Jessie x64, but should work on Ubuntu.
    • Ubuntu has different package names for a lot of the packages, you need to find and change them
  • Logfiles/Benchmark Results are stored compressed as .ydc Files

Development Information

  • Development specific details on databases and elastic infrastructures can be found on their specific files (see links at the beginning)
  • Adding a New Database