Tengu is a flexible and dynamic data platform that can adapt to your needs. Tengu makes it possible to quickly set up and test a number of different data technologies such as Hadoop, Spark, Storm and Kafka. Tengu gives users the ability to connect different data technologies in order to get a complete distributed data solution. Tengu makes it possible to integrate with existing data and automation infrastructure. Existing automation artifacts such as Chef cookbooks and Puppet scripts can be reused to ease the transition to and integration with Tengu.
Tengu enables a number of use-cases:
- Tengu provides a sandbox that enables guided experimentation with data tools. Companies that want to get started with big data technologies can use Tengu to build and test different big data solutions to see what fits their use-case.
- Companies that want to integrate big data tools into their existing infrastructure can use Tengu to manage the integration between big data tools and their existing data infrastructure. Because Tengu is automation-tool independent, it can easily reuse existing automation artifacts such as Chef cookbooks and Puppet scripts.
- Developers that want to accelerate adoption of their data tools can capture the operational knowledge of their data tool in Tengu to allow fast and easy deployment and integration.
When you integrate your data tool into Tengu, user can very easily integrate your tool in their workflows. By using Juju at its core, Tengu enables you to connect your data tool to the wide range of tools already available in Tengu.
When you write a Juju Charm, you capture the best practice operational knowledge of your data tool and enable others to reuse this knowledge. The first step is to capture the knowledge of how to install and configure your data tool. This enables users to rapidly deploy and test your data tool. The second step is to capture the knowledge of the options for communication and collaboration with other data tools. When you implement existing interfaces in your Charm, users can swap out existing data tools and replace them by your data tool. This enables very fast adoption of new data tools.
The Juju docs contain a good high-level overview of what Juju is. Are you planning to integrate you data tool and want some guidance? Don't hesitate to contact us! You can contact me at [email protected].
This repository contains all the Open Source Charms, Bundles, Interfaces and Layers of the Tengu platform. This repository is the upstream for the Tengu Charms available in the Charm Store.
Most of the Tengu services, like rest2jfed and tengu-instance-admin, are contained in their Charms in this repository and do not have a seperate upstream project.
This repository does not contain the required bundles to deploy the Tengu platform as a whole, since those bundles contain "private" config options like private keys. Those bundles can be accessed in the private Tengu repository if you have the correct credentials.
Tengu uses Juju at its core. Juju is used to
This git repository doesn't contain big files like binaries and tarballs. You have to download these binaries before you can deploy Charms from this repository. The tengu downloadbigfiles tool downloads the binaries and puts them in the correct folder. This tool is installed by default on a Hauchiwa and is run once during installation.
If you add a <filename>.source file in a folder, tengu downloadbigfiles will download the source to that folder as <filename>. The <filename>.source file either specifies a url or a download command. The <filename>.source file can optionally specify that the downloaded file should be extracted. This is the format:
<url> # url to download resource from
[<action>] # action to execute after download. Currently only 'extract' is supported
or
command: <download command> # command to execute in order to download resource
[<action>] # action to execute after download. Currently only 'extract' is supported
The second format can be used when you need to add custom cookies and headers to the download request. This is for example needed when downloading oracle java as used in the rest2jfed charm.
Both the resulting Charms and the source layers of layered Charms are located in this repository. Changes to layered Charms have to be made to the corresponding layers in their charms/layers/<layerName> directory. Do not make changes to the charm in charms/<series>/<charmName> directory if it is a layered charm! Changes will be overwritten after rebuilding the Charm!
After editing the layers, you can regenerate the layer by running charm generate <top-layer-name>. It is advised to do this in a python virtualenv to avoid contamination between your host, the charm tools and the layer dependencies.
sudo apt-get install python-virtualenv
virtualenv charm-tools # create the virtualenv
. charm-tools/bin/activate # go inside the virtualenv
pip install -U charm-tools pip # install charm-tools
Note: Layered Charms run python 3 be default and install pip3 as default pip installation.
I'm using Atom on Ubuntu with some extentions. Here's how I installed everything:
# Install Atom:
sudo add-apt-repository ppa:webupd8team/atom
sudo apt-get update
sudo apt-get install atom
# Install python package manager
sudo apt install python-pip python3-pip
# Pyton linting (code checking) for both python 2 and python 3
sudo pip2 install pylint
sudo pip3 install pylint
apm install linter
apm install linter-pylint
mkdir ~/bin
nano ~/bin/pylint and add:
#!/bin/bash
if [[ $(head -n 1 "${@: -1}") == *python3* ]]
then
pylint3 --extension-pkg-whitelist=lxml,netifaces "$@"
else
pylint2 --extension-pkg-whitelist=lxml,netifaces "$@"
fi
nano ~/bin/pylint2 and add:
#!/usr/bin/python2
# EASY-INSTALL-ENTRY-SCRIPT: 'pylint','console_scripts','pylint'
__requires__ = 'pylint'
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.exit(
load_entry_point('pylint', 'console_scripts', 'pylint')()
)
nano ~/bin/pylint3 and add:
#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'pylint','console_scripts','pylint'
__requires__ = 'pylint'
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.exit(
load_entry_point('pylint', 'console_scripts', 'pylint')()
)
and finally: chmod u+x ~/bin/pylint ~/bin/pylint2 ~/bin/pylint3. Log out and log back in to save the changes.
Charm tools: helper toos to Charm.
# Juju Charm tools
sudo apt-get install charm-tools
sudo apt install python-pip
pip install charmhelpers
When running juju debug-hooks, you enter a tmux session. The default tmux bindings on Ubuntu are a bit strange. ctrl-a is the default command. To enable sane mouse scrolling set set-window-option -g mode-mouse on in ~/.tmux.conf of the server.
pull PR from github
git pull origin pull/$PR_NUM/head
prettyprint json output
| python -m json.tool
grep and get text around match
cat log | grep -A10 <searchterm> # Next 10 lines
cat log | grep -B10 <searchterm> # Previous 10 lines