Installation

Let’s get Eskapade up and running! In order to make this as easy as possible, we provide both a Docker image and a virtual machine where everything you need is installed and working properly. Alternatively, you can download the repository and run it on your own machine.

This manual is written for Linux systems, but Eskapade also runs fine on macOS systems.

Eskapade on your own machine

Eskapade can be installed as any other Python package with easy_install or pip. To get started, simply do:

$ pip install Eskapade

We have verified that this works on Linux 16.04 and MacOS based machines.

Or check out the code from out github repository:

$ git clone git@github.com:KaveIO/Eskapade.git
$ pip install -e Eskapade/

where the code is installed in editable mode (option -e).

You can now use Eskapade in Python with:

import eskapade

Congratulations, you are now ready to use Eskapade!

See the other parts of the documentation for specific usage.

Eskapade with Docker

Consistent environments for Eskapade development and use can be created with docker containers. Containers are created and managed by Docker. An Eskapade container contains a pre-installed Eskapade setup and runs out-of-the-box. It is possible to mount your customised Eskapade code for development inside the container.

The instructions below show how one can use a locally checked-out version of Eskapade in combination with this Docker image. This combination is a very convenient way of actively working with and/or developing code in Eskapade.

By default, Eskapade code is executed as root user in the container. It is possible, however, to run with reduced user privileges inside containers through user mapping with the Docker host, as decribed in the last section.

Required software

Docker installation instructions can be found here: https://docs.docker.com/install/.

Eskapade source code is optional (for development purposes), you can check it out locally with the command:

git clone git@github.com:KaveIO/Eskapade.git eskapade

Getting Eskapade docker images

From DockerHub

The official Eskapade docker image is provided on DockerHub.

$  docker pull kave/eskapade-env:0.8

This will download the kave/eskapade-env:0.8 image locally. Downloading this docker image can take a minute or two.

Building from scratch

To build the docker image from scratch using the Eskapade source code, do:

$  cd eskapade/docker/eskapade-env && sh create_docker.sh

This will produce the kave/eskapade-env:0.8 image locally.

Spinning up docker containers

Out-of-the-box

From this image, containers with the Eskapade environment set up, can be run out-of-the-box:

$  docker run -p 8888:8888 -it kave/eskapade-env:0.8

Where port 8888 is forwarded to the docker host to make Jupyter notebook available (below).

E.g. one can now do:

cd /opt/eskapade
eskapade_run --help

and run any Eskapade code. See the Tutorial section for examples.

Exit the (docker) bash shell with:

exit

See section After you exit Docker (below) for cleaning up obsolete docker processes.

Mounting source code

$  docker run -v <ESKAPADE>:/opt/eskapade -p 8888:8888 -it kave/eskapade-env:0.8

Where <ESKAPADE> specifies the path of the Eskapade source code on the docker host, and where /opt/eskapade is the location of the Eskapade source code inside the container.

NOTE: in case you mount a clean installation of the Eskapade source code, you have to (re-)build the libraries by executing:

$ pip install -e /opt/eskapade

Running as non-root user

For increased security in a production environment, it is recommended to run Eskapade code inside the container as non-root user. The Dockerfile in the eskapade-user directory provides an additional user-mapping layer to the eskapade-env image: it creates a esdev user that has its own virtual Python environment with Eskapade installed. The mapping of user id’s between Docker host and container ensure that proper permissions are propogated when writing/reading to the mounted volume with Eskapade code.

To obtain a centrally produced Eskapade image, use:

$ docker pull kave/eskapade-usr:0.8

Or build the Eskapade docker image with esdev user installation, from scratch:

$  cd docker/eskapade-usr && docker build -t kave/eskapade-usr:0.8 .

This will produce the kave/eskapade-usr:0.8 image.

From this image, containers with the Eskapade environment set up, can be run out-of-the-box:

$ docker run -e HOST_USER_ID=$(id -u) -e HOST_USER_GID=$(id -g) -p 8888:8888 -it kave/eskapade-usr:0.8

The first time you run this command it will likely take some time. The HOST_USER_ID and HOST_USER_GID environment variables are used to dynamically map user- and group id’s between the host and Docker container, ensuring proper read/write permissions.

Remapping the user id

To prevent the remapping of user and group id from happening the next time you boot up the image, open another shell:

$ docker ps

Copy the top CONTAINER-ID string, matching the running instance of the kave/eskapade-usr:0.8 image, and then paste it:

$ docker commit CONTAINER-ID kave/eskapade-usr:0.8

Next time when you run:

$ docker run -e HOST_USER_ID=$(id -u) -e HOST_USER_GID=$(id -g) -p 8888:8888 -it kave/eskapade-usr:0.8

the remapping of user and group id should no longer happen.

Mounting source code

Containers with the user-specific Eskapade environment setup can be run out-of-the-box, and with your own mounted (customised) source code, using:

$  docker run -e HOST_USER_ID=$(id -u) -e HOST_USER_GID=$(id -g) -v <ESKAPADE>:/home/esdev/eskapade -p 8888:8888 -it kave/eskapade-usr:0.8

Where <ESKAPADE> specifies the path of the Eskapade source code.

NOTE: in case you mount a clean installation of the Eskapade source code, you have to (re-)build the libraries by executing:

$ pip install -e /home/esdev/eskapade

This combination is a great way of using and developing Eskapade code.

Consider adding a permanent alias to your local ~/.bashrc or ~/.bash_profile file:

alias eskapade_docker='docker run -e HOST_USER_ID=$(id -u) -e HOST_USER_GID=$(id -g) -v <ESKAPADE>:/home/esdev/eskapade -p 8888:8888 -it kave/eskapade-usr:0.8'

So the next time, in a fresh shell, you can simply run the command eskapade_docker.

Starting Jupyter notebook

To run the Jupyter notebook on port 8888 from the docker environment:

cd /opt/eskapade
jupy &

And press enter twice to return to the shell prompt.

The command jupy & starts up Jupyter notebook in the background on port 8888 and pipes the output to the log file nohup.out.

In your local browser then go to address:

localhost:8888/

And you will see the familiar Jupyter environment. In case you get asked for a password, take a look at the tail end of the file nohup.out, where you will see the exact url address that you need to go to.

E.g. you can now do import eskapade (shift-enter) to get access to the Eskapade library.

Be sure to run jupy & from a directory that is mounted in the docker container, such as /opt/eskapade. In this way any notebook(s) you create are kept after you exit the docker run.

After you exit Docker

Every time you want to have a clean Docker environment, run the following commands:

# --- 1. remove all exited docker processes
docker ps -a | grep Exited | awk '{print "docker stop "$1 "; docker rm "$1}' | sh

# --- 2. remove all failed docker image builts
docker images | grep "<none>" | awk '{print "docker rmi "$3}' | sh

# --- 3. remove dangling volume mounts
docker volume ls -qf dangling=true | awk '{print "docker volume rm "$1}' | sh

To automate this, we advise you put these commands in an executable docker_cleanup.sh script.

Eskapade on a virtual machine

Consistent environments for Eskapade development and use are created with virtual machines. Machines are created and managed by Vagrant, with VirtualBox as a provider.

Getting started

Required software

To build and run Eskapade boxes, Vagrant and VirtualBox are required. Vagrant can be downloaded at https://www.vagrantup.com/downloads.html. For Ubuntu (and other Debian-based systems), download the Vagrant Debian package, move to where you want to install this and run this command to install:

dpkg -i vagrant_<VERSION>_<ARCHITECTURE>.deb

where the version and architecture are in the file name.

VirtualBox can be downloaded and installed by following the instructions on https://www.virtualbox.org/wiki/Downloads. Make sure you install the latest version from the VirtualBox repository instead of an older version from the repositories of your system.

Repository

The repository, which contains the code to build your virtual environment, is hosted on github. Clone it to your machine with:

$ git clone git@github.com:KaveIO/Eskapade.git

All code below is executed in the root of this repository, unless otherwise stated.

Virtual environment

The environment box contains Ubuntu and the KAVE Toolbox, including Anaconda, Spark, and ROOT.

You can either build this box yourself, or download the image. In order to build it yourself, run the following commands. It might take a while to build (up to two hours!)

cd vagrant/dev
vagrant up

Alternatively, in order to download and import its image, do:

cd vagrant/dev_image
vagrant box add eskapade-dev.json
vagrant up

Eskapade users and developers log in as the user esdev on localhost, by default on port 2222:

ssh -p 2222 esdev@localhost

This user can log in with the password esdev.

By default the local directory /path/to/your/local/eskapade is mounted (containing your local Eskapade repository) under /opt/eskapade in your vagrant machine. You can now edit the files in this directory, either locally or in the (vagrant) bash shell, and any updates to these files will be kept after exiting the virtual machine.

You are now ready to use Eskapade!

The next time…

Simply do:

cd vagrant/dev
vagrant up

or:

cd vagrant/dev_image
vagrant up

depending on whether you built vagrant yourself or downloaded the image.

Then you can access it via ssh (password esdev):

ssh -p 2222 esdev@localhost

Easy log-in

To make logging in easier, the key pair vagrant/dev/ssh/esdev_id_rsa.pub, vagrant/dev/ssh/esdev_id_rsa can be used, and an example SSH configuration is provided in vagrant/dev/ssh/config. Put these files in your ~/.ssh/:

cp vagrant/dev/ssh/* ~/.ssh/

You can then log in using the command:

ssh esdevbox

Vagrant boxes

Boxes are built and started with the command vagrant up in the directory of the Vagrantfile describing the box. A box can be restarted by executing vagrant reload. The virtual machines are administered by the vagrant user, which logs in by running vagrant ssh in the directory of the Vagrantfile. The vagrant user has root access to the system by password-less sudo.

Starting Jupyter notebook

To run the Jupyter notebook on port 8888 from the vagrant machine:

cd /opt/eskapade
jupy &

And press enter twice to return to the shell prompt.

The command jupy & starts up Jupyter notebook in the background on port 8888 and pipes the output to the log file nohup.out.

In your local browser then go to address:

localhost:8888/

And you will see the familiar Jupyter environment.

E.g. you can now do import eskapade (shift-enter) to get access to the Eskapade library.

Be sure to run jupy & from a directory that is mounted in the vagrant machine, such as /opt/eskapade. In this way any notebook(s) you create are kept after you exit the docker run.

Requirements

Eskapade requires Python 3 and some libraries, which can be found in setup.py at the root of the repository.

Eskapade can be installed as any other Python package with easy_install or pip:

$ pip install /path/to/eskapade

Alternatively, consider installing KaveToolbox version 3.6 or higher. To install the released version:

$ yum -y install wget curl tar zip unzip gzip python
$ wget http://repos:kaverepos@repos.kave.io/noarch/KaveToolbox/3.6-Beta/kavetoolbox-installer-3.6-Beta.sh
$ sudo bash kavetoolbox-installer-3.6-Beta.sh [--quiet]

(–quiet is for a quieter install, remove the brackets!)

If anaconda is already installed in your machine, consider creating a conda virtual environment with Python 3.6 to install all the requirements and Eskapade itself to avoid collisions:

$ conda create -n eskapade_env36 python=3.6 anaconda

Then you can activate it as follows:

$ source activate eskapade_env36

More information about conda virtual environments can be found here

Installing Eskapade on macOS

To install eskapade on macOS, see our macOS appendix.