uptasticsearch

Introduction

This project tackles the issue of getting data out of Elasticsearch and into a tabular format in R.

Table of contents

1. How it Works
2. Installation
   1. R
   2. Python
3. Usage Examples
   1. Get a Batch of Documents
   2. Aggregation Results
4. Next Steps
   1. Auth Support
5. Running Tests Locally
6. Regenerating the Documentation Site

How it Works

The core functionality of this package is the es_search function. This returns a data.table containing the parsed result of any given query. Note that this includes aggs queries.

Installation

R

Releases of this package can be installed from CRAN:

install.packages('uptasticsearch')

To use the development version of the package, which has the newest changes, you can install directly from GitHub

devtools::install_github("UptakeOpenSource/uptasticsearch", subdir = "r-pkg")

Python

This package is not currently available on PyPi. To build the development version from source, clone this repo, then :

cd py-pkg
pip install .

Usage Examples

The examples presented here pertain to a fictional Elasticsearch index holding some information on a movie theater business.

Example 1: Get a Batch of Documents

The most common use case for this package will be the case where you have an ES query and want to get a data frame representation of many resulting documents.

In the example below, we use uptasticsearch to look for all survey results in which customers said their satisfaction was "low" or "very low" and mentioned food in their comments.

library(uptasticsearch)

# Build your query in an R string
qbody <- '{
  "query": {
    "filtered": {
      "filter": {
        "bool": {
          "must": [
            {
              "exists": {
                "field": "customer_comments"
              }
            },
            {
              "terms": {
                "overall_satisfaction": ["very low", "low"]
              }
            }
          ]
        }
      }
    },
    "query": {
      "match_phrase": {
        "customer_comments": "food"
      }
    }
  }
}'

# Execute the query, parse into a data.table
commentDT <- es_search(
    es_host = 'http://mydb.mycompany.com:9200'
    , es_index = "survey_results"
    , query_body = qbody
    , scroll = "1m"
    , n_cores = 4
)

Example 2: Aggregation Results

Elasticsearch ships with a rich set of aggregations for creating summarized views of your data. uptasticsearch has built-in support for these aggregations.

In the example below, we use uptasticsearch to create daily timeseries of summary statistics like total revenue and average payment amount.

library(uptasticsearch)

# Build your query in an R string
qbody <- '{
  "query": {
    "filtered": {
      "filter": {
        "bool": {
          "must": [
            {
              "exists": {
                "field": "pmt_amount"
              }
            }
          ]
        }
      }
    }
  },
  "aggs": {
    "timestamp": {
      "date_histogram": {
        "field": "timestamp",
        "interval": "day"
      },
      "aggs": {
        "revenue": {
          "extended_stats": {
            "field": "pmt_amount"
          }
        }
      }
    }
  },
  "size": 0
}'

# Execute the query, parse result into a data.table
revenueDT <- es_search(
    es_host = 'http://mydb.mycompany.com:9200'
    , es_index = "transactions"
    , size = 1000
    , query_body = qbody
    , n_cores = 1
)

In the example above, we used the date_histogram and extended_stats aggregations. es_search has built-in support for many other aggregations and combinations of aggregations, with more on the way. Please see the table below for the current status of the package. Note that names of the form "agg1 - agg2" refer to the ability to handled aggregations nested inside other aggregations.

Agg type	R support?
"cardinality"	YES
"date_histogram"	YES
date_histogram - cardinality	YES
date_histogram - extended_stats	YES
date_histogram - histogram	YES
date_histogram - percentiles	YES
date_histogram - significant_terms	YES
date_histogram - stats	YES
date_histogram - terms	YES
"extended_stats"	YES
"histogram"	YES
"percentiles"	YES
"significant terms"	YES
"stats"	YES
"terms"	YES
terms - cardinality	YES
terms - date_histogram	YES
terms - date_histogram - cardinality	YES
terms - date_histogram - extended_stats	YES
terms - date_histogram - histogram	YES
terms - date_histogram - percentiles	YES
terms - date_histogram - significant_terms	YES
terms - date_histogram - stats	YES
terms - date_histogram - terms	YES
terms - extended_stats	YES
terms - histogram	YES
terms - percentiles	YES
terms - significant_terms	YES
terms - stats	YES
terms - terms	YES

Auth Support

uptasticsearch does not currently support queries with authentication. This will be added in future versions.

Running Tests Locally

When developing on this package, you may want to run Elasticsearch locally to speed up the testing cycle. We've provided some gross bash scripts at the root of this repo to help!

To run the code below, you will need Docker. Note that I've passed an argument to setup_local.sh indicating the major version of ES I want to run. If you don't do that, this script will just run the most recent major version of Elasticsearch. Look at the source code of setup_local.sh for a list of the valid arguments.

# Start up Elasticsearch on localhost:9200 and seed it with data
./setup_local.sh 5.5

# Run tests
make test_r

# Get test coverage and generate coverage report
make coverage_r

# Tear down the container and remove testing files
./cleanup_local.sh

Regenerating the Documentation Site

This project uses Github Pages to host a documentation site:

https://uptakeopensource.github.io/uptasticsearch/

This documentation needs to be periodically, manually updated. To generate the new files for an "update the site" PR, just run the following:

make gh_pages