Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[timeseries] Add initial support for elasticsearch #99 #164

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ WORKDIR /opt/openwisp/tests/
ENV NAME=openwisp-monitoring \
PYTHONBUFFERED=1 \
INFLUXDB_HOST=influxdb \
REDIS_HOST=redis
REDIS_HOST=redis \
ELASTICSEARCH_HOST=es01
CMD ["sh", "docker-entrypoint.sh"]
EXPOSE 8000
41 changes: 39 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ Available Features
* Collects and displays `device status <#device-status>`_ information like uptime, RAM status, CPU load averages,
Interface properties and addresses, WiFi interface status and associated clients,
Neighbors information, DHCP Leases, Disk/Flash status
* Collection of monitoring information in a timeseries database (`InfluxDB <https://www.influxdata.com/>`_ and `Elasticsearch <https://www.elastic.co/elasticsearch/>`_ are currently supported)
* Monitoring charts for uptime, packet loss, round trip time (latency), associated wifi clients, interface traffic,
RAM usage, CPU load, flash/disk usage
* Charts can be viewed at resolutions of 1 day, 3 days, a week, a month and a year
Expand Down Expand Up @@ -108,6 +109,8 @@ beforehand.
In case you prefer not to use Docker you can `install InfluxDB <https://docs.influxdata.com/influxdb/v1.8/introduction/install/>`_
and Redis from your repositories, but keep in mind that the version packaged by your distribution may be different.

If you wish to use ``Elasticsearch`` for storing and retrieving timeseries data then `install Elasticsearch <https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_.

Install spatialite and sqlite:

.. code-block:: shell
Expand Down Expand Up @@ -165,6 +168,20 @@ Follow the setup instructions of `openwisp-controller
'PORT': '8086',
}

In case, you wish to use ``Elasticsearch`` for timeseries data storage and retrieval,
make use of the following settings

.. code-block:: python

TIMESERIES_DATABASE = {
'BACKEND': 'openwisp_monitoring.db.backends.elasticsearch',
'USER': 'openwisp',
'PASSWORD': 'openwisp',
'NAME': 'openwisp2',
'HOST': 'localhost',
'PORT': '9200',
}

``urls.py``:

.. code-block:: python
Expand Down Expand Up @@ -461,6 +478,9 @@ This data is only used to assess the recent status of devices, keeping
it for a long time would not add much benefit and would cost a lot more
in terms of disk space.

**Note**: In case you use ``Elasticsearch`` then time shall be taken as integral multiple of a day.
That means the time ``36h0m0s`` shall be interpreted as ``24h0m0s`` (integral multiple of a day).

``OPENWISP_MONITORING_AUTO_PING``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down Expand Up @@ -764,18 +784,30 @@ MB (megabytes) instead of GB (Gigabytes) you can use:
"SUM(rx_bytes) / 1000000 AS download FROM {key} "
"WHERE time >= '{time}' AND content_type = '{content_type}' "
"AND object_id = '{object_id}' GROUP BY time(1d)"
)
),
'elasticsearch': _make_query({
'upload': {'sum': {'field': 'points.fields.tx_bytes'}},
'download': {'avg': {'field': 'points.fields.rx_bytes'}},
})
},
}
}

# Please declare the operations separately in case you use elasticsearch as done below
OPENWISP_MONITORING_ADDITIONAL_CHART_OPERATIONS = {
'upload': {'operator': '/', 'value': 1000000},
'download': {'operator': '/', 'value': 1000000},
}

Or if you want to define a new chart configuration, which you can then
call in your custom code (eg: a custom check class), you can do so as follows:

.. code-block:: python

from django.utils.translation import gettext_lazy as _

from openwisp_monitoring.db.backends.elasticsearch import _make_query

OPENWISP_MONITORING_CHARTS = {
'ram': {
'type': 'line',
Expand All @@ -789,7 +821,12 @@ call in your custom code (eg: a custom check class), you can do so as follows:
"MEAN(buffered) AS buffered FROM {key} WHERE time >= '{time}' AND "
"content_type = '{content_type}' AND object_id = '{object_id}' "
"GROUP BY time(1d)"
)
),
'elasticsearch': _make_query({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why don't you change the code so that calling this functon from the configuration is not necessary?
You can loop over the data structure and call it when it's initialized, this way we make things easy for users and we avoid them come to complain to us in the support channels 😂

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that can be done and I have done the same for built-in charts. _make_query is just a utility function which will update the aggregation for a default_chart_query defined in openwisp_monitoring.db.backends.elasticsearch.queries. So queries returned via _make_query will always retain the structure of default_chart_query.

I wanted to leave the option of directly using a dsl query with timeseries_db.query, exactly like how we can query InfluxDB directly using the same function.

A full dsl query will look like this,

{'query': {'nested': {'path': 'tags',
   'query': {'bool': {'must': [{'match': {'tags.object_id': {'query': '9a39a5ae-146b-4a50-b113-f9381b8c1721'}}},
      {'match': {'tags.content_type': {'query': 'config.device'}}}]}}}},
 '_source': False,
 'size': 0,
 'aggs': {'GroupByTime': {'nested': {'path': 'points',
    'aggs': {'set_range': {'filter': {'range': {'points.time': {'from': 'now-1d/d',
         'to': 'now/d'}}},
      'aggs': {'time': {'date_histogram': {'field': 'points.time',
         'fixed_interval': '10m',
         'format': 'date_time_no_millis',
         'order': {'_key': 'desc'},
         'time_zone': 'Asia/Kolkata'},
        'aggs': {'nest': {'nested': {'path': 'points.fields',
           'aggs': {'CPU_load': {'avg': {'field': 'points.fields.cpu_usage'}}}}}}}}}}}}}}

So, if we make _make_query as a compulsion, we might be cutting down a user's freedom to query via DSL. Personally, I would like to give user this freedom (this would enable him to just put a query like above in chart configuration and it will work) 😄.

'total': {'avg': {'field': 'points.fields.total'}},
'free': {'avg': {'field': 'points.fields.free'}},
'buffered': {'avg': {'field': 'points.fields.buffered'}},
})
},
}
}
Expand Down
48 changes: 48 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ services:
depends_on:
- influxdb
- redis
- es01
- es02

influxdb:
image: influxdb:1.8-alpine
Expand All @@ -22,6 +24,45 @@ services:
INFLUXDB_DB: openwisp2
INFLUXDB_USER: openwisp
INFLUXDB_USER_PASSWORD: openwisp
# clustered version of elasticsearch is used as that might be used in production
es01:
Copy link
Contributor

@PabloCastellano PabloCastellano Jul 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to run Elasticsearch in a High Available environment. Testing HA capabilities is ElasticSearch's job 😃 . We can simply make sure that setting up a multi-nodes cluster works but IMHO it is enough for us to run tests in only one instance.

WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree though there are some problems that I was facing with elasticsearch docker due to which too I am using two nodes 😅. Can you please check out if it's possible to run elasticsearch on a single port (I am not sure about this as I could not :/ ) and then I can adapt. Thanks!

image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
container_name: es01
environment:
- "node.name=es01"
- "discovery.seed_hosts=es02"
- "cluster.initial_master_nodes=es01,es02"
- "cluster.name=openwisp2"
- "bootstrap.memory_lock=true"
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
container_name: es02
environment:
- "node.name=es02"
- "discovery.seed_hosts=es01"
- "cluster.initial_master_nodes=es01,es02"
- "cluster.name=openwisp2"
- "bootstrap.memory_lock=true"
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata02:/usr/share/elasticsearch/data
networks:
- esnet

redis:
image: redis:5.0-alpine
Expand All @@ -31,3 +72,10 @@ services:

volumes:
influxdb-data: {}
esdata01:
driver: local
esdata02:
driver: local

networks:
esnet:
4 changes: 1 addition & 3 deletions openwisp_monitoring/db/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
from .backends import timeseries_db

chart_query = timeseries_db.queries.chart_query
default_chart_query = timeseries_db.queries.default_chart_query
device_data_query = timeseries_db.queries.device_data_query

__all__ = ['timeseries_db', 'chart_query', 'default_chart_query', 'device_data_query']
__all__ = ['timeseries_db', 'chart_query']
3 changes: 2 additions & 1 deletion openwisp_monitoring/db/backends/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@ def load_backend_module(backend_name=TIMESERIES_DB['BACKEND'], module=None):
except ImportError as e:
# The database backend wasn't found. Display a helpful error message
# listing all built-in database backends.
builtin_backends = ['influxdb']
builtin_backends = ['influxdb', 'elasticsearch']
raise e
if backend_name not in [
f'openwisp_monitoring.db.backends.{b}' for b in builtin_backends
]:
Expand Down
3 changes: 3 additions & 0 deletions openwisp_monitoring/db/backends/elasticsearch/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from .queries import _make_query

__all__ = ['_make_query']
Loading