[timeseries] Add initial support for elasticsearch #99

[nepython]

Checks:

• I have manually tested the proposed changes
• I have written new test cases to avoid regressions (if necessary)
• I have updated the documentation (e.g. README.rst)

Closes #99, #68

Need Help
There is one test (test_get_device_metrics_csv) failing on travis (it's probably caused by cardinality aggregation counting None from date_histogram as a value). I am unable to find a solution to this. In real case this should be rare as can be confirmed by the passing of test_wifi_hostapd.

PS: Query posted on https://discuss.elastic.co/t/cardinality-aggregation-always-returning-one-extra-count/243462

[nemesifier]

Good progress @nepython, see my comments below.

[nemesifier]

why don't you change the code so that calling this functon from the configuration is not necessary?
You can loop over the data structure and call it when it's initialized, this way we make things easy for users and we avoid them come to complain to us in the support channels 😂

[nepython]

Yes that can be done and I have done the same for built-in charts. _make_query is just a utility function which will update the aggregation for a default_chart_query defined in openwisp_monitoring.db.backends.elasticsearch.queries. So queries returned via _make_query will always retain the structure of default_chart_query.

I wanted to leave the option of directly using a dsl query with timeseries_db.query, exactly like how we can query InfluxDB directly using the same function.

A full dsl query will look like this,

{'query': {'nested': {'path': 'tags',
   'query': {'bool': {'must': [{'match': {'tags.object_id': {'query': '9a39a5ae-146b-4a50-b113-f9381b8c1721'}}},
      {'match': {'tags.content_type': {'query': 'config.device'}}}]}}}},
 '_source': False,
 'size': 0,
 'aggs': {'GroupByTime': {'nested': {'path': 'points',
    'aggs': {'set_range': {'filter': {'range': {'points.time': {'from': 'now-1d/d',
         'to': 'now/d'}}},
      'aggs': {'time': {'date_histogram': {'field': 'points.time',
         'fixed_interval': '10m',
         'format': 'date_time_no_millis',
         'order': {'_key': 'desc'},
         'time_zone': 'Asia/Kolkata'},
        'aggs': {'nest': {'nested': {'path': 'points.fields',
           'aggs': {'CPU_load': {'avg': {'field': 'points.fields.cpu_usage'}}}}}}}}}}}}}}

So, if we make _make_query as a compulsion, we might be cutting down a user's freedom to query via DSL. Personally, I would like to give user this freedom (this would enable him to just put a query like above in chart configuration and it will work) 😄.

[PabloCastellano]

We don't need to run Elasticsearch in a High Available environment. Testing HA capabilities is ElasticSearch's job 😃 . We can simply make sure that setting up a multi-nodes cluster works but IMHO it is enough for us to run tests in only one instance.

WDYT?

[nepython]

I agree though there are some problems that I was facing with elasticsearch docker due to which too I am using two nodes 😅. Can you please check out if it's possible to run elasticsearch on a single port (I am not sure about this as I could not :/ ) and then I can adapt. Thanks!

[nepython]

Currently, things to do:~

• Fix cardinality aggregation returning an extra value for wifi_clients causing test_get_device_metrics_csv to fail.
• Elasticsearch tests are currently consuming ~120s to run. This can be reduced to 75s by retaining indices. There are other optimizations too which shall be implemented together in [timeseries] Optimize elasticsearch #168.

[nemesifier]

Currently, things to do:~

• Fix cardinality aggregation returning an extra value for wifi_clients causing test_get_device_metrics_csv to fail.

Please resolve this asap.

• Elasticsearch tests are currently consuming ~120s to run. This can be reduced to 75s by retaining indices. There are other optimizations too which shall be implemented together in [timeseries] Optimize elasticsearch #168.

We may do this in a second step. We make it work first, then we optimize.

[nepython]

Currently, things to do:~

• Fix cardinality aggregation returning an extra value for wifi_clients causing test_get_device_metrics_csv to fail.

Please resolve this asap.

I need help, mentioned in opening comment 😬. I am still trying to resolve but couldn't find any fix, will question this on elasticsearch discussion forums if I am unable to in the next 1-2 days.

• Elasticsearch tests are currently consuming ~120s to run. This can be reduced to 75s by retaining indices. There are other optimizations too which shall be implemented together in [timeseries] Optimize elasticsearch #168.

We may do this in a second step. We make it work first, then we optimize.

Ok

[nemesifier]

Currently, things to do:~

• Fix cardinality aggregation returning an extra value for wifi_clients causing test_get_device_metrics_csv to fail.

Please resolve this asap.

I need help, mentioned in opening comment . I am still trying to resolve but couldn't find any fix, will question this on elasticsearch discussion forums if I am unable to in the next 1-2 days.

I see that you mentioned it, can you please write a detailed explanation of what is going on and provide links to the external info you mention?

[nemesifier]

@nepython how's it going with fixing the build here?
Please focus on completing this ⚠️

[nepython]

@nepython how's it going with fixing the build here?
Please focus on completing this ⚠️

@nemesisdesign, definitely I am working to fix the build, working mostly on https://github.com/nepython/openwisp-monitoring/tree/issues/99-add-initial-support-second-timeseries_db to try out minor changes but so far no good. I asked a query on S/O in the morning (due to no response on elasticsearch discussion forum 😶), it can be found at https://stackoverflow.com/questions/63590693/elasticsearch-cardinality-aggregation-returning-one-extra-count.

As far as I can visualize, the clients Chart sometimes returns an extra count and sometimes it doesn't. In the current travis build one test is failing due to this. Locally, that one test is always passing now (but it was failing earlier 5-6 days ago just don't know why it is passing now :/).

What I am trying to figure out right now is why sometimes the clients charts returns an extra count and correct count the other times. Is this even an issue on our end or on elasticsearch's (unsure)? I posted a detailed status on the PR and a request too to provide me with some JSON data or access to any instance in which the user is ready to experiment with elasticsearch (as you had mentioned in our last call) few days back on the IM channel https://gitter.im/openwisp/openwisp-monitoring?at=5f3f709f49148b41c9619185 and reminded the same personally but didn't receive any response 😓.

Right now if we are in a hurry to merge this PR, we can do two things:~

1. Disable clients chart for elasticsearch and state it in README as Work in Progress.
2. We can include the code related to clients chart in dev with a short note related to this problem of clients chart in a section say Known Issues and open up an issue for the same, while skipping that one test for now and adding a comment above it that this needs to be fixed.

This is a big add-on and something I have worked really hard upon so would really like to see this getting merged though even after a month I am not able to solve this one single issue (which is blocking everything else) :(. I am determined to fix this issue even after GSoC and see this module getting released and be helpful to others 😃.