Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database plugin - feature request: Only write to DB on minimal change #285

Closed
smaiLee opened this issue Sep 26, 2018 · 10 comments
Closed

Database plugin - feature request: Only write to DB on minimal change #285

smaiLee opened this issue Sep 26, 2018 · 10 comments
Labels
plugins Issue relates to a plugin

Comments

@smaiLee
Copy link
Contributor

smaiLee commented Sep 26, 2018

To reduce superfluous data it would be nice if logging of values to the database could be prevented if not a minimal difference to the last logged value is reached.

Use cases:

  • Values sent by physical sensors (like temperature, brightness, distance or flow sensor) often wobble in short intervals between two values (e.g. 24.1°C and 24.2°C and back again).
    Unfortunately not every sending device allows to prevent this.
  • Sometimes an exact value is needed for live operations but for logging less precission would be sufficient.
@smaiLee smaiLee changed the title Database plugin: Only write to DB on minimal change Database plugin - feature request: Only write to DB on minimal change Sep 26, 2018
@onkelandy
Copy link
Member

Would be a nice feature indeed. I'd suggest one global "threshold" defined in the plugin.yaml (default: 0) and the possibility to overwrite it for each item individually.

@msinn msinn added the plugins Issue relates to a plugin label Mar 26, 2019
smaiLee added a commit to smaiLee/plugins that referenced this issue Jan 5, 2020
@msinn
Copy link
Member

msinn commented Jun 26, 2020

e.g. 24.1°C and 24.2°C and back again

That's not a simple request.
The first step for this example would be to ignore the 24.2 value because it is only 0,1 higher than the previous value. So far so good...

If the next value would be 24.1 again, all is fine, but what if not?

If the next value would be 25.0, is would be written to the database, but the value 24.2 would be lost completely

What to do If the next value would be 24.3?

  • Following the rule rule, 24.3 would be written, but the value 24.2 would be lost completely
  • Even more decisions would have to be made, if the database plugin would remember the 24.2 somehow. Should 24.3 be written to database or should it be ignored because it is only 0.1 higher than the previous value?

Another problem would be, that the item value and the database value would differ.

My question is: Wouldn't it be better to handle the wobbling in the plugin that receives the data?

@onkelandy
Copy link
Member

As far as I understood smai he'd need a precise value in shng in general but not necessarily in the "log" of the database. Meaning it should/could be integrated in the db plugin.

I guess when you can configure the threshold for items individually it would be totally fine when some values are not written at all, wobbling or not. meaning 24.1 is written and the current value is checked against that value. 24.2 gets ignored and forgotten but 24.3 is written again. It's more or less some "precision" attribute that you can set for items.

the other option would be to really "just" prevent wobbling meaning the plugin would need to remember 24.2 and the timestamp and write that into the db as soon as the next value is not 24.1 again. Not sure if that's the main usecase.

@msinn
Copy link
Member

msinn commented Jun 26, 2020

To prvent wobbling, someone would have to come up with a global definition. Otherwise 10 people have 12 ideas what wobbling is for them and the database plugin has to be changed over and over. (That is, why I would prefer not to implement wobbling-prevention in the database plugin but have users in need of wobbling-prevention implement it in the plugin creating the data or in a logic.

Is Wobbling:

  • a change of 0.1 units?
  • a change in the last digit of the value (and what if the number of digits keeps changing)
  • how should th wobbling center be determined?
  • if in the example above the wobbling center would be 24.1 and it can fluctuate 0.1 units, the case could arise that the value changes from 24.1 -> 24.0 -> 24.2. Now we have a change of 0.2 but both values are 0.1 around the wobbling center. How should it be handled?

@onkelandy
Copy link
Member

Maybe wobbling is a special issue indeed that should be handled somewhere else.
Nevertheless some "simple" threshold for the db plugin would be a nice feature I think.

Example like above, based on an attribute "database_threshold: 0.2" or similiar:
start with 24.1 -> db entry, no matter what threshold is defined as shng just started
24.2 -> ignore
24.0 -> ignore
23.9 -> db entry
24.0 -> ignore
24.1 -> db entry
25.0 -> db entry
24.9 -> ignore
24.8 -> db entry
etc.

That makes the plugins work as usual but limits db entries. Maybe at shutdown the current value should be written regardless.

@msinn
Copy link
Member

msinn commented Jun 26, 2020

I give you an example why "simple" threshold is not a good idea. Here is an example similar to the one you gave:

Example based on an attribute "database_threshold: 0.2" or similiar:
start with 24.1 -> db entry, no matter what threshold is defined as shng just started
24.2 -> ignore
24.3 -> ignore
24.4 -> ignore
24.5 -> ignore
24.6 -> ignore
24.7 -> ignore
24.8 -> ignore
24.9 -> ignore
etc.

@onkelandy
Copy link
Member

Comparison has to be made with the last entry in db.
Meaning there is 24.1 in the db -> 24.3 gets written as the difference is 0.2 (threshold 0.2 = minimum difference between written and current)

@msinn
Copy link
Member

msinn commented Jun 26, 2020

But to check that, it would imply a database read for every item change...

@onkelandy
Copy link
Member

or keep the value in memory (dict in the plugin). No idea if that is a good idea concerning resources

@msinn
Copy link
Member

msinn commented Jan 30, 2022

This issue has been moved to the plugin repo: smarthomeNG/plugins#564

@msinn msinn closed this as completed Jan 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
plugins Issue relates to a plugin
Projects
None yet
Development

No branches or pull requests

3 participants