Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store database in chunks to prevent running out of space #29

Open
Dawnkai opened this issue Dec 25, 2023 · 0 comments
Open

Store database in chunks to prevent running out of space #29

Dawnkai opened this issue Dec 25, 2023 · 0 comments
Labels
backend Tasks related to backend non-essential Tasks that are not required, but would be nice to have

Comments

@Dawnkai
Copy link
Owner

Dawnkai commented Dec 25, 2023

If the data from the API is saved on the local drive, eventually we will run out of space (especially since this is meant to run on IoT devices, which have little disk space to begin with). It would be nice to save data in chunks. I see it like this:

  • Specify the max amount of measurements that can be stored in a single database file.
  • If the database reaches max limit of measurements, create a new database file and from now on save data to the new database.
  • Rename old database to a value that will allow the system to find what date range is contained in that database.
  • When fetching old data, use the old database (if the request is contained in the date range of that database) - and update only that old database, not the new one - unless the date range isn't covered by previous database. Don't insert data into the old database.
  • If the request requires data contained in multiple databases, fetch and merge the results.
  • Allow the user to specify max amount of database files. If the amount of database files exceeds this limit, remove the oldest database.

Thanks to this the user can ensure that they will not run out of space when the system will be running for a long time (which is kind of what IoT does anyway).

@Dawnkai Dawnkai added backend Tasks related to backend non-essential Tasks that are not required, but would be nice to have labels Dec 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Tasks related to backend non-essential Tasks that are not required, but would be nice to have
Projects
None yet
Development

No branches or pull requests

1 participant