Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prefer ethdb, try to migrate sqlite to ethdb #223

Open
GrapeBaBa opened this issue Nov 17, 2024 · 3 comments
Open

Prefer ethdb, try to migrate sqlite to ethdb #223

GrapeBaBa opened this issue Nov 17, 2024 · 3 comments
Assignees

Comments

@GrapeBaBa
Copy link
Member

No description provided.

@fearlessfe
Copy link

When use ethdb, the contentId and contentValue are stored as key and value. distance = contentId xor nodeId

  1. It's hard to get the distance. When the capacity of db is full, the farthest distance is setted to radius, there is no simple way to get the farthest distance
  2. When the db is full and we should delete the content order by distance desc until the size is below the capacity

For the first problem, we can trace the farthest distance when the new data inserted in the db.

For the second problem, we should cache the top N distance in order, and keep the distance -> value size, so we can know how many items should be deleted.

@r4f4ss
Copy link

r4f4ss commented Nov 18, 2024

Please bear in mind that I am not familiarized with ethdb and its subjacents databases like pebble.

  1. Pebble, for instance, have the same interface as RocksDB, that have more documentation available. This link can indirectly helps you: https://stackoverflow.com/questions/73670453/rocksdb-range-query-on-numbers
  2. It is a possibility to use SQLite as subjacent database with ethdb, just not sure if it is the best choice.
  3. You may have to implement this search functionality in ethdb.

@fearlessfe
Copy link

when the capacity is full, we must delete the top n farthest item to free some storage. In pebble, we can use iterator to get the last item. So we can use distance = contentId xor nodeId as key, so we can easily delete the farthest items.

For the storage size, because of the LSM model, delete item is just add a label, the storage space is not freed, it hard to know the real storage size. So we will maintain the size, len(key) + len(value) will be treat as the storage size. For permance, the size will be store in pebble every 5 seconds. other time, it just update in memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants