Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor rg-ipfs queue #141

Open
dariusc93 opened this issue Jan 14, 2023 · 1 comment
Open

Refactor rg-ipfs queue #141

dariusc93 opened this issue Jan 14, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request improvement Improve on existing feature P-High High Priority refactoring Major change to features

Comments

@dariusc93
Copy link
Contributor

dariusc93 commented Jan 14, 2023

The current queue is setup as a map which would check for peers connection status and send messages accordingly if they are online, however this would create a bottleneck with large amounts of messages and events stored in queue and would not be scalable since we check one peer at a time.

The queue should be refactored to be its own struct that would handle each peer as a entry with a separate task performing checks on each entry. When we wish to add something to queue for that specific peer, we would use a function to the queue that would send the data to the entry, If there one available. If there is no entry for said peer then one would be created. Inside each entry would contain a separate tokio task that would run and would perform a check on the peer connectivity. Once the peer is confirmed to be online, we would deliver all the entries supplied. After everything was delivered, we would signal that the queue entry is empty and the queue, which would also have its own task, would check each entry for its status and after seeing any items marked empty would begin a timer which would mark the removal the entry at the end of the timer.

Each entry would be saved in its own file that would be load into its own entry points and begin the task after loading it into memory.

Notes:

  • Each entry will be encrypted on file but would be decrypted when loading it prior to the task starting
  • Loading each entry should be non-blocking (meaning each entry should be loaded in a separate task)
  • If an entry is invalid, corrupted, or damaged, to skip it (and eventually override it)
  • When a conversation is deleted and there is an entry referencing a conversation, those items in queue should be removed
  • If a message is reference in the entry and it is deleted in the conversation, any entries referencing the message should be remove from queue.
  • (Optional) The task in the messaging store should signal the queue to shutdown all entry task. To do this, we could notify, a channel, or an atomicbool to signal each entry task to kill its loop so the task can end. This should only be done if we want a graceful shutdown, though not required as once the data is sent to the queue entry, it should automatically save from within that task.
  • (Optional) Though not implemented yet, have a base so offline services are available, it would utilize that first but if its unable to utilize it for whatever reason to fallback to localize queue.
@dariusc93 dariusc93 added enhancement New feature or request improvement Improve on existing feature refactoring Major change to features P-High High Priority blocked Blocked by another feature, request or issue. labels Jan 14, 2023
@dariusc93 dariusc93 removed the blocked Blocked by another feature, request or issue. label Jan 19, 2023
@dariusc93 dariusc93 self-assigned this Jan 20, 2023
@dariusc93
Copy link
Contributor Author

With #217 came changes that introduce a request/response style system. This system could possibly be used to replace the queue itself by requesting messages from since a point to other users in the chat, in which the user would respond with any messages, however this comes with a problem in detecting deleted or edited messages when making a request.

Possible solution in the event we decide to go this route:
1, Create a new internal message that is ignored by the interface but is used to reference specific events such as pinning, unpinning, editing and deleting and pass those along to point to messages that were changed in some way.
2. Queue those specific events instead and request the queue from the user for those events prior to requesting any messages.

@dariusc93 dariusc93 added the question Further information is requested label Jun 16, 2023
@dariusc93 dariusc93 removed the question Further information is requested label Dec 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request improvement Improve on existing feature P-High High Priority refactoring Major change to features
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant