-
-
Notifications
You must be signed in to change notification settings - Fork 89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Mark reactions as IMAP-seen in marknoticed_chat() (#6210) #6213
base: main
Are you sure you want to change the base?
Conversation
Since you marked it as a draft, I take it that it's not ready to be reviewed yet? |
a089199
to
b71c681
Compare
Now it's ready and seems working, but probably needs more tests. Was a draft because i needed to optimise out saving unnecessary columns to db. It's interesting that sent out reactions were already hidden messages, didn't know about that. |
Ideally there will be a test that marks a reaction as seen on one device and checks that a MsgsNoticed event is emitted on another device (and check that the test fails without this PR here), in addition to the unit tests. If this is not feasible, would be good to manually test that this event is sent. Edit: If, during testing, it turns out that it's harder to do than expected, it would be fine to just ignore the issue, see deltachat/deltachat-android#3377 (comment) |
I think it's not so difficult to write a jsonrpc Python test for that and it would be good to have incoming reactions in the |
de2b5a1
to
d9482dc
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice that not a lot of added complexity was necessary
src/receive_imf.rs
Outdated
|
||
state = if seen | ||
|| fetching_existing_messages | ||
|| is_mdn | ||
|| is_reaction | ||
|| chat_id_blocked == Blocked::Yes | ||
{ | ||
state = if seen || fetching_existing_messages || is_mdn || chat_id_blocked == Blocked::Yes { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change leads to IncomingMsg instead of MsgsChanged being emitted, so that the UI shows a notification "Hocuri: ❤️" if I reacted with a heart emoji. Which we don't want, since we instead to show the "Hocuri reacted ❤️ to "I like your avatar!"" message in the UI.
So, this needs to be:
state = if seen || fetching_existing_messages || is_mdn || chat_id_blocked == Blocked::Yes {
MessageState::InSeen
} else if is_reaction {
MessageState::InNoticed
} else {
MessageState::InFresh
};
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, this is a bug which requires a separate test it seems.
But again, making reactions InNoticed
to only make marknoticed_chat()
faster doesn't look conceptually correct. Reactions are still a user-noticeable info, so there's smth to "notice". And wrt the bug you found, rather we shouldn't emit IncomingMsg
for hidden messages. But not sure, need to try and see if the tests doesn't break at least. EDIT: Tried, the Rust tests continue to work at least.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A PR extending the existing tests to catch this bug: #6257. Checked that the modified tests fail for this PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Incoming reactions aren't really InFresh either, because InFresh means that the message is counted in the unread-counter and is shown in notifications. And info messages are also immediately added as InNoticed, because that's what they semantically are.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not necessarily, get_fresh_msg_cnt()
includes the and m.hidden=0
SQL condition, so if we need, we can make incoming reactions hidden InFresh messages. As for "is shown in notifications", now this is true for incoming reactions, that's also why i think they should be InFresh. If we add them as InNoticed and show notifications for them, that would be different from incoming messages. So, my suggestion is:
- Add incoming reactions as as hidden InFresh messages.
- Add other incoming hidden messages as InSeen. But looking at
receive_imf.rs
i don't see such messages even exist. But then i don't understand why we haveand m.hidden=0
checks for incoming messages in SQL everywhere. So this needs investigation. - Add a db migration for existing hidden InFresh messages. At least just in case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From a code perspective, we can make them InFresh, it requires one more special case (in order not to send an IncomingMsg), but that doesn't make any difference, really.
What does make a difference is whether we introduce 1ms of delay (1ms on quite a new phone, that is - older phones may easily need 3 times as long). Doing this just once isn't noticeable, but if we do it often (and without necessarily needing to) then we will end up with slow code (and markseen is called every time a chat is opened). Measuring this isn't hard, you can also do it - if you find a way that is just as fast, I'm fine with that, too. You may need to measure on a phone, not on Desktop, because on a fast PC the difference may not be noticeable.
Add a db migration for existing hidden InFresh messages. At least just in case.
We shouldn't do database migrations "just in case", since there is always some chance that they create problems, and we don't have any indication that it will solve problems.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From a code perspective, we can make them InFresh, it requires one more special case (in order not to send an IncomingMsg)
Done in 110f0d3, doesn't look complex.
What does make a difference is whether we introduce 1ms of delay
I'm going to profile that, but anyway, there's a new SQL statement selecting hidden messages (actually reactions), so i'm afraid it will always add some delay (NB: message::markseen_msgs()
is a no-op if msg_ids
is empty, so it's not a source of the problem). In #6213 (comment) you suggested a solution:
This can be fixed by writing
WHERE state=?
here and replacingMessageState::InFresh, MessageState::InSeen
withMessageState::InNoticed
below.
but i don't understand how it works -- just because you simplify the SQL statement (which is unlikely) or because you don't select other unnecessary messages and don't pass them to message::markseen_msgs()
? But there should be no other hidden InFresh
messages currently (NB: Only looking at current receive_imf.rs
, i'm not sure they were never created in the past).
We shouldn't do database migrations "just in case", since there is always some chance that they create problems
See "NB" above -- if hidden InFresh
messages were created in the past, they will be selected and passed to message::markseen_msgs()
which we should avoid. So, probably this db migration will be a no-op, but as we can't be sure (we can't inspect all the previously released versions, really), better to add it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but i don't understand how it works -- just because you simplify the SQL statement (which is unlikely) or because you don't select other unnecessary messages and don't pass them to
message::markseen_msgs()
?
I don't know, my guess would be that it's because the simplification in the SQL statement enabled some optimization.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe it works because we have msgs_index7 ON msgs (state, hidden, chat_id)
. If so, i should replace state>=? AND state<?
with an OR
condition containing just InFresh
and InNoticed
.
EDIT: Maybe just with state=<InFresh>
because InNoticed
reactions are normally a result of mark_old_messages_as_noticed()
which is called from imap
when new messages arrive, but that doesn't need synchronisation using the \Seen
flag because that happens on all devices anyway.
EDIT: This will lead to not all reactions marked as seen on IMAP, but i think it's ok, better to optimise here.
BTW, this should be changed now: deltachat-core-rust/src/chat.rs Lines 1411 to 1413 in e121fc1
We have hidden messages now, but they shouldn't affect how other messages are ordered, so |
This is a code path for protection messages. Protection messages should be sorted under any hidden messages, incl. reactions, and we already have outgoing reactions as hidden messages. The comment just says why we don't add As for the |
110f0d3
to
3e8b050
Compare
src/chat.rs
Outdated
let mut stmt = conn.prepare( | ||
"SELECT id FROM msgs | ||
WHERE state=? AND hidden=1 AND chat_id=? | ||
ORDER BY id DESC LIMIT 100", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now this statement should be fast as we have msgs_index7 ON msgs (state, hidden, chat_id)
, but maybe we want LIMIT 1
here because the effect is the same anyway. This is what i don't understand:
- Suppose one device received reaction A.
- And another device -- A and B.
- The user clicks on "Mark as read" on the first device.
- On the second device
MsgsNoticed
is emitted which results in removing a notification for reaction B. - Then the user has a chance to see the notification for reaction B on the first device only.
CC @r10s
Seems the same problem exists for "usual" messages because MsgsNoticed
only contains ChatId
. Didn't know about that. I hope this is just my misunderstanding.
EDIT: But if i'm right and there were no user complaints for years, this isn't a big problem and may be solved later. I changed the LIMIT in the SQL to 1, this doesn't really matter. Some reactions will be marked as seen, some won't, this isn't important.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you measure it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This SELECT is already fast enough, but i added another commit to optimize the UPDATE statement. On my machine for some random chat now it takes 59.086µs vs 41.202µs w/o this PR, so i think the degradation is acceptable. The whole function took 231.989µs. So, even though the hidden
value set is [0, 1]
, it must be checked explicitly in statements to employ the corresponding db index.
3e8b050
to
4e257c5
Compare
Removed the unnecessary db migration because even if InFresh hidden messages exist currently, now they might affect the logic only once resulting in marking the last such message as |
4e257c5
to
27a1244
Compare
// `mark_old_messages_as_noticed()` which happens on all devices anyway. Also we limit | ||
// this to one message because the effect is the same anyway. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would have assumed the previous limit of 100 to be more efficient, because all reactions need to be marked as seen eventually, and it's more efficient to mark all of them at once (although I do like having a limit at all, because otherwise the UI may become unresponsive when too many messages are marked as seen at once)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need to mark all reactions as seen on IMAP, we only want to trigger MsgsNoticed
on other devices, so we only select the last hidden message (reaction) and check if it's InFresh
. Anyway we don't send MDNs for reactions. If a reactions is already InNoticed
or InSeen
, it normally has the same state on other devices, so there's nothing to do, that's why i added filter()
below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need to mark all reactions as seen on IMAP
It's true that we don't need to, but this PR here as-is will mark all reactions as seen eventually after the user opened the chat repeatedly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, i added LIMIT 1
, but i select InFresh
, InNoticed
and InSeen
hidden messages (reactions) and then check that the selected reaction is InFresh
(the filter()
call below).
For uniformity it would be better to mark all reactions as seen eventually, but it's an extra work on IMAP that i suggest to avoid at least for now.
src/chat.rs
Outdated
AND hidden=0 | ||
AND chat_id=?;", | ||
(MessageState::InNoticed, MessageState::InFresh, chat_id), | ||
) | ||
.await? | ||
== 0 | ||
{ | ||
return Ok(()); | ||
} else { | ||
let conn_fn = |conn: &mut rusqlite::Connection| { | ||
// This is to trigger emitting `MsgsNoticed` on other devices when reactions are noticed | ||
// locally. We filter out `InNoticed` messages because they are normally a result of | ||
// `mark_old_messages_as_noticed()` which happens on all devices anyway. Also we limit | ||
// this to one message because the effect is the same anyway. | ||
// | ||
// Even if `message::markseen_msgs()` fails then, in the worst case other devices won't | ||
// emit `MsgsNoticed` and app notifications won't be removed. The bigger problem is that | ||
// another device may have more reactions received and not yet seen notifications are | ||
// removed from it, but the same problem already exists for "usual" messages, so let's | ||
// not solve it for now. | ||
let mut stmt = conn.prepare( | ||
"SELECT id, state FROM msgs | ||
WHERE (state=? OR state=? OR state=?) | ||
AND hidden=1 | ||
AND chat_id=? | ||
ORDER BY id DESC LIMIT 1", | ||
)?; | ||
let id_to_markseen = stmt | ||
.query_row( | ||
( | ||
MessageState::InFresh, | ||
MessageState::InNoticed, | ||
MessageState::InSeen, | ||
chat_id, | ||
), | ||
|row| { | ||
let id: MsgId = row.get(0)?; | ||
let state: MessageState = row.get(1)?; | ||
Ok((id, state)) | ||
}, | ||
) | ||
.optional()? | ||
.filter(|&(_, state)| state == MessageState::InFresh) | ||
.map(|(id, _)| id); | ||
|
||
// NB: Enumerate `hidden` values to employ `msgs_index7`. | ||
let nr_msgs_noticed = conn.execute( | ||
"UPDATE msgs | ||
SET state=? | ||
WHERE state=? | ||
AND (hidden=0 OR hidden=1) | ||
AND chat_id=?", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why mark hidden messages as Noticed here when they will be marked as InSeen anyway?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FTR: I only added OR hidden=1
to this SQL UPDATE
. Only the last hidden message needs to be marked as InSeen
, the effect on other devices doesn't depend on number of hidden messages marked as seen. I added LIMIT 1
to the statement above just to optimise, w/o limit it won't scale and also we don't want to mark old hidden messages as seen because we want to only trigger MsgsNoticed
on other devices if we have new fresh messages. So, anyway all hidden messages can't be marked as seen on IMAP, even with LIMIT
and in several iterations to preserve UI responsiveness
27a1244
to
92e8898
Compare
@Hocuri An alternative to |
6c90599
to
3aa1e03
Compare
Currently (without this PR) reactions are never marked as seen on IMAP, right? |
Yes
The same is true for |
3aa1e03
to
00122b8
Compare
00122b8
to
e71b17c
Compare
I split this into two commit for easier reviewing at least. |
The problem is that |
After digging through the PR for a bit more, I now understand the new logic to be:
However, it took me quite long to understand this logic even though I know the surrounding code quite good right now, so, for someone who doesn't know the surrounding code it will be super hard. Moreover, the increased complexity doesn't make everything perfect, e.g. consider the following scenario:
So, in order to a) reduce the complexity and b) fix this problem and c) slightly reduce the performance of
|
Good catch, but this is a bug that already exists in the current code for "usual" messages because
The first commit doesn't add any complexity, it rather unifies things for all reactions, so let's talk about
Probably you want to say "slightly improve". Yes, if incoming reactions are added as But despite of the all advantages of reactions being added as
Anyway |
Before, received reactions went to the trash chat. Make them hidden chat messages as already done for sent reactions, just for unification. Incoming received reactions remain `InSeen`, so no changes are needed to `chat::marknoticed_chat()` et al.
When a reaction notification is shown in the UIs, there's an option "Mark Read", but the UIs are unaware of reactions message ids, so the UIs just call `marknoticed_chat()` in this case. We don't want to introduce reactions message ids to the UIs (at least currently), but let's make received reactions `InFresh` so that the existing `\Seen` flag synchronisation mechanism works for them, and mark the last fresh hidden incoming message (reaction) in the chat as seen in `chat::marknoticed_chat()` to trigger emitting `MsgsNoticed` on other devices. There's a problem though that another device may have more reactions received and not yet seen notifications are removed from it when handling `MsgsNoticed`, but the same problem already exists for "usual" messages, so let's not solve it for now.
e71b17c
to
1076f37
Compare
See commit messages.
Close #6210
For
DC_CHAT_ID_ARCHIVED_LINK
this does nothing, marking all reactions as seen in the archive doesn't seem useful.mark_old_messages_as_noticed()
only marks reactions asInNoticed
, this function is called fromimap
when new messages arrive, so it is anyway called on all devices, no synchronisation is needed.DONE:
marknoticed_chat()
.