"Message does not exist"

[RobertKirchner9]

This is not an existential threat to my well-being, just a mild annoyance, but ElectronMail, since the last update, has been intermittently flashing a red warning, "Message does not exist". The warning appears at random moments and seems to have no consequences.

[vladimiry]

If you have the local store option enabled for the mail account, it might potentially be that Proton has changed some internals that the app's "local store" feature didn't take into the account doing mails syncing. I didn't notice the listed warning at my side though yet.

[ylluminarious]

I've noticed this problem as well, but it seems to only be on a single account and not others.

I'd be happy to help diagnose this further if necessary @vladimiry

[ylluminarious]

@vladimiry I noticed this error occurring again on the same account, and I simultaneously noticed the following messages in log.log:

[2024-05-04 22:29:04.508] [error] src/electron-preload/webview/lib/const.ts [preload: webview/primary] src/electron-preload/webview/primary/api/db-patch/index.ts buildDbPatchRetryPipeline {
  name: 'StatusCodeError',
  message: 'Message does not exist',
  status: 422,
  responseUrl: 'https://mail-api.proton.me/mail/v4/messages/<wiped-out>',
  responseStatusText: '',
  dataCode: 2501,
  dataError: 'Message does not exist'
}
[2024-05-04 22:29:04.509] [error] src/electron-preload/webview/lib/const.ts [preload: webview/primary] src/electron-preload/webview/primary/api/index.ts [pubsub-to-rpc-api] [provider] notification.error {"name":"buildDbPatch","channel":"electron-mail:webview-api:primary","payloadType":"request","uid":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"} {
  name: 'StatusCodeError',
  message: 'Message does not exist',
  status: 422,
  responseUrl: 'https://mail-api.proton.me/mail/v4/messages/<wiped-out>',
  responseStatusText: '',
  dataCode: 2501,
  dataError: 'Message does not exist'
}

Does that help shed any light on what the root cause of this issue is?

[ylluminarious]

@vladimiry Another thing I noticed when re-examining this issue is that all of the messages in the local store for this account are extremely out-of-date, when compared to the current messages in the online store. The newest messages in the local store are from about 6 months ago, which is when the latest ElectronMail update came out. It seems that no new messages have been getting saved in the local store since that time...

Perhaps this issue could be mitigated if we could somehow delete the entire local store for this specific account and then trigger a totally fresh re-sync of the local store from the online store?

Is it possible to surgically purge the local store of a singular account without deleting the corresponding messages from the actual online account?

[vladimiry]

The app should re-sync the messages, regardless how old the local data is, and I believe it was trying to do so, but faced the Message does not exist error for some reason. Apparently, some time ago, Proton changed something that the app didn't take into the account. This error is not added to the "recoverable" list in the app, and so messages sync process never completes (and starts again each time). Technically, the app is not supposed to download non-existing messages, and it was so before.

The workaround is adding this error to the "recoverable" list for the sync process, so the error gets skipped and process continuous. This would be a dirty, but probably valid hack for this case at this time. But a proper option is to make the app not to download such "shadow messages", which would be hard to do not knowing what exactly was changed by Proton.

Perhaps this issue could be mitigated if we could somehow delete the entire local store for this specific account and then trigger a totally fresh re-sync of the local store from the online store?

Worth trying. The "local store" is a per-account feature. So you could disable this toggle for the problematic account, restart the app (the "local store"-related data will be removed for this account at the app startup), and enable it again.

Adding "provider change" label as this error didn't occur before and only started to happen with recent app update. The app itself didn't change the related/syncing logic.

[ylluminarious]

@vladimiry Thank you very much for clarifying the nature of this bug. That makes a lot more sense now.

Regarding your suggested workaround:

... you could disable this toggle for the problematic account, restart the app (the "local store"-related data will be removed for this account at the app startup), and enable it again.

I went ahead and tried what you suggested here, and I can confirm that this indeed works. It re-created my local store for the problematic account and now I see that the local store is matching the online store.

Thanks so much for the suggestion. It seems this workaround is sufficient for my needs at present until a more permanent fix can be implemented here.

[babebort]

@vladimiry Thank you very much for clarifying the nature of this bug. That makes a lot more sense now.

Regarding your suggested workaround:

... you could disable this toggle for the problematic account, restart the app (the "local store"-related data will be removed for this account at the app startup), and enable it again.

I went ahead and tried what you suggested here, and I can confirm that this indeed works. It re-created my local store for the problematic account and now I see that the local store is matching the online store.

Thanks so much for the suggestion. It seems this workaround is sufficient for my needs at present until a more permanent fix can be implemented here.

Should I just switch off and toggle on the local store setting?

[vladimiry]

Should I just switch off and toggle on the local store setting?

If the error is persistent, not a one-off case, then this would be the way. But I remember you got many messages, so the long "bootstrap/initial data load" will be triggered again. So this is up to you, either completely stopping using "local store" or re-creating it (by disabling the toggle, restarting the app, and enabling it back).

I don't think I will be ready to put effort into this issue, at least until I start facing the issue in my own.

[babebort]

May I need to delete database.bin ?

Should I just switch off and toggle on the local store setting?

If the error is persistent, not a one-off case, then this would be the way. But I remember you got many messages, so the long "bootstrap/initial data load" will be triggered again. So this is up to you, either completely stopping using "local store" or re-creating it (by disabling the toggle, restarting the app, and enabling it back).

I don't think I will be ready to put effort into this issue, at least until I start facing the issue in my own.

[vladimiry]

May I need to delete database.bin ?

It keeps the local data for all the accounts. To trigger full local store re-creating for a specific account, it should be enough to disable the "local store" thing for that account, restarting the app and enabling it again.

But I'd like you not to do that for a while, so you could test a fix. See details below.

There are two parts of the "local store" data filling. The first is a bootstrap/initial data fetch, and the second is a partial data updates according to the changes. The seconds part is a process of handling Proton's events queue. The queue represents all the changes ever happened with the account data.

So there is a valid case when the app detected a bunch of queued events to process. But some message removing might happen during the detected queued events processing by the app. And if the removed message also lies in the "being processed queued events" list, the original "422 / Message does not exist" error is expected to happen.

So a simple fix would be in ignoring that "422 / Message does not exist" error. There is no risk of data loss in the "local store" since ever removed message won't get back with the same ID.

To recap, I'm going to enable the described fix and share a new build here. This expected to happen later today or tomorrow.

[vladimiry]

@babebort can you locate the log.log file in the settings folder and share some 422 errors here as listed in #661 (comment)? So I could extend the ignored errors whitelist.

So a simple fix would be in ignoring that "422 / Message does not exist" error. There is no risk of data loss in the "local store" since ever removed message won't get back with the same ID.

Actually, the described logic is already implemented in the app. But the combination of "status: 422" and "dataCode: 2501", as listed in #661 (comment), is not there yet.

[vladimiry]

But I'd like you not to do that for a while, so you could test a fix.

I'm going to enable the described fix and share a new build here.

If someone needs it, the build with the fix candidate is available here.

Not sure if the build handles all the reported cases, as the log.log error records of the recent report have not been provided.

[babebort]

@babebort can you locate the log.log file in the settings folder and share some 422 errors here as listed in #661 (comment)? So I could extend the ignored errors whitelist.

So a simple fix would be in ignoring that "422 / Message does not exist" error. There is no risk of data loss in the "local store" since ever removed message won't get back with the same ID.

Actually, the described logic is already implemented in the app. But the combination of "status: 422" and "dataCode: 2501", as listed in #661 (comment), is not there yet.

log.log

Here is it, but also my concern is that keeping signed in is not working anymore, could you take a look it ?

i was using low-latency kernel and after rebooting to generic I faced it!
image

and I don't know how but a few seconds later my sign-in disappeared...

[babebort]

kindly remove my log.log if there is mentioned of some my sensitive data

[babebort]

[vladimiry]

kindly remove my log.log if there is mentioned of some my sensitive data

Thanks for the log data. Actually, I only asked for the issue-related error lines. Sharing the log in full here was your choice. Anyway, there should be no really sensitive information as the app doesn't log such. Besides, the app takes some measurements of wiping some unique data/IDs (you can see some <wiped-out> stuff in the log file, but the wiping is not perfect yet).

Same as posted before in #661 (comment):

  status: 422,
  responseUrl: 'https://mail-api.proton.me/mail/v4/messages/<wiped-out>',
  responseStatusText: '',
  dataCode: 2501,

So I think the build shared in #661 (comment) should be gracefully handling the "Message does not exist" error, if the implemented logic is good enough.

Here is it, but also my concern is that keeping signed in is not working anymore, could you take a look it ?

message: 'Invocation timeout of calling "loadDatabase" method on "electron-mail:ipcMain-api" channel with 300000ms timeout',

According to the logs, the sign-in is ok, but the consequent database loading operation doesn't get completed in 5 minutes (300000ms), which doesn't look normal. You could increase the timeouts.databaseLoading value in the config.json file, like 2x, but I don't think the app would be really usable with such long startup time. I guess you got many email messages in the account and very slow computer. There is no proper way of increasing the "large local data"-related logic performance other than handling #571, which I didn't start working on.

[babebort]

After loading with a low-latency kernel app is signed again...

[babebort]

many email messages in the account and very slow computer

its true