Hebrew (all RTL?) Language Error #5834

spacemule · 2021-12-13T01:03:41Z

Expected behavior

Emails should sync and appear.

Actual behavior

Syncing is stuck indefinitely due to emails in my inbox that contain Hebrew. Folders without Hebrew sync fine.

Running occ mail:account:sync provides the following output:

In ImapToDbSynchronizer.php line 249:
                                                                                                                         
  Sync failed for 5:INBOX: mb_convert_encoding(): Argument #3 ($from_encoding) contains invalid encoding "iso-8859-8-i"  
                                                                                                                         

In String.php line 160:
                                                                                                
  mb_convert_encoding(): Argument #3 ($from_encoding) contains invalid encoding "iso-8859-8-i"

Mail app

Mail app version: (see apps admin page, e.g. 0.5.3)
1.11.1

Mailserver or service: (e.g. Outlook, Yahoo, Gmail, Exchange,...)
Mailbox.org

Server configuration

Operating system: (e.g. Debian 8)
ppenSUSE MicroOS

Web server: (e.g. Apache, Nginx,...)
Nginx

Database: (e.g. MariaDB, SQLite or PostgreSQL)
MariaDB

PHP version: (e.g. 7.0)
Whatever is in the docker image nextcloud:fpm

Nextcloud Version: (see admin page, e.g. 13.0.2)
23.0.0.10

Client configuration

Browser: (e.g. Firefox 48)
Firefox 94.0.2

Operating system: (e.g. Arch Linux)
openSUSE Tumbleweed

Thanks!!

The text was updated successfully, but these errors were encountered:

ChristophWurst · 2021-12-15T15:57:44Z

I find similar logs in other tickets. #4925 (comment) for example.

xinstein · 2021-12-16T09:41:02Z

I encountered this using Chinese, GB1803 0,
Preventing me from using the mail app entirely

miaulalala · 2021-12-20T14:04:38Z

Looks like \Horde_Mime_Part::getCharset is returning values that are not always compatible with PHP charsets, as it uses the Mime Header content type label to extract this information from an email. iso-8859-8-i is invalid in PHP, but iso-8859-8 would be a valid charset.

@xinstein can you post the complete error message please?

miaulalala · 2021-12-20T14:32:29Z

@spacemule would you be able to send me an email as .eml attachment to [email protected] with hebrew content? I am currenly testing a fix, but I'm not sure just copy pasting a hebrew lorem ipsum is enough.

spacemule · 2021-12-21T20:33:03Z

I sent three as an attachment. They're from a domain that matches my username here in case they end up in spam.

Thanks!

miaulalala · 2021-12-22T16:41:30Z

@spacemule Thanks for the emails! I tested my changes with them and it looks ok.

miaulalala · 2021-12-22T16:41:43Z

Who here is able to check out this branch here: #5882 and test the changes, too?

miaulalala · 2022-01-03T08:22:59Z

@spacemule if you're not busy I would apprechiate if you could test these changes for me 🙏

@xinstein you too if possible ☺️

miaulalala · 2022-01-04T10:48:42Z

@xinstein can you please send me your nextcloud.log to [email protected]? You can find it in your data folder.

spacemule · 2022-01-08T19:05:44Z

@miaulalala I'm working on this now. Unfortunately, I'm not much of a web developer, and I cannot find the documentation to install this branch manually. I'm assuming it's occ app:install, but is it expecting a tarball or a filesystem directory?

xinstein · 2022-01-09T03:36:15Z

@spacemule simply git-cloning into the custom_apps folder will do.

Also, I might have mis-stated my situation.

The error message I encountered is the same as in the first post ( mb_convert_encoding and like ).
The error messages I posted later are raised when I test #5882 (1.20alpha1), after I git clone and enable, it crashes itself as well as my nextcloud settings page.
I probably should have pasted them in that pr.

miaulalala · 2022-01-10T12:56:10Z

@spacemule xinstein is right. Do you know how to git clone the app? Then you can use git to checkout the branch fix/detect-char-encoding

xinstein · 2022-01-10T13:52:16Z

@spacemule You might also need to chown the cloned folder to the user running the nextcloud instance

doc75 · 2022-02-01T21:34:32Z

@miaulalala I just installed the patch #5882 to see if the error is disappearing on my side (was in the cron job for me). I'll let you know after a while to be sure it does not come back.

doc75 · 2022-02-02T19:47:13Z

@miaulalala I cannot see anymore the error after applying the patch. And it appears again after removing the patch.

But when the patch is active I still have sync error raised (not sure if this is related to the patch or not, but the stack does not mention the modified file by the patch). I am attaching the log in case it is relevant:

Log of error raised (patch active)

{
  "reqId": "j6fDPFnTgXCL2H4pYMt8",
  "level": 3,
  "time": "2022-02-02T15:25:06+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "mail",
  "method": "",
  "url": "--",
  "message": "Cron mail sync failed for account {accountId}",
  "userAgent": "--",
  "version": "22.2.3.0",
  "exception": {
    "Exception": "OCA\\Mail\\Exception\\ServiceException",
    "Message": "Sync failed for 3:INBOX: syntax error, unexpected token \"use\", expecting \",\" or \";\"",
    "Code": 0,
    "Trace": [
      {
        "file": "/var/www/html/custom_apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php",
        "line": 135,
        "function": "sync",
        "class": "OCA\\Mail\\Service\\Sync\\ImapToDbSynchronizer",
        "type": "->",
        "args": [
          {
            "__class__": "OCA\\Mail\\Account"
          },
          {
            "id": 43,
            "__class__": "OCA\\Mail\\Db\\Mailbox"
          },
          {
            "__class__": "OC\\AppFramework\\ScopedPsrLogger"
          },
          42,
          null,
          false,
          true
        ]
      },
      {
        "file": "/var/www/html/custom_apps/mail/lib/BackgroundJob/SyncJob.php",
        "line": 109,
        "function": "syncAccount",
        "class": "OCA\\Mail\\Service\\Sync\\ImapToDbSynchronizer",
        "type": "->",
        "args": [
          {
            "__class__": "OCA\\Mail\\Account"
          },
          {
            "__class__": "OC\\AppFramework\\ScopedPsrLogger"
          }
        ]
      },
      {
        "file": "/var/www/html/lib/public/BackgroundJob/Job.php",
        "line": 79,
        "function": "run",
        "class": "OCA\\Mail\\BackgroundJob\\SyncJob",
        "type": "->",
        "args": [
          {
            "accountId": 3
          }
        ]
      },
      {
        "file": "/var/www/html/lib/public/BackgroundJob/TimedJob.php",
        "line": 63,
        "function": "execute",
        "class": "OCP\\BackgroundJob\\Job",
        "type": "->",
        "args": [
          {
            "__class__": "OC\\BackgroundJob\\JobList"
          },
          {
            "__class__": "OC\\Log"
          }
        ]
      },
      {
        "file": "/var/www/html/cron.php",
        "line": 127,
        "function": "execute",
        "class": "OCP\\BackgroundJob\\TimedJob",
        "type": "->",
        "args": [
          {
            "__class__": "OC\\BackgroundJob\\JobList"
          },
          {
            "__class__": "OC\\Log"
          }
        ]
      }
    ],
    "File": "/var/www/html/custom_apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php",
    "Line": 249,
    "Previous": {
      "Exception": "ParseError",
      "Message": "syntax error, unexpected token \"use\", expecting \",\" or \";\"",
      "Code": 0,
      "Trace": [
        {
          "file": "/var/www/html/lib/composer/composer/ClassLoader.php",
          "line": 346,
          "function": "Composer\\Autoload\\includeFile",
          "args": [
            "/var/www/html/custom_apps/mail/lib//Model/IMAPMessage.php"
          ]
        },
        {
          "file": "/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php",
          "line": 249,
          "function": "loadClass",
          "class": "Composer\\Autoload\\ClassLoader",
          "type": "->",
          "args": [
            "OCA\\Mail\\Model\\IMAPMessage"
          ]
        },
        {
          "function": "OCA\\Mail\\IMAP\\{closure}",
          "class": "OCA\\Mail\\IMAP\\MessageMapper",
          "type": "->",
          "args": [
            "*** sensitive parameters replaced ***"
          ]
        },
        {
          "file": "/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php",
          "line": 256,
          "function": "array_map",
          "args": [
            {
              "__class__": "Closure"
            },
            [
              "*** sensitive parameter replaced ***",
              {
                "__class__": "Horde_Imap_Client_Data_Fetch"
              },
              {
                "__class__": "Horde_Imap_Client_Data_Fetch"
              }
            ]
          ]
        },
        {
          "file": "/var/www/html/custom_apps/mail/lib/IMAP/Sync/Synchronizer.php",
          "line": 99,
          "function": "findByIds",
          "class": "OCA\\Mail\\IMAP\\MessageMapper",
          "type": "->",
          "args": [
            {
              "cacheFields": [
                "HICenv",
                "HICflags",
                "HIChdrs",
                "HICdate",
                "HICsize",
                "And 1 more entries, set log level to debug to see all entries"
              ],
              "changed": true,
              "statuscache": true,
              "__class__": "Horde_Imap_Client_Socket"
            },
            "INBOX",
            [
              77205,
              77206,
              77208
            ]
          ]
        },
        {
          "file": "/var/www/html/custom_apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php",
          "line": 354,
          "function": "sync",
          "class": "OCA\\Mail\\IMAP\\Sync\\Synchronizer",
          "type": "->",
          "args": [
            {
              "cacheFields": [
                "HICenv",
                "HICflags",
                "HIChdrs",
                "HICdate",
                "HICsize",
                "And 1 more entries, set log level to debug to see all entries"
              ],
              "changed": true,
              "statuscache": true,
              "__class__": "Horde_Imap_Client_Socket"
            },
            {
              "__class__": "OCA\\Mail\\IMAP\\Sync\\Request"
            },
            8
          ]
        },
        {
          "file": "/var/www/html/custom_apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php",
          "line": 230,
          "function": "runPartialSync",
          "class": "OCA\\Mail\\Service\\Sync\\ImapToDbSynchronizer",
          "type": "->",
          "args": [
            {
              "__class__": "OCA\\Mail\\Account"
            },
            {
              "id": 43,
              "__class__": "OCA\\Mail\\Db\\Mailbox"
            },
            {
              "__class__": "OC\\AppFramework\\ScopedPsrLogger"
            },
            42,
            null
          ]
        },
        {
          "file": "/var/www/html/custom_apps/mail/lib/Service/Sync/ImapToDbSynchronizer.php",
          "line": 135,
          "function": "sync",
          "class": "OCA\\Mail\\Service\\Sync\\ImapToDbSynchronizer",
          "type": "->",
          "args": [
            {
              "__class__": "OCA\\Mail\\Account"
            },
            {
              "id": 43,
              "__class__": "OCA\\Mail\\Db\\Mailbox"
            },
            {
              "__class__": "OC\\AppFramework\\ScopedPsrLogger"
            },
            42,
            null,
            false,
            true
          ]
        },
        {
          "file": "/var/www/html/custom_apps/mail/lib/BackgroundJob/SyncJob.php",
          "line": 109,
          "function": "syncAccount",
          "class": "OCA\\Mail\\Service\\Sync\\ImapToDbSynchronizer",
          "type": "->",
          "args": [
            {
              "__class__": "OCA\\Mail\\Account"
            },
            {
              "__class__": "OC\\AppFramework\\ScopedPsrLogger"
            }
          ]
        },
        {
          "file": "/var/www/html/lib/public/BackgroundJob/Job.php",
          "line": 79,
          "function": "run",
          "class": "OCA\\Mail\\BackgroundJob\\SyncJob",
          "type": "->",
          "args": [
            {
              "accountId": 3
            }
          ]
        },
        {
          "file": "/var/www/html/lib/public/BackgroundJob/TimedJob.php",
          "line": 63,
          "function": "execute",
          "class": "OCP\\BackgroundJob\\Job",
          "type": "->",
          "args": [
            {
              "__class__": "OC\\BackgroundJob\\JobList"
            },
            {
              "__class__": "OC\\Log"
            }
          ]
        },
        {
          "file": "/var/www/html/cron.php",
          "line": 127,
          "function": "execute",
          "class": "OCP\\BackgroundJob\\TimedJob",
          "type": "->",
          "args": [
            {
              "__class__": "OC\\BackgroundJob\\JobList"
            },
            {
              "__class__": "OC\\Log"
            }
          ]
        }
      ],
      "File": "/var/www/html/custom_apps/mail/lib/Model/IMAPMessage.php",
      "Line": 57
    },
    "CustomMessage": "Cron mail sync failed for account {accountId}"
  }
}

miaulalala · 2022-02-07T14:37:07Z

Thanks everyone for testing this!

@doc75 I'm not sure where that error is coming from, can you have a look at your /lib/Service/Sync/ImapToDbSynchronizer.php file and possibly copy&paste the sync function here? I have looked through the code and it's not apparent to me why this fails.

doc75 · 2022-02-08T07:44:03Z

Here is the content of the file ImapToDbSynchronizer.php:

        public function sync(Account $account,
                                                 Mailbox $mailbox,
                                                 LoggerInterface $logger,
                                                 int $criteria = Horde_Imap_Client::SYNC_NEWMSGSUIDS | Horde_Imap_Client::SYNC_FLAGSUIDS | Horde_Imap_Client::SYNC_VANISHEDUIDS,
                                                 array $knownUids = null,
                                                 bool $force = false,
                                                 bool $batchSync = false): void {
                if ($mailbox->getSelectable() === false) {
                        return;
                }

                if ($force || ($criteria & Horde_Imap_Client::SYNC_NEWMSGSUIDS)) {
                        $logger->debug("Locking mailbox " . $mailbox->getId() . " for new messages sync");
                        $this->mailboxMapper->lockForNewSync($mailbox);
                }
                if ($force || ($criteria & Horde_Imap_Client::SYNC_FLAGSUIDS)) {
                        $logger->debug("Locking mailbox " . $mailbox->getId() . " for changed messages sync");
                        $this->mailboxMapper->lockForChangeSync($mailbox);
                }
                if ($force || ($criteria & Horde_Imap_Client::SYNC_VANISHEDUIDS)) {
                        $logger->debug("Locking mailbox " . $mailbox->getId() . " for vanished messages sync");
                        $this->mailboxMapper->lockForVanishedSync($mailbox);
                }

                try {
                        if ($force
                                || $mailbox->getSyncNewToken() === null
                                || $mailbox->getSyncChangedToken() === null
                                || $mailbox->getSyncVanishedToken() === null) {
                                $logger->debug("Running initial sync for " . $mailbox->getId());
                                $this->runInitialSync($account, $mailbox, $logger);
                        } else {
                                try {
                                        $logger->debug("Running partial sync for " . $mailbox->getId());
                                        $this->runPartialSync($account, $mailbox, $logger, $criteria, $knownUids);
                                } catch (UidValidityChangedException $e) {
                                        $logger->warning('Mailbox UID validity changed. Wiping cache and performing full sync for ' . $mailbox->getId());
                                        $this->resetCache($account, $mailbox);
                                        $logger->debug("Running initial sync for " . $mailbox->getId() . " after cache reset");
                                        $this->runInitialSync($account, $mailbox, $logger);
                                } catch (MailboxDoesNotSupportModSequencesException $e) {
                                        $logger->warning('Mailbox does not support mod-sequences error occured. Wiping cache and performing full sync for ' . $mailbox->getId(), [
                                                'exception' => $e,
                                        ]);
                                        $this->resetCache($account, $mailbox);
                                        $logger->debug("Running initial sync for " . $mailbox->getId() . " after cache reset - no mod-sequences error");
                                        $this->runInitialSync($account, $mailbox, $logger);
                                }
                        }
                } catch (ServiceException $e) {
                        // Just rethrow, don't wrap into another exception
                        throw $e;
                } catch (Throwable $e) {
                        throw new ServiceException('Sync failed for ' . $account->getId() . ':' . $mailbox->getName() . ': ' . $e->getMessage(), 0, $e);
                } finally {
                        if ($force || ($criteria & Horde_Imap_Client::SYNC_VANISHEDUIDS)) {
                                $logger->debug("Unlocking mailbox " . $mailbox->getId() . " from vanished messages sync");
                                $this->mailboxMapper->unlockFromVanishedSync($mailbox);
                        }
                        if ($force || ($criteria & Horde_Imap_Client::SYNC_FLAGSUIDS)) {
                                $logger->debug("Unlocking mailbox " . $mailbox->getId() . " from changed messages sync");
                                $this->mailboxMapper->unlockFromChangedSync($mailbox);
                        }
                        if ($force || ($criteria & Horde_Imap_Client::SYNC_NEWMSGSUIDS)) {
                                $logger->debug("Unlocking mailbox " . $mailbox->getId() . " from new messages sync");
                                $this->mailboxMapper->unlockFromNewSync($mailbox);
                        }
                }

                if (!$batchSync) {
                        $this->dispatcher->dispatchTyped(
                                new SynchronizationEvent(
                                        $account,
                                        $this->logger
                                )
                        );
                }
        }

miaulalala · 2023-02-13T15:46:11Z

@spacemule can you test the newest Mail (2.2.)? I tested today and this is what emails look like for me now:

spacemule · 2023-02-15T17:25:07Z

I don't have a nextcloud instance anymore, but I'd be happy to forward some emails.

…

On February 13, 2023 5:46:23 PM GMT+02:00, Anna ***@***.***> wrote: @spacemule can you test the newest Mail (2.2.)? I tested today and this is what emails look like for me now: ![image](https://user-images.githubusercontent.com/7427347/218504538-eb3a02fe-46dd-48de-8880-9ba01c344302.png) -- Reply to this email directly or view it on GitHub: #5834 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

miaulalala · 2023-02-16T12:31:52Z

Ok I'mm going to close this as fixed then. Feel free to open another ticket if you run into more issues.

spacemule added 0. to triage bug labels Dec 13, 2021

miaulalala mentioned this issue Dec 20, 2021

Loading messages fails on some folders with mb_convert_encoding() error #5845

Closed

miaulalala mentioned this issue Dec 20, 2021

Change how charsets are detected for emails #5882

Merged

This comment has been minimized.

Sign in to view

miaulalala added 2. developing and removed 0. to triage labels Jan 4, 2022

miaulalala self-assigned this Jan 4, 2022

This comment has been minimized.

Sign in to view

adarnimrod mentioned this issue Mar 3, 2022

mb_convert_encoding does not support windows-1250. #1037

Open

ChristophWurst added this to 💌 📅 👥 Groupware team Mar 24, 2022

ChristophWurst moved this to 📄 To do (5-20 entries) in 💌 📅 👥 Groupware team Mar 24, 2022

ChristophWurst moved this from 📄 To do (5-20 entries) to 🏗️ In progress in 💌 📅 👥 Groupware team Mar 24, 2022

miaulalala mentioned this issue May 11, 2022

Mails with i.e. greek characters not correctly displayed #2921

Closed

miaulalala mentioned this issue Aug 16, 2022

Add imip processing in the background #6812

Merged

1 task

ChristophWurst added 1. to develop and removed 2. developing labels Sep 5, 2022

ChristophWurst unassigned miaulalala Sep 5, 2022

miaulalala self-assigned this Feb 13, 2023

miaulalala closed this as completed Feb 16, 2023

github-project-automation bot moved this from 🏗️ In progress to ☑️ Done in 💌 📅 👥 Groupware team Feb 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hebrew (all RTL?) Language Error #5834

Hebrew (all RTL?) Language Error #5834

spacemule commented Dec 13, 2021

ChristophWurst commented Dec 15, 2021

xinstein commented Dec 16, 2021

miaulalala commented Dec 20, 2021

miaulalala commented Dec 20, 2021

spacemule commented Dec 21, 2021

miaulalala commented Dec 22, 2021

miaulalala commented Dec 22, 2021

miaulalala commented Jan 3, 2022

This comment has been minimized.

miaulalala commented Jan 4, 2022

This comment has been minimized.

This comment has been minimized.

spacemule commented Jan 8, 2022

xinstein commented Jan 9, 2022 •

edited

Loading

miaulalala commented Jan 10, 2022

xinstein commented Jan 10, 2022

doc75 commented Feb 1, 2022

doc75 commented Feb 2, 2022

miaulalala commented Feb 7, 2022

doc75 commented Feb 8, 2022

miaulalala commented Feb 13, 2023

spacemule commented Feb 15, 2023 via email

miaulalala commented Feb 16, 2023

Hebrew (all RTL?) Language Error #5834

Hebrew (all RTL?) Language Error #5834

Comments

spacemule commented Dec 13, 2021

Expected behavior

Actual behavior

Mail app

Server configuration

Client configuration

ChristophWurst commented Dec 15, 2021

xinstein commented Dec 16, 2021

miaulalala commented Dec 20, 2021

miaulalala commented Dec 20, 2021

spacemule commented Dec 21, 2021

miaulalala commented Dec 22, 2021

miaulalala commented Dec 22, 2021

miaulalala commented Jan 3, 2022

This comment has been minimized.

miaulalala commented Jan 4, 2022

This comment has been minimized.

This comment has been minimized.

spacemule commented Jan 8, 2022

xinstein commented Jan 9, 2022 • edited Loading

miaulalala commented Jan 10, 2022

xinstein commented Jan 10, 2022

doc75 commented Feb 1, 2022

doc75 commented Feb 2, 2022

miaulalala commented Feb 7, 2022

doc75 commented Feb 8, 2022

miaulalala commented Feb 13, 2023

spacemule commented Feb 15, 2023 via email

miaulalala commented Feb 16, 2023

xinstein commented Jan 9, 2022 •

edited

Loading