cron runs into several out-of-memory issues after 25 upgrade #7495

blizzz · 2022-09-23T08:20:10Z

⚠️ This issue respects the following points: ⚠️

This is a bug, not a question or a configuration/webserver/proxy issue.
This issue is not already reported on Github (I've searched it).
Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
Nextcloud Server is running on 64bit capable CPU, PHP and OS.
I agree to follow Nextcloud's Code of Conduct.

Bug description

cron.php runs long and eventually dies with "Allowed memory size … exhausted" after upgrade to NC 25 (RC1). This happens a couple time before it settles down and works as expected.

But, what might be broken by the jobs that were terminated?

Steps to reproduce

Run NC 24
Upgrade to NC 25 RC1
Watch the logs

Expected behavior

Background job run nicely and do not cause memory exhaustion failures (with a sensible memory limit set of course – 512 MB in this case).

Installation method

No response

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 22.2.3 to 23.0.1)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

Default user-backend (database)
LDAP/ Active Directory
SSO - SAML
Other

Configuration report

No response

List of activated Apps

Enabled:
  - activity: 2.17.0
  - admin_audit: 1.15.0
  - announcementcenter: 6.4.0-beta.1
  - calendar: 4.0.0-alpha.1
  - calendar_resource_management: 0.3.1
  - circles: 25.0.0-alpha.0
  - cloud_federation_api: 1.8.0
  - collectives: 2.0.0-beta1
  - comments: 1.15.0
  - contacts: 5.0.0-alpha.3
  - contactsinteraction: 1.6.0
  - dashboard: 7.5.0
  - dav: 1.24.0
  - deck: 1.8.0-beta.1
  - droneci_fast_lane: 1.0.2
  - end_to_end_encryption: 1.11.0-beta.1
  - external: 4.1.0
  - federatedfilesharing: 1.15.0
  - federation: 1.15.0
  - files: 1.20.0
  - files_lock: 24.0.1
  - files_pdfviewer: 2.6.0
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - files_videoplayer: 1.14.0
  - firstrunwizard: 2.14.0
  - forms: 3.0.0-beta.1
  - groupfolders: 13.0.0-beta.1
  - guests: 2.3.0
  - integration_discourse: 1.0.4
  - integration_github: 1.0.5
  - integration_mastodon: 1.0.3
  - integration_reddit: 1.0.5
  - integration_twitter: 1.0.3
  - integration_zammad: 1.0.4
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - mail: 2.0.0-beta.7
  - nextcloud_announcements: 1.14.0
  - notes: 4.5.1
  - notifications: 2.13.1
  - notify_push: 0.4.0
  - oauth2: 1.13.0
  - password_policy: 1.15.0
  - photos: 1.7.1
  - privacy: 1.9.0
  - provisioning_api: 1.15.0
  - related_resources: 1.0.0
  - richdocuments: 7.0.0-beta.1
  - settings: 1.7.0
  - sharebymail: 1.15.0
  - spreed: 15.0.0-rc.1
  - support: 1.8.0
  - survey_client: 1.13.0
  - suspicious_login: 4.3.0
  - systemtags: 1.15.0
  - tasks: 0.14.4
  - text: 3.6.0
  - theming: 2.0.0
  - twofactor_backupcodes: 1.14.0
  - twofactor_nextcloud_notification: 3.5.0-beta.1
  - twofactor_totp: 7.0.0
  - twofactor_webauthn: 1.0.0-beta.2
  - updatenotification: 1.15.0
  - user_ldap: 1.15.0
  - user_status: 1.5.0
  - viewer: 1.9.0
  - weather_status: 1.5.0
  - workflowengine: 2.7.0
  - zammad_organisation_management: 1.1.3

Nextcloud Signing status

No response

Nextcloud Logs

{
  "reqId": "haa62QjoTzDjoMYNIS3B",
  "level": 3,
  "time": "2022-09-22T20:25:16+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "PHP",
  "method": "",
  "url": "--",
  "message": "Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) at /var/www/cloud.nextcloud.com/nextcloud/lib/private/Files/Cache/Cache.php#189",
  "userAgent": "--",
  "version": "25.0.0.13",
  "data": {
    "app": "PHP"
  }
}
{
  "reqId": "GIzL5W5DtvfF28BhVlSY",
  "level": 3,
  "time": "2022-09-22T20:33:54+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "PHP",
  "method": "",
  "url": "--",
  "message": "Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) at /var/www/cloud.nextcloud.com/nextcloud/apps/mail/vendor/bytestream/horde-stream/lib/Horde/Stream.
php#402",
  "userAgent": "--",
  "version": "25.0.0.13",
  "data": {
    "app": "PHP"
  }
}
{
  "reqId": "OAv7s2QKDBmji4a7atpL",
  "level": 3,
  "time": "2022-09-22T20:43:33+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "PHP",
  "method": "",
  "url": "--",
  "message": "Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) at /var/www/cloud.nextcloud.com/nextcloud/apps/mail/vendor/bytestream/horde-stream/lib/Horde/Stream.
php#402",
  "userAgent": "--",
  "version": "25.0.0.13",
  "data": {
    "app": "PHP"
  }
}
{
  "reqId": "ctTckN9gOVl7nl5NCFSL",
  "level": 3,
  "time": "2022-09-22T20:52:57+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "PHP",
  "method": "",
  "url": "--",
  "message": "Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) at /var/www/cloud.nextcloud.com/nextcloud/lib/public/AppFramework/Db/Entity.php#260",
  "userAgent": "--",
  "version": "25.0.0.13",
  "data": {
    "app": "PHP"
  }
}
{
  "reqId": "sEIPa3IEcW3xxCnA9904",
  "level": 3,
  "time": "2022-09-22T21:53:15+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "PHP",
  "method": "",
  "url": "--",
  "message": "Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) at /var/www/cloud.nextcloud.com/nextcloud/apps/mail/vendor/bytestream/horde-stream/lib/Horde/Stream.php#402",
  "userAgent": "--",
  "version": "25.0.0.13",
  "data": {
    "app": "PHP"
  }
}
{
  "reqId": "EikaXlu6xkK9ySBmlq0c",
  "level": 3,
  "time": "2022-09-22T22:05:59+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "PHP",
  "method": "",
  "url": "--",
  "message": "Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) at /var/www/cloud.nextcloud.com/nextcloud/apps/mail/vendor/bytestream/horde-stream/lib/Horde/Stream.php#402",
  "userAgent": "--",
  "version": "25.0.0.13",
  "data": {
    "app": "PHP"
  }
}
{
  "reqId": "faU8qCKZ8lBUsWawvuJy",
  "level": 3,
  "time": "2022-09-22T22:07:12+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "PHP",
  "method": "",
  "url": "--",
  "message": "Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) at /var/www/cloud.nextcloud.com/nextcloud/apps/mail/vendor/bytestream/horde-stream/lib/Horde/Stream.php#402",
  "userAgent": "--",
  "version": "25.0.0.13",
  "data": {
    "app": "PHP"
  }
}

Additional info

No response

blizzz · 2022-09-23T08:25:44Z

T'is a bit hard to debug for we don't have the state anymore, and the exhaustion could happen anyplace. For mail appears most often, you might suspect it is on of the jobs happening there.

blizzz · 2022-09-23T08:26:26Z

@ChristophWurst @GretaD for your evaluation ^

szaimen · 2022-10-20T22:02:33Z

Is this still happening?

blizzz · 2022-10-21T07:23:54Z

Cannot say, have no big instance to upgrade.

szaimen · 2022-10-21T07:31:05Z

Ah, I might have misread. So the problem happens directly after the upgrade and then never again?

blizzz · 2022-10-21T07:40:30Z

~~a few times and then not anymore as far is i observed, yes.~~

It actually comes back

{"reqId":"EVqCzeoPNesWJtSbrvR5","level":3,"time":"2022-10-20T21:56:05+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) at /var/www/cloud.nextcloud.com/nextcloud/apps/mail/vendor/bytestream/horde-stream/lib/Horde/Stream.php#402","userAgent":"--","version":"25.0.0.18","data":{"app":"PHP"}}

I recently overread someone talking of memory leaks on Horde. Maybe this is related? cc @miaulalala

blizzz · 2022-10-21T07:43:00Z

had to update my last comment ^

st3iny · 2024-08-13T08:34:14Z

From #6280

[PHP] Error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 8192 bytes) at /var/www/nextcloud/apps/mail/vendor/pear-pear.horde.org/Horde_Stream/Horde/Stream.php#203

at 2022-04-21T12:10:29+00:00

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 8192 bytes) in /var/www/nextcloud/apps/mail/vendor/pear-pear.horde.org/Horde_Stream/Horde/Stream.php on line 203
PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 1052672 bytes) in /var/www/nextcloud/apps/mail/lib/Cache/Cache.php on line 126

st3iny · 2024-08-13T08:35:49Z

I think this has been addressed in the meantime. Please reopen otherwise.

See #9581, #7494 and #7485

blizzz added bug 0. Needs triage labels Sep 23, 2022

szaimen transferred this issue from nextcloud/server Oct 25, 2022

st3iny added 0. to triage and removed 0. Needs triage labels Aug 13, 2024

st3iny mentioned this issue Aug 13, 2024

[Bug]: Horde_Stream/Horde/Stream.php on line 203 OOM #6280

Closed

8 tasks

st3iny closed this as completed Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cron runs into several out-of-memory issues after 25 upgrade #7495

cron runs into several out-of-memory issues after 25 upgrade #7495

blizzz commented Sep 23, 2022 •

edited

Loading

blizzz commented Sep 23, 2022

blizzz commented Sep 23, 2022

szaimen commented Oct 20, 2022

blizzz commented Oct 21, 2022

szaimen commented Oct 21, 2022

blizzz commented Oct 21, 2022 •

edited

Loading

blizzz commented Oct 21, 2022

st3iny commented Aug 13, 2024

st3iny commented Aug 13, 2024

cron runs into several out-of-memory issues after 25 upgrade #7495

cron runs into several out-of-memory issues after 25 upgrade #7495

Comments

blizzz commented Sep 23, 2022 • edited Loading

⚠️ This issue respects the following points: ⚠️

Bug description

Steps to reproduce

Expected behavior

Installation method

Operating system

PHP engine version

Web server

Database engine version

Is this bug present after an update or on a fresh install?

Are you using the Nextcloud Server Encryption module?

What user-backends are you using?

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

Additional info

blizzz commented Sep 23, 2022

blizzz commented Sep 23, 2022

szaimen commented Oct 20, 2022

blizzz commented Oct 21, 2022

szaimen commented Oct 21, 2022

blizzz commented Oct 21, 2022 • edited Loading

blizzz commented Oct 21, 2022

st3iny commented Aug 13, 2024

st3iny commented Aug 13, 2024

blizzz commented Sep 23, 2022 •

edited

Loading

blizzz commented Oct 21, 2022 •

edited

Loading