Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JMSPublishFilter: improve memory footprint and impose a limit #142

Merged
merged 7 commits into from
May 7, 2024

Conversation

eolivelli
Copy link
Collaborator

@eolivelli eolivelli commented May 7, 2024

Motivation:
The JMSPublishFilter can allocate an unbounded amount of memory (especially String objects), this can lead to OutOfMemory errors.

Changes:
This is a refactor that reduces the amount of memory retained to handle the queue of messages in the JMSPublishFilter.
Previously the queue contained MessageMetadata, with all the properties as Strings, now we keep only 1 Netty ByteBuf, and the parsing of the Metadata is done on the processing thread, 1 message at a time per subscription.

Therefore there is a limit on the amount of memory retained for the queue, when the limit is reached the producers start to be throttled.

  • new metric (gauge): pulsar_jmsfilter_processing_memory: bytes used for MessageMetadata waiting for processing
  • new metric (gauge): pulsar_jmsfilter_processing_pending_operations: number of pending operations
  • new configuration: jmsFiltersOnPublishMaxMemoryMB, this is the maximum amount of memory to retain for the operations in the queue, this is heap memory

@eolivelli eolivelli changed the title MSPublishFilters: improve memory footprint MSPublishFilters: improve memory footprint and impose a limit May 7, 2024
@eolivelli eolivelli changed the title MSPublishFilters: improve memory footprint and impose a limit JMSPublishFilter: improve memory footprint and impose a limit May 7, 2024
@eolivelli eolivelli merged commit 05a0021 into master May 7, 2024
1 of 3 checks passed
@eolivelli eolivelli deleted the jms-publish-filter-memory branch May 7, 2024 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant