Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding documentation for the regex topic monitor config #1140

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Adding documentation for the regex topic monitor config #1140

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/learn/documentation/versioned/jobs/samza-configurations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ These are the basic properties for setting up a Samza application.
|job.changelog.system|inherited from job.default.system|This property is required if you would like to override the system defined in `job.default.system` for the changelog. The changelog will be used with the stream specified in `stores.store-name.changelog` config. You can override this system by specifying both the system and the stream in `stores.store-name.changelog`.|
|job.coordinator.system|inherited from job.default.system|This property is required if you would like to override the system defined in `job.default.system` for coordination. The **_system-name_** to use for creating and maintaining the Coordinator Stream.|
|job.config.rewriter.<br>**_rewriter-name_**.class|(none)|You can optionally define configuration rewriters, which have the opportunity to dynamically modify the job configuration before the job is started. For example, this can be useful for pulling configuration from an external configuration management system, or for determining the set of input streams dynamically at runtime. The value of this property is a fully-qualified Java classname which must implement [ConfigRewriter](../api/javadocs/org/apache/samza/config/ConfigRewriter.html). Samza ships with these rewriters by default:<br><br>`org.apache.samza.config.RegExTopicGenerator`<br>When consuming from Kafka, this allows you to consume all Kafka topics that match some regular expression (rather than having to list each topic explicitly). This rewriter has additional configuration.<br><br>`org.apache.samza.config.EnvironmentConfigRewriter`<br>This rewriter takes environment variables that are prefixed with `SAMZA_` and adds them to the configuration, overriding previous values where they exist. The keys are lowercased and underscores are converted to dots.|
|job.coordinator.monitor-input-regex.frequency.ms| 300000| If using the `org.apache.samza.config.RegExTopicGenerator` as a rewriter (using `job.config.rewriter.**_rewriter-name_**.class`) in your Samza-YARN job, this configuration allows you to change the default monitoring frequency for detecting new topics matching the input regex. To disable monitoring for new topics set this value to < 0. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reword: s/configuration/property so that the tone is consistent with the rest of the config docs

make docs concise: "This property sets the interval to poll for new topics matching the input-regex."

unless answered elsewhere, clarify these questions:

  1. what's the behavior when new topics matching the regex are discovered?
  2. why would/should a Samza customer care?

|job.config.rewriters|(none)|If you have defined configuration rewriters, you need to list them here, in the order in which they should be applied. The value of this property is a comma-separated list of **_rewriter-name_** tokens.|
|job.config.rewriter.<br>**_rewriter-name_**.system|(none)|Set this property to the `system-name` of the Kafka system from which you want to consume all matching topics.|
|job.config.rewriter.<br>**_rewriter-name_**.regex|(none)|A regular expression specifying which topics you want to consume within the Kafka system `job.config.rewriter.*.system`. Any topics matched by this regular expression will be consumed in addition to any topics you specify in your application.|
Expand Down