Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide CentralDogmaPropertySupplier that can read YAML #173

Open
maediy opened this issue Nov 1, 2022 · 2 comments
Open

Provide CentralDogmaPropertySupplier that can read YAML #173

maediy opened this issue Nov 1, 2022 · 2 comments

Comments

@maediy
Copy link

maediy commented Nov 1, 2022

The properties files on centraldogma must be written in JSON format for now.
I want to leave some comments on the properties files, but JSON doesn't support comment-out.
So, I hope that decaton-centraldogma supports YAML which supports comment-out.

@ocadaruma
Copy link
Member

related: line/centraldogma#151

@Yang-33
Copy link
Contributor

Yang-33 commented Dec 19, 2024

I've thought about this and come up with two workarounds. I'm not sure if anyone is following the issue, but let me share them:

(1) Implement a PropertySupplier that supports YAML
decaton user can implement a YAML version of CentralDogmaPropertySupplier. By receiving the Central Dogma file as text and reading it with a YAML parser, YAML support can be achieved.
The downside is that the user must take responsibility for maintenance. Additionally, there is no guarantee that the YAML version will continue to be usable with future changes to decaton.

probably we? can submit PR with this approach to this repository...?

(2) Support YAML/JSON5 to JSON conversion in user land with CI or scripts
Central Dogma users utilize the mirroring feature with Git, especially with GitHub, which is convenient. By setting up CI or scripts to create files for decaton from YAML/JSON5, comments can be used.
For example, we can convert YAML/JSON5 files to a comment-free state with CI or scripts, place them somewhere, and have decaton read them.
The downside is the risk (a kind of operational error?) that users might change the files intended for decaton after conversion.
If we define a CI job to run scripts to covert, and check no diff between original file(YAML/JSON5) and created file(JSON), there may be no operation error.

(Example)

# See wiki.example.com at first!
decaton.partition.concurrency: 1 # All tasks are processed after calling deferCompletion
decaton.processing.rate.per.partition: -1
decaton.max.pending.records: 2000 # message is not small, this is to avoid OOME. Do not consume many messages
decaton.group.rebalance.timeout.ms: 5000
decaton.processing.shutdown.timeout.ms: 5000 # must be same as above
decaton.ignore.keys:
  - "user-1" # abuser
  - "user-2" # QA
$ yq -o=json decaton-some-topic.yml decaton-some-topic.json

{
  "decaton.partition.concurrency": 1,
  "decaton.processing.rate.per.partition": -1,
  "decaton.max.pending.records": 2000,
  "decaton.group.rebalance.timeout.ms": 5000,
  "decaton.processing.shutdown.timeout.ms": 5000,
  "decaton.ignore.keys": [
    "user-1",
    "user-2"
  ]
}

(this assumes that user app with decaton reads decaton-some-topic.json)

I hope this is helpful.

It seems the use of Central Dogma in decaton is very simple. Even without using Central Dogma's JSON support, decaton can achieve this issue by receiving the file as a text file and using an object mapper to interpret it as JSON or YAML (probably). I hope this!
I've known and used decaton over three years(i love decaton), and I've often wanted to add notes or comments about ignored keys to the decaton files in Central Dogma to explain history or caution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants