Issues with Syncing Data from MongoDB Replica Set to Two Different Elasticsearch Clusters

[toBetterMans]

Hello,

I am currently working on synchronizing data from a MongoDB replica set to two different Elasticsearch clusters using Monstache. I have successfully synced the data to elasticsearchA, but I'm facing issues when trying to connect to a second Elasticsearch cluster.

In my configuration, I have set up the elasticsearch-urls as follows:

toml
elasticsearch-urls = [
"http://localhost:9200", # First Elasticsearch server
"http://other-elasticsearch:9200" # Second Elasticsearch server
]
However, I am unable to establish a connection to the second Elasticsearch address.

Could you please provide guidance on how to properly configure Monstache to sync data to both Elasticsearch clusters? Are there any specific settings or troubleshooting steps I should consider?

Thank you for your assistance!

[rwynn]

The Elasticsearch URLs in the configuration are meant to be endpoints in the same cluster. I think you would need to run 2 different monstache processes to sync to 2 clusters.

[toBetterMans]

Dear rwynn,

Thank you very much for your prompt response to my issue. I appreciate your assistance.

I have a few more questions regarding the synchronization process. In my MongoDB, I have three databases, each containing several collections divided by month, such as a.2024-11 and b.2024-12. I would like to know how to synchronize all this data to a new Elasticsearch cluster.

Currently, my MongoDB is set up as a replica set. I’ve tested the synchronization, and I can confirm that new data modifications in MongoDB are successfully synced to the new Elasticsearch cluster. However, I am facing challenges with syncing historical data.

Could you please advise on how to approach this issue? Any guidance you could provide would be greatly appreciated.
My Config.toml

mongo-url = "mongodb://mongo:27017/?replicaSet=rs0"
elasticsearch-urls = ["http://user:password@es:9200"]
direct-read-namespaces = ["a.a.2021-01","a.a.2021-02","b.b.2021-01","b.b.2021-02","c.c.2021-01","c.c.2021-02"]
change-stream-namespaces = ["a","b","c"]
direct-read-no-timeout = true
index-as-update = true
gzip = true
stats = true
index-stats = true
elasticsearch-max-conns = 8 
dropped-collections = true
dropped-databases = true
replay = true
resume = true 
resume-write-unsafe = true
resume-name = "anj"
verbose = true
cluster-name = 'docker-cluster' 
exit-after-direct-reads = true

[logs]
info = "/home/monstache/info.log"
#trace = "/home/monstache/trace.log"
error = "/home/monstache/error.log"
warn = "/home/monstache/warn.log"

This direct-read-namespaces can i use direct-read-namespaces =["a.a.*"]?
Thank you once again for your help!

Best regards.

[toBetterMans]

My start log:

INFO 2024/12/25 01:43:12 Started monstache version 6.7.0
INFO 2024/12/25 01:43:12 Go version go1.14
INFO 2024/12/25 01:43:12 MongoDB go driver v1.3.5
INFO 2024/12/25 01:43:12 Elasticsearch go driver 7.0.18
INFO 2024/12/25 01:43:12 Successfully connected to MongoDB version 4.0.20
INFO 2024/12/25 01:43:12 Successfully connected to Elasticsearch version 8.12.2
INFO 2024/12/25 01:43:12 Sending systemd READY=1
INFO 2024/12/25 01:43:12 Joined cluster docker-cluster
INFO 2024/12/25 01:43:12 Pausing work for cluster docker-cluster