how to use gtm to sync to a mongo primary #21
Comments
|
i also have another issue with direct read in my printOp go routine i simply use the code you have as an example to print operations i expect this code to print until the collection is fully received then since the channel is nil the default case will run and the routine will exit however the routine hits the default case immediately |
|
Hopefully this will help... main.go package main
import (
"context"
"flag"
"fmt"
"log"
"time"
"github.com/rwynn/gtm"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func readContext(ctx *gtm.OpCtx, done chan bool) {
var errsDone, opsDone bool
var opCnt, errCnt int
for !errsDone || !opsDone {
select {
case op, open := <-ctx.OpC:
if op == nil && !open {
opsDone = true
break
}
opCnt++
break
case err, open := <-ctx.ErrC:
if err == nil && !open {
errsDone = true
break
}
errCnt++
break
}
}
fmt.Printf("Processed %d ops and %d errs\n", opCnt, errCnt)
close(done)
}
func main() {
var mongoURL string
flag.StringVar(&mongoURL, "url", "mongodb://localhost:27017", "MongoDB connection URL")
flag.Parse()
log.Printf("Connecting to MongoDB at %s", mongoURL)
client, err := mongo.NewClient(options.Client().ApplyURI(mongoURL))
if err != nil {
log.Fatalln(err)
}
if err = client.Connect(context.Background()); err != nil {
log.Fatalln(err)
}
defer client.Disconnect(context.Background())
done := make(chan bool)
gtmCtx := gtm.Start(client, >m.Options{
DirectReadNs: []string{"test.test"},
ChangeStreamNs: []string{"test.test"},
MaxAwaitTime: time.Duration(10) * time.Second,
OpLogDisabled: true,
})
go readContext(gtmCtx, done)
gtmCtx.DirectReadWg.Wait()
gtmCtx.Stop()
<-done
fmt.Println("All done")
}go.mod |
|
awesome |
|
i ran the code and saw something interesting with operation print: without operation print: |
|
i did a bit more investigation after the direct read is done you have closed the ctx which i think kills the channel and causes the |
|
I think it may be related to the use of buffered channels in gtm. Try updating the readContext function I posted and give the name op to the 2 _ vars. Then only set the 2 done flags if... op == nil && !open Due to buffering I think that the channel is closed but still has data on it. When the channel is empty the zero value of nil for a pointer will start getting returned. |
|
Also to your previous question. With Change stream NS set, yes you will also receive changes to the collections that are happening. There are methods to check if the event source is direct read or change event. Make that an empty slice if you don’t want changes. You will probably want OplogDisabled always set to true. This project is old and thus supports reading directly from the oplog. But you don’t want to do that anymore since change streams. |
|
i updated my original posted code to account for the buffering. |
|
i tried the new code and just added fmt.Println(op) and fmt.Println(err) but still i lose some operations it should have 2918601 ops |
|
Can you post the full code that does not work? Also, does it work with or without the |
|
works correctly without the print |
|
Even when I add the That's really a mystery to me why that Print would be affecting the count. |
|
i,m using 1.13.1 on centos7 |
|
i tried with go 1.13.4 but no luck i also used another server with more resources to make sure there are no bottlenecks there is it the same for you? |
|
on the new server running without the print returns different results (some correct and some not correct) |
|
here is my mongo logs . looks like latency issues |
|
i ran the code couple more times on both servers |
|
I think I found the mistake on my part. It was silly. The condition for the for !errsDone || !opsDone {Notice the I think |
|
looks like that was it |
|
i checked a couple more times and it is working :) |
i,m trying to use gtm to sync two mongo instances.
first instance has been running for a while and has roughly 50GB of data.
my question is if i do a direct read and start writing to the secondary database what can i do about the changes made to the primary database while i,m reading the data since it can change after i read it
my own idea is to store the op-log while doing the direct read and then apply it to the second database.
i understand that there are already tools for this but this is part of a bigger project and i need to achieve this via code.
any better options?
The text was updated successfully, but these errors were encountered: