Move retry logic to mongochangestream.
#19
Conversation
There was a problem hiding this comment.
I think this will accidentally result in there not being any retries in mongo2crate.
I was looking at this recently, and I noticed that there are a couple of try/catches in syncData.ts (example) around Crate operations, so if they fail, we're catching the error and emitting an "error" event instead. In that case, because no error is thrown, mongochangestream would not retry.
I think we need to remove the try/catches too, but it would also be good to ensure that only one "error" event is emitted at the end after the retries, which I think indicates that we need to emit the error event in mongochangestream. See my WIP PR: smartprocure/mongochangestream#30
| const redis = new Redis({ keyPrefix: 'testing:' }) | ||
| // MongoDB | ||
| const mongoClient = await MongoClient.connect( | ||
| process.env.MONGO_CONN as string |
There was a problem hiding this comment.
This isn't a blocker, but it would be helpful to document in the README how to set up your environment and run the tests.
| // Wait for the change stream events to be processed | ||
| await setTimeout(ms('6s')) | ||
| const countResponse = await crate.query( | ||
| `SELECT COUNT(*) FROM ${sync.qualifiedName} WHERE "createdAt" >= '${date.toISOString()}'` |
There was a problem hiding this comment.
Wouldn't they all be equal to date?
| @@ -14,6 +12,7 @@ import { | |||
| } from './crate/util.js' | |||
|
|
|||
| const debug = _debug('mongo2crate:crate') | |||
| debug.log = console.log.bind(console) | |||
There was a problem hiding this comment.
What's the purpose of this line?
| @@ -132,6 +136,7 @@ export const initSync = ( | |||
| } | |||
| } catch (e) { | |||
| emit('error', { error: e, changeStream: true }) | |||
There was a problem hiding this comment.
Just checking my understanding: in the event of an error, we'll emit an error event here in mongo2crate (potentially multiple times, once for each retry), then a processError event in mongochangestream once all the retries are exhausted.
mongochangestream- Move retry logic tomongochangestream.immutableoption.