Big Export Failure: Deadline exceeded

[super9xman]

Hello,

I have around 350k documents in 1 of my collections and I am trying to export this data out of firestore...However, I run into the below error. Is there a way to get batch export? I tried --chunk, but it does not work for exports...

Any other solution? Please advice..

Error:

Failure: { Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded
at Object.exports.createStatusError (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\common.js:91:15)
at ClientReadableStream._emitStatusIfDone (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client.js:233:26)
at ClientReadableStream._receiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client.js:211:8)
at Object.onReceiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:1272:15)
at InterceptingListener._callNext (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:568:42)
at InterceptingListener.onReceiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:618:8)
at C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:1029:24
code: 4,
metadata: Metadata { _internal_repr: {} },
details: 'Deadline Exceeded' }
PS C:\Users\JD\Desktop\Learn\practice\firestore-migrator> fire-migrate export --no-subcolls allquotes.json quotes
Getting selected collections...
Failure: { Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded
at Object.exports.createStatusError (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\common.js:91:15)
at ClientReadableStream._emitStatusIfDone (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client.js:233:26)
at ClientReadableStream._receiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client.js:211:8)
at Object.onReceiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:1272:15)
at InterceptingListener._callNext (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:568:42)
at InterceptingListener.onReceiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:618:8)
at C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:1029:24
code: 4,
metadata: Metadata { _internal_repr: {} },
details: 'Deadline Exceeded' }

[stildalf]

Hi super9xman,

Do you have an idea on which bounds we're hitting here, based on Firestore's quotas? Number of records, request size, timeout, etc.

You're right, fire-migrate only batches uploads/writes for the moment. I'll see if I can put a PR together for --chunk on exports.