Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[firestore-bigquery-export] Add lifecycle event #1303

Merged
merged 23 commits into from
Oct 17, 2023

Conversation

joehan
Copy link
Collaborator

@joehan joehan commented Nov 16, 2022

Adds lifecycle event to export existing documents to BigQuery

TODO

  • Docs review from @kevinthecheung - done in firebaseprivate
  • Update package-lock.json once [email protected] is published
  • Code review
  • Republish and test to ensure that nothing broke while switching to published deps
  • Decide whether to deprecate or fully remove old import script
  • Test this lifecycle hook more thoroughly - I don't have a great understanding of how the BigQuery import script should behave, so I'd like an extra look at this. Do we need to make more changes to make the import code idempotent?
  • Should we add an onConfigure hook here too?

@joehan joehan requested a review from a team as a code owner November 16, 2022 19:43
@joehan joehan requested a review from ifielker November 16, 2022 19:49
@dackers86
Copy link
Member

dackers86 commented Nov 18, 2022

@joehan

onConfigure

This event would be very useful, ideally we could use this to move the tables creation/update into the events. With this we would no longer be required to create/update the tables on each record written, which would support a few performance issues that exist in the change-tracker!

This improvement would also be added to the onInstall, onUpdate events.

Import script

It is my understanding that the import script is the only other library that uses the change-tracker dependency. Not relying on this would help with current maintenance and release constraints. I should mention this is a complex dependency. The import script also has a fair amount of users https://www.npmjs.com/package/@firebaseextensions/fs-bq-import-collection.

firestore-bigquery-export/PREINSTALL.md Show resolved Hide resolved
firestore-bigquery-export/README.md Show resolved Hide resolved
firestore-bigquery-export/README.md Show resolved Hide resolved
firestore-bigquery-export/extension.yaml Outdated Show resolved Hide resolved
firestore-bigquery-export/extension.yaml Show resolved Hide resolved
firestore-bigquery-export/functions/src/index.ts Outdated Show resolved Hide resolved
firestore-bigquery-export/functions/tsconfig.json Outdated Show resolved Hide resolved
@dackers86
Copy link
Member

All lifecycle hooks are included in this update.

For update and reconfiguration, the option is there to not perform the backfill. The question could be whether this enough for users to know that this will cause a new import if the value is not update?

@dackers86
Copy link
Member

The best discussed option here is still still allow the Import script, as this is currently built into many developers workflows.

This could be still deprecated in the future

@dackers86 dackers86 merged commit 7dcb382 into next Oct 17, 2023
8 checks passed
@dackers86 dackers86 deleted the lifecycle.firestore-bigquery-export branch October 17, 2023 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants