Skip to content

Commit

Permalink
Merge pull request #1776 from firebase/@invertase/refactor-import-script
Browse files Browse the repository at this point in the history
@invertase/refactor import script
  • Loading branch information
cabljac authored Oct 23, 2023
2 parents f21c598 + 65eb14d commit e235c31
Show file tree
Hide file tree
Showing 16 changed files with 1,801 additions and 587 deletions.
8 changes: 6 additions & 2 deletions firestore-bigquery-export/guides/IMPORT_EXISTING_DOCUMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,15 @@ You may pause and resume the import script from the last batch at any point.
- If document changes occur in the time between installing the extension and running the import script.
- If you run the import script multiple times over the same collection.

- You cannot use wildcard notation in the collection path (i.e. `/collection/{document}/sub_collection}`). Instead, you can use a [collectionGroup](https://firebase.google.com/docs/firestore/query-data/queries#collection-group-query) query. To use a `collectionGroup` query, provide the collection name value as `${COLLECTION_PATH}`, and set `${COLLECTION_GROUP_QUERY}` to `true`. For example, if you are trying to import `/collection/{document}/sub_collection`, the value for the `${COLLECTION_PATH}` should be provided as `sub_collection`. Keep in mind that if you have another sub collection with the same name (e.g. `/collection2/{document}/sub_collection`, that will be imported too.
- You can use wildcard notation in the collection path. Suppose, for example, you have collections `users/user1/pets` and `users/user2/pets`, but also `admins/admin1/pets`. If you set `${COLLECTION_GROUP_QUERY}` to `true` and provide the collection path as `${users/{uid}/pets}`, the import script will import the former two collections but not the later, and will populate the `path_params` column of the big query table with the relevant `uid`s.

- You can also use a [collectionGroup](https://firebase.google.com/docs/firestore/query-data/queries#collection-group-query) query. To use a `collectionGroup` query, provide the collection name value as `${COLLECTION_PATH}`, and set `${COLLECTION_GROUP_QUERY}` to `true`. For example, if you are trying to import `/collection/{document}/sub_collection`, the value for the `${COLLECTION_PATH}` should be provided as `sub_collection`. Keep in mind that if you have another sub collection with the same name (e.g. `/collection2/{document}/sub_collection`, that will be imported too.

- Warning: The import operation is not idempotent; running it twice, or running it after documents have been imported will likely produce duplicate data in your bigquery table.

Warning: A `collectionGroup` query will target every collection in your Firestore project with the provided `${COLLECTION_PATH}`. For example, if you have 10,000 documents with a sub-collection named: `landmarks`, the import script will query every document in 10,000 `landmarks` collections.
You can also use a simple [collectionGroup](https://firebase.google.com/docs/firestore/query-data/queries#collection-group-query) query. To use a `collectionGroup` query, provide the collection name value as `${COLLECTION_PATH}`, and set `${COLLECTION_GROUP_QUERY}` to `true`.

Warning: A `collectionGroup` query will target every collection in your Firestore project with the provided `${COLLECTION_PATH}`. For example, if you have 10,000 documents with a sub-collection named: `landmarks`, the import script will query every document in 10,000 `landmarks` collections.

### Run the script

Expand Down
Loading

0 comments on commit e235c31

Please sign in to comment.