Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Sync Feature #467

Merged
merged 3 commits into from
Jun 4, 2024
Merged

Fix Sync Feature #467

merged 3 commits into from
Jun 4, 2024

Conversation

mit-27
Copy link
Contributor

@mit-27 mit-27 commented May 29, 2024

Implemented Changes

  • Added an initial sync that triggers when a new connection is created, syncing all objects based on the current connection.
  • Added an object order sync that corrects and sets the appropriate foreign attributes within a unified object.
  • Demo Link

Copy link

vercel bot commented May 29, 2024

@mit-27 is attempting to deploy a commit to the Panora Team on Vercel.

A member of the Team first needs to authorize it.

Copy link

changeset-bot bot commented May 29, 2024

⚠️ No Changeset found

Latest commit: 66aedfb

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Contributor

coderabbitai bot commented May 29, 2024

Walkthrough

The recent updates focus on enhancing the synchronization feature across various modules and services within the project. Key changes include integrating the CoreSyncService for initial synchronization, updating error handling in utility functions, and modifying the API reference documentation. Additionally, the connector update script has been refined, and the GitlabCommentMapper class has been improved for better handling of user and ticket IDs.

Changes

File Path Change Summary
.../connections.controller.ts Integrated CoreSyncService for initial sync, updated handleCallback method, added logging and redirect.
.../connections.module.ts Added new modules for CRM, ticketing, and syncing services, included CoreSyncService provider.
.../sync/sync.service.ts Added initialSync method, updated getSyncStatus method, imported necessary services and types.
.../ticketing/@lib/@utils/index.ts Modified error handling logic for tcg_contact retrieval.
.../utils/types/original/original.ticketing.ts Updated type declarations for OriginalAccountInput, OriginalCollectionInput, etc.
docs/api-reference/introduction.mdx Updated base URL information, changed tab title from "US Servers" to "Cloud," removed "EU Servers" tab.
scripts/connectorUpdate.js Updated variable names and file paths for connectors.
.../ticketing/comment/services/gitlab/mappers.ts Updated assignment of opts with user_id and ticket_id in GitlabCommentMapper class.

Sequence Diagram(s) (Beta)

sequenceDiagram
    participant Client
    participant ConnectionsController
    participant CoreSyncService
    participant CRMService
    participant AccountingService
    participant MarketingAutomationService
    participant TicketingService

    Client ->> ConnectionsController: handleCallback()
    ConnectionsController ->> CRMService: await crmService.handleCallback()
    ConnectionsController ->> AccountingService: await accountingService.handleCallback()
    ConnectionsController ->> MarketingAutomationService: await marketingAutomationService.handleCallback()
    ConnectionsController ->> TicketingService: await ticketingService.handleCallback()
    ConnectionsController ->> CoreSyncService: initialSync()
    CoreSyncService ->> Database: Sync Data
    CoreSyncService -->> ConnectionsController: Sync Complete
    ConnectionsController -->> Client: Redirect with status
Loading

Assessment against linked issues

Objective Addressed Explanation
Initial sync should trigger after connection creation (#431)
Maintain sync order based on object associations (#431)

Poem

In the code where sync does dance,
We’ve made enhancements, took a chance.
With services now in perfect tune,
Data flows like a gentle monsoon.
Bugs are squashed, the sync is bright,
Our codebase shines in the moonlight. 🌙


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Summary

  • Introduced initial sync feature in connections.controller.ts for new connections
  • Added multiple CRM and ticketing modules in connections.module.ts
  • Implemented initialSync method in CoreSyncService for comprehensive sync
  • Modified error handling in Utils class to return undefined for missing tcg_contact
  • Improved immutability in GitlabCommentMapper by using the spread operator

@@ -42,9 +46,107 @@ export class CoreSyncService {
this.logger.setContext(CoreSyncService.name);
}

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment explaining the purpose of the initialSync method.

@@ -126,7 +126,7 @@ export class Utils {
id_tcg_collection: uuid,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning undefined instead of throwing an error changes the method's behavior. Ensure that all calling code handles undefined appropriately.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits Files that changed from the base of the PR and between 233555d and a214ec5.
Files ignored due to path filters (1)
  • packages/api/swagger/swagger-spec.json is excluded by !**/*.json
Files selected for processing (5)
  • packages/api/src/@core/connections/connections.controller.ts (5 hunks)
  • packages/api/src/@core/connections/connections.module.ts (2 hunks)
  • packages/api/src/@core/sync/sync.service.ts (3 hunks)
  • packages/api/src/ticketing/@lib/@utils/index.ts (1 hunks)
  • packages/api/src/ticketing/comment/services/gitlab/mappers.ts (2 hunks)
Additional Context Used
Biome (47)
packages/api/src/@core/connections/connections.controller.ts (20)

55-55: Decorators are not valid here.


56-56: Decorators are not valid here.


57-57: Decorators are not valid here.


58-58: Decorators are not valid here.


136-136: Decorators are not valid here.


137-137: Decorators are not valid here.


138-138: Decorators are not valid here.


139-139: Decorators are not valid here.


140-140: Decorators are not valid here.


141-141: Decorators are not valid here.


142-142: Decorators are not valid here.


143-143: Decorators are not valid here.


161-161: Decorators are not valid here.


75-75: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.


161-161: Unexpected any. Specify a different type.


8-9: All these imports are only used as types.


9-10: All these imports are only used as types.


10-11: All these imports are only used as types.


12-13: All these imports are only used as types.


14-15: All these imports are only used as types.

packages/api/src/@core/sync/sync.service.ts (20)

83-88: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.


116-121: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.


150-152: This code is unreachable


188-188: Do not use template literals if interpolation and special-character handling are not needed.


1-2: All these imports are only used as types.


4-5: All these imports are only used as types.


6-7: All these imports are only used as types.


7-8: All these imports are only used as types.


8-9: All these imports are only used as types.


9-10: All these imports are only used as types.


10-11: All these imports are only used as types.


11-12: All these imports are only used as types.


12-13: All these imports are only used as types.


13-14: All these imports are only used as types.


14-15: All these imports are only used as types.


15-16: All these imports are only used as types.


16-17: All these imports are only used as types.


17-18: All these imports are only used as types.


18-19: All these imports are only used as types.


19-20: All these imports are only used as types.

packages/api/src/ticketing/@lib/@utils/index.ts (1)

2-2: A Node.js builtin module should be imported with the node: protocol.

packages/api/src/ticketing/comment/services/gitlab/mappers.ts (6)

65-65: This variable implicitly has the any type.


1-1: All these imports are only used as types.


1-5: All these imports are only used as types.


5-6: All these imports are only used as types.


6-7: All these imports are only used as types.


9-10: All these imports are only used as types.

Additional comments not posted (8)
packages/api/src/@core/connections/connections.module.ts (3)

10-26: Ensure that all newly added modules are properly initialized and used within the system.


35-50: Verify that all modules are correctly imported and utilized in the imports array of the @Module decorator.


52-52: Adding CoreSyncService to providers ensures that it can be injected where needed in this module. Good encapsulation and modularity.

packages/api/src/ticketing/comment/services/gitlab/mappers.ts (1)

87-87: The use of the spread operator to include user_id and ticket_id in opts ensures that the existing properties are not overwritten, which is a good practice for maintaining data integrity.

Also applies to: 105-109

packages/api/src/ticketing/@lib/@utils/index.ts (1)

129-129: The early return pattern used here prevents further execution if res is null, which is a good practice for avoiding errors in subsequent lines of code.

packages/api/src/@core/connections/connections.controller.ts (2)

20-20: The integration of CoreSyncService in the constructor and its use in the handleCallback method aligns with the PR objectives to trigger an initial sync upon a new connection. This is crucial for ensuring data consistency from the start.

Also applies to: 38-38


Line range hint 54-128: The handleCallback method is well-structured with clear error handling and appropriate redirects. The use of switch on vertical to handle different connection types is a good practice, ensuring that each type is handled according to its specific needs.

packages/api/src/@core/sync/sync.service.ts (1)

49-140: The initialSync method is comprehensive and well-structured. It handles synchronization tasks for different services based on the vertical. The use of asynchronous tasks and error handling within the sync operations is a good practice, ensuring robustness and reliability.

Copy link

gitguardian bot commented May 29, 2024

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
9763120 Triggered Generic High Entropy Secret 66aedfb docs/mint.json View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits Files that changed from the base of the PR and between a214ec5 and 4732ca9.
Files selected for processing (2)
  • packages/api/src/@core/connections/connections.controller.ts (5 hunks)
  • packages/api/src/@core/sync/sync.service.ts (3 hunks)
Additional Context Used
Biome (40)
packages/api/src/@core/connections/connections.controller.ts (20)

55-55: Decorators are not valid here.


56-56: Decorators are not valid here.


57-57: Decorators are not valid here.


58-58: Decorators are not valid here.


127-127: Decorators are not valid here.


128-128: Decorators are not valid here.


129-129: Decorators are not valid here.


130-130: Decorators are not valid here.


131-131: Decorators are not valid here.


132-132: Decorators are not valid here.


133-133: Decorators are not valid here.


134-134: Decorators are not valid here.


152-152: Decorators are not valid here.


75-75: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.


152-152: Unexpected any. Specify a different type.


8-9: All these imports are only used as types.


9-10: All these imports are only used as types.


10-11: All these imports are only used as types.


12-13: All these imports are only used as types.


14-15: All these imports are only used as types.

packages/api/src/@core/sync/sync.service.ts (20)

83-88: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.


116-121: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.


144-146: This code is unreachable


182-182: Do not use template literals if interpolation and special-character handling are not needed.


1-2: All these imports are only used as types.


4-5: All these imports are only used as types.


6-7: All these imports are only used as types.


7-8: All these imports are only used as types.


8-9: All these imports are only used as types.


9-10: All these imports are only used as types.


10-11: All these imports are only used as types.


11-12: All these imports are only used as types.


12-13: All these imports are only used as types.


13-14: All these imports are only used as types.


14-15: All these imports are only used as types.


15-16: All these imports are only used as types.


16-17: All these imports are only used as types.


17-18: All these imports are only used as types.


18-19: All these imports are only used as types.


19-20: All these imports are only used as types.

Additional comments not posted (12)
packages/api/src/@core/connections/connections.controller.ts (8)

20-20: Added import for CoreSyncService.

This import is necessary for the new functionality introduced in the handleCallback method.


38-38: Added coreSyncService as a private property.

This is a good practice for dependency injection and will be used in the handleCallback method.


54-54: Method handleCallback is now asynchronous.

Changing this method to async is appropriate since it now includes asynchronous operations such as initialSync.


76-76: Call to crmConnectionsService.handleCRMCallBack within the CRM case of the switch statement.

This is a necessary call to handle CRM-specific callback logic.


89-89: Call to accountingConnectionsService.handleAccountingCallBack within the Accounting case of the switch statement.

Proper handling of the accounting callback as per the new sync requirements.


101-101: Call to marketingAutomationConnectionsService.handleMarketingAutomationCallBack within the Marketing Automation case of the switch statement.

Ensures that marketing automation callbacks are handled correctly.


109-109: Call to ticketingConnectionsService.handleTicketingCallBack within the Ticketing case of the switch statement.

Correct implementation for handling ticketing callbacks.


118-118: Integration of initialSync in the handleCallback method.

This integration is crucial for initiating the sync process once the callback is handled.

packages/api/src/@core/sync/sync.service.ts (4)

3-5: Added imports for ConnectorCategory, CrmObject, and ENGAGEMENTS_TYPE.

These imports are necessary for the new functionality in the initialSync method.


28-28: Constructor now includes PrismaService and various sync services.

This setup is essential for the operations performed in the initialSync method.


49-138: Implementation of the initialSync method.

The method is well-structured and handles different verticals appropriately. However, consider adding a comment explaining the purpose of this method for better maintainability.


49-49: Add a comment explaining the purpose of the initialSync method.

This comment was previously suggested by another reviewer and is still valid.

@mit-27 mit-27 force-pushed the fix-sync-feature branch from 4732ca9 to 66aedfb Compare May 29, 2024 06:37
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits Files that changed from the base of the PR and between 4732ca9 and 66aedfb.
Files ignored due to path filters (3)
  • docs/logo/fav.png is excluded by !**/*.png, !**/*.png
  • docs/logo/favicon.png is excluded by !**/*.png, !**/*.png
  • docs/mint.json is excluded by !**/*.json
Files selected for processing (5)
  • docs/api-reference/introduction.mdx (1 hunks)
  • packages/api/scripts/connectorUpdate.js (3 hunks)
  • packages/api/src/@core/connections/connections.controller.ts (6 hunks)
  • packages/api/src/@core/sync/sync.service.ts (3 hunks)
  • packages/api/src/@core/utils/types/original/original.ticketing.ts (4 hunks)
Files skipped from review due to trivial changes (1)
  • docs/api-reference/introduction.mdx
Additional Context Used
Biome (78)
packages/api/scripts/connectorUpdate.js (18)

10-10: A Node.js builtin module should be imported with the node: protocol.


11-11: A Node.js builtin module should be imported with the node: protocol.


12-12: A Node.js builtin module should be imported with the node: protocol.


75-75: Template literals are preferred over string concatenation.


79-120: Prefer for...of instead of forEach.


162-197: Prefer for...of instead of forEach.


201-205: Template literals are preferred over string concatenation.


251-269: Prefer for...of instead of forEach.


253-253: Template literals are preferred over string concatenation.


260-260: This regular expression contains unclear uses of consecutive spaces.


263-263: Template literals are preferred over string concatenation.


284-284: Change to an optional chain.


295-301: Prefer for...of instead of forEach.


35-35: This let declares a variable that is only assigned once.


71-71: Reassigning a function parameter is confusing.


140-140: This let declares a variable that is only assigned once.


279-279: This let declares a variable that is only assigned once.


285-285: This let declares a variable that is only assigned once.

packages/api/src/@core/connections/connections.controller.ts (20)

55-55: Decorators are not valid here.


56-56: Decorators are not valid here.


57-57: Decorators are not valid here.


58-58: Decorators are not valid here.


125-125: Decorators are not valid here.


126-126: Decorators are not valid here.


127-127: Decorators are not valid here.


128-128: Decorators are not valid here.


129-129: Decorators are not valid here.


130-130: Decorators are not valid here.


131-131: Decorators are not valid here.


132-132: Decorators are not valid here.


150-150: Decorators are not valid here.


75-75: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.


150-150: Unexpected any. Specify a different type.


8-9: All these imports are only used as types.


9-10: All these imports are only used as types.


10-11: All these imports are only used as types.


12-13: All these imports are only used as types.


14-15: All these imports are only used as types.

packages/api/src/@core/sync/sync.service.ts (20)

83-88: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.


116-121: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.


144-146: This code is unreachable


182-182: Do not use template literals if interpolation and special-character handling are not needed.


1-2: All these imports are only used as types.


4-5: All these imports are only used as types.


6-7: All these imports are only used as types.


7-8: All these imports are only used as types.


8-9: All these imports are only used as types.


9-10: All these imports are only used as types.


10-11: All these imports are only used as types.


11-12: All these imports are only used as types.


12-13: All these imports are only used as types.


13-14: All these imports are only used as types.


14-15: All these imports are only used as types.


15-16: All these imports are only used as types.


16-17: All these imports are only used as types.


17-18: All these imports are only used as types.


18-19: All these imports are only used as types.


19-20: All these imports are only used as types.

packages/api/src/@core/utils/types/original/original.ticketing.ts (20)

2-6: All these imports are only used as types.


6-7: All these imports are only used as types.


7-8: All these imports are only used as types.


8-12: All these imports are only used as types.


12-16: All these imports are only used as types.


16-20: All these imports are only used as types.


20-24: All these imports are only used as types.


24-28: All these imports are only used as types.


28-32: All these imports are only used as types.


32-36: All these imports are only used as types.


36-40: All these imports are only used as types.


40-44: All these imports are only used as types.


44-48: All these imports are only used as types.


48-52: All these imports are only used as types.


52-56: All these imports are only used as types.


56-60: All these imports are only used as types.


60-64: All these imports are only used as types.


64-68: All these imports are only used as types.


68-72: All these imports are only used as types.


72-76: All these imports are only used as types.

Additional comments not posted (11)
packages/api/src/@core/connections/connections.controller.ts (5)

20-20: Added import for CoreSyncService.

This import is necessary for the new functionality related to synchronization.


38-38: Added coreSyncService to the constructor.

This is essential for accessing the synchronization service within this controller.


54-54: Converted handleCallback to an async function.

This change supports the asynchronous operations within the method, particularly the new sync functionality.


76-76: Added async calls to handle callbacks for different services.

These additions are part of enhancing the callback handling by integrating various services.

Also applies to: 87-87, 99-99, 107-107


116-116: Added a call to initialSync in handleCallback.

This is a crucial addition for initiating the synchronization process as described in the PR objectives.

packages/api/src/@core/utils/types/original/original.ticketing.ts (1)

158-160: Updated type definitions with vertical bars.

This change clarifies the type options, making the code more readable and maintainable.

Also applies to: 183-185, 224-226, 256-258

packages/api/src/@core/sync/sync.service.ts (3)

3-5: Added necessary imports for new functionality.

These imports are essential for the new synchronization features implemented in this service.


28-28: Added PrismaService to the constructor.

This is necessary for database operations within the synchronization processes.


49-138: Implemented initialSync method.

This method is well-structured and aligns with the PR objectives to handle initial synchronization based on the provided vertical.

packages/api/scripts/connectorUpdate.js (2)

279-279: Updated enumName to baseConnectors.

This change aligns with the updated file paths and naming conventions used in the project.


331-333: Updated file paths for providers and enums.

These updates are necessary to reflect the new directory structure for connectors.

@naelob naelob merged commit 498efc4 into panoratech:main Jun 4, 2024
8 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix: Sync Feature
2 participants