refactor: split trade desk into audience and realtime conversions destinations

[Gauravudia]

What are the changes introduced in this PR?

• Split the Trade Desk into audience and real time conversions destinations. Where audience destination will be used for syncing audience from warehouse source through rETL flow and real time conversions destination will be used for tracking conversions through even stream flow.
• Created the separate folders the_trade_desk and the_trade_desk_real_time_conversions
• Moved the real time conversion on processor
• Transformer proxy is not required for real time conversions.

What is the related Linear task?

Resolves INT-1558

Please explain the objectives of your changes below

Put down any required details on the broader aspect of your changes. If there are any dependent changes, mandatorily mention them here

webapp and integration config PR's for separating out trade desk destination.
https://github.com/rudderlabs/rudder-webapp/pull/6057
rudderlabs/rudder-integrations-config#1213
rudderlabs/rudder-integrations-config#1212

Any changes to existing capabilities/behaviour, mention the reason & what are the changes ?

N/A

Any new dependencies introduced with this change?

N/A

Any new generic utility introduced or modified. Please explain the changes.

N/A

Any technical or performance related pointers to consider with the change?

N/A

---

Developer checklist

• My code follows the style guidelines of this project

• No breaking changes are being introduced.

• All related docs linked with the PR?

• All changes manually tested?

• Any documentation changes needed with this change?

• Is the PR limited to 10 file changes?

• Is the PR limited to one linear task?

• Are relevant unit and component test-cases added?

Reviewer checklist

• Is the type of change in the PR title appropriate as per the changes?

• Verified that there are no credentials or confidential data exposed with the changes.

Summary by CodeRabbit

• New Features
  • Introduced integration with The Trade Desk real-time conversions API, including support for various e-commerce events and custom properties.
• Refactor
  • Streamlined utility functions focusing on essential tasks like TTL calculation, endpoint retrieval, and signature header generation.
  • Refactored proxy request handling to improve header management and response logic.
• Bug Fixes
  • Removed unsupported event type 'track' and related functionalities to enhance stability.
• Chores
  • Cleaned up test suites by removing outdated tests and adding new ones for real-time conversions integration.
• Documentation
  • Updated sample data and constants to align with The Trade Desk real-time conversions integration.

[coderabbitai]

Important

Auto Review Skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Walkthrough

The overall changes involve a significant refactoring of the integration with The Trade Desk in a codebase, focusing on streamlining and optimizing the handling of event tracking and real-time conversions. Support for certain event types and specific functionalities has been removed, with a notable shift towards enhancing real-time conversions through new configurations and utility functions. This restructuring indicates a move towards a more focused and efficient integration approach with The Trade Desk's API, improving payload preparation, event handling, and data processing.

Changes

File Path	Summary
src/cdk/v2/.../the_trade_desk/config.js and src/cdk/v2/.../the_trade_desk/transformRecord.js	Removed support for 'track' event type, various constants, real-time conversion endpoint, and configuration retrieval. Also, removed validateConfig function and related checks.
src/cdk/v2/.../the_trade_desk/utils.js and src/cdk/v2/.../the_trade_desk/utils.test.js	Refactored utility functions focusing on basic operations like calculating TTL, getting endpoints, and generating signature headers. Restructured tests to focus on individual functions.
src/cdk/v2/.../the_trade_desk_real_time_conversions/config.js, src/cdk/v2/.../the_trade_desk_real_time_conversions/utils.js, src/cdk/v2/.../the_trade_desk_real_time_conversions/utils.test.js	Introduced new configurations and utility functions for The Trade Desk real-time conversions API integration, focusing on payload preparation, event handling, and data enrichment.
src/v0/.../the_trade_desk/networkHandler.js	Refactored proxy request handling with focus on header management and response logic.
test/integrations/.../the_trade_desk/*.ts	Removed outdated configurations and data related to Trade Desk destination, including trackerId and specific event tracking details.
test/integrations/.../the_trade_desk_real_time_conversions/*.ts	Added new constants, sample data, and configurations for testing The Trade Desk real-time conversions integration.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

---

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit-tests for this file.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit tests for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository from git and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit tests.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• The JSON schema for the configuration file is available here.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

[devops-github-rudderstack]

Test report for this run is available at: https://test-integrations-dev.s3.amazonaws.com/integrations-test-reports/rudder-transformer/3091/test-report.html

[coderabbitai]

@Gauravudia: It seems like your comment might have been cut off. Did you want a review of the pull request, or was there something specific you needed assistance with? Please let me know how I can assist you further.

[codecov]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (8bf55d0) 87.21% compared to head (114560c) 87.19%.

Files	Patch %	Lines
...c/v0/destinations/the_trade_desk/networkHandler.js	80.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3091      +/-   ##
===========================================
- Coverage    87.21%   87.19%   -0.02%     
===========================================
  Files          537      538       +1     
  Lines        29243    29191      -52     
  Branches      6962     6953       -9     
===========================================
- Hits         25503    25452      -51     
+ Misses        3393     3392       -1     
  Partials       347      347              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[devops-github-rudderstack]

Test report for this run is available at: https://test-integrations-dev.s3.amazonaws.com/integrations-test-reports/rudder-transformer/3091/test-report.html

[Gauravudia]

@coderabbitai review

[coderabbitai]

Review Status

Actionable comments generated: 11

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 6a364fb and 0c45214.

Files ignored due to path filters (4)

• src/cdk/v2/destinations/the_trade_desk/rtWorkflow.yaml is excluded by: !**/*.yaml
• src/cdk/v2/destinations/the_trade_desk_real_time_conversions/data/TTDCommonConfig.json is excluded by: !**/*.json
• src/cdk/v2/destinations/the_trade_desk_real_time_conversions/data/TTDItemConfig.json is excluded by: !**/*.json
• src/cdk/v2/destinations/the_trade_desk_real_time_conversions/procWorkflow.yaml is excluded by: !**/*.yaml

Files selected for processing (14)

• src/cdk/v2/destinations/the_trade_desk/config.js (2 hunks)
• src/cdk/v2/destinations/the_trade_desk/transformRecord.js (3 hunks)
• src/cdk/v2/destinations/the_trade_desk/utils.js (2 hunks)
• src/cdk/v2/destinations/the_trade_desk/utils.test.js (2 hunks)
• src/cdk/v2/destinations/the_trade_desk_real_time_conversions/config.js (1 hunks)
• src/cdk/v2/destinations/the_trade_desk_real_time_conversions/utils.js (1 hunks)
• src/cdk/v2/destinations/the_trade_desk_real_time_conversions/utils.test.js (1 hunks)
• src/v0/destinations/the_trade_desk/networkHandler.js (2 hunks)
• test/integrations/destinations/the_trade_desk/common.ts (2 hunks)
• test/integrations/destinations/the_trade_desk/delivery/data.ts (2 hunks)
• test/integrations/destinations/the_trade_desk/network.ts (2 hunks)
• test/integrations/destinations/the_trade_desk/router/data.ts (6 hunks)
• test/integrations/destinations/the_trade_desk_real_time_conversions/common.ts (1 hunks)
• test/integrations/destinations/the_trade_desk_real_time_conversions/processor/data.ts (1 hunks)

Files skipped from review due to trivial changes (1)

• test/integrations/destinations/the_trade_desk_real_time_conversions/common.ts

Additional comments: 12

src/cdk/v2/destinations/the_trade_desk/config.js (1)

• 1-1: LGTM! The changes align with the PR's objectives to focus on audience syncing by removing support for 'track' events and related constants.

test/integrations/destinations/the_trade_desk/common.ts (1)

• 3-15: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [1-1]

LGTM! The updates to the test configurations align with the changes in the destination's structure and functionality.

src/cdk/v2/destinations/the_trade_desk/utils.js (1)

• 1-1: LGTM! The refactoring of utility functions aligns with the PR's objectives to streamline and simplify the destination for audience syncing.

src/cdk/v2/destinations/the_trade_desk/utils.test.js (1)

• 1-1: LGTM! The updates to the test file align with the refactoring of utility functions, ensuring relevant and focused test coverage.

src/cdk/v2/destinations/the_trade_desk_real_time_conversions/config.js (1)

• 1-1: LGTM! The configurations introduced are essential for the focused handling of real-time conversion events in the new destination structure.

test/integrations/destinations/the_trade_desk/network.ts (1)

• 1-1: LGTM! The updates to the test file align with the changes in the destination's structure and functionality, ensuring accurate test coverage.

src/cdk/v2/destinations/the_trade_desk/transformRecord.js (1)

• 1-1: LGTM! The simplification and adjustments in the file align with the destination's refocused functionality on audience syncing.

src/v0/destinations/the_trade_desk/networkHandler.js (1)

• 8-35: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [1-1]

LGTM! The refactoring enhances the destination's security, reliability, and error handling, aligning with best practices.

test/integrations/destinations/the_trade_desk/delivery/data.ts (1)

• 5-10: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [1-1]

LGTM! The updates to the test data align with the new destination structure and functionality, ensuring accurate test coverage.

src/cdk/v2/destinations/the_trade_desk_real_time_conversions/utils.js (1)

• 1-1: LGTM! The utility functions introduced are essential for the focused handling of real-time conversion events in the new destination structure.

test/integrations/destinations/the_trade_desk_real_time_conversions/processor/data.ts (2)

• 482-899: The 'Custom event' test case demonstrates the use of custom properties mapping. Ensure that the mapping logic is thoroughly tested in other parts of the codebase, as this test case only verifies the output structure but not the mapping functionality itself.
• 906-978: The 'Mapped standard trade desk event' test case changes the event name from 'custom event abc' to 'direction' based on the eventsMapping configuration. Verify that the mapping functionality works as expected across various event types and scenarios.

[koladilip]

Overall looks good, what was the main reason for this? is it came from product or more of tech decision?

[Gauravudia]

Overall looks good, what was the main reason for this? is it came from product or more of tech decision?

Not feasible to support multiple dest types in a single destination due to UI limitation.

[devops-github-rudderstack]

Test report for this run is available at: https://test-integrations-dev.s3.amazonaws.com/integrations-test-reports/rudder-transformer/3091/test-report.html

[devops-github-rudderstack]

Test report for this run is available at: https://test-integrations-dev.s3.amazonaws.com/integrations-test-reports/rudder-transformer/3091/test-report.html

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues

Measures
0 Security Hotspots
99.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[krishna2020]

Overall looks good, what was the main reason for this? is it came from product or more of tech decision?

We need some research on how to support multiple use-cases of a destination within one definition - this is not there as of today. And research needs UI layout changes along with backend handling.
The way we have rETL flow of VDM screen showing different objects support, the same kind of behaviour need to be explored in Event Stream. This needs some time to figure out e2e flow. So for immediate resolution, going with separation now.