feat: add structured logging #2968

sanpj2292 · 2024-01-05T13:43:09Z

What are the changes introduced in this PR?

Adding structured logging to the service. We have used winston for this purpose.

What is the related Linear task?

Resolves INT-1087

Please explain the objectives of your changes below

Put down any required details on the broader aspect of your changes. If there are any dependent changes, mandatorily mention them here

Any changes to existing capabilities/behaviour, mention the reason & what are the changes ?

N/A

Any new dependencies introduced with this change?

N/A

Any new generic utility introduced or modified. Please explain the changes.

N/A

Any technical or performance related pointers to consider with the change?

N/A

Developer checklist

My code follows the style guidelines of this project
No breaking changes are being introduced.
All related docs linked with the PR?
All changes manually tested?
Any documentation changes needed with this change?
Is the PR limited to 10 file changes?
Is the PR limited to one linear task?
Are relevant unit and component test-cases added?

Reviewer checklist

Is the type of change in the PR title appropriate as per the changes?
Verified that there are no credentials or confidential data exposed with the changes.

Signed-off-by: Sai Sankeerth <[email protected]>

coderabbitai · 2024-01-05T13:43:17Z

Important

Auto Review Skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit-tests for this file.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit tests for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository from git and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit tests.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
The JSON schema for the configuration file is available here.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

…to feat.struct-log

codecov · 2024-01-05T13:49:21Z

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (b1df337) 87.28% compared to head (7c12872) 87.29%.
Report is 45 commits behind head on develop.

Files	Patch %	Lines
src/services/destination/postTransformation.ts	75.00%	4 Missing ⚠️
src/logger/structured-logger.ts	90.62%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2968      +/-   ##
===========================================
+ Coverage    87.28%   87.29%   +0.01%     
===========================================
  Files          872      873       +1     
  Lines        29500    29561      +61     
  Branches      6865     6884      +19     
===========================================
+ Hits         25749    25806      +57     
- Misses        3405     3409       +4     
  Partials       346      346

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Sai Sankeerth <[email protected]>

koladilip · 2024-01-06T16:36:36Z

We need to have discussion on this @sanpj2292 , few questions:

Are we planning to send a child logger with all the context to destination or source handlers? If yes how does that work?
I thought from root level module we need keep passing a logger to child modules functions with more context, but there is no child logger being used here. Since the context changes per request we need to create child loggers per request that may be costly (not sure if that is really concern or not) else we need to do something golang does with context, we can create child context from parent and so on, we can keep one logger at module level but context keeps changing.

sanpj2292 · 2024-01-08T06:39:35Z

Are we planning to send a child logger with all the context to destination or source handlers? If yes how does that work?

Most of use-case regarding logging surrounds around having a good logger that could tell us clearly about what's happening in the service. The logger requirement at destination/source level is quite less at this point. But when we consider the logger to be used at destination/source level currently this is the thought process.

I thought from root level module we need keep passing a logger to child modules functions with more context, but there is no child logger being used here. Since the context changes per request we need to create child loggers per request that may be costly (not sure if that is really concern or not) else we need to do something golang does with context, we can create child context from parent and so on, we can keep one logger at module level but context keeps changing.

We have to look into the possibility of having a single parent logger with changing context.

As per our discussion offline, will try to implement that version & we can look at it in detail

Signed-off-by: Sai Sankeerth <[email protected]>

sonarqubecloud · 2024-01-08T10:50:03Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

koladilip · 2024-01-08T14:32:50Z

src/services/destination/nativeIntegration.ts

@@ -113,8 +121,12 @@ export class NativeIntegrationDestinationService implements DestinationService {
          tags.FEATURES.ROUTER,
        );
        try {
+          const metadataWithSvcCtx = {
+            ...requestMetadata,
+            serviceContext: MiscService.getLoggableData(metaTO.errorDetails),


this is more of request context right, service context might be misleading?

The thought process behind naming was to reflect the context at service level.

may be check with others and choose the one with less confusing

src/services/destination/nativeIntegration.ts

koladilip · 2024-01-08T17:28:06Z

src/services/destination/nativeIntegration.ts

+        metaTO.metadata = event.metadata;
+        const metadataWithSvcCtx = {
+          ...requestMetadata,
+          serviceContext: MiscService.getLoggableData(metaTO.errorDetails),


why only from error details? we should use

destinationType, event.metadata?.destinationId, event.metadata?.workspaceId, tags.FEATURES.PROCESSOR,

We are storing the information like destinationType, destinationId etc,. at service layer level in this type. We are just re-using the same type.

devops-github-rudderstack · 2024-01-29T01:23:02Z

This PR is considered to be stale. It has been open for 20 days with no further activity thus it is going to be closed in 7 days. To avoid such a case please consider removing the stale label manually or add a comment to the PR.

feat: add structured logging

f35c5de

Signed-off-by: Sai Sankeerth <[email protected]>

sanpj2292 requested review from sivashanmukh and a team as code owners January 5, 2024 13:43

Merge branch 'develop' of github.com:rudderlabs/rudder-transformer in…

f55b3c6

…to feat.struct-log

chore: add log-level selection support via env

c20c677

Signed-off-by: Sai Sankeerth <[email protected]>

feat: make loggable data available at handler level

7c12872

Signed-off-by: Sai Sankeerth <[email protected]>

koladilip reviewed Jan 8, 2024

View reviewed changes

devops-github-rudderstack added the Stale label Jan 29, 2024

devops-github-rudderstack closed this Feb 5, 2024

devops-github-rudderstack deleted the feat.struct-log branch April 9, 2024 01:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add structured logging #2968

feat: add structured logging #2968

sanpj2292 commented Jan 5, 2024 •

edited

Loading

coderabbitai bot commented Jan 5, 2024 •

edited

Loading

Auto Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

CodeRabbit Discord Community

codecov bot commented Jan 5, 2024 •

edited

Loading

koladilip commented Jan 6, 2024 •

edited

Loading

sanpj2292 commented Jan 8, 2024

sonarqubecloud bot commented Jan 8, 2024

koladilip Jan 8, 2024

sanpj2292 Jan 8, 2024

koladilip Jan 8, 2024

koladilip Jan 8, 2024

sanpj2292 Jan 8, 2024

devops-github-rudderstack commented Jan 29, 2024

feat: add structured logging #2968

feat: add structured logging #2968

Conversation

sanpj2292 commented Jan 5, 2024 • edited Loading

What are the changes introduced in this PR?

What is the related Linear task?

Please explain the objectives of your changes below

Any changes to existing capabilities/behaviour, mention the reason & what are the changes ?

Any new dependencies introduced with this change?

Any new generic utility introduced or modified. Please explain the changes.

Any technical or performance related pointers to consider with the change?

Developer checklist

Reviewer checklist

coderabbitai bot commented Jan 5, 2024 • edited Loading

Auto Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

CodeRabbit Discord Community

codecov bot commented Jan 5, 2024 • edited Loading

Codecov Report

koladilip commented Jan 6, 2024 • edited Loading

sanpj2292 commented Jan 8, 2024

sonarqubecloud bot commented Jan 8, 2024

Quality Gate passed

koladilip Jan 8, 2024

Choose a reason for hiding this comment

sanpj2292 Jan 8, 2024

Choose a reason for hiding this comment

koladilip Jan 8, 2024

Choose a reason for hiding this comment

koladilip Jan 8, 2024

Choose a reason for hiding this comment

sanpj2292 Jan 8, 2024

Choose a reason for hiding this comment

devops-github-rudderstack commented Jan 29, 2024

sanpj2292 commented Jan 5, 2024 •

edited

Loading

coderabbitai bot commented Jan 5, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)

codecov bot commented Jan 5, 2024 •

edited

Loading

koladilip commented Jan 6, 2024 •

edited

Loading