Merge remote-tracking branch 'origin/master' into Issue-32

Signed-off-by: Jesse Szwedko <[email protected]>
DataDog · May 8, 2024 · 59f32d6 · 59f32d6
2 parents c6efa4f + b485d1d
commit 59f32d6
Show file tree

Hide file tree

Showing 27 changed files with 1,239 additions and 235 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -0,0 +1,9 @@
+# See https://help.github.com/articles/about-codeowners/ for syntax
+# Rules are matched bottom-to-top, so one team can own subdirectories
+# and another the rest of the directory.
+
+
+*                     @DataDog/agent-processing-and-routing
+
+# Documentation
+*.md                  @DataDog/documentation
diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml
@@ -0,0 +1,44 @@
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: [ master ]
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'csharp' ]
+        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
+        # Learn more about CodeQL language support at https://git.io/codeql-language-support
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v3
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v2
+      with:
+        languages: ${{ matrix.language }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+        # queries: ./path/to/local/query, your-org/your-repo/queries@main
+
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@v2
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v2
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
@@ -0,0 +1,25 @@
+name: dotnet package
+
+on: [push]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        dotnet-version:
+          - '5.0.x'
+
+    steps:
+      - uses: actions/checkout@v3
+      - name: Setup .NET Core SDK ${{ matrix.dotnet-version }}
+        uses: actions/setup-dotnet@v3
+        with:
+          dotnet-version: ${{ matrix.dotnet-version }}
+      - name: Install dependencies
+        run: dotnet restore
+      - name: Build
+        run: dotnet build --configuration Release --no-restore
+      - name: Test
+        run: dotnet test --no-restore --verbosity normal
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,54 @@
 # Changelog
 
+## 0.5.2
+* Improve the exception thrown when a request fails or retry times out. Thanks @cdhunt
+
+## 0.5.1
+* Allow using custom HttpClient and Url. Thanks @tomasz-soltysik
+
+## 0.5.0
+* Split logs larger than the max message size
+* Make max message size configurable
+* Raise max payload size to `5MB`
+
+## 0.4.0
+* Add support for custom `ITextFormatter`.
+
+## 0.3.8
+* Update Serilog.Sinks.PeriodicBatching to `3.0.0`. Thanks @dominikus1993 
+
+## 0.3.7
+* Bump Newtonsoft.Json to `13.0.1`
+
+## 0.3.6
+
+* Remove unused UTF8 property #66. Thanks @jnyrup  
+* Support Datadog V2 intake API 
+* .NET 5 and .NET 6 support #68. Thanks @AndrewMD5  
+* Remove remaining traces of net6.0 #72. Thanks @jnyrup 
+
+## 0.3.5
+
+* Handle TCP disconnection leading to missing logs #64. Thanks @lokalan.
+
+## 0.3.4
+
+* Add more net framework targets to solve incompatibility issues with NuGet version resolving to "nearest" .net framework vs compiler targets
+
+## 0.3.3
+
+* Add a handler called when an exception occurs (#51).
+* Minor refactoring (#42). Thanks @SychevIgor
+
+## 0.3.2
+
+* Allow batch size, period and queue limit to be set (#41). Thanks @benfoster.
+
+## 0.3.1
+
+* Remove excessive selflog
+* Cleanup dependencies
+
 ## 0.3.0
 
 * HTTP is now the default sending method for logs (#24). This is a **breaking change** for

diff --git a/README.md b/README.md
@@ -1,16 +1,19 @@
 # Serilog.Sinks.Datadog.Logs
 
-A Serilog sink that send events and logs staight away to Datadog. By default the sink sends logs over HTTPS
+A Serilog sink that send events and logs straight away to Datadog. By default the sink sends logs over HTTPS
 
 **Package** - [Serilog.Sinks.Datadog.Logs](http://nuget.org/packages/serilog.sinks.datadog.logs)
-| **Platforms** - .NET 4.5.1, netstandard1.3, netstandard2.0
+| **Platforms** - .NET 4.5, .NET 4.6.1, .NET 4.7.2, netstandard1.3, netstandard2.0
 
 Note: For other .NET versions, ensure that the default TLS version used is `1.2`
 
 ```csharp
-var log = new LoggerConfiguration()
+using (var log = new LoggerConfiguration()
     .WriteTo.DatadogLogs("<API_KEY>")
-    .CreateLogger();
+    .CreateLogger())
+{
+    // Some code
+}
 ```
 
 By default the logs are forwarded to Datadog via **HTTPS** on port 443 to the US site.
@@ -24,7 +27,7 @@ You can also add the following properties (source, service, host, tags) to the S
 
 ```csharp
 var config = new DatadogConfiguration(url: "intake.logs.datadoghq.com", port: 10516, useSSL: true, useTCP: true);
-var log = new LoggerConfiguration()
+using (var log = new LoggerConfiguration()
     .WriteTo.DatadogLogs(
         "<API_KEY>",
         source: "<SOURCE_NAME>",
@@ -33,52 +36,67 @@ var log = new LoggerConfiguration()
         tags: new string[] {"<TAG_1>:<VALUE_1>", "<TAG_2>:<VALUE_2>"},
         configuration: config
     )
-    .CreateLogger();
+    .CreateLogger())
+{
+    // Some code
+}
 ```
 
 ## Example
 
 Sending the following log:
 
 ```csharp
-var log = new LoggerConfiguration()
+using (var log = new LoggerConfiguration()
     .WriteTo.DatadogLogs("<API_KEY>")
-    .CreateLogger();
+    .CreateLogger())
+{    
+    // An example
+    var position = new { Latitude = 25, Longitude = 134 };
+    var elapsedMs = 34;
 
+    log.Information("Processed {@Position} in {Elapsed:000} ms.", position, elapsedMs);
+}
+```
+or
+```csharp
+Log.Logger = new LoggerConfiguration()
+    .WriteTo.DatadogLogs("<API_KEY>")
+    .CreateLogger();
+
 // An example
 var position = new { Latitude = 25, Longitude = 134 };
 var elapsedMs = 34;
 
-log.Information("Processed {@Position} in {Elapsed:000} ms.", position, elapsedMs);
+Log.Information("Processed {@Position} in {Elapsed:000} ms.", position, elapsedMs);
+Log.CloseAndFlush();
 ```
 
 In the platform, the log looks like as the following JSON Object:
 
 ```json
 {
+    "message": "Processed { Latitude: 25, Longitude: 134 } in 034 ms.",
     "MessageTemplate": "Processed {@Position} in {Elapsed:000} ms.",
-    "Level": "Information",
-    "Timestamp": "2016-09-02T15:02:29.648Z",
-    "Renderings": {
-        "Elapsed": [{
-            "Format": "000",
-            "Rendering": "034"
-        }]
-    },
+    "timestamp": "2022-11-23T09:48:56.0262350-05:00",
+    "level": "Information",
     "Properties": {
         "Position": {
             "Latitude": 25,
             "Longitude": 134
         },
         "Elapsed": 34
-    }
+    },
+    "Renderings": [
+        "034"
+    ]
 }
 ```
 
 ## Configuration from `appsettings.json`
 
 Since 0.2.0, you can configure the Datadog sink by using an `appsettings.json` file with
-the [Serilog.Setting.Configuration](https://github.com/serilog/serilog-settings-configuration) package.
+the [Serilog.Settings.Configuration](https://github.com/serilog/serilog-settings-configuration) package.
 
 In the `"Serilog.WriteTo"` array, add an entry for `DatadogLogs`. An example is shown below:
 
@@ -95,6 +113,12 @@ In the `"Serilog.WriteTo"` array, add an entry for `DatadogLogs`. An example is
         "source": "<SOURCE_NAME>",
         "host": "<HOST_NAME>",
         "tags": ["<TAG_1>:<VALUE_1>", "<TAG_2>:<VALUE_2>"],
+        "configuration" : {
+          "url": "intake.logs.datadoghq.com", 
+          "port": 10516, 
+          "useSSL": true, 
+          "useTCP": true
+        }
       }
     }
   ],
@@ -105,6 +129,52 @@ In the `"Serilog.WriteTo"` array, add an entry for `DatadogLogs`. An example is
 }
 ```
 
+**NOTE:** the `configuration` section is optional so that you may override the defaults. 
+
+## Using a custom log formatter
+You can implement a [custom `ITextFormatter` ](https://github.com/serilog/serilog/blob/dev/src/Serilog/Formatting/ITextFormatter.cs)and pass it to the sink to change the format of your logs. This is useful if you want to add/remove/modify fields from the final JSON payload, or emit non-json logs to Datadog. 
+
+There are several options for implementing custom formatters. The easiest way is to use [Serilog-expressions](https://github.com/serilog/serilog-expressions). Below is an example of a Serilog-expression `ITextFormatter` that drops the `MessageTemplate` field: 
+
+```C#
+public class DatadogJsonNoTemplateFormatter: ExpressionTemplate
+{
+    public DatadogJsonNoTemplateFormatter() : base(@"{ {
+        Timestamp: @t,
+        level: @l,
+        message: @m, 
+        Properties: {..@p},
+        Renderings: @r}
+    }") {}
+}
+```
+
+If you cannot use Serilog-expressions due to framework compatibility - you can implement your own with `JsonValueFormatter` or a default implementation ex: [serilog-formatting-compact](https://github.com/serilog/serilog-formatting-compact)
+
+
+## Support Configuration Options
+
+[`DatadogLogs`](https://github.com/DataDog/serilog-sinks-datadog-logs/blob/master/src/Serilog.Sinks.Datadog.Logs/Configuration/Extensions/System.Configuration/LoggerConfigurationDatadogLogsExtensions.cs#L40) supports the following arguments:
+
+| argument                 | Type                   | Description                                             |
+| ------------------------ | ---------------------- | ------------------------------------------------------- |
+| `apiKey`                 | `string`               | Your Datadog API key.                                   |
+| `source`                 | `string`               | The integration name.                                   |
+| `service`                | `string`               | The service name.                                       |
+| `host`                   | `string`               | The host name.                                          |
+| `tags`                   | `string[]`             | Custom tags.                                            |
+| `configuration`          | `DatadogConfiguration` | The Datadog logs client configuration.                  |
+| `logLevel`               | `LogEventLevel`        | The minimum log level for the sink.                     |
+| `batchSizeLimit`         | `int`                  | The maximum number of events to emit in a single batch. |
+| `batchPeriod`            | `TimeSpan`             | The time to wait before emitting a new event batch.     |
+| `queueLimit`             | `int`                  | Maximum number of events to hold in the sink's internal queue, or `null` for an unbounded queue. The default is `10000` |
+| `exceptionHandler`       | `Action<Exception>`    | This function is called when an exception occurs when using `DatadogConfiguration.UseTCP=false` (the default configuration). |
+| `detectTCPDisconnection` | `bool`                 | Detect when the TCP connection is lost and recreate a new connection. |
+| `formatter`              | `ITextFormatter`       | A custom formatter implementation to change the format of the logs |
+| `maxMessageSize`         | `int`                  | The maximum size in bytes of a message before it is split into chunks |
+
+**NOTE:** if `maxMessageSize` [exceeds the documented API limit of 1MB](https://docs.datadoghq.com/api/latest/logs/) - any payloads larger than 1MB will be dropped by the intake. 
+
 ## How to build the NuGet package
 
 Bump the version in `src/Serilog.Sinks.Datadog.Logs.csproj` and merge your branch
@@ -118,3 +188,9 @@ msbuild Serilog.Sinks.Datadog.Logs.sln /t:pack /p:Configuration=Release
 ```
 
 You can find the `.nupkg` file at `src/Serilog.Sinks.Datadog.Logs/bin/Release/Serilog.Sinks.Datadog.Logs.<version>.nupkg`
+
+## Serilog.Sinks.Async	
+As `Serilog.Sinks.Datadog.Logs` implements [Serilog.Sinks.PeriodicBatching](https://github.com/serilog/serilog-sinks-periodicbatching), using [Serilog.Sinks.Async](https://github.com/serilog/serilog-sinks-async) is not recommended.	
+
+From [Serilog.Sinks.Async documentation](https://github.com/serilog/serilog-sinks-async#serilogsinksasync---):	
+> Note: many of the network-based sinks (CouchDB, Elasticsearch, MongoDB, Seq, Splunk...) already perform asynchronous batching natively and do not benefit from this wrapper.
diff --git a/Serilog.Sinks.Datadog.Logs.sln b/Serilog.Sinks.Datadog.Logs.sln
@@ -1,10 +1,15 @@
+
 Microsoft Visual Studio Solution File, Format Version 12.00
-# Visual Studio Version 16
-VisualStudioVersion = 16.0.29418.71
+# Visual Studio 15
+VisualStudioVersion = 15.0.26124.0
 MinimumVisualStudioVersion = 15.0.26124.0
 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{D0A5A0BA-56F9-4C56-8785-8C9F23A3B411}"
 EndProject
-Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Serilog.Sinks.Datadog.Logs", "src\Serilog.Sinks.Datadog.Logs\Serilog.Sinks.Datadog.Logs.csproj", "{897105F7-58A7-4849-98A8-EB8EDD450E47}"
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Serilog.Sinks.Datadog.Logs", "src\Serilog.Sinks.Datadog.Logs\Serilog.Sinks.Datadog.Logs.csproj", "{897105F7-58A7-4849-98A8-EB8EDD450E47}"
+EndProject
+Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tests", "tests", "{CDDB8CB1-42E1-4722-8D83-4BB298ED5D16}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Serilog.Sinks.Datadog.Logs.Tests", "tests\Serilog.Sinks.Datadog.Logs.Tests\Serilog.Sinks.Datadog.Logs.Tests.csproj", "{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}"
 EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
@@ -15,27 +20,37 @@ Global
 		Release|x64 = Release|x64
 		Release|x86 = Release|x86
 	EndGlobalSection
+	GlobalSection(SolutionProperties) = preSolution
+		HideSolutionNode = FALSE
+	EndGlobalSection
 	GlobalSection(ProjectConfigurationPlatforms) = postSolution
 		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
 		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Debug|Any CPU.Build.0 = Debug|Any CPU
-		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Debug|x64.ActiveCfg = Debug|Any CPU
-		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Debug|x64.Build.0 = Debug|Any CPU
-		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Debug|x86.ActiveCfg = Debug|Any CPU
-		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Debug|x86.Build.0 = Debug|Any CPU
+		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Debug|x64.ActiveCfg = Debug|x64
+		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Debug|x64.Build.0 = Debug|x64
+		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Debug|x86.ActiveCfg = Debug|x86
+		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Debug|x86.Build.0 = Debug|x86
 		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Release|Any CPU.Build.0 = Release|Any CPU
-		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Release|x64.ActiveCfg = Release|Any CPU
-		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Release|x64.Build.0 = Release|Any CPU
-		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Release|x86.ActiveCfg = Release|Any CPU
-		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Release|x86.Build.0 = Release|Any CPU
-	EndGlobalSection
-	GlobalSection(SolutionProperties) = preSolution
-		HideSolutionNode = FALSE
+		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Release|x64.ActiveCfg = Release|x64
+		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Release|x64.Build.0 = Release|x64
+		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Release|x86.ActiveCfg = Release|x86
+		{897105F7-58A7-4849-98A8-EB8EDD450E47}.Release|x86.Build.0 = Release|x86
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Debug|x64.ActiveCfg = Debug|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Debug|x64.Build.0 = Debug|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Debug|x86.Build.0 = Debug|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Release|Any CPU.Build.0 = Release|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Release|x64.ActiveCfg = Release|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Release|x64.Build.0 = Release|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Release|x86.ActiveCfg = Release|Any CPU
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C}.Release|x86.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(NestedProjects) = preSolution
 		{897105F7-58A7-4849-98A8-EB8EDD450E47} = {D0A5A0BA-56F9-4C56-8785-8C9F23A3B411}
-	EndGlobalSection
-	GlobalSection(ExtensibilityGlobals) = postSolution
-		SolutionGuid = {DAA72944-174B-45F4-92A5-FCEB78A03E00}
+		{54D06DEA-0F0B-4D2A-ADDA-2F5221EE036C} = {CDDB8CB1-42E1-4722-8D83-4BB298ED5D16}
 	EndGlobalSection
 EndGlobal