[Monitor Ingestion] Beta 1 library for monitor logs ingestion (#22394)

rush.json

@@ -511,6 +511,11 @@
       "projectFolder": "sdk/monitor/monitor-query",
       "versionPolicyName": "client"
     },
+    {
+      "packageName": "@azure/monitor-ingestion",
+      "projectFolder": "sdk/monitor/monitor-ingestion",
+      "versionPolicyName": "client"
+    },
     {
       "packageName": "@azure/dev-tool",
       "projectFolder": "common/tools/dev-tool",

sdk/monitor/ci.yml

@@ -39,3 +39,5 @@ extends:
         safeName: azuremonitoropentelemetryexporter
       - name: azure-monitor-query
         safeName: azuremonitorquery
+      - name: azure-monitor-ingestion
+        safeName: azuremonitoringestion

sdk/monitor/monitor-ingestion/.nycrc

@@ -0,0 +1,19 @@
+{
+    "include": [
+      "dist-esm/src/**/*.js"
+    ],
+    "exclude": [
+      "**/*.d.ts",
+      "dist-esm/src/generated/*"
+    ],
+    "reporter": [
+      "text-summary",
+      "html",
+      "cobertura"
+    ],
+    "exclude-after-remap": false,
+    "sourceMap": true,
+    "produce-source-map": true,
+    "instrument": true,
+    "all": true
+  }

sdk/monitor/monitor-ingestion/CHANGELOG.md

@@ -0,0 +1,7 @@
+# Release History
+
+## 1.0.0-beta.1 (2022-07-07)
+
+### Features Added
+
+- First beta release for the @azure/monitor-ingestion library.

sdk/monitor/monitor-ingestion/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2022 Microsoft
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sdk/monitor/monitor-ingestion/README.md

@@ -0,0 +1,205 @@
+# Azure Monitor Ingestion client library for JS
+
+The Azure Monitor Ingestion client library is used to send custom logs to [Azure Monitor][azure_monitor_overview].
+
+This library allows you to send data from virtually any source to supported built-in tables or to custom tables that you create in Log Analytics workspace. You can even extend the schema of built-in tables with custom columns.
+
+**Resources:**
+* [Source code](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/monitor/monitor-ingestion/src)
+* [Package (NPM)](https://www.npmjs.com/)
+* [Service documentation][azure_monitor_overview]
+* [Change log](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/monitor/monitor-ingestion/CHANGELOG.md)
+
+## Getting started
+
+### Prerequisites
+
+- An [Azure subscription](https://azure.microsoft.com/free)
+- A [Data Collection Endpoint](https://docs.microsoft.com/azure/azure-monitor/essentials/data-collection-endpoint-overview)
+- A [Data Collection Rule](https://docs.microsoft.com/azure/azure-monitor/essentials/data-collection-rule-overview)
+- A [Log Analytics workspace](https://docs.microsoft.com/azure/azure-monitor/logs/log-analytics-workspace-overview)
+
+### Install the package
+
+Install the Azure Monitor Ingestion client library for JS with [npm](https://www.npmjs.com/):
+
+```bash
+npm install @azure/monitor-ingestion
+```
+
+### Authenticate the client
+
+An authenticated client is required to ingest data. To authenticate, create an instance of a [TokenCredential](https://docs.microsoft.com/javascript/api/@azure/core-auth/tokencredential?view=azure-node-latest) class (see [@azure/identity](https://www.npmjs.com/package/@azure/identity) for `DefaultAzureCredential` and other `TokenCredential` implementations). Pass it to the constructor of your client class.
+
+To authenticate, the following example uses `DefaultAzureCredential` from the [@azure/identity](https://www.npmjs.com/package/@azure/identity) package:
+
+```ts
+import { DefaultAzureCredential } from "@azure/identity";
+import { LogsIngestionClient } from "@azure/monitor-ingestion";
+
+import * as dotenv from "dotenv";
+dotenv.config();
+
+const logsIngestionEndpoint = process.env.LOGS_INGESTION_ENDPOINT || "logs_ingestion_endpoint";
+
+const credential = new DefaultAzureCredential();
+const logsIngestionClient = new LogsIngestionClient(logsIngestionEndpoint, credential);
+```
+
+## Key concepts
+
+### Data Collection Endpoint
+
+Data Collection Endpoints (DCEs) allow you to uniquely configure ingestion settings for Azure Monitor. [This 
+article][data_collection_endpoint] provides an overview of data collection endpoints including their contents and 
+structure and how you can create and work with them.
+
+### Data Collection Rule
+
+Data collection rules (DCR) define data collected by Azure Monitor and specify how and where that data should be sent or
+stored. The REST API call must specify a DCR to use. A single DCE can support multiple DCRs, so you can specify a
+different DCR for different sources and target tables.
+
+The DCR must understand the structure of the input data and the structure of the target table. If the two don't match,
+it can use a transformation to convert the source data to match the target table. You may also use the transform to
+filter source data and perform any other calculations or conversions.
+
+For more details, refer to [Data collection rules in Azure Monitor][data_collection_rule].
+
+### Log Analytics workspace tables
+
+Custom logs can send data to any custom table that you create and to certain built-in tables in your Log Analytics 
+workspace. The target table must exist before you can send data to it. The following built-in tables are currently supported:
+
+- [CommonSecurityLog](https://docs.microsoft.com/azure/azure-monitor/reference/tables/commonsecuritylog)
+- [SecurityEvents](https://docs.microsoft.com/azure/azure-monitor/reference/tables/securityevent)
+- [Syslog](https://docs.microsoft.com/azure/azure-monitor/reference/tables/syslog)
+- [WindowsEvents](https://docs.microsoft.com/azure/azure-monitor/reference/tables/windowsevent)
+
+## Examples
+
+- [Upload custom logs](#upload-custom-logs)
+- [Verify logs](#verify-logs)
+
+You can familiarize yourself with different APIs using [Samples](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/monitor/Azure.Monitor.Ingestion/samples).
+
+### Upload custom logs
+
+You can create a client and call the client's `Upload` method. Take note of the data ingestion [limits](https://docs.microsoft.com/azure/azure-monitor/service-limits#custom-logs).
+
+```js
+const { DefaultAzureCredential } = require("@azure/identity");
+const { LogsIngestionClient } = require("@azure/monitor-ingestion");
+
+require("dotenv").config();
+
+async function main() {
+  const logsIngestionEndpoint = process.env.LOGS_INGESTION_ENDPOINT || "logs_ingestion_endpoint";
+  const ruleId = process.env.DATA_COLLECTION_RULE_ID || "data_collection_rule_id";
+  const streamName = process.env.STREAM_NAME || "data_stream_name";
+  const credential = new DefaultAzureCredential();
+  const client = new LogsIngestionClient(logsIngestionEndpoint, credential);
+  const logs = [
+    {
+      Time: "2021-12-08T23:51:14.1104269Z",
+      Computer: "Computer1",
+      AdditionalContext: "context-2",
+    },
+    {
+      Time: "2021-12-08T23:51:14.1104269Z",
+      Computer: "Computer2",
+      AdditionalContext: "context",
+    },
+  ];
+  const result = await client.upload(ruleId, streamName, logs);
+  if (result.uploadStatus !== "Success") {
+    console.log("Some logs have failed to complete ingestion. Upload status=", result.uploadStatus);
+    for (const errors of result.errors) {
+      console.log(`Error - ${JSON.stringify(errors.responseError)}`);
+      console.log(`Log - ${JSON.stringify(errors.failedLogs)}`);
+    }
+  }
+}
+main().catch((err) => {
+  console.error("The sample encountered an error:", err);
+  process.exit(1);
+});
+
+module.exports = { main };
+```
+### Verify logs
+
+You can verify that your data has been uploaded correctly by using the [@azure/monitor-query](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/monitor/monitor-query/README.md#install-the-package) library. Run the [Upload custom logs](#upload-custom-logs) sample first before verifying the logs. 
+
+```js
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+/**
+ * @summary Demonstrates how to run query against a Log Analytics workspace to verify if the logs were uploaded
+ */
+
+const { DefaultAzureCredential } = require("@azure/identity");
+const { LogsQueryClient } = require("@azure/monitor-query");
+
+const monitorWorkspaceId = process.env.MONITOR_WORKSPACE_ID || "workspace_id";
+const tableName = process.env.TABLE_NAME || "table_name";
+require("dotenv").config();
+
+async function main() {
+  const credential = new DefaultAzureCredential();
+  const logsQueryClient = new LogsQueryClient(credential);
+  const queriesBatch = [
+    {
+      workspaceId: monitorWorkspaceId,
+      query: tableName + " | count;",
+      timespan: { duration: "P1D" },
+    },
+  ];
+
+  const result = await logsQueryClient.queryBatch(queriesBatch);
+  if (result[0].status === "Success") {
+    console.log("Table entry count: ", JSON.stringify(result[0].tables));
+  } else {
+    console.log(
+      `Some error encountered while retrieving the count. Status = ${result[0].status}`,
+      JSON.stringify(result[0])
+    );
+  }
+}
+
+main().catch((err) => {
+  console.error("The sample encountered an error:", err);
+  process.exit(1);
+});
+
+module.exports = { main };
+
+```
+## Troubleshooting
+
+### Logging
+
+Enabling logging may help uncover useful information about failures. To see a log of HTTP requests and responses, set the `AZURE_LOG_LEVEL` environment variable to `info`. Alternatively, logging can be enabled at runtime by calling `setLogLevel` in the `@azure/logger`:
+
+```ts
+import { setLogLevel } from "@azure/logger";
+
+setLogLevel("info");
+```
+
+For detailed instructions on how to enable logs, see the [@azure/logger package docs](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/core/logger).
+
+## Next steps
+To learn more about Azure Monitor, see the [Azure Monitor service documentation][azure_monitor_overview].
+
+## Contributing
+
+If you'd like to contribute to this library, please read the [contributing guide](https://github.com/Azure/azure-sdk-for-js/blob/main/CONTRIBUTING.md) to learn more about how to build and test the code.
+
+<!-- LINKS -->
+[azure_monitor_overview]: https://docs.microsoft.com/azure/azure-monitor/overview
+[data_collection_endpoint]: https://docs.microsoft.com/azure/azure-monitor/essentials/data-collection-endpoint-overview
+[data_collection_rule]: https://docs.microsoft.com/azure/azure-monitor/essentials/data-collection-rule-overview
+
+![Impressions](https://azure-sdk-impressions.azurewebsites.net/api/impressions/azure-sdk-for-js/sdk/monitor/monitor-ingestion/README.png)