diff --git a/.gitbook.yaml b/.gitbook.yaml index f763fd123..6d105b687 100644 --- a/.gitbook.yaml +++ b/.gitbook.yaml @@ -56,6 +56,7 @@ redirects: output/chronicle: ./pipeline/outputs/chronicle.md output/cloudwatch: ./pipeline/outputs/cloudwatch.md output/datadog: ./pipeline/outputs/datadog.md + output/doris: ./pipeline/outputs/doris.md output/es: ./pipeline/outputs/elasticsearch.md output/fabric: ./pipeline/outputs/azure_kusto.md output/file: ./pipeline/outputs/file.md diff --git a/SUMMARY.md b/SUMMARY.md index 432b8fff3..b4d9fb6fb 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -170,6 +170,7 @@ * [Azure Logs Ingestion API](pipeline/outputs/azure_logs_ingestion.md) * [Counter](pipeline/outputs/counter.md) * [Datadog](pipeline/outputs/datadog.md) + * [Doris](pipeline/outputs/doris.md) * [Dynatrace](pipeline/outputs/dynatrace.md) * [Elasticsearch](pipeline/outputs/elasticsearch.md) * [File](pipeline/outputs/file.md) diff --git a/administration/configuring-fluent-bit/README.md b/administration/configuring-fluent-bit/README.md index 794d41950..38ad77146 100644 --- a/administration/configuring-fluent-bit/README.md +++ b/administration/configuring-fluent-bit/README.md @@ -100,6 +100,7 @@ Outputs bigquery Send events to BigQuery via streaming insert counter Records counter datadog Send events to DataDog HTTP Event Collector + doris Apache Doris es Elasticsearch exit Exit after a number of flushes (test purposes) file Generate log file diff --git a/administration/transport-security.md b/administration/transport-security.md index 4443cfd70..dca847495 100644 --- a/administration/transport-security.md +++ b/administration/transport-security.md @@ -32,6 +32,7 @@ The following **output** plugins can take advantage of the TLS feature: * [Azure Logs Ingestion API](../pipeline/outputs/azure_logs_ingestion.md) * [BigQuery](../pipeline/outputs/bigquery.md) * [Datadog](../pipeline/outputs/datadog.md) +* [Doris](../pipeline/outputs/doris.md) * [Elasticsearch](../pipeline/outputs/elasticsearch.md) * [Forward](../pipeline/outputs/forward.md) * [GELF](../pipeline/outputs/gelf.md) diff --git a/installation/sources/build-and-install.md b/installation/sources/build-and-install.md index 99674abf8..16c5bc1c0 100644 --- a/installation/sources/build-and-install.md +++ b/installation/sources/build-and-install.md @@ -188,6 +188,7 @@ The _output plugins_ gives the capacity to flush the information to some externa | [FLB\_OUT\_COUNTER](../../pipeline/outputs/counter.md) | Enable Counter output plugin | On | | [FLB\_OUT\_CLOUDWATCH\_LOGS](../../pipeline/outputs/cloudwatch.md) | Enable Amazon CloudWatch output plugin | On | | [FLB\_OUT\_DATADOG](../../pipeline/outputs/datadog.md) | Enable Datadog output plugin | On | +| [FLB\_OUT|_DORIS](../../pipeline/outputs/doris.md) | Enable Apache Doris output plugin | On | | [FLB\_OUT\_ES](../../pipeline/outputs/elasticsearch.md) | Enable [Elastic Search](http://www.elastic.co) output plugin | On | | [FLB\_OUT\_FILE](../../pipeline/outputs/file.md) | Enable File output plugin | On | | [FLB\_OUT\_KINESIS\_FIREHOSE](../../pipeline/outputs/firehose.md) | Enable Amazon Kinesis Data Firehose output plugin | On | diff --git a/pipeline/outputs/doris.md b/pipeline/outputs/doris.md new file mode 100644 index 000000000..e8d19e675 --- /dev/null +++ b/pipeline/outputs/doris.md @@ -0,0 +1,98 @@ +--- +description: Send logs to Apache Doris +--- + +# Apache Doris + +The **doris** output plugin lets you ingest your records into an +[Apache Doris](https://doris.apache.org) database. To use this plugin, you must have an +operational Doris service running in your environment. + +## Configuration Parameters + +| Key | Description | Default | +| :--- | :--- | :--- | +| `host` | HTTP address of the target Doris frontend (fe) or frontend (be). | `127.0.0.1` | +| `port` | HTTP port of the target Doris frontend (fe) or frontend (be). | `8030` | +| `user` | Username for Doris access. | _none_ | +| `password` | Password for Doris access. | _none_ | +| `database` | The target Doris database. | _none_ | +| `table` | The target Doris table. | _none_ | +| `label_prefix` | Label prefix of Doris stream load, the final generated Label is {label_prefix}\_{timestamp}\_{uuid}. | `fluentbit` | +| `time_key` | The name of the time key in the output record. | `date` | +| `header` | Headers of Doris stream load. Multiple headers can be set. See [Doris stream load](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual) for details. | _none_ | +| `log_request` | Whether to output Doris Stream Load request and response metadata in logs for troubleshooting. | `true` | +| `log_progress_interval` | The time interval in seconds to calculate and output the speed in the log. Set to 0 to disable this type of logging. | `10` | +| `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | + +### TLS / SSL + +The Doris output plugin supports TLS/SSL. See [TLS/SSL](../../administration/transport-security.md) +for more details about the supported properties and general configuration. + +## Get started + +To insert records into a Doris database, run the plugin from the command line or define a configuration file: + +### Command Line + +The Doris plugin can read the parameters from the command through the `-p` argument, + +as shown in the following example: + +```shell copy +fluent-bit -i cpu -t cpu -o doris \ + -m '*' \ + -p host=127.0.0.1 \ + -p port=8030 \ + -p user=admin \ + -p password=admin \ + -p database=d_fb \ + -p table=t_fb \ + -p header='columns date, cpu_p, log=cast(cpu_p as string)' +``` + +### Configuration File + +In your main configuration file, append the following `Input` and `Output` sections. + +{% tabs %} +{% tab title="fluent-bit.conf" %} +```python copy +[INPUT] + Name cpu + Tag cpu + +[OUTPUT] + name doris + match * + host 127.0.0.1 + port 8030 + user admin + password admin + database d_fb + table t_fb + header columns date, cpu_p, log=cast(cpu_p as string) +``` +{% endtab %} + +{% tab title="fluent-bit.yaml" %} +```yaml copy +pipeline: + inputs: + - name: cpu + tag: cpu + outputs: + - name: doris + match: '*' + host: 127.0.0.1 + port: 8030 + user: admin + password: admin + database: d_fb + table: t_fb + header: + - columns date, cpu_p, log=cast(cpu_p as string) +``` +{% endtab %} +{% endtabs %} diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index 18b032117..a49cb70e8 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -36,6 +36,7 @@ deprovisioning deprovisions Devo DogStatsD +Doris downsample downsampled downsamples