diff --git a/README.md b/README.md index 1402fb5d..ec74a51b 100644 --- a/README.md +++ b/README.md @@ -88,20 +88,20 @@ __Advanced Configuration Features:__ This is recommended for dev environments only. (Default: false) * `SKIP_SSL_VALIDATION_SPLUNK`: Skips SSL certificate validation for connection to Splunk. Secure communications will not check SSL certificates against a trusted certificate authority. (Default: false) This is recommended for dev environments only. -* `FIREHOSE_SUBSCRIPTION_ID`: Tags nozzle events with a Firehose subscription id. See https://docs.pivotal.io/pivotalcf/1-11/loggregator/log-ops-guide.html. (Default: splunk-firehose) +* `FIREHOSE_SUBSCRIPTION_ID`: Tags nozzle events with a Firehose subscription id. See https://docs.vmware.com/en/VMware-Tanzu-Application-Service/6.0/tas-for-vms/log-ops-guide.html. (Default: splunk-firehose) * `FIREHOSE_KEEP_ALIVE`: Keep alive duration for the Firehose consumer. (Default: 25s) * `ADD_APP_INFO`: Enrich raw data with app info. A comma separated list of app metadata (AppName,OrgName,OrgGuid,SpaceName,SpaceGuid). (Default: "") * `ADD_TAGS`: Add additional tags from envelope to splunk event. (Default: false) - (Please note: Adding tags / Enabling this feature may slightly impact the performance due to the increased event size) + (Please note: Enabling this feature may slightly impact the performance due to the increased event size) * `IGNORE_MISSING_APP`: If the application is missing, then stop repeatedly querying application info from Cloud Foundry. (Default: true) * `MISSING_APP_CACHE_INVALIDATE_TTL`: How frequently the missing app info cache invalidates (in s/m/h. For example, 3600s or 60m or 1h). (Default: 0s) (see below for more details) * `APP_CACHE_INVALIDATE_TTL`: How frequently the app info local cache invalidates (in s/m/h. For example, 3600s or 60m or 1h). (Default: 0s) (see below for more details) * `ORG_SPACE_CACHE_INVALIDATE_TTL`: How frequently the org and space cache invalidates (in s/m/h. For example, 3600s or 60m or 1h). (Default: 72h) * `APP_LIMITS`: Restrict to APP_LIMITS the most updated apps per request when populating the app metadata cache. keep it 0 to update all the apps. (Default: 0) * `BOLTDB_PATH`: Bolt database path. (Default: cache.db) -* `EVENTS`: A comma separated list of events to include. It is a required field. Possible values: ValueMetric,CounterEvent,Error,LogMessage,HttpStartStop,ContainerMetric. If no eventtype is selected, nozzle will automatically select LogMessage to keep the nozzle running. (Default: "ValueMetric,CounterEvent,ContainerMetric") +* `EVENTS`: A comma separated list of events to include. It is a required field. Possible values: ValueMetric,CounterEvent,Error,LogMessage,HttpStartStop,ContainerMetric. If no event type is selected, nozzle will automatically select LogMessage to keep the nozzle running. (Default: "ValueMetric,CounterEvent,ContainerMetric") * `EXTRA_FIELDS`: Extra fields to annotate your events with (format is key:value,key:value). (Default: "") -* `FLUSH_INTERVAL`: Time interval (in s/m/h. For example, 3600s or 60m or 1h) for flushing queue to Splunk regardless of CONSUMER_QUEUE_SIZE. Protects against stale events in low throughput systems. (Default: 5s) +* `FLUSH_INTERVAL`: Time interval (in s/m/h. For example, 3600s or 60m or 1h) for flushing queue to Splunk regardless of `CONSUMER_QUEUE_SIZE`. Protects against stale events in low throughput systems. (Default: 5s) * `CONSUMER_QUEUE_SIZE`: Sets the internal consumer queue buffer size. Events will be pushed to Splunk after queue is full. (Default: 10000) * `HEC_BATCH_SIZE`: Set the batch size for the events to push to HEC (Splunk HTTP Event Collector). (Default: 100) * `HEC_RETRIES`: Retry count for sending events to Splunk. After expiring, events will begin dropping causing data loss. (Default: 5) @@ -112,8 +112,8 @@ This is recommended for dev environments only. * `SPLUNK_METRIC_INDEX`: Index in which metric data will be ingested when monitoring module is enabled * `SELECTED_MONITORING_METRICS`: Name of the metrics that you want to monitor and add using comma seprated values. List of the metrics that are supported in the metrics modules are given below * `REFRESH_SPLUNK_CONNECTION`: If set to true, PCF will periodically refresh connection to Splunk (how often depends on KEEP_ALIVE_TIMER value). If set to false connection will be kept alive and reused. (Default: false) -* `KEEP_ALIVE_TIMER`: Time after which connection to Splunk will be refreshed, if REFRESH_SPLUNK_CONNECTION is set to true (in s/m/h. For example, 3600s or 60m or 1h). (Default: 30s) -* `MEMORY_BALLAST_SIZE`: Size of memory allocated to reduce GC cycles. Default is 0, Size should be less than the total memory. +* `KEEP_ALIVE_TIMER`: Time after which connection to Splunk will be refreshed, if `REFRESH_SPLUNK_CONNECTION` is set to true (in s/m/h. For example, 3600s or 60m or 1h). (Default: 30s) +* `MEMORY_BALLAST_SIZE`: Size of memory allocated to reduce GC cycles. Size should be less than the total memory. Default is 0. __About app cache params:__ diff --git a/splunknozzle/config.go b/splunknozzle/config.go index 32e823d3..7d80a68d 100644 --- a/splunknozzle/config.go +++ b/splunknozzle/config.go @@ -99,7 +99,7 @@ func NewConfigFromCmdFlags(version, branch, commit, buildos string) *Config { OverrideDefaultFromEnvar("SKIP_SSL_VALIDATION_CF").Default("false").BoolVar(&c.SkipSSLCF) kingpin.Flag("skip-ssl-validation-splunk", "Skip cert validation (for dev environments"). OverrideDefaultFromEnvar("SKIP_SSL_VALIDATION_SPLUNK").Default("false").BoolVar(&c.SkipSSLSplunk) - kingpin.Flag("subscription-id", "Id for the subscription."). + kingpin.Flag("firehose-subscription-id", "Id for the subscription."). OverrideDefaultFromEnvar("FIREHOSE_SUBSCRIPTION_ID").Default("splunk-firehose").StringVar(&c.SubscriptionID) kingpin.Flag("firehose-keep-alive", "Keep Alive duration for the firehose consumer"). OverrideDefaultFromEnvar("FIREHOSE_KEEP_ALIVE").Default("25s").DurationVar(&c.KeepAlive) diff --git a/tile/tile.yml b/tile/tile.yml index b71144e2..4dcbd957 100644 --- a/tile/tile.yml +++ b/tile/tile.yml @@ -1,7 +1,7 @@ --- name: splunk-nozzle label: Splunk Firehose Nozzle for Vmware Tanzu -description: Forward firehose logs & metrics to Splunk +description: Forward firehose logs and metrics to Splunk icon_file: icon.png apply_open_security_group: true # Apply open security group, default: false @@ -28,26 +28,27 @@ forms: - name: splunk_host type: string label: HTTP Event Collector Endpoint URL - description: HTTP Event Collector endpoint URL. + description: Splunk HTTP Event Collector endpoint URL. It is required parameter. - name: splunk_token type: secret label: HTTP Event Collector Token - description: Splunk HTTP Event Collector token. + description: Splunk HTTP Event Collector token. It is required parameter. - name: skip_ssl_validation_splunk type: boolean label: Skip SSL Validation default: false description: Skip SSL certificate validation for connection to Splunk. Secure communications will not check SSL certificates against a trusted Certificate Authority. Skipping SSL validation in production environment is not recommended. - - name: splunk_index + - name: splunk_index #It's a required parameter in config.go and doesn't have default, but here it has default value type: string label: Index - description: The name of the Splunk index that events will be sent to. WARNING:Setting an invalid index will cause events to be lost. + description: The name of the Splunk index that events will be sent to. It is required parameter. WARNING:Setting an invalid index will cause events to be lost. default: main - name: splunk_logging_index type: string label: Logging Index - description: The name of the Splunk index where logging events will be sent to. WARNING:Setting an invalid index will cause events to be lost. When not provided, all logging events will be forwarded to the splunk_index. + description: The name of the Splunk index where logging events will be sent to. When not provided, all logging events will be forwarded to the splunk_index. WARNING:Setting an invalid index will cause events to be lost. default: "" + optional: true - name: cf-config label: Cloud Foundry Settings @@ -56,15 +57,15 @@ forms: - name: api_endpoint type: string label: API Endpoint - description: Cloud Foundry API endpoint. + description: Cloud Foundry API endpoint. It is required parameter. - name: client_id type: string label: Client ID - description: CF UAA client ID + description: CF UAA Client ID. It is required parameter. - name: client_secret type: secret label: Client Secret - description: CF UAA client secret + description: CF UAA Client Secret. It is required parameter. - name: skip_ssl_validation_cf type: boolean label: Skip SSL Validation @@ -75,11 +76,8 @@ forms: label: Event Types default: [ - "HttpStartStop", - "LogMessage", "ValueMetric", "CounterEvent", - "Error", "ContainerMetric", ] options: @@ -104,13 +102,14 @@ forms: - name: scale_out_nozzle type: integer label: Scale Out Nozzle - description: Scale out Splunk nozzle. Recommendation:Run 2 or more nozzles for high availability. + description: Adds more Splunk nozzle components to spread the load. Recommendation:Run 2 or more nozzles for high availability. default: 2 - name: firehose_subscription_id type: string label: Firehose Subscription ID - description: Unique subscription ID to nozzle. Firehose balances across socket connections with the same ID. + description: Unique subscription ID to tag nozzle events. Firehose distribute data evenly across all instances if you use the same subscription ID on each nozzle instance. optional: true + default: splunk-firehose - name: add_app_info type: multi_select_options label: Add App Information @@ -126,12 +125,12 @@ forms: label: SpaceName - name: SpaceGuid label: SpaceGuid - description: Enriches event raw data with application metadata + description: Enriches raw data in events with application metadata - name: add_tags type: boolean label: Add Tags default: false - description: Add additional tags from envelope to Splunk Event. + description: Add additional tags from envelope to Splunk Event. WARNING:Enabling this feature may slightly impact the performance due to the increased event size. - name: extra_fields type: string label: Additional Fields @@ -140,7 +139,7 @@ forms: - name: hec_retries type: integer label: HEC Retries - default: 3 + default: 5 description: The retry count for sending events to the Splunk platform. Events not successfully sent after this number of retries will be dropped, causing data loss. - name: hec_batch_size type: integer @@ -151,7 +150,7 @@ forms: type: integer label: HEC Workers default: 8 - description: The number of concurrent workers sending data to Splunk HTTP Event Collector. Scale this number to your Splunk platform data collection capacity accordingly. + description: The number of concurrent workers sending data to Splunk HTTP Event Collector. Scaling this number will improve your Splunk platform data collection capacity. - name: consumer_queue_size type: integer label: Consumer Queue Size @@ -159,9 +158,9 @@ forms: description: The internal consumer queue buffer size. Events will be sent to your Splunk platform after queue is full. - name: status_monitor_interval type: string - label: Monitor Queue Pressure + label: Status Monitor Interval default: 0s - description: Time interval (in s/m/h. For example, 3600s or 60m or 1h) for monitoring memory queue pressure. Use to help with back-pressure insights. (Increases CPU load. Use for insights purposes only) Default is 0s (Disabled). + description: Time interval (in s/m/h. For example, 3600s or 60m or 1h) for Enabling Monitoring - metric data of insights with in the connectors. (Increases CPU load. Use for insights purposes only) Default is 0s (Disabled). - name: flush_interval type: string label: Flush Interval @@ -171,42 +170,42 @@ forms: type: string label: Missing App Cache Invalidate TTL default: 0s - description: Time interval (in s/m/h. For example, 3600s or 60m or 1h) between refreshing the missing app info cache. Set to 0s to maintain cache until nozzle restart. + description: Time interval (in s/m/h. For example, 3600s or 60m or 1h) between refreshing the cache of missed application information. Set to 0s to maintain cache until nozzle restart. - name: app_cache_invalidate_ttl type: string label: App Cache Invalidate TTL default: 0s - description: Time interval (in s/m/h. For example, 3600s or 60m or 1h) between refreshing the app info local cache. Set to 0s to only populate the cache during startup or restart of the nozzle. + description: Time interval (in s/m/h. For example, 3600s or 60m or 1h) between refreshing the application information local cache. Set to 0s to only populate the cache during startup or restart of the nozzle. - name: org_space_cache_invalidate_ttl type: string label: Org Space Cache Invalidate TTL default: 72h - description: Time interval (in s/m/h. For example, 3600s or 60m or 1h) between refreshing the org and space cache. Defaults to 72h. + description: Time interval (in s/m/h. For example, 3600s or 60m or 1h) between refreshing the organization and space cache. Defaults to 72h. - name: app_limits type: integer label: App Limits default: 0 - description: The number of apps for which metadata is gathered when refreshing the app metadata cache (order based on app creation date). Set to 0 to remove limit. + description: The number of applications for which metadata is gathered when refreshing the application metadata cache (order is based on application creation date). Set to 0 to remove limit and update all of the applications. - name: nozzle_memory type: string label: Nozzle Memory - description: Nozzle memory in MB. + description: Nozzle memory in MB. Default is 256 MB. default: 256M - name: firehose_keep_alive type: string label: Firehose Keep Alive - description: Keep alive duration (in sec) for the firehose consumer. + description: Keep alive duration (in seconds) for the firehose consumer. default: 25s - name: enable_event_tracing type: boolean label: Enable Event Tracing default: false - description: Enables data loss tracing. + description: Enables tracing of data loss. Splunk events will now contain a UUID, Splunk Nozzle Event Counts, and a Subscription ID for Splunk correlation searches. - name: ignore_missing_app type: boolean label: Ignore Missing App default: true - description: Do not trigger an app metadata refresh when encountering data from an app without info in the app metadata cache. + description: If the application is missing, then stop repeatedly querying for information from Cloud Foundry. - name: selected_monitoring_metrics type: multi_select_options label: Metric types @@ -256,27 +255,29 @@ forms: label: nozzle.cache.boltdb.hit - name: nozzle.cache.boltdb.miss label: nozzle.cache.boltdb.miss - description: Comma separated list of metrics that user want to visualize + description: Comma separated list of metrics that user want to monitor - name: splunk_metric_index type: string label: Splunk Metric Index - description: Index where Metric Data will be indexed + description: Index where Metric Data will be ingested when monitoring module is enabled optional: true - name: refresh_splunk_connection type: boolean label: Refresh Splunk connection description: Periodically refresh connection to Splunk optional: true + default: false - name: keep_alive_timer type: string label: Keep alive timer - description: Interval used to close and refresh connection to Splunk + description: Time after which connection to Splunk will be refreshed, if Refresh Splunk Connection is set to true (in s/m/h. For example, 3600s or 60m or 1h). optional: true + default: 30s - name: memory_ballast_size type: integer label: Memory Ballast Size default: 0 - description: Size of memory allocated to reduce GC cycles. Default is 0, Size should be less than the total memory. + description: Size of memory allocated to reduce GC cycles. Size should be less than the total memory. Default is 0. migration: | if (typeof properties['properties']['.properties.add_app_info']['value'] == "boolean") {