config-parameters.md

description
Various types of configuration parameters explained with their types, descriptions, and default values

Configuration Parameters

This document describes the various configuration parameters for the config.yaml file. You can fine-tune them to suit your application's needs.

{% hint style="info" %} These configuration parameters are also applicable for the config.toml file, in case you have an older RudderStack deployment. {% endhint %}

{% hint style="success" %} To best understand some of the terms covered in this guide, we recommend going through the RudderStack architecture first. {% endhint %}

Global Parameters

Parameter name	Type	Description	Default value
maxProcess	Int	Number of parallel threads used in server. Should be set to number of cores.	12
gwDBRetention	String	The time for which the events are stored in the Gateway database after they have been processed. Examples: 1h, 30m, 35s, etc.	0h
routerDBRetention	String	The time for which the events are stored in the Router database after they have been processed. Examples: 1h, 30m, 45s, etc.	0h
enableProcessor	Boolean	This variable enables or disables the Processor module. It will be set to false when running in Degraded mode.	true
enableRouter	Boolean	This variable enables or disables the Router module. Will be set to false when running in Degraded mode.	true
enableStats	Boolean	This variable enables or disables stats.	true

{% hint style="info" %} For more information on the Processor, Gateway, and Router modules of the RudderStack backend, refer to the Architecture. {% endhint %}

[Gateway]

Parameter name	Type	Description	Default value
webPort	Int	The port on which the RudderStack server runs.	8080
maxUserWebRequestWorkerProcess	Int	RudderStack spawns this specified number of processes to consume the events at a user level.	64
maxDBWriterProcess	Int	As requests come in to the Gateway and are batched, RudderStack runs the maxDBWriterProcess writers to send these batches to the database and the Configuration Backend.	256
CustomVal	String	For the creation of a job in the backend PostgreSQL database (jobsDB), the value of this variable will be assigned to the CustomVal column.	GW
maxBatchSize	Int	The batch size used in the Gateway. The requests are batched up to this size before writing to the database.	32
batchTimeout	String	In case the request batches do not meet the maxBatchSize, the batches are are sent to the database and the Configuration Backend in this interval.	20ms
maxReqSizeInKB	Int	An error message ("Request size exceeds max limit") is thrown for a particular request is when its size in KB crosses this value_._	4000
enableDedup	Boolean	Enables or disables deduplication of events. RudderStack uses message_id to de-dup. The duplicate events are dropped at the Gateway.	false
dedupWindow	String	Events with the same message_id within this timeframe are considered duplicate and are dropped.	3600s
enableRateLimit	Boolean	Rate limits the number of requests accepted by the Gateway. This is used for running the RudderStack-hosted service.	false

[SourceDebugger]

Parameter name	Type	Description	Default value
disableEventUploads	Boolean	Enables or disables the event schema upload.	false
maxBatchSize	Int	The maximum size of the live events batch sent to the Configuration Backend.	32
maxESQueueSize	Int	The maximum size of the live events queue in the memory.	1024
maxRetry	Int	The maximum number of attempts RudderStack makes to upload the request batches, in case of any errors.	3
batchTimeout	String	In case the request batches do not meet the maxBatchSize, the request batches are uploaded to Configuration Backend in this time interval.	2s
retrySleep	String	In case of errors while uploading the request batches, RudderStack waits for this time interval before retrying. This is done until the maxRetry limit is reached.	100ms

[JobsDB]

Parameter name	Type	Description	Default value
jobDoneMigrateThres	Float64	If (deletedJobsCount / totalJobsCount) > jobDoneMigrateThres, RudderStack migrates the table of the jobs that have been processed.	0.8
jobStatusMigrateThres	Float64	If (statusCount / totalCount) > jobStatusMigrateThres, RudderStack migrates the table of jobs that have been processed.	5
maxDSSize	Int	The maximum size of a table. If the maximum size is reached, RudderStack migrates all the jobs in that table.	100000
maxMigrateOnce	Int	The maximum number of tables that can be migrated together.	10
maxTableSizeInMB	Int	The maximum size of the tables in MB. The tables are migrated if they cross this limit.	300
migrateDSLoopSleepDuration	String	The time RudderStack waits before migrating the datasets.	30s
maxMigrateDSProbe	Int64	The maximum number of tables probed to find if they can be migrated.	10
addNewDSLoopSleepDuration	String	The time RudderStack waits before adding a dataset.	5s
backupCheckSleepDuration	String	RudderStack waits for this time interval before checking if a dataset needs backing up.	5s
enableBackup	Boolean	Enables or disables the backup. This is set to false in the Degraded mode.	true

[Router]

Parameter name	Type	Description	Default value
jobQueryBatchSize	Int	The size of a jobs batch to get from the database. This includes the retry list, processing list, unprocessed list, or executed list.	10000
updateStatusBatchSize	Int	The minimum size needed to update the status of a batch of jobs.	1000
readSleep	String	The time RudderStack waits before fetching the next jobs batch from the database, in case the length of unprocessed and retry list is 0.	1000ms
noOfWorkers	Int	RudderStack starts this number of workers to send events to the destinations.	64
noOfJobsPerChannel	Int	The number of jobs a channel in each worker can contain.	1000
maxSleep	String	The time to wait when the response status code is not 200. This is to give RudderStack some time before the next retry.	60s
minSleep	String	The time to sleep when the response status code is 200.	0s
maxStatusUpdateWait	String	The time to sleep before ending one round of stats collection.	5s
useTestSink	Boolean	Runs internal tests if set to true.	false
maxFailedCountForJob	Int	The maximum number of times a job can fail before it is marked as aborted.	8
guaranteeUserEventOrder	Boolean	RudderStack maintains the order of user events if set to true.	true
retryTimeWindow	String	The minimum retry window in case of 5XX, 429 errors.	180m
minRetryBackoff	String	The minimum time before the next retry in case of 5XX, 429 errors.	10s
maxRetryBackoff	String	The maximum allowed time between the errors in case of 5XX, 429 errors.	300s

[BatchRouter]

Parameter name	Type	Description	Default value
mainLoopSleep	String	The timeout while running the main loop.	2s
noOfWorkers	Int	The number of workers to batch jobs before deletion.	8
jobQueryBatchSize	Int	The number of events picked up from the batch router's database (Jobs DB) in each query.	100000
uploadFreq	String	The frequency with which the batch router dumps the events to the storage destinations.	30s
maxFailedCountForJob	Int	The maximum number of times a job can fail before marking it as aborted.	128

[Warehouse]

{% hint style="info" %} You can ignore these settings if you don't have any warehouse destinations configured in RudderStack. {% endhint %}

Parameter name	Type	Description	Default value
stagingFilesTable	String	Table name of the staging files.	wh_staging_files
loadFilesTable	String	Table name of load files.	wh_load_files
uploadsTable	String	Table name of uploads.	wh_uploads
schemasTable	String	Table name of schemas.	wh_schemas
uploadFreq	String	The frequency of the upload in seconds.	1800s
noOfWorkers	Int	Number of concurrent writes to the warehouse.	8
mainLoopSleep	String	The time RudderStack waits between multiple warehouse writes.	5s
stagingFilesBatchSize	Int	The batch size of the staging files.	960

[Processor]

Variable name	Type	Description	Default value
loopSleep	String	In case the length of the user jobs process queue is 0 or the unprocessed and retry list is empty, RudderStack sleeps for this specified time.	10ms
maxLoopSleep	String	Maximum loop sleep time for the Processor.	5000ms
dbReadBatchSize	Int	The total number of events to get as a batch from the database.	10000
transformBatchSize	Int	Batch size of the events added to the request queue before sending them for transformation.	100
userTransformBatchSize	Int	Batch size of the events added to request queue before sending them to the custom transformation server. Note: This is used only when a user transformation function is connected to a destination.	200
sessionThresholdEvents	Int	The minimum number of events needed to be process further.	20
sessionThreshold	String	The minimum time needed before a new session is created.	10s
maxChanSize	Int	The maximum channel size for the request and response queue in the transformer.	2048
processSessions	Boolean	If set to true, the status of the job in the database is updated.	false
numTransformWorker	Int	Specifies the number of Go transform workers.	8
maxRetry	Int	The maximum number of times a transformer retries hitting the API in case of an error.	30
retrySleep	String	The sleep time in case of a transformer error while hitting the API, before retrying.	100ms

[BackendConfig]

Variable name	Type	Description	Default value
pollInterval	String	The frequency of updating data from the Configuration Backend.	5s
configFromFile	Boolean	When set to true, RudderStack reads the backend workspace configuration from a JSON file instead of fetching it from the API.	false
configJSONPath	String	The path of the JSON file which contains the backend workspace configuration.	/etc/rudderstack /workspaceConfig.json

[RateLimit]

Variable name	Type	Description	Default value
eventLimit	Int64	The maximum number of events to be allowed in a time interval.	1000
rateLimitWindow	String	The rolling time interval used to limit the allowed number of events.	60m
noOfBucketsInWindow	Int32	The number of buckets rateLimitWindow is broken down into.	12

[Diagnostics]

Parameter name	Type	Description	Default value
enableDiagnosis	Boolean	RudderStack sends the server diagnostics report to the user. Disabling this will disable sending all diagnostics information.	true
gatewayTimePeriod	String	The time interval to send the Gateway requests report	60s
routerTimePeriod	String	The time interval to send the Router requests report.	60s
batchRouterTimePeriod	String	The time interval to send the Batch router requests report.	10m
enableServerStartMetric	Boolean	Sends the server start event.	true
enableConfigIdentifyMetric	Boolean	Sends the workspace config received event.	true
enableServerStartedMetric	Boolean	Sends the successful server start event.	true
enableConfigProcessedMetric	Boolean	Sends the workspace config details.	true
enableGatewayMetric	Boolean	Sends the Gateway request metrics.	true
enableRouterMetric	Boolean	Sends the Router request metrics.	true
enableBatchRouterMetric	Boolean	Sends the Batch Router request metrics.	true
enableDestinationFailuresMetric	Boolean	Sends the destination failures metrics.	true

Contact Us

For more information on any of the configuration parameters discussed in this guide, you can contact us or start a conversation on our Slack channel.