From 58ed1a8e0a2c4afc76a7d405761d127c041c9c90 Mon Sep 17 00:00:00 2001 From: "shaoyue.chen" Date: Mon, 27 May 2024 10:42:49 +0800 Subject: [PATCH] auto generated Signed-off-by: shaoyue.chen --- .../reference/sys_config/configure_common.md | 583 ++++++- .../sys_config/configure_datacoord.md | 900 +++++++++-- .../sys_config/configure_datanode.md | 553 ++++++- .../en/reference/sys_config/configure_etcd.md | 287 +++- site/en/reference/sys_config/configure_gpu.md | 59 + .../en/reference/sys_config/configure_grpc.md | 228 +++ .../sys_config/configure_indexcoord.md | 102 ++ .../sys_config/configure_indexnode.md | 128 +- .../reference/sys_config/configure_kafka.md | 145 -- .../sys_config/configure_knowhere.md | 36 - .../sys_config/configure_localstorage.md | 36 +- site/en/reference/sys_config/configure_log.md | 75 +- .../sys_config/configure_messagechannel.md | 433 ----- .../sys_config/configure_metastore.md | 30 + .../reference/sys_config/configure_minio.md | 249 ++- site/en/reference/sys_config/configure_mq.md | 115 ++ .../en/reference/sys_config/configure_nats.md | 266 --- .../reference/sys_config/configure_natsmq.md | 279 ++++ .../reference/sys_config/configure_proxy.md | 550 +++++-- .../reference/sys_config/configure_pulsar.md | 95 +- .../sys_config/configure_querycoord.md | 834 +++++++++- .../sys_config/configure_querynode.md | 756 ++++++++- .../sys_config/configure_quota_limits.md | 736 --------- .../sys_config/configure_quotaandlimits.md | 1428 +++++++++++++++++ .../reference/sys_config/configure_rocksmq.md | 93 +- .../sys_config/configure_rootcoord.md | 154 +- .../en/reference/sys_config/configure_tikv.md | 205 +++ site/en/reference/sys_config/configure_tls.md | 65 + .../reference/sys_config/configure_trace.md | 108 ++ .../sys_config/system_configuration.md | 305 ++-- 30 files changed, 7282 insertions(+), 2551 deletions(-) create mode 100644 site/en/reference/sys_config/configure_gpu.md create mode 100644 site/en/reference/sys_config/configure_grpc.md create mode 100644 site/en/reference/sys_config/configure_indexcoord.md delete mode 100644 site/en/reference/sys_config/configure_kafka.md delete mode 100644 site/en/reference/sys_config/configure_knowhere.md delete mode 100644 site/en/reference/sys_config/configure_messagechannel.md create mode 100644 site/en/reference/sys_config/configure_metastore.md create mode 100644 site/en/reference/sys_config/configure_mq.md delete mode 100644 site/en/reference/sys_config/configure_nats.md create mode 100644 site/en/reference/sys_config/configure_natsmq.md delete mode 100644 site/en/reference/sys_config/configure_quota_limits.md create mode 100644 site/en/reference/sys_config/configure_quotaandlimits.md create mode 100644 site/en/reference/sys_config/configure_tikv.md create mode 100644 site/en/reference/sys_config/configure_tls.md create mode 100644 site/en/reference/sys_config/configure_trace.md diff --git a/site/en/reference/sys_config/configure_common.md b/site/en/reference/sys_config/configure_common.md index 88608b9fc..b474fa3ca 100644 --- a/site/en/reference/sys_config/configure_common.md +++ b/site/en/reference/sys_config/configure_common.md @@ -2,15 +2,12 @@ id: configure_common.md related_key: configure group: system_configuration.md -summary: Learn how to configure common parameters of Milvus. -title: Common Configurations +summary: Learn how to configure common for Milvus. --- -# Common Configurations +# common-related Configurations -This topic introduces the common configurations of Milvus. -Under this section, you can configure the default names of partition and index, and the Time Travel (data retention) span of Milvus. ## `common.defaultPartitionName` @@ -23,15 +20,93 @@ Under this section, you can configure the default names of partition and index, - Name of the default partition when a collection is created. - "_default" + +
  • default partition name for a collection
    • + _default + ## `common.defaultIndexName` - +
    + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • default index name
    		• _default_idx
    + + +## `common.entityExpiration` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Entity expiration in seconds, CAUTION -1 means never expire
    		• -1
    + + +## `common.indexSliceSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB
    		• 16
    + + +## `common.threadCoreCoefficient.highPriority` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • This parameter specify how many times the number of threads is the number of cores in high priority pool
    		• 10
    + + +## `common.threadCoreCoefficient.middlePriority` + + @@ -40,15 +115,17 @@ Under this section, you can configure the default names of partition and index, - - + +
    Description
    Name of the index when it is created with name unspecified."_default_idx" +
  • This parameter specify how many times the number of threads is the number of cores in middle priority pool
    		• 5
    -## `common.retentionDuration` - +## `common.threadCoreCoefficient.lowPriority` + +
    @@ -58,14 +135,422 @@ Under this section, you can configure the default names of partition and index, - +
  • This parameter specify how many times the number of threads is the number of cores in low priority pool
    • + + + +
    Description
    -
  • The retention duration of the deleted data allowed for Time Travel.
    • -
  • Unit: Second
    • -     		432000 1
    + + +## `common.buildIndexThreadPoolRatio` + + + + + + + + + + + +
    DescriptionDefault Value
    0.75
    + +## `common.DiskIndex.MaxDegree` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    56
    + + +## `common.DiskIndex.SearchListSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    100
    + + +## `common.DiskIndex.PQCodeBudgetGBRatio` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    0.125
    + + +## `common.DiskIndex.BuildNumThreadsRatio` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    1
    + + +## `common.DiskIndex.SearchCacheBudgetGBRatio` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    0.1
    + + +## `common.DiskIndex.LoadNumThreadRatio` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    8
    + + +## `common.DiskIndex.BeamWidthRatio` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    4
    + + +## `common.gracefulTime` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • milliseconds. it represents the interval (in ms) by which the request arrival time needs to be subtracted in the case of Bounded Consistency.
    		• 5000
    + + +## `common.gracefulStopTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • seconds. it will force quit the server if the graceful stop process is not completed during this time.
    		• 1800
    + + +## `common.storageType` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • please adjust in embedded Milvus: local, available values are [local, remote, opendal], value minio is deprecated, use remote instead
    		• remote
    + + +## `common.simdType` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Default value: auto
    • +
  • Valid values: [auto, avx512, avx2, avx, sse4_2]
    • +
  • This configuration is only used by querynode and indexnode, it selects CPU instruction set for Searching and Index-building.
    		• auto
    + + +## `common.security.authorizationEnabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `common.security.superUsers` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The superusers will ignore some system check processes,
    • +
  • like the old password verification when updating the credential
    • + + +## `common.security.tlsMode` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    0
    + + +## `common.session.ttl` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • ttl value when session granting a lease to register service
    		• 30
    + + +## `common.session.retryTimes` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • retry times when session sending etcd requests
    		• 30
    + + +## `common.locks.metrics.enable` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • whether gather statistics for metrics locks
    		• false
    + + +## `common.locks.threshold.info` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • minimum milliseconds for printing durations in info level
    		• 500
    + + +## `common.locks.threshold.warn` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • minimum milliseconds for printing durations in warn level
    		• 1000
    + + +## `common.storage.scheme` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    s3
    + + +## `common.storage.enablev2` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + ## `common.ttMsgEnabled` @@ -77,8 +562,68 @@ Under this section, you can configure the default names of partition and index, - - + + + + +
    Whether to disable the internal time messaging mechanism for the system. If disabled (set to `false`), the system will not allow DML operations, including insertion, deletion, queries, and searches. This helps Milvus-CDC synchronize incremental data.false +
  • Whether the instance disable sending ts messages
    		• true
    + + +## `common.traceLogMode` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • trace request info
    		• 0
    + + +## `common.bloomFilterSize` + + + + + + + + + + + + -
    DescriptionDefault Value
    +
  • bloom filter initial size
    		• 100000
    \ No newline at end of file + + + +## `common.maxBloomFalsePositive` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • max false positive rate for bloom filter
    		• 0.001
    + + diff --git a/site/en/reference/sys_config/configure_datacoord.md b/site/en/reference/sys_config/configure_datacoord.md index 61c69e220..743236d95 100644 --- a/site/en/reference/sys_config/configure_datacoord.md +++ b/site/en/reference/sys_config/configure_datacoord.md @@ -2,22 +2,510 @@ id: configure_datacoord.md related_key: configure group: system_configuration.md -summary: Learn how to configure data coordinator of Milvus. -title: Data Coordinator-related Configurations +summary: Learn how to configure dataCoord for Milvus. --- -# Data Coordinator-related Configurations +# dataCoord-related Configurations -This topic introduces the data coordinator-related configurations of Milvus. -Data coordinator (data coord) manages the topology of data nodes, maintains metadata, and triggers flush, compact, and other background data operations. -Under this section, you can configure data coord address, segment settings, compaction, garbage collection, etc. +## `dataCoord.channel.watchTimeoutInterval` + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Timeout on watching channels (in seconds). Datanode tickler update watch progress will reset timeout timer.
    		• 300
    + + +## `dataCoord.channel.balanceWithRpc` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether to enable balance with RPC, default to use etcd watch
    		• true
    + + +## `dataCoord.channel.legacyVersionWithoutRPCWatch` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Datanodes <= this version are considered as legacy nodes, which doesn't have rpc based watch(). This is only used during rolling upgrade where legacy nodes won't get new channels
    		• 2.4.0
    + + +## `dataCoord.channel.balanceSilentDuration` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The duration after which the channel manager start background channel balancing
    		• 300
    + + +## `dataCoord.channel.balanceInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The interval with which the channel manager check dml channel balance status
    		• 360
    + + +## `dataCoord.channel.checkInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The interval in seconds with which the channel manager advances channel states
    		• 1
    + + +## `dataCoord.channel.notifyChannelOperationTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Timeout notifing channel operations (in seconds).
    		• 5
    + + +## `dataCoord.segment.maxSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Maximum size of a segment in MB
    		• 1024
    + + +## `dataCoord.segment.diskSegmentMaxSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Maximun size of a segment in MB for collection which has Disk index
    		• 2048
    + + +## `dataCoord.segment.sealProportion` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    0.12
    + + +## `dataCoord.segment.assignmentExpiration` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The time of the assignment expiration in ms
    		• 2000
    + + +## `dataCoord.segment.allocLatestExpireAttempt` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The time attempting to alloc latest lastExpire from rootCoord after restart
    		• 200
    + + +## `dataCoord.segment.maxLife` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The max lifetime of segment in seconds, 24*60*60
    		• 86400
    + + +## `dataCoord.segment.maxIdleTime` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • If a segment didn't accept dml records in maxIdleTime and the size of segment is greater than
    • +
  • minSizeFromIdleToSealed, Milvus will automatically seal it.
    • +
  • The max idle time of segment in seconds, 10*60.
    		• 600
    + + +## `dataCoord.segment.minSizeFromIdleToSealed` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The min size in MB of segment which can be idle from sealed.
    		• 16
    + + +## `dataCoord.segment.maxBinlogFileNumber` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The max number of binlog file for one segment, the segment will be sealed if
    • +
  • the number of binlog file reaches to max value.
    		• 32
    + + +## `dataCoord.segment.smallProportion` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The segment is considered as "small segment" when its # of rows is smaller than
    		• 0.5
    + + +## `dataCoord.segment.compactableProportion` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • (smallProportion * segment max # of rows).
    • +
  • A compaction will happen on small segments if the segment after compaction will have
    		• 0.85
    + + +## `dataCoord.segment.expansionRate` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • over (compactableProportion * segment max # of rows) rows.
    • +
  • MUST BE GREATER THAN OR EQUAL TO !!!
    • +
  • During compaction, the size of segment # of rows is able to exceed segment max # of rows by (expansionRate-1) * 100%.
    		• 1.25
    + + +## `dataCoord.autoUpgradeSegmentIndex` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • whether auto upgrade segment index to index engine's version
    		• false
    + + +## `dataCoord.enableCompaction` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Enable data segment compaction
    		• true
    + + +## `dataCoord.compaction.enableAutoCompaction` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    true
    + + +## `dataCoord.compaction.indexBasedCompaction` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    true
    + + +## `dataCoord.compaction.rpcTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    10
    + + +## `dataCoord.compaction.maxParallelTaskNum` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    10
    + + +## `dataCoord.compaction.workerMaxParallelTaskNum` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    2
    -## `dataCoord.address` - +## `dataCoord.compaction.levelzero.forceTrigger.minSize` + +
    @@ -27,17 +515,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • The minmum size in bytes to force trigger a LevelZero Compaction, default as 8MB
    • +
    Description
    -
  • TCP/IP address of data coordinator.
    • -
  • Data coordinator monitors all IPv4 addresses if this parameter is set as 0.0.0.0.
    • -     		localhost 8388608
    -## `dataCoord.port` - +## `dataCoord.compaction.levelzero.forceTrigger.maxSize` + +
    @@ -46,15 +533,17 @@ Under this section, you can configure data coord address, segment settings, comp - - + +
    Description
    TCP port of data coordinator.13333 +
  • The maxmum size in bytes to force trigger a LevelZero Compaction, default as 64MB
    		• 67108864
    -## `dataCoord.grpc.serverMaxRecvSize` - +## `dataCoord.compaction.levelzero.forceTrigger.deltalogMinNum` + +
    @@ -64,17 +553,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • The minimum number of deltalog files to force trigger a LevelZero Compaction
    • +
    Description
    -
  • The maximum size of each RPC request that the data coord can receive.
    • -
  • Unit: Byte
    • -     		2147483647 10
    -## `dataCoord.grpc.serverMaxSendSize` - +## `dataCoord.compaction.levelzero.forceTrigger.deltalogMaxNum` + +
    @@ -84,18 +572,34 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • The maxmum number of deltalog files to force trigger a LevelZero Compaction, default as 30
    • +
    Description
    -
  • The maximum size of each respond the data coord can send when receiving an RPC request.
    • -
  • Unit: Byte
    • -     		2147483647 30
    -## `dataCoord.grpc.clientMaxRecvSize` +## `dataCoord.enableGarbageCollection` - +
    + + + + + + + + + + + + +
    DescriptionDefault Value
    true
    + + +## `dataCoord.gc.interval` + + @@ -105,18 +609,54 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • gc interval in seconds
    • +
    Description
    -
  • The maximum size of each respond that the data coord can receive when sending an RPC request.
    • -
  • Unit: Byte
    • -     		104857600 3600
    -## `dataCoord.grpc.clientMaxSendSize` +## `dataCoord.gc.missingTolerance` - +
    + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • file meta missing tolerance duration in seconds, default to 24hr(1d)
    		• 86400
    + + +## `dataCoord.gc.dropTolerance` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • file belongs to dropped entity tolerance duration in seconds. 3600
    		• 10800
    + + +## `dataCoord.gc.removeConcurrent` + + @@ -126,17 +666,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • number of concurrent goroutines to remove dropped s3 objects
    • +
    Description
    -
  • The maximum size of each RPC request that the data coord can send.
    • -
  • Unit: Byte
    • -     		104857600 32
    -## `dataCoord.activeStandby.enabled` - +## `dataCoord.gc.scanInterval` + +
    @@ -146,16 +685,34 @@ Under this section, you can configure data coord address, segment settings, comp +
  • garbage collection scan residue interval in hours
    • + + + +
    Description
    - Whether the dataCoord works in active-standby mode. - 168
    + + +## `dataCoord.enableActiveStandby` + + + + + + + + + + +
    DescriptionDefault Value
    false
    -## `dataCoord.replicas` - +## `dataCoord.brokerTimeout` + +
    @@ -165,16 +722,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • 5000ms, dataCoord broker rpc timeout
    • +
    Description
    - Number of dataCoord pods. This is required if `dataCoord.activeStandby.enabled` is set to `true`. - 1 5000
    -## `dataCoord.enableCompaction` - +## `dataCoord.autoBalance` + +
    @@ -184,18 +741,16 @@ Under this section, you can configure data coord address, segment settings, comp +
  • Enable auto balance
    • Description
    -
  • Switch value to control if to enable segment compaction.
    • -
  • Compaction merges small-size segments into a large segment, and clears the entities deleted beyond the rentention duration of Time Travel.
    • -     		true
    -## `dataCoord.enableGarbageCollection` +## `dataCoord.checkAutoBalanceConfigInterval` - +
    @@ -205,16 +760,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • the interval of check auto balance config
    • +
    Description
    - Switch value to control if to enable garbage collection to clear the discarded data in MinIO or S3 service. - true 10
    -## `dataCoord.segment.maxSize` - +## `dataCoord.import.filesPerPreImportTask` + +
    @@ -224,18 +779,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • The maximum number of files allowed per pre-import task.
    • +
    Description
    -
  • The maximum size of a segment.
    • -
  • Unit: MB
    • -
  • datacoord.segment.maxSize and datacoord.segment.sealProportion together determine if a segment can be sealed.
    • -     		512 2
    -## `dataCoord.segment.sealProportion` - +## `dataCoord.import.taskRetention` + +
    @@ -245,18 +798,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • The retention period in seconds for tasks in the Completed or Failed state.
    • +
    Description
    -
  • The minimum proportion to datacoord.segment.maxSize to seal a segment.
    • -
  • datacoord.segment.maxSize and datacoord.segment.sealProportion together determine if a segment can be sealed.
    • -     		0.23 10800
    -## `dataCoord.segment.assignmentExpiration` +## `dataCoord.import.maxSizeInMBPerImportTask` - +
    @@ -266,18 +817,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • To prevent generating of small segments, we will re-group imported files. This parameter represents the sum of file sizes in each group (each ImportTask).
    • +
    Description
    -
  • Expiration time of the segment assignment.
    • -
  • Unit: ms
    • -     		2000 6144
    -## `dataCoord.compaction.enableAutoCompaction` +## `dataCoord.import.scheduleInterval` - +
    @@ -287,18 +836,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • The interval for scheduling import, measured in seconds.
    • +
    Description
    -
  • Switch value to control if to enable automatic segment compaction during which data coord locates and merges compactable segments in the background.
    • -
  • This configuration takes effect only when dataCoord.enableCompaction is set as true.
    • -     		true 2
    -## `dataCoord.gc.interval` +## `dataCoord.import.checkIntervalHigh` - +
    @@ -308,18 +855,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • The interval for checking import, measured in seconds, is set to a high frequency for the import checker.
    • +
    Description
    -
  • The interval at which data coord performs garbage collection.
    • -
  • Unit: Second
    • -
  • This configuration takes effect only when dataCoord.enableGarbageCollection is set as true.
    • -     		3600 2
    -## `dataCoord.gc.missingTolerance` - +## `dataCoord.import.checkIntervalLow` + +
    @@ -329,19 +874,16 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • The interval for checking import, measured in seconds, is set to a low frequency for the import checker.
    • +
    Description
    -
  • The retention duration of the unrecorded binary log (binlog) files.
    • -
  • Setting a reasonably large value for this parameter avoids erroneously deleting the newly created binlog files that lack metadata.
    • -
  • Unit: Second
    • -
  • This configuration takes effect only when dataCoord.enableGarbageCollection is set as true.
    • -     		86400 120
    -## `dataCoord.gc.dropTolerance` - +## `dataCoord.import.maxImportFileNumPerReq` + +
    @@ -351,11 +893,157 @@ Under this section, you can configure data coord address, segment settings, comp - +
  • The maximum number of files allowed per single import request.
    • + + + +
    Description
    -
  • The retention duration of the binlog files of the deleted segments before they are cleared.
    • -
  • Unit: Second
    • -
  • This configuration takes effect only when dataCoord.enableGarbageCollection is set as true.
    • -     		86400 1024
    + + +## `dataCoord.import.waitForIndex` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Indicates whether the import operation waits for the completion of index building.
    		• true
    + + +## `dataCoord.gracefulStopTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • seconds. force stop node without graceful stop
    		• 5
    + + +## `dataCoord.ip` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • if not specified, use the first unicastable address
    • + + +## `dataCoord.port` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    13333
    + + +## `dataCoord.grpc.serverMaxSendSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    536870912
    + + +## `dataCoord.grpc.serverMaxRecvSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    268435456
    + + +## `dataCoord.grpc.clientMaxSendSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    268435456
    + + +## `dataCoord.grpc.clientMaxRecvSize` + + + + + + + + + + + +
    DescriptionDefault Value
    536870912
    + + diff --git a/site/en/reference/sys_config/configure_datanode.md b/site/en/reference/sys_config/configure_datanode.md index f41c960e6..6b2145572 100644 --- a/site/en/reference/sys_config/configure_datanode.md +++ b/site/en/reference/sys_config/configure_datanode.md @@ -2,22 +2,35 @@ id: configure_datanode.md related_key: configure group: system_configuration.md -summary: Learn how to configure data node of Milvus. -title: Data Node-related Configurations +summary: Learn how to configure dataNode for Milvus. --- -# Data Node-related Configurations +# dataNode-related Configurations -This topic introduces the data node-related configurations of Milvus. -Data node retrieves incremental log data by subscribing to the log broker, processes mutation requests, and packs log data into log snapshots and stores them in the object storage. -Under this section, you can configure data node port, etc. +## `dataNode.dataSync.flowGraph.maxQueueLength` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Maximum length of task queue in flowgraph
    		• 16
    -## `dataNode.port` +## `dataNode.dataSync.flowGraph.maxParallelism` - +
    @@ -26,15 +39,17 @@ Under this section, you can configure data node port, etc. - - + +
    Description
    TCP port of data node.21124 +
  • Maximum number of tasks executed in parallel in the flowgraph
    		• 1024
    -## `dataNode.grpc.serverMaxRecvSize` - +## `dataNode.dataSync.maxParallelSyncMgrTasks` + +
    @@ -44,17 +59,16 @@ Under this section, you can configure data node port, etc. - +
  • The max concurrent sync task number of datanode sync mgr globally
    • +
    Description
    -
  • The maximum size of each RPC request that the data node can receive.
    • -
  • Unit: Byte
    • -     		2147483647 256
    -## `dataNode.grpc.serverMaxSendSize` - +## `dataNode.dataSync.skipMode.enable` + +
    @@ -64,17 +78,16 @@ Under this section, you can configure data node port, etc. - +
  • Support skip some timetick message to reduce CPU usage
    • +
    Description
    -
  • The maximum size of each respond the data node can send when receiving an RPC request.
    • -
  • Unit: Byte
    • -     		2147483647 true
    -## `dataNode.grpc.clientMaxRecvSize` - +## `dataNode.dataSync.skipMode.skipNum` + +
    @@ -84,17 +97,16 @@ Under this section, you can configure data node port, etc. - +
  • Consume one for every n records skipped
    • +
    Description
    -
  • The maximum size of each respond that the data node can receive when sending an RPC request.
    • -
  • Unit: Byte
    • -     		104857600 4
    -## `dataNode.grpc.clientMaxSendSize` - +## `dataNode.dataSync.skipMode.coldTime` + +
    @@ -104,17 +116,16 @@ Under this section, you can configure data node port, etc. - +
  • Turn on skip mode after there are only timetick msg for x seconds
    • +
    Description
    -
  • The maximum size of each RPC request that the data node can send.
    • -
  • Unit: Byte
    • -     		104857600 60
    -## `dataNode.dataSync.flowGraph.maxQueueLength` - +## `dataNode.segment.insertBufSize` + +
    @@ -124,19 +135,16 @@ Under this section, you can configure data node port, etc. - +
  • Max buffer size to flush for a single segment.
    • +
    Description
    -
  • The maximum size of task queue cache in flow graph in data node.
    • -
  • Unit: MB
    • -
  • Data node uses flow graph to subscribe to and organize the message flow.
    • -     		1024 16777216
    -## `dataNode.flush.insertBufSize` +## `dataNode.segment.deleteBufBytes` - +
    @@ -146,11 +154,462 @@ Under this section, you can configure data node port, etc. +
  • Max buffer size in bytes to flush del for a single channel, default as 16MB
    • Description
    -
  • The maximum size of each binlog file in a segment buffered in memory. Binlog files whose size exceeds this value are then flushed to MinIO or S3 service.
    • -
  • Unit: Byte
    • -
  • Setting this parameter too small causes the system to store a small amount of data too frequently. Setting it too large increases the system's demand for memory.
    • -     		16777216
    + + +## `dataNode.segment.syncPeriod` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The period to sync segments if buffer is not empty.
    		• 600
    + + +## `dataNode.memory.forceSyncEnable` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Set true to force sync if memory usage is too high
    		• true
    + + +## `dataNode.memory.forceSyncSegmentNum` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • number of segments to sync, segments with top largest buffer will be synced.
    		• 1
    + + +## `dataNode.memory.checkInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the interal to check datanode memory usage, in milliseconds
    		• 3000
    + + +## `dataNode.memory.forceSyncWatermark` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • memory watermark for standalone, upon reaching this watermark, segments will be synced.
    		• 0.5
    + + +## `dataNode.timetick.byRPC` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    true
    + + +## `dataNode.timetick.interval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    500
    + + +## `dataNode.channel.workPoolSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • specify the size of global work pool of all channels
    • +
  • if this parameter <= 0, will set it as the maximum number of CPUs that can be executing
    • +
  • suggest to set it bigger on large collection numbers to avoid blocking
    		• -1
    + + +## `dataNode.channel.updateChannelCheckpointMaxParallel` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • specify the size of global work pool for channel checkpoint updating
    • +
  • if this parameter <= 0, will set it as 10
    		• 10
    + + +## `dataNode.channel.updateChannelCheckpointInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the interval duration(in seconds) for datanode to update channel checkpoint of each channel
    		• 60
    + + +## `dataNode.channel.updateChannelCheckpointRPCTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • timeout in seconds for UpdateChannelCheckpoint RPC call
    		• 20
    + + +## `dataNode.channel.maxChannelCheckpointsPerPRC` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The maximum number of channel checkpoints per UpdateChannelCheckpoint RPC.
    		• 128
    + + +## `dataNode.channel.channelCheckpointUpdateTickInSeconds` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The frequency, in seconds, at which the channel checkpoint updater executes updates.
    		• 10
    + + +## `dataNode.import.maxConcurrentTaskNum` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The maximum number of import/pre-import tasks allowed to run concurrently on a datanode.
    		• 16
    + + +## `dataNode.import.maxImportFileSizeInGB` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The maximum file size (in GB) for an import file, where an import file refers to either a Row-Based file or a set of Column-Based files.
    		• 16
    + + +## `dataNode.import.readBufferSizeInMB` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The data block size (in MB) read from chunk manager by the datanode during import.
    		• 16
    + + +## `dataNode.compaction.levelZeroBatchMemoryRatio` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The minimal memory ratio of free memory for level zero compaction executing in batch mode
    		• 0.05
    + + +## `dataNode.gracefulStopTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • seconds. force stop node without graceful stop
    		• 1800
    + + +## `dataNode.ip` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • if not specified, use the first unicastable address
    • + + +## `dataNode.port` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    21124
    + + +## `dataNode.grpc.serverMaxSendSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    536870912
    + + +## `dataNode.grpc.serverMaxRecvSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    268435456
    + + +## `dataNode.grpc.clientMaxSendSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    268435456
    + + +## `dataNode.grpc.clientMaxRecvSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    536870912
    + + diff --git a/site/en/reference/sys_config/configure_etcd.md b/site/en/reference/sys_config/configure_etcd.md index fcc732218..bc77ce16f 100644 --- a/site/en/reference/sys_config/configure_etcd.md +++ b/site/en/reference/sys_config/configure_etcd.md @@ -3,18 +3,11 @@ id: configure_etcd.md related_key: configure group: system_configuration.md summary: Learn how to configure etcd for Milvus. -title: etcd-related Configurations --- # etcd-related Configurations -This topic introduces the etcd-related configurations of Milvus. etcd is the metadata engine supporting Milvus' metadata storage and access. - -In this section, you can configure etcd endpoints, relevant key prefixes, etc. - -
    -To share an etcd instance among multiple Milvus instances, you need to change etcd.rootPath to a unique value for each of the Milvus instances. For details, refer to Operation FAQs. -
    +Related configuration of etcd, used to store Milvus metadata & service discovery. ## `etcd.endpoints` @@ -27,11 +20,7 @@ To share an etcd instance among multiple Milvus instances, you need to change - -
  • Endpoints used to access etcd service. You can change this parameter as the endpoints of your own etcd cluster.
    • -
  • Environment variable: ETCD_ENDPOINTS
    • -
  • etcd preferentially acquires valid address from environment variable ETCD_ENDPOINTS when Milvus is started.
    • - + localhost:2379 @@ -50,17 +39,13 @@ To share an etcd instance among multiple Milvus instances, you need to change -
  • Root prefix of the key to where Milvus stores data in etcd.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -
  • To share an etcd instance among multiple Milvus instances, consider changing this to a different value for each Milvus instance before you start them. For details, see Operation FAQs.
    • -
  • Set an easy-to-identify root path for Milvus if etcd service already exists.
    • -
  • Changing this for an already running Milvus instance may result in failures to read legacy data.
    • - +
  • The root path where data is stored in etcd
    • by-dev + ## `etcd.metaSubPath` @@ -73,10 +58,7 @@ To share an etcd instance among multiple Milvus instances, you need to change +
  • metaRootPath = rootPath + '/' + metaSubPath
    • @@ -95,13 +77,264 @@ To share an etcd instance among multiple Milvus instances, you need to change +
  • kvRootPath = rootPath + '/' + kvSubPath
    • -
  • Sub-prefix of the key to where Milvus stores metadata-related information in etcd.
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		meta
    -
  • Sub-prefix of the key to where Milvus stores timestamps in etcd.
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended not to change this parameter if there is no specific reason.
    • -     		kv
    +## `etcd.log.level` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
    		• info
    + + +## `etcd.log.path` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • path is one of:
    • +
  • - "default" as os.Stderr,
    • +
  • - "stderr" as os.Stderr,
    • +
  • - "stdout" as os.Stdout,
    • +
  • - file path to append server logs to.
    • +
  • please adjust in embedded Milvus: /tmp/milvus/logs/etcd.log
    		• stdout
    + + +## `etcd.ssl.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether to support ETCD secure connection mode
    		• false
    + + +## `etcd.ssl.tlsCert` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • path to your cert file
    		• /path/to/etcd-client.pem
    + + +## `etcd.ssl.tlsKey` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • path to your key file
    		• /path/to/etcd-client-key.pem
    + + +## `etcd.ssl.tlsCACert` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • path to your CACert file
    		• /path/to/ca.pem
    + + +## `etcd.ssl.tlsMinVersion` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • TLS min version
    • +
  • Optional values: 1.0, 1.1, 1.2, 1.3。
    • +
  • We recommend using version 1.2 and above.
    		• 1.3
    + + +## `etcd.requestTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Etcd operation timeout in milliseconds
    		• 10000
    + + +## `etcd.use.embed` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether to enable embedded Etcd (an in-process EtcdServer).
    		• false
    + + +## `etcd.data.dir` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Embedded Etcd only. please adjust in embedded Milvus: /tmp/milvus/etcdData/
    		• default.etcd
    + + +## `etcd.auth.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether to enable authentication
    		• false
    + + +## `etcd.auth.userName` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • username for etcd authentication
    • + + +## `etcd.auth.password` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • password for etcd authentication
    • + + diff --git a/site/en/reference/sys_config/configure_gpu.md b/site/en/reference/sys_config/configure_gpu.md new file mode 100644 index 000000000..720dd287c --- /dev/null +++ b/site/en/reference/sys_config/configure_gpu.md @@ -0,0 +1,59 @@ +--- +id: configure_gpu.md +related_key: configure +group: system_configuration.md +summary: Learn how to configure gpu for Milvus. +--- + +# gpu-related Configurations + +#when using GPU indexing, Milvus will utilize a memory pool to avoid frequent memory allocation and deallocation. + +#here, you can set the size of the memory occupied by the memory pool, with the unit being MB. + +#note that there is a possibility of Milvus crashing when the actual memory demand exceeds the value set by maxMemSize. + +#if initMemSize and MaxMemSize both set zero, + +#milvus will automatically initialize half of the available GPU memory, + +#maxMemSize will the whole available GPU memory. + +## `gpu.initMemSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Gpu Memory Pool init size
    • + + +## `gpu.maxMemSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Gpu Memory Pool Max size
    • + + diff --git a/site/en/reference/sys_config/configure_grpc.md b/site/en/reference/sys_config/configure_grpc.md new file mode 100644 index 000000000..b9d86edef --- /dev/null +++ b/site/en/reference/sys_config/configure_grpc.md @@ -0,0 +1,228 @@ +--- +id: configure_grpc.md +related_key: configure +group: system_configuration.md +summary: Learn how to configure grpc for Milvus. +--- + +# grpc-related Configurations + + + +## `grpc.log.level` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    WARNING
    + + +## `grpc.gracefulStopTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • second, time to wait graceful stop finish
    		• 10
    + + +## `grpc.client.compressionEnabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `grpc.client.dialTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    200
    + + +## `grpc.client.keepAliveTime` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    10000
    + + +## `grpc.client.keepAliveTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    20000
    + + +## `grpc.client.maxMaxAttempts` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    10
    + + +## `grpc.client.initialBackoff` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    0.2
    + + +## `grpc.client.maxBackoff` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    10
    + + +## `grpc.client.minResetInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    1000
    + + +## `grpc.client.maxCancelError` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    32
    + + +## `grpc.client.minSessionCheckInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    200
    + + diff --git a/site/en/reference/sys_config/configure_indexcoord.md b/site/en/reference/sys_config/configure_indexcoord.md new file mode 100644 index 000000000..61560edfd --- /dev/null +++ b/site/en/reference/sys_config/configure_indexcoord.md @@ -0,0 +1,102 @@ +--- +id: configure_indexcoord.md +related_key: configure +group: system_configuration.md +summary: Learn how to configure indexCoord for Milvus. +--- + +# indexCoord-related Configurations + + + +## `indexCoord.bindIndexNodeMode.enable` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `indexCoord.bindIndexNodeMode.address` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    localhost:22930
    + + +## `indexCoord.bindIndexNodeMode.withCred` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `indexCoord.bindIndexNodeMode.nodeID` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    0
    + + +## `indexCoord.segment.minSegmentNumRowsToEnableIndex` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • It's a threshold. When the segment num rows is less than this value, the segment will not be indexed
    		• 1024
    + + diff --git a/site/en/reference/sys_config/configure_indexnode.md b/site/en/reference/sys_config/configure_indexnode.md index 5cedf6c0a..962e5a087 100644 --- a/site/en/reference/sys_config/configure_indexnode.md +++ b/site/en/reference/sys_config/configure_indexnode.md @@ -2,22 +2,34 @@ id: configure_indexnode.md related_key: configure group: system_configuration.md -summary: Learn how to configure index node of Milvus. -title: Index Node-related Configurations +summary: Learn how to configure indexNode for Milvus. --- -# Index Node-related Configurations +# indexNode-related Configurations -This topic introduces the index node-related configurations of Milvus. -Index node builds indexes for vectors. -Under this section, you can configure index node port, etc. +## `indexNode.scheduler.buildParallel` + + + + + + + + + + + + + +
    DescriptionDefault Value
    1
    -## `indexNode.port` - +## `indexNode.enableDisk` + +
    @@ -26,15 +38,35 @@ Under this section, you can configure index node port, etc. - - + +
    Description
    TCP port of index node.21121 +
  • enable index node build disk vector index
    		• true
    -## `indexNode.grpc.serverMaxRecvSize` - +## `indexNode.maxDiskUsagePercentage` + +
    + + + + + + + + + + + + +
    DescriptionDefault Value
    95
    + + +## `indexNode.ip` + + @@ -44,14 +76,31 @@ Under this section, you can configure index node port, etc. - +
  • if not specified, use the first unicastable address
    • + + + +
    Description
    -
  • The maximum size of each RPC request that the index node can receive.
    • -
  • Unit: Byte
    • -     		2147483647
    + + +## `indexNode.port` + + + + + + + + + + + +
    DescriptionDefault Value
    21121
    + ## `indexNode.grpc.serverMaxSendSize` @@ -63,18 +112,16 @@ Under this section, you can configure index node port, etc. - - + +
    -
  • The maximum size of each respond the index node can send when receiving an RPC request.
    • -
  • Unit: Byte
    • -     		2147483647536870912
    -## `indexNode.grpc.clientMaxRecvSize` - +## `indexNode.grpc.serverMaxRecvSize` + +
    @@ -83,15 +130,13 @@ Under this section, you can configure index node port, etc. - - + +
    Description
    -
  • The maximum size of each respond that the index node can receive when sending an RPC request.
    • -
  • Unit: Byte
    • -     		104857600268435456
    + ## `indexNode.grpc.clientMaxSendSize` @@ -103,11 +148,28 @@ Under this section, you can configure index node port, etc. - - + +
    -
  • The maximum size of each RPC request that the index node can send.
    • -
  • Unit: Byte
    • -     		104857600268435456
    + + +## `indexNode.grpc.clientMaxRecvSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    536870912
    + + diff --git a/site/en/reference/sys_config/configure_kafka.md b/site/en/reference/sys_config/configure_kafka.md deleted file mode 100644 index 27a5af631..000000000 --- a/site/en/reference/sys_config/configure_kafka.md +++ /dev/null @@ -1,145 +0,0 @@ ---- -id: configure_kafka.md -related_key: configure -group: system_configuration.md -summary: Learn how to configure Kafka for Milvus cluster. -title: Kafka-related Configurations ---- - -# Kafka-related Configurations - -This topic introduces the Kafka-related configurations of Milvus. - -Kafka is the underlying engine supporting Milvus cluster's reliable storage and publication/subscription of message streams. - -Under this section, you can configure Kafka producer, consumer, sasl information and etc. - -## `kafka.producer.client.id` - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Producer Client ID of Kafka service.
    • -
    - -## `kafka.consumer.client.id` - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Consumer Client ID of Kafka service.
    • -
    - -## `kafka.brokerList` - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • List of broker addresses of Kafka service.
    • -
  • Each value is separated from each other by comma
    • -
  • Eg: localhost1:9092,localhost2:9092,localhost3:9092
    • -
    - -## `kafka.saslUsername` - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Username of Kafka service if Simple Authentication and Security Layer is enabled.
    • -
    - -## `kafka.saslPassword` - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Password of Kafka service if Simple Authentication and Security Layer is enabled.
    • -
    - -## `kafka.saslMechanisms` - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Mechanisms of Kafka service if Simple Authentication and Security Layer is enabled.
    • -
  • Valid values maybe: GSSAPI, PLAIN, SCRAM, OAUTHBEARER.
    • -
    - -## `kafka.securityProtocol` - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Security protocol of Kafka service.
    • -
  • Valid values maybe: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
    • -
    diff --git a/site/en/reference/sys_config/configure_knowhere.md b/site/en/reference/sys_config/configure_knowhere.md deleted file mode 100644 index 12baead16..000000000 --- a/site/en/reference/sys_config/configure_knowhere.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -id: configure_knowhere.md -related_key: configure -group: system_configuration.md -summary: Learn how to configure Knowhere-related parameters of Milvus. -title: Knowhere-related Configurations ---- - -# Knowhere-related Configurations - -This topic introduces the knowhere-related configurations of Milvus. - -[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus. - -Under this section, you can configure the default SIMD instruction set type of the system. - -## `knowhere.simdType` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • The CPU instruction set which can accelerate vector search and index building.
    • -
  • Options: auto, avx512, avx2, avx, and sse4_2
    • -     		auto
    - diff --git a/site/en/reference/sys_config/configure_localstorage.md b/site/en/reference/sys_config/configure_localstorage.md index 504291c89..d5b363824 100644 --- a/site/en/reference/sys_config/configure_localstorage.md +++ b/site/en/reference/sys_config/configure_localstorage.md @@ -2,17 +2,11 @@ id: configure_localstorage.md related_key: configure group: system_configuration.md -summary: Learn how to configure local storage of Milvus. -title: Local Storage-related Configurations +summary: Learn how to configure localStorage for Milvus. --- -# Local Storage-related Configurations +# localStorage-related Configurations -This topic introduces the local storage-related configurations of Milvus. - -Milvus stores the vector data in local storage during search or query to avoid repetitive access to MinIO or S3 service. - -Under this section, you can enable local storage, and configure the path, etc. ## `localStorage.path` @@ -27,32 +21,10 @@ Under this section, you can enable local storage, and configure the path, etc. -
  • Local path to where vector data are stored during a search or a query to avoid repetitve access to MinIO or S3 service.
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -
  • This configuration takes effect only when localStorage.enabled is set as true.
    • - - /var/lib/milvus/data +
  • please adjust in embedded Milvus: /tmp/milvus/data/
    • + /tmp/milvus/data/ -## `localStorage.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    - Switch value to control if to enable the local storage of the vector data to avoid repetitve access to MinIO or S3 service. - true
    diff --git a/site/en/reference/sys_config/configure_log.md b/site/en/reference/sys_config/configure_log.md index f6161167a..5e41af530 100644 --- a/site/en/reference/sys_config/configure_log.md +++ b/site/en/reference/sys_config/configure_log.md @@ -2,18 +2,12 @@ id: configure_log.md related_key: configure group: system_configuration.md -summary: Learn how to configure log of Milvus. -title: Log-related Configurations +summary: Learn how to configure log for Milvus. --- -# Log-related Configurations - -This topic introduces the log-related configurations of Milvus. - -Using Milvus generates a collection of logs. By default, Milvus uses logs to record information at debug or even higher level for standard output (stdout) and standard error (stderr). - -Under this section, you can configure the system log output. +# log-related Configurations +Configures the system log output. ## `log.level` @@ -27,15 +21,13 @@ Under this section, you can configure the system log output. -
  • Milvus log level.
    • -
  • Option: debug, info, warn, error, panic, and fatal
    • -
  • It is recommended to use debug level under test and development environments, and info level in production environment.
    • - - debug +
  • Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
    • + info + ## `log.file.rootPath` @@ -48,16 +40,13 @@ Under this section, you can configure the system log output. - +
  • root dir path to put logs, default "" means no log file will print. please adjust in embedded Milvus: /tmp/milvus/logs
    • +
    -
  • Root path to the log files.
    • -
  • The default value is set empty, indicating to output log files to standard output (stdout) and standard error (stderr).
    • -
  • If this parameter is set to a valid local path, Milvus writes and stores log files in this path.
    • -
  • Set this parameter as the path that you have permission to write.
    • -     		""
    + ## `log.file.maxSize` @@ -70,14 +59,13 @@ Under this section, you can configure the system log output. +
  • MB
    • -
  • The maximum size of a log file.
    • -
  • Unit: MB
    • -     		300
    + ## `log.file.maxAge` @@ -90,16 +78,13 @@ Under this section, you can configure the system log output. +
  • Maximum time for log retention in day.
    • -
  • The maximum retention time before a log file is automatically cleared.
    • -
  • Unit: day
    • -
  • The minimum value is 1.
    • -
  • This parameter takes effect only after the valid log.file.rootPath is set.
    • -     		10
    + ## `log.file.maxBackups` @@ -111,17 +96,13 @@ Under this section, you can configure the system log output. - +
    -
  • The maximum number of log files to back up.
    • -
  • Unit: day
    • -
  • The minimum value is 1.
    • -
  • This parameter takes effect only after the valid log.file.rootPath is set.
    • -     		20
    + ## `log.format` @@ -134,11 +115,29 @@ Under this section, you can configure the system log output. - +
  • text or json
    • +
    -
  • Milvus log format.
    • -
  • Option: text and JSON
    • -     		text text
    + +## `log.stdout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Stdout enable or not
    		• true
    + + diff --git a/site/en/reference/sys_config/configure_messagechannel.md b/site/en/reference/sys_config/configure_messagechannel.md deleted file mode 100644 index 77b32eb6a..000000000 --- a/site/en/reference/sys_config/configure_messagechannel.md +++ /dev/null @@ -1,433 +0,0 @@ ---- -id: configure_messagechannel.md -related_key: configure -group: system_configuration.md -summary: Learn how to configure message channels of Milvus. -title: Message Channel-related Configurations ---- - -# Message Channel-related Configurations - -This topic introduces the message channel-related configurations of Milvus. - -Under this section, you can configure the message channel name prefixes and component subscription name prefixes. - -
    -
  • To share a Pulsar instance with multi-tenancy enabled among multiple Milvus instances, you need to change pulsar.tenant or pulsar.namespace to a unique value for each of the Milvus instances.
    • -
  • To share a Pulsar instance with multi-tenancy disabled among multiple Milvus instances, you need to change msgChannel.chanNamePrefix.cluster to a unique value for each of the Milvus instances.
    • -For details, refer to Operation FAQs. -
    - -## `msgChannel.chanNamePrefix.cluster` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Root name prefix of the channel when a message channel is created.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -
  • To share a Pulsar instance among multiple Milvus instances, consider changing this to a name rather than the default one for each Milvus instance before you start them. For details, see Operational FAQs.
    • -     		"by-dev"
    - -## `msgChannel.chanNamePrefix.rootCoordTimeTick` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the root coord publishes time tick messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.rootCoordTimeTick}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"rootcoord-timetick"
    - -## `msgChannel.chanNamePrefix.rootCoordStatistics` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the root coord publishes its own statistics messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.rootCoordStatistics}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"rootcoord-statistics"
    - -## `msgChannel.chanNamePrefix.rootCoordDml` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the root coord publishes Data Manipulation Language (DML) messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.rootCoordDml}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"rootcoord-dml"
    - -## `msgChannel.chanNamePrefix.rootCoordDelta` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the root coord publishes data deletion messages in sealed segment.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.rootCoordDelta}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"rootcoord-delta"
    - -## `msgChannel.chanNamePrefix.search` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the proxy publishes search messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.search}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"search"
    - -## `msgChannel.chanNamePrefix.searchResult` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the query node publishes search result messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.searchResult}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"searchResult"
    - -## `msgChannel.chanNamePrefix.proxyTimeTick` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the proxy publishes time tick messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.proxyTimeTick}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"proxyTimeTick"
    - -## `msgChannel.chanNamePrefix.queryTimeTick` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the query node publishes time tick messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.queryTimeTick}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"queryTimeTick"
    - -## `msgChannel.chanNamePrefix.queryNodeStats` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the query node publishes its own statistics messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.queryNodeStats}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"query-node-stats"
    - -## `msgChannel.chanNamePrefix.dataCoordInsertChannel` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the data coord publishes data insert messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.dataCoordInsertChannel}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"insert-channel-"
    - -## `msgChannel.chanNamePrefix.dataCoordStatistic` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the data coord publishes its own statistics messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.dataCoordStatistic}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"datacoord-statistics-channel"
    - -## `msgChannel.chanNamePrefix.dataCoordTimeTick` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the data coord publishes time tick messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.dataCoordTimeTick}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"datacoord-timetick-channel"
    - -## `msgChannel.chanNamePrefix.dataCoordSegmentInfo` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Sub-name prefix of the message channel where the data coord publishes segment information messages.
    • -
  • The complete channel name prefix is ${msgChannel.chanNamePrefix.cluster}-${msgChannel.chanNamePrefix.dataCoordSegmentInfo}
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"segment-info-channel"
    - - -## `msgChannel.subNamePrefix.rootCoordSubNamePrefix` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Subscription name prefix of the root coord.
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"rootCoord"
    - -## `msgChannel.subNamePrefix.proxySubNamePrefix` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Subscription name prefix of the proxy.
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"proxy"
    - -## `msgChannel.subNamePrefix.queryNodeSubNamePrefix` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Subscription name prefix of the query node.
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"queryNode"
    - -## `msgChannel.subNamePrefix.dataNodeSubNamePrefix` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Subscription name prefix of the data node.
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"dataNode"
    - -## `msgChannel.subNamePrefix.dataCoordSubNamePrefix` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Subscription name prefix of the data coord.
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -     		"dataCoord"
    - diff --git a/site/en/reference/sys_config/configure_metastore.md b/site/en/reference/sys_config/configure_metastore.md new file mode 100644 index 000000000..521c7d0dc --- /dev/null +++ b/site/en/reference/sys_config/configure_metastore.md @@ -0,0 +1,30 @@ +--- +id: configure_metastore.md +related_key: configure +group: system_configuration.md +summary: Learn how to configure metastore for Milvus. +--- + +# metastore-related Configurations + + + +## `metastore.type` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Default value: etcd, Valid values: [etcd, tikv]
    		• etcd
    + + diff --git a/site/en/reference/sys_config/configure_minio.md b/site/en/reference/sys_config/configure_minio.md index 9870f56a6..58358c416 100644 --- a/site/en/reference/sys_config/configure_minio.md +++ b/site/en/reference/sys_config/configure_minio.md @@ -2,21 +2,14 @@ id: configure_minio.md related_key: configure group: system_configuration.md -summary: Learn how to configure MinIO for Milvus. -title: MinIO-related Configurations +summary: Learn how to configure minio for Milvus. --- -# MinIO-related Configurations +# minio-related Configurations -This topic introduces the MinIO-related configurations of Milvus. +Related configuration of MinIO/S3/GCS or any other service supports S3 API, which is responsible for data persistence for Milvus. -Milvus supports MinIO and Amazon S3 as the storage engine for data persistence of insert log files and index files. Whereas MinIO is the de facto standard for S3 compatibility, you can configure S3 parameters directly under MinIO section. - -In this section, you can configure MinIO or S3 address, relevant access keys, etc. - -
    -To share a MinIO instance among multiple Milvus instances, you need to change minio.bucketName or minio.rootPath to a unique value for each of the Milvus instances. For details, refer to Operation FAQs. -
    +We refer to the storage service as MinIO/S3 in the following description for simplicity. ## `minio.address` @@ -30,13 +23,7 @@ To share a MinIO instance among multiple Milvus instances, you need to change -
  • IP address of MinIO or S3 service.
    • -
  • Environment variable: MINIO_ADDRESS
    • -
  • minio.address and minio.port together generate the valid access to MinIO or S3 service.
    • -
  • MinIO preferentially acquires the valid IP address from the environment variable MINIO_ADDRESS when Milvus is started.
    • -
  • Default value applies when MinIO or S3 is running on the same network with Milvus.
    • -
  • Milvus 2.0 does not support secure access to MinIO or S3 service. Future releases will support secure access to MinIO.
    • - +
  • Address of MinIO/S3
    • localhost @@ -55,16 +42,13 @@ To share a MinIO instance among multiple Milvus instances, you need to change -
  • Port of MinIO or S3 service.
    • -
  • Environment variable: MINIO_ADDRESS
    • -
  • minio.address and minio.port together generate the valid access to MinIO or S3 service.
    • -
  • MinIO preferentially acquires the valid port from the environment variable MINIO_ADDRESS when Milvus is started.
    • - +
  • Port of MinIO/S3
    • 9000 + ## `minio.accessKeyID` @@ -77,12 +61,7 @@ To share a MinIO instance among multiple Milvus instances, you need to change +
  • accessKeyID of MinIO/S3
    • @@ -101,17 +80,13 @@ To share a MinIO instance among multiple Milvus instances, you need to change +
  • MinIO/S3 encryption string
    • -
  • Access key ID that MinIO or S3 issues to user for authorized access.
    • -
  • Environment variable: MINIO_ACCESS_KEY_ID or minio.accessKeyID
    • -
  • minio.accessKeyID and minio.secretAccessKey together are used for identity authentication to access the MinIO or S3 service.
    • -
  • This configuration must be set identical to the environment variable MINIO_ACCESS_KEY_ID, which is necessary for starting MinIO or S3.
    • -
  • The default value applies to MinIO or S3 service that started with the default docker-compose.yml file.
    • -     		minioadmin
    -
  • Secret key used to encrypt the signature string and verify the signature string on server. It must be kept strictly confidential and accessible only to the MinIO or S3 server and users.
    • -
  • Environment variable: MINIO_SECRET_ACCESS_KEY or minio.secretAccessKey
    • -
  • minio.accessKeyID and minio.secretAccessKey together are used for identity authentication to access the MinIO or S3 service.
    • -
  • This configuration must be set identical to the environment variable MINIO_SECRET_ACCESS_KEY, which is necessary for starting MinIO or S3.
    • -
  • The default value applies to MinIO or S3 service that started with the default docker-compose.yml file.
    • -     		minioadmin
    + ## `minio.useSSL` @@ -124,14 +99,32 @@ To share a MinIO instance among multiple Milvus instances, you need to change +
  • Access to MinIO/S3 with SSL
    • -
  • Switch value to control if to access the MinIO or S3 service through SSL.
    • -     		false
    +## `minio.ssl.tlsCACert` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • path to your CACert file
    		• /path/to/public.crt
    + + ## `minio.bucketName` @@ -144,14 +137,8 @@ To share a MinIO instance among multiple Milvus instances, you need to change - +
  • Bucket name in MinIO/S3
    • +
    -
  • Name of the bucket where Milvus stores data in MinIO or S3.
    • -
  • Milvus 2.0.0 does not support storing data in multiple buckets.
    • -
  • Bucket with this name will be created if it does not exist. If the bucket already exists and is accessible, it will be used directly. Otherwise, there will be an error.
    • -
  • To share an MinIO instance among multiple Milvus instances, consider changing this to a different value for each Milvus instance before you start them. For details, see Operation FAQs.
    • -
  • The data will be stored in the local Docker if Docker is used to start the MinIO service locally. Ensure that there is sufficient storage space.
    • -
  • A bucket name is globally unique in one MinIO or S3 instance.
    • -     		"a-bucket" a-bucket
    @@ -169,13 +156,173 @@ To share a MinIO instance among multiple Milvus instances, you need to change -
  • Root prefix of the key to where Milvus stores data in MinIO or S3.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -
  • To share an MinIO instance among multiple Milvus instances, consider changing this to a different value for each Milvus instance before you start them. For details, see Operation FAQs.
    • -
  • Set an easy-to-identify root key prefix for Milvus if etcd service already exists.
    • -
  • Changing this for an already running Milvus instance may result in failures to read legacy data.
    • - +
  • The root path where the message is stored in MinIO/S3
    • files + + +## `minio.useIAM` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether to useIAM role to access S3/GCS instead of access/secret keys
    • +
  • For more information, refer to
    • +
  • aws: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html
    • +
  • gcp: https://cloud.google.com/storage/docs/access-control/iam
    • +
  • aliyun (ack): https://www.alibabacloud.com/help/en/container-service-for-kubernetes/latest/use-rrsa-to-enforce-access-control
    • +
  • aliyun (ecs): https://www.alibabacloud.com/help/en/elastic-compute-service/latest/attach-an-instance-ram-role
    		• false
    + + +## `minio.cloudProvider` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Cloud Provider of S3. Supports: "aws", "gcp", "aliyun".
    • +
  • You can use "aws" for other cloud provider supports S3 API with signature v4, e.g.: minio
    • +
  • You can use "gcp" for other cloud provider supports S3 API with signature v2
    • +
  • You can use "aliyun" for other cloud provider uses virtual host style bucket
    • +
  • When useIAM enabled, only "aws", "gcp", "aliyun" is supported for now
    		• aws
    + + +## `minio.iamEndpoint` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Custom endpoint for fetch IAM role credentials. when useIAM is true & cloudProvider is "aws".
    • +
  • Leave it empty if you want to use AWS default endpoint
    • + + +## `minio.logLevel` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Log level for aws sdk log. Supported level: off, fatal, error, warn, info, debug, trace
    		• fatal
    + + +## `minio.region` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Specify minio storage system location region
    • + + +## `minio.useVirtualHost` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether use virtual host mode for bucket
    		• false
    + + +## `minio.requestTimeoutMs` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • minio timeout for request time in milliseconds
    		• 10000
    + + +## `minio.listObjectsMaxKeys` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The maximum number of objects requested per batch in minio ListObjects rpc,
    • +
  • 0 means using oss client by default, decrease these configration if ListObjects timeout
    		• 0
    + + diff --git a/site/en/reference/sys_config/configure_mq.md b/site/en/reference/sys_config/configure_mq.md new file mode 100644 index 000000000..724a5fcc0 --- /dev/null +++ b/site/en/reference/sys_config/configure_mq.md @@ -0,0 +1,115 @@ +--- +id: configure_mq.md +related_key: configure +group: system_configuration.md +summary: Learn how to configure mq for Milvus. +--- + +# mq-related Configurations + +Milvus supports four MQ: rocksmq(based on RockDB), natsmq(embedded nats-server), Pulsar and Kafka. + +You can change your mq by setting mq.type field. + +If you don't set mq.type field as default, there is a note about enabling priority if we config multiple mq in this file. + +1. standalone(local) mode: rocksmq(default) > natsmq > Pulsar > Kafka + +2. cluster mode: Pulsar(default) > Kafka (rocksmq and natsmq is unsupported in cluster mode) + +## `mq.type` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Default value: "default"
    • +
  • Valid values: [default, pulsar, kafka, rocksmq, natsmq]
    		• default
    + + +## `mq.enablePursuitMode` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Default value: "true"
    		• true
    + + +## `mq.pursuitLag` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • time tick lag threshold to enter pursuit mode, in seconds
    		• 10
    + + +## `mq.pursuitBufferSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • pursuit mode buffer size in bytes
    		• 8388608
    + + +## `mq.mqBufSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MQ client consumer buffer length
    		• 16
    + + diff --git a/site/en/reference/sys_config/configure_nats.md b/site/en/reference/sys_config/configure_nats.md deleted file mode 100644 index 915339077..000000000 --- a/site/en/reference/sys_config/configure_nats.md +++ /dev/null @@ -1,266 +0,0 @@ ---- -id: configure_nats.md -related_key: configure -group: system_configuration.md -summary: Learn how to configure NATs for Milvus standalone. -title: NATS-related Configurations ---- - -# NATS-related Configurations - -This topic introduces the NATs-related configurations of Milvus. - -NATS is a message-oriented middleware that allows data exchange between applications and services, segmented in the form of messages. Milvus uses NATS as a underlying engine for reliable storage and pub/sub of message streams. You can use it as an alternative to RocksMQ. - -Under this section, you can configure message size, retention time and size, etc. - -## `natsmq.server.port` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    - Listening port of the NATS server. - 4222
    - -## `natsmq.server.storeDir` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    - JetStream storage path. - /var/lib/milvus/nats
    - -## `natsmq.server.maxFileStore` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    - Maximum size of the file storage. - 17179869184 (16 GB)
    - -## `natsmq.server.maxPayload` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    - Maximum size of the payload per message in bytes. - 8388608 (8 MB)
    - -## `natsmq.server.maxPending` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    - Maximum buffer size per client connection in bytes. - 67108864 (64 MB)
    - -## `natsmq.server.initializeTimeout` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    - Timeout duration for NATs to initialize in milliseconds. - 4000 (4 seconds)
    - -## `natsmq.monitor.debug` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    - Whether to enable debug logs - false
    - -## `natsmq.monitor.logTime` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    - Whether to include timestamps in debug logs. - true
    - -## `natsmq.monitor.logFile` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Storage path of the log files generated.
    • -
  • If left unspecified, no log files are to be generated.
    • -     		N/A
    - -## `natsmq.monitor.logSizeLimit` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum size per log file in bytes.
    • -
  • If it is set to 0, no limit applies.
    • -     		0
    - -## `natsmq.rentention.maxAge` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    - Maximum age per message in the P-channel in minutes. - 4320 (3 days)
    - -## `natsmq.rentention.maxBytes` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum age per message in the P-channel in minutes.
    • -
  • If it is left unspecified, no limit applies.
    • -     		N/A
    - -## `natsmq.rentention.maxMsgs` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of messages per P-channel.
    • -
  • If it is left unspecified, no limit applies.
    • -     		N/A
    diff --git a/site/en/reference/sys_config/configure_natsmq.md b/site/en/reference/sys_config/configure_natsmq.md new file mode 100644 index 000000000..540dc6f73 --- /dev/null +++ b/site/en/reference/sys_config/configure_natsmq.md @@ -0,0 +1,279 @@ +--- +id: configure_natsmq.md +related_key: configure +group: system_configuration.md +summary: Learn how to configure natsmq for Milvus. +--- + +# natsmq-related Configurations + +natsmq configuration. + +more detail: https://docs.nats.io/running-a-nats-service/configuration + +## `natsmq.server.port` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Port for nats server listening
    		• 4222
    + + +## `natsmq.server.storeDir` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Directory to use for JetStream storage of nats
    		• /var/lib/milvus/nats
    + + +## `natsmq.server.maxFileStore` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Maximum size of the 'file' storage
    		• 17179869184
    + + +## `natsmq.server.maxPayload` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Maximum number of bytes in a message payload
    		• 8388608
    + + +## `natsmq.server.maxPending` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Maximum number of bytes buffered for a connection Applies to client connections
    		• 67108864
    + + +## `natsmq.server.initializeTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • waiting for initialization of natsmq finished
    		• 4000
    + + +## `natsmq.server.monitor.trace` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • If true enable protocol trace log messages
    		• false
    + + +## `natsmq.server.monitor.debug` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • If true enable debug log messages
    		• false
    + + +## `natsmq.server.monitor.logTime` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • If set to false, log without timestamps.
    		• true
    + + +## `natsmq.server.monitor.logFile` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Log file path relative to .. of milvus binary if use relative path
    		• /tmp/milvus/logs/nats.log
    + + +## `natsmq.server.monitor.logSizeLimit` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Size in bytes after the log file rolls over to a new one
    		• 536870912
    + + +## `natsmq.server.retention.maxAge` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Maximum age of any message in the P-channel
    		• 4320
    + + +## `natsmq.server.retention.maxBytes` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • How many bytes the single P-channel may contain. Removing oldest messages if the P-channel exceeds this size
    • + + +## `natsmq.server.retention.maxMsgs` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • How many message the single P-channel may contain. Removing oldest messages if the P-channel exceeds this limit
    • + + diff --git a/site/en/reference/sys_config/configure_proxy.md b/site/en/reference/sys_config/configure_proxy.md index a5278b7b3..4bf42de6c 100644 --- a/site/en/reference/sys_config/configure_proxy.md +++ b/site/en/reference/sys_config/configure_proxy.md @@ -2,22 +2,35 @@ id: configure_proxy.md related_key: configure group: system_configuration.md -summary: Learn how to configure proxy of Milvus. -title: Proxy-related Configurations +summary: Learn how to configure proxy for Milvus. --- -# Proxy-related Configurations +# proxy-related Configurations -This topic introduces the proxy-related configurations of Milvus. +Related configuration of proxy, used to validate client requests and reduce the returned results. -Proxy is the access layer of the system and endpoint to users. It validates client requests and reduces the returned results. +## `proxy.timeTickInterval` -Under this section, you can configure proxy port, system limits, etc. + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • ms, the interval that proxy synchronize the time tick
    		• 200
    -## `proxy.port` +## `proxy.healthCheckTimeout` - +
    @@ -26,16 +39,17 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    TCP port of proxy.19530 +
  • ms, the interval that to do component healthy check
    		• 3000
    -## `proxy.grpc.serverMaxRecvSize` +## `proxy.msgStream.timeTick.bufSize` - +
    @@ -44,19 +58,16 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    -
  • The maximum size of each RPC request that the proxy can receive.
    • -
  • Unit: Byte
    • -     		536870912512
    -## `proxy.grpc.serverMaxSendSize` +## `proxy.maxNameLength` - +
    @@ -66,17 +77,16 @@ Under this section, you can configure proxy port, system limits, etc. - +
  • Maximum length of name for a collection or alias
    • +
    Description
    -
  • The maximum size of each respond the proxy can send when receiving an RPC request.
    • -
  • Unit: Byte
    • -     		536870912 255
    -## `proxy.grpc.clientMaxRecvSize` - +## `proxy.maxFieldNum` + +
    @@ -86,18 +96,18 @@ Under this section, you can configure proxy port, system limits, etc. - +
  • Maximum number of fields in a collection.
    • +
  • As of today (2.2.0 and after) it is strongly DISCOURAGED to set maxFieldNum >= 64.
    • +
  • So adjust at your risk!
    • +
    Description
    -
  • The maximum size of each respond that the proxy can receive when sending an RPC request.
    • -
  • Unit: Byte
    • -     		104857600 64
    -## `proxy.grpc.clientMaxSendSize` +## `proxy.maxVectorFieldNum` - +
    @@ -107,18 +117,16 @@ Under this section, you can configure proxy port, system limits, etc. - +
  • Maximum number of vector fields in a collection.
    • +
    Description
    -
  • The maximum size of each RPC request that the proxy can send.
    • -
  • Unit: Byte
    • -     		104857600 4
    -## `proxy.timeTickInterval` +## `proxy.maxShardNum` - +
    @@ -128,17 +136,16 @@ Under this section, you can configure proxy port, system limits, etc. - +
  • Maximum number of shards in a collection
    • +
    Description
    -
  • The interval at which proxy synchronizes the time tick.
    • -
  • Unit: ms
    • -     		200 16
    -## `proxy.msgStream.timeTick.bufSize` - +## `proxy.maxDimension` + +
    @@ -148,16 +155,16 @@ Under this section, you can configure proxy port, system limits, etc. - +
  • Maximum dimension of a vector
    • +
    Description
    - The maximum number of messages can be buffered in the timeTick message stream of the proxy when producing messages. - 512 32768
    -## `proxy.maxNameLength` - +## `proxy.ginLogging` + +
    @@ -166,15 +173,18 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The maximum length of the name or alias that can be created in Milvus, including the collection name, collection alias, partition name, and field name.255 +
  • Whether to produce gin logs.\n
    • +
  • please adjust in embedded Milvus: false
    		• true
    -## `proxy.maxFieldNum` - +## `proxy.ginLogSkipPaths` + +
    @@ -183,15 +193,17 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The maximum number of field can be created when creating in a collection. 64 +
  • skip url path for gin log
    		• /
    -## `proxy.maxDimension` - +## `proxy.maxTaskNum` + +
    @@ -200,15 +212,17 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The maximum number of dimensions of a vector can have when creating in a collection.32768 +
  • max task number of proxy task queue
    		• 1024
    -## `proxy.maxShardNum` - +## `proxy.mustUsePartitionKey` + +
    @@ -217,15 +231,17 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The maximum number of shards can be created when creating in a collection.64 +
  • switch for whether proxy must use partition key for the collection
    		• false
    -## `proxy.maxTaskNum` - +## `proxy.accessLog.enable` + +
    @@ -234,32 +250,36 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The maximum number of tasks in the task queue of the proxy.1024 +
  • if use access log
    		• false
    -## `proxy.maxVectorFieldNum` - +## `proxy.accessLog.minioEnable` + +
    - + - - + +
    DescriptionDefault ValueDefault Value
    The maximum number of vector fields that can be specified in a collection. Value range: [1, 10].4 +
  • if upload sealed access log file to minio
    		• false
    -## `proxy.accessLog.enable` - +## `proxy.accessLog.localPath` + +
    @@ -268,12 +288,13 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    Whether to enable the access log feature.False/tmp/milvus_access
    + ## `proxy.accessLog.filename` @@ -285,15 +306,17 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    The name of the access log file. If you leave this parameter empty, access logs will be printed to stdout.Empty string +
  • Log filename, leave empty to use stdout.
    		•
    -## `proxy.accessLog.localPath` - +## `proxy.accessLog.maxSize` + +
    @@ -302,15 +325,17 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The local folder path where the access log file is stored. This parameter can be specified when proxy.accessLog.filename is not empty./tmp/milvus_access +
  • Max size for a single file, in MB.
    		• 64
    -## `proxy.accessLog.maxSize` - +## `proxy.accessLog.cacheSize` + +
    @@ -320,14 +345,13 @@ Under this section, you can configure proxy port, system limits, etc. - +
  • Size of log of memory cache, in B
    • +
    Description
    -
  • The maximum size allowed for a single access log file. If the log file size reaches this limit, a rotation process will be triggered. This process seals the current access log file, creates a new log file, and clears the contents of the original log file.
    • -
  • Unit: MB
    • -     		64 10240
    + ## `proxy.accessLog.rotatedTime` @@ -340,17 +364,16 @@ Under this section, you can configure proxy port, system limits, etc. +
  • Max time for single access log file in seconds
    • -
  • The maximum time interval allowed for rotating a single access log file. Upon reaching the specified time interval, a rotation process is triggered, resulting in the creation of a new access log file and sealing of the previous one.
    • -
  • Unit: seconds
    • -     		0
    -## `proxy.accessLog.maxBackups` - +## `proxy.accessLog.remotePath` + +
    @@ -359,15 +382,17 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The maximum number of sealed access log files that can be retained. If the number of sealed access log files exceeds this limit, the oldest one will be deleted.8 +
  • File path in minIO
    		• access_log/
    -## `proxy.accessLog.minioEnable` - +## `proxy.accessLog.remoteMaxTime` + +
    @@ -376,15 +401,17 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    Whether to upload local access log files to MinIO. This parameter can be specified when proxy.accessLog.filename is not empty.False +
  • Max time for log file in minIO, in hours
    		• 0
    -## `proxy.accessLog.remotePath` - +## `proxy.accessLog.formatters.base.format` + +
    @@ -393,15 +420,16 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The path of the object storage for uploading access log files.access_log/[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost]
    -## `proxy.accessLog.remoteMaxTime` - +## `proxy.accessLog.formatters.query.format` + +
    @@ -410,15 +438,297 @@ Under this section, you can configure proxy port, system limits, etc. - - + + + + +
    Description
    The time interval allowed for uploading access log files. If the upload time of a log file exceeds this interval, the file will be deleted. Setting the value to 0 disables this feature.0[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr]
    + + +## `proxy.accessLog.formatters.query.methods` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    Query,Search,Delete
    + + +## `proxy.connectionCheckIntervalSeconds` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the interval time(in seconds) for connection manager to scan inactive client info
    		• 120
    + + +## `proxy.connectionClientInfoTTLSeconds` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • inactive client info TTL duration, in seconds
    		• 86400
    + + +## `proxy.maxConnectionNum` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the max client info numbers that proxy should manage, avoid too many client infos
    		• 10000
    + + +## `proxy.gracefulStopTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • seconds. force stop node without graceful stop
    		• 30
    + + +## `proxy.slowQuerySpanInSeconds` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • query whose executed time exceeds the `slowQuerySpanInSeconds` can be considered slow, in seconds.
    		• 5
    + + +## `proxy.http.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether to enable the http server
    		• true
    + + +## `proxy.http.debug_mode` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether to enable http server debug mode
    		• false
    + + +## `proxy.http.port` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • high-level restful api
    • + + +## `proxy.http.acceptTypeAllowInt64` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • high-level restful api, whether http client can deal with int64
    		• true
    + + +## `proxy.http.enablePprof` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether to enable pprof middleware on the metrics port
    		• true
    + + +## `proxy.ip` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • if not specified, use the first unicastable address
    • + + +## `proxy.port` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    19530
    + + +## `proxy.internalPort` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    19529
    + + +## `proxy.grpc.serverMaxSendSize` + + + + + + + + + + + +
    DescriptionDefault Value
    268435456
    -## `proxy.accessLog.base.format` - +## `proxy.grpc.serverMaxRecvSize` + +
    @@ -427,15 +737,16 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The log format with dynamic metrics. This format applies to all methods by default. For more information about metrics, see Configure Access Logs.Empty string67108864
    -## `proxy.accessLog..format` - +## `proxy.grpc.clientMaxSendSize` + +
    @@ -444,15 +755,16 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The custom log format that you configure for specific methods. This parameter works together with proxy.accessLog.<custom_formatter_name>.methods.Empty string268435456
    -## `proxy.accessLog..methods` - +## `proxy.grpc.clientMaxRecvSize` + +
    @@ -461,8 +773,10 @@ Under this section, you can configure proxy port, system limits, etc. - - + +
    Description
    The methods to which the custom formatter applies. This parameter works together with proxy.accessLog.<custom_formatter_name>.format.Empty string67108864
    + + diff --git a/site/en/reference/sys_config/configure_pulsar.md b/site/en/reference/sys_config/configure_pulsar.md index 594e25cf6..20820e9f7 100644 --- a/site/en/reference/sys_config/configure_pulsar.md +++ b/site/en/reference/sys_config/configure_pulsar.md @@ -2,24 +2,12 @@ id: configure_pulsar.md related_key: configure group: system_configuration.md -summary: Learn how to configure Pulsar for Milvus cluster. -title: Pulsar-related Configurations +summary: Learn how to configure pulsar for Milvus. --- -# Pulsar-related Configurations - -This topic introduces the Pulsar-related configurations of Milvus. - -Pulsar is the underlying engine supporting Milvus cluster's reliable storage and publication/subscription of message streams. - -Under this section, you can configure Pulsar address, the message size, etc. - -
    -
  • To share a Pulsar instance with multi-tenancy enabled among multiple Milvus instances, you need to change pulsar.tenant or pulsar.namespace to a unique value for each of the Milvus instances.
    • -
  • To share a Pulsar instance with multi-tenancy disabled among multiple Milvus instances, you need to change msgChannel.chanNamePrefix.cluster to a unique value for each of the Milvus instances.
    • -For details, refer to Operation FAQs. -
    +# pulsar-related Configurations +Related configuration of pulsar, used to manage Milvus logs of recent mutation operations, output streaming log, and provide log publish-subscribe services. ## `pulsar.address` @@ -33,12 +21,7 @@ For details, refer to -
  • Port of Pulsar service.
    • -
  • Environment variable: PULSAR_ADDRESS
    • -
  • pulsar.address and pulsar.port together generate the valid access to Pulsar.
    • -
  • Pulsar preferentially acquires the valid port from the environment variable PULSAR_ADDRESS when Milvus is started.
    • - +
  • Port of Pulsar
    • 6650 + ## `pulsar.webport` @@ -79,14 +59,13 @@ For details, refer to @@ -99,16 +78,13 @@ For details, refer to @@ -120,15 +96,13 @@ For details, refer to Operation FAQs. - +
    public
    + ## `pulsar.namespace` @@ -140,11 +114,48 @@ For details, refer to Operation FAQs. - +
    default
    + + +## `pulsar.requestTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • pulsar client global request timeout in seconds
    		• 60
    + + +## `pulsar.enableClientMetrics` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether to register pulsar client metrics into milvus metrics path.
    		• false
    + + diff --git a/site/en/reference/sys_config/configure_querycoord.md b/site/en/reference/sys_config/configure_querycoord.md index d24965fc9..55f59af65 100644 --- a/site/en/reference/sys_config/configure_querycoord.md +++ b/site/en/reference/sys_config/configure_querycoord.md @@ -2,22 +2,52 @@ id: configure_querycoord.md related_key: configure group: system_configuration.md -summary: Learn how to configure query coordinator of Milvus. -title: Query Coordinator-related Configurations +summary: Learn how to configure queryCoord for Milvus. --- -# Query Coordinator-related Configurations +# queryCoord-related Configurations -This topic introduces the query coordinator-related configurations of Milvus. +Related configuration of queryCoord, used to manage topology and load balancing for the query nodes, and handoff from growing segments to sealed segments. -Query coordinator (query coord) manages topology and load-balancing of the query nodes, and handoff operation from growing segments to sealed segments. +## `queryCoord.taskMergeCap` -Under this section, you can configure query coord address, auto handoff, auto load-balancing, etc. + + + + + + + + + + + + + +
    DescriptionDefault Value
    1
    + + +## `queryCoord.taskExecutionCap` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    256
    -## `queryCoord.address` +## `queryCoord.autoHandoff` - +
    @@ -27,17 +57,16 @@ Under this section, you can configure query coord address, auto handoff, auto lo - +
  • Enable auto handoff
    • +
    Description
    -
  • TCP/IP address of query coordinator.
    • -
  • Query coordinator monitors all IPv4 addresses if this parameter is set as 0.0.0.0.
    • -     		localhost true
    -## `queryCoord.port` - +## `queryCoord.autoBalance` + +
    @@ -46,15 +75,17 @@ Under this section, you can configure query coord address, auto handoff, auto lo - - + +
    Description
    TCP port of query coordinator.19531 +
  • Enable auto balance
    		• true
    -## `queryCoord.activeStandby.enabled` - +## `queryCoord.autoBalanceChannel` + +
    @@ -64,16 +95,16 @@ Under this section, you can configure query coord address, auto handoff, auto lo - +
  • Enable auto balance channel
    • +
    Description
    - Whether the queryCoord works in active-standby mode. - false true
    -## `queryCoord.replicas` - +## `queryCoord.balancer` + +
    @@ -83,16 +114,16 @@ Under this section, you can configure query coord address, auto handoff, auto lo - +
  • auto balancer used for segments on queryNodes
    • +
    Description
    - Number of queryCoord pods. This is required if `queryCoord.activeStandby.enabled` is set to `true`. - 1 ScoreBasedBalancer
    -## `queryCoord.autoHandoff` - +## `queryCoord.globalRowCountFactor` + +
    @@ -102,17 +133,16 @@ Under this section, you can configure query coord address, auto handoff, auto lo - +
  • the weight used when balancing segments among queryNodes
    • +
    Description
    -
  • Switch value to control if to automatically replace a growing segment with the corresponding indexed sealed segment when the growing segment reaches the sealing threshold.
    • -
  • If this parameter is set false, Milvus simply searches the growing segments with brute force.
    • -     		true 0.1
    -## `queryCoord.autoBalance` - +## `queryCoord.scoreUnbalanceTolerationFactor` + +
    @@ -122,13 +152,32 @@ Under this section, you can configure query coord address, auto handoff, auto lo - +
  • the least value for unbalanced extent between from and to nodes when doing balance
    • + + + +
    Description
    - Switch value to control if to automatically balance the memory usage among query nodes by distributing segment loading and releasing operations evenly. - true 0.05
    + + +## `queryCoord.reverseUnBalanceTolerationFactor` + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the largest value for unbalanced extent between from and to nodes after doing balance
    		• 1.3
    + ## `queryCoord.overloadedMemoryThresholdPercentage` @@ -141,13 +190,13 @@ Under this section, you can configure query coord address, auto handoff, auto lo +
  • The threshold percentage that memory overload
    • - The threshold of memory usage (in percentage) in a query node to trigger the sealed segment balancing. - 90
    + ## `queryCoord.balanceIntervalSeconds` @@ -159,10 +208,7 @@ Under this section, you can configure query coord address, auto handoff, auto lo - + @@ -180,17 +226,16 @@ Under this section, you can configure query coord address, auto handoff, auto lo - +
    -
  • The interval at which query coord balances the memory usage among query nodes.
    • -
  • Unit: Second
    • -     		60
    - The threshold of memory usage difference (in percentage) between any two query nodes to trigger the sealed segment balancing. - 30
    -## `queryCoord.grpc.serverMaxRecvSize` - +## `queryCoord.rowCountFactor` + +
    @@ -200,17 +245,16 @@ Under this section, you can configure query coord address, auto handoff, auto lo - +
  • the row count weight used when balancing segments among queryNodes
    • +
    Description
    -
  • The maximum size of each RPC request that the query coord can receive.
    • -
  • Unit: Byte
    • -     		2147483647 0.4
    -## `queryCoord.grpc.serverMaxSendSize` - +## `queryCoord.segmentCountFactor` + +
    @@ -220,18 +264,16 @@ Under this section, you can configure query coord address, auto handoff, auto lo - +
  • the segment count weight used when balancing segments among queryNodes
    • +
    Description
    -
  • The maximum size of each respond the query coord can send when receiving an RPC request.
    • -
  • Unit: Byte
    • -     		2147483647 0.4
    -## `queryCoord.grpc.clientMaxRecvSize` +## `queryCoord.globalSegmentCountFactor` - +
    @@ -241,17 +283,239 @@ Under this section, you can configure query coord address, auto handoff, auto lo - +
  • the segment count weight used when balancing segments among queryNodes
    • +
    Description
    -
  • The maximum size of each respond that the query coord can receive when sending an RPC request.
    • -
  • Unit: Byte
    • -     		104857600 0.1
    -## `queryCoord.grpc.clientMaxSendSize` - +## `queryCoord.segmentCountMaxSteps` + +
    + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • segment count based plan generator max steps
    		• 50
    + + +## `queryCoord.rowCountMaxSteps` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • segment count based plan generator max steps
    		• 50
    + + +## `queryCoord.randomMaxSteps` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • segment count based plan generator max steps
    		• 10
    + + +## `queryCoord.growingRowCountWeight` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the memory weight of growing segment row count
    		• 4
    + + +## `queryCoord.balanceCostThreshold` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the threshold of balance cost, if the difference of cluster's cost after executing the balance plan is less than this value, the plan will not be executed
    		• 0.001
    + + +## `queryCoord.checkSegmentInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    1000
    + + +## `queryCoord.checkChannelInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    1000
    + + +## `queryCoord.checkBalanceInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    10000
    + + +## `queryCoord.checkIndexInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    10000
    + + +## `queryCoord.channelTaskTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • 1 minute
    		• 60000
    + + +## `queryCoord.segmentTaskTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • 2 minute
    		• 120000
    + + +## `queryCoord.distPullInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    500
    + + +## `queryCoord.heartbeatAvailableInterval` + + @@ -261,10 +525,438 @@ Under this section, you can configure query coord address, auto handoff, auto lo - +
  • 10s, Only QueryNodes which fetched heartbeats within the duration are available
    • + -
    Description
    -
  • The maximum size of each RPC request that the query coord can send.
    • -
  • Unit: Byte
    • -     		104857600 10000
    \ No newline at end of file + + + +## `queryCoord.loadTimeoutSeconds` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    600
    + + +## `queryCoord.distRequestTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the request timeout for querycoord fetching data distribution from querynodes, in milliseconds
    		• 5000
    + + +## `queryCoord.heatbeatWarningLag` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the lag value for querycoord report warning when last heatbeat is too old, in milliseconds
    		• 5000
    + + +## `queryCoord.checkHandoffInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    5000
    + + +## `queryCoord.enableActiveStandby` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `queryCoord.checkInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    1000
    + + +## `queryCoord.checkHealthInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • 3s, the interval when query coord try to check health of query node
    		• 3000
    + + +## `queryCoord.checkHealthRPCTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • 100ms, the timeout of check health rpc to query node
    		• 2000
    + + +## `queryCoord.brokerTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • 5000ms, querycoord broker rpc timeout
    		• 5000
    + + +## `queryCoord.collectionRecoverTimes` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • if collection recover times reach the limit during loading state, release it
    		• 3
    + + +## `queryCoord.observerTaskParallel` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the parallel observer dispatcher task number
    		• 16
    + + +## `queryCoord.checkAutoBalanceConfigInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the interval of check auto balance config
    		• 10
    + + +## `queryCoord.checkNodeSessionInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the interval(in seconds) of check querynode cluster session
    		• 60
    + + +## `queryCoord.gracefulStopTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • seconds. force stop node without graceful stop
    		• 5
    + + +## `queryCoord.enableStoppingBalance` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • whether enable stopping balance
    		• true
    + + +## `queryCoord.channelExclusiveNodeFactor` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the least node number for enable channel's exclusive mode
    		• 4
    + + +## `queryCoord.cleanExcludeSegmentInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the time duration of clean pipeline exclude segment which used for filter invalid data, in seconds
    		• 60
    + + +## `queryCoord.ip` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • if not specified, use the first unicastable address
    • + + +## `queryCoord.port` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    19531
    + + +## `queryCoord.grpc.serverMaxSendSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    536870912
    + + +## `queryCoord.grpc.serverMaxRecvSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    268435456
    + + +## `queryCoord.grpc.clientMaxSendSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    268435456
    + + +## `queryCoord.grpc.clientMaxRecvSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    536870912
    + + diff --git a/site/en/reference/sys_config/configure_querynode.md b/site/en/reference/sys_config/configure_querynode.md index ca5f3b47a..0af33d27d 100644 --- a/site/en/reference/sys_config/configure_querynode.md +++ b/site/en/reference/sys_config/configure_querynode.md @@ -2,22 +2,399 @@ id: configure_querynode.md related_key: configure group: system_configuration.md -summary: Learn how to configure query node of Milvus. -title: Query Node-related Configurations +summary: Learn how to configure queryNode for Milvus. --- -# Query Node-related Configurations +# queryNode-related Configurations -This topic introduces the query node-related configurations of Milvus. +Related configuration of queryNode, used to run hybrid search between vector and scalar data. + +## `queryNode.stats.publishInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Interval for querynode to report node information (milliseconds)
    		• 1000
    + + +## `queryNode.segcore.knowhereThreadPoolNumRatio` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The number of threads in knowhere's thread pool. If disk is enabled, the pool size will multiply with knowhereThreadPoolNumRatio([1, 32]).
    		• 4
    + + +## `queryNode.segcore.knowhereScoreConsistency` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • used for zilliz-cloud ; please ignore it for open source.
    		• false
    + + +## `queryNode.segcore.chunkRows` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The number of vectors in a chunk.
    		• 128
    + + +## `queryNode.segcore.interimIndex.enableIndex` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Enable segment build with index to accelerate vector search when segment is in growing or binlog.
    		• true
    + + +## `queryNode.segcore.interimIndex.nlist` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • temp index nlist, recommend to set sqrt(chunkRows), must smaller than chunkRows/8
    		• 128
    + + +## `queryNode.segcore.interimIndex.nprobe` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • nprobe to search small index, based on your accuracy requirement, must smaller than nlist
    		• 16
    + + +## `queryNode.segcore.interimIndex.memExpansionRate` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • extra memory needed by building interim index
    		• 1.15
    + + +## `queryNode.segcore.interimIndex.buildParallelRate` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • the ratio of building interim index parallel matched with cpu num
    		• 0.5
    + + +## `queryNode.loadMemoryUsageFactor` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The multiply factor of calculating the memory usage while loading segments
    		• 1
    + + +## `queryNode.enableDisk` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • enable querynode load disk index, and search on disk index
    		• false
    + + +## `queryNode.maxDiskUsagePercentage` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    95
    + + +## `queryNode.cache.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    true
    + + +## `queryNode.cache.memoryLimit` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • 2 GB, 2 * 1024 *1024 *1024
    		• 2147483648
    + + +## `queryNode.cache.readAheadPolicy` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The read ahead policy of chunk cache, options: `normal, random, sequential, willneed, dontneed`
    		• willneed
    + + +## `queryNode.cache.warmup` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • options: async, sync, off.
    • +
  • Specifies the necessity for warming up the chunk cache.
    • +
  • 1. If set to "sync" or "async," the original vector data will be synchronously/asynchronously loaded into the
    • +
  • chunk cache during the load process. This approach has the potential to substantially reduce query/search latency
    • +
  • for a specific duration post-load, albeit accompanied by a concurrent increase in disk usage;
    • +
  • 2. If set to "off," original vector data will only be loaded into the chunk cache during search/query.
    		• async
    + + +## `queryNode.mmap.mmapEnabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Enable mmap for loading data
    		• false
    + + +## `queryNode.lazyload.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Enable lazyload for loading data
    		• false
    + + +## `queryNode.lazyload.waitTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • max wait timeout duration in milliseconds before start to do lazyload search and retrieve
    		• 30000
    -Query node performs hybrid search of vector and scalar data on both incremental and historical data. -Under this section, you can configure query node port, graceful time, etc. +## `queryNode.lazyload.requestResourceTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • max timeout in milliseconds for waiting request resource for lazy load, 5s by default
    		• 5000
    -## `queryNode.gracefulTime` +## `queryNode.lazyload.requestResourceRetryInterval` - +
    @@ -27,20 +404,16 @@ Under this section, you can configure query node port, graceful time, etc. - +
  • retry interval in milliseconds for waiting request resource for lazy load, 2s by default
    • +
    Description
    -
  • The minimum time before the newly inserted data can be searched.
    • -
  • Unit: ms
    • -
  • Milvus executes a search request directly when the search message timestamp is earlier the query node system time.
    • -
  • When the search message timestamp is later than the query node system time, the Milvus waits until that the time diferrence between query node system time and the timestamp is less than this parameter, and then executes the search request.
    • -     		0 2000
    -## `queryNode.port` +## `queryNode.lazyload.maxRetryTimes` - +
    @@ -49,16 +422,17 @@ Under this section, you can configure query node port, graceful time, etc. - - + +
    Description
    TCP port of query node.21123 +
  • max retry times for lazy load, 1 by default
    		• 1
    -## `queryNode.grpc.serverMaxRecvSize` +## `queryNode.lazyload.maxEvictPerRetry` - +
    @@ -68,17 +442,106 @@ Under this section, you can configure query node port, graceful time, etc. - +
  • max evict count for lazy load, 1 by default
    • +
    Description
    -
  • The maximum size of each RPC request that the query node can receive.
    • -
  • Unit: Byte
    • -     		2147483647 1
    -## `queryNode.grpc.serverMaxSendSize` - +## `queryNode.grouping.enabled` + +
    + + + + + + + + + + + + +
    DescriptionDefault Value
    true
    + + +## `queryNode.grouping.maxNQ` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    1000
    + + +## `queryNode.grouping.topKMergeRatio` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    20
    + + +## `queryNode.scheduler.receiveChanSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    10240
    + + +## `queryNode.scheduler.unsolvedQueueSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    10240
    + + +## `queryNode.scheduler.maxReadConcurrentRatio` + + @@ -88,17 +551,20 @@ Under this section, you can configure query node port, graceful time, etc. - +
  • maxReadConcurrentRatio is the concurrency ratio of read task (search task and query task).
    • +
  • Max read concurrency would be the value of hardware.GetCPUNum * maxReadConcurrentRatio.
    • +
  • It defaults to 2.0, which means max read concurrency would be the value of hardware.GetCPUNum * 2.
    • +
  • Max read concurrency must greater than or equal to 1, and less than or equal to hardware.GetCPUNum * 100.
    • +
  • (0, 100]
    • +
    Description
    -
  • The maximum size of each respond the query node can send when receiving an RPC request.
    • -
  • Unit: Byte
    • -     		2147483647 1
    -## `queryNode.grpc.clientMaxRecvSize` - +## `queryNode.scheduler.cpuRatio` + +
    @@ -108,17 +574,59 @@ Under this section, you can configure query node port, graceful time, etc. - +
  • ratio used to estimate read task cpu usage.
    • +
    Description
    -
  • The maximum size of each respond that the query node can receive when sending an RPC request.
    • -
  • Unit: Byte
    • -     		104857600 10
    -## `queryNode.grpc.clientMaxSendSize` - +## `queryNode.scheduler.maxTimestampLag` + +
    + + + + + + + + + + + + +
    DescriptionDefault Value
    86400
    + + +## `queryNode.scheduler.scheduleReadPolicy.name` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • fifo: A FIFO queue support the schedule.
    • +
  • user-task-polling:
    • +
  • The user's tasks will be polled one by one and scheduled.
    • +
  • Scheduling is fair on task granularity.
    • +
  • The policy is based on the username for authentication.
    • +
  • And an empty username is considered the same user.
    • +
  • When there are no multi-users, the policy decay into FIFO"
    		• fifo
    + + +## `queryNode.scheduler.scheduleReadPolicy.taskQueueExpire` + + @@ -128,17 +636,16 @@ Under this section, you can configure query node port, graceful time, etc. - +
  • Control how long (many seconds) that queue retains since queue is empty
    • +
    Description
    -
  • The maximum size of each RPC request that the query node can send.
    • -
  • Unit: Byte
    • -     		104857600 60
    -## `queryNode.replicas` - +## `queryNode.scheduler.scheduleReadPolicy.enableCrossUserGrouping` + +
    @@ -147,17 +654,17 @@ Under this section, you can configure query node port, graceful time, etc. - - + +
    Description
    - The number of in-memory replicas of data segments that are created on query nodes when a collection is loaded. In a Standalone deployment, the maximum value is 1. For more information, refer to In-Memory Replica. - 1 +
  • Enable Cross user grouping when using user-task-polling policy. (Disable it if user's task can not merge each other)
    		• false
    -## `queryNode.stats.publishInterval` - +## `queryNode.scheduler.scheduleReadPolicy.maxPendingTaskPerUser` + +
    @@ -167,14 +674,13 @@ Under this section, you can configure query node port, graceful time, etc. - +
  • Max pending task per user in scheduler
    • +
    Description
    -
  • The interval that query node publishes the node statistics information, including segment status, cpu usage, memory usage, health status, etc.
    • -
  • Unit: ms
    • -     		1000 1024
    + ## `queryNode.dataSync.flowGraph.maxQueueLength` @@ -187,18 +693,16 @@ Under this section, you can configure query node port, graceful time, etc. - +
  • Maximum length of task queue in flowgraph
    • +
    -
  • The maximum size of task queue cache in flow graph in query node.
    • -
  • Unit: MB
    • -
  • Query node uses flow graph to subscribe to and organize the message flow.
    • -     		1024 16
    -## `queryNode.segcore.chunkRows` - +## `queryNode.dataSync.flowGraph.maxParallelism` + +
    @@ -208,16 +712,16 @@ Under this section, you can configure query node port, graceful time, etc. +
  • Maximum number of tasks executed in parallel in the flowgraph
    • Description
    - Row count by which Segcore divides a segment into chunks. - 1024
    -## `queryNode.segcore.InterimIndex` - +## `queryNode.enableSegmentPrune` + +
    @@ -227,15 +731,119 @@ Under this section, you can configure query node port, graceful time, etc. - +
  • use partition prune function on shard delegator
    • + + + +
    Description
    - Whether to create a temporary index for growing segments and sealed segments not yet indexed, improving search performance.
    -
    • - Milvus will eventually seals and indexes all segments, but enabling this optimizes search performance for immediate queries following data insertion. -
      • -
    • - This defaults to `true`, indicating that Milvus creates temporary index for growing segments and the sealed segments that are not indexed upon searches. -
    -     		true false
    + + +## `queryNode.ip` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • if not specified, use the first unicastable address
    • + + +## `queryNode.port` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    21123
    + + +## `queryNode.grpc.serverMaxSendSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    536870912
    + + +## `queryNode.grpc.serverMaxRecvSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    268435456
    + + +## `queryNode.grpc.clientMaxSendSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    268435456
    + + +## `queryNode.grpc.clientMaxRecvSize` + + + + + + + + + + + +
    DescriptionDefault Value
    536870912
    + + diff --git a/site/en/reference/sys_config/configure_quota_limits.md b/site/en/reference/sys_config/configure_quota_limits.md deleted file mode 100644 index cb8f1430e..000000000 --- a/site/en/reference/sys_config/configure_quota_limits.md +++ /dev/null @@ -1,736 +0,0 @@ ---- -id: configure_quota_limits.md -related_key: configure -group: system_configuration.md -summary: Learn how to configure quotas and limitations. -title: Quota- and Limit-related configurations ---- - -# Quota- and Limit-related configurations - -This topic introduces the configuration items related to quotas and limits in Milvus. - -Some of these configuration items are used to set thresholds for Milvus to proactively throttle DDL/DML/DQL requests related to collections, partitions, indexes, etc. - -Some of them are used to set backpressure signals that force Milvus to lower the rate of DDL/DML/DQL requests. - -## `quotaAndLimits.limits.maxCollectionNumPerDB` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Maximum number of collections per database.64
    - -## `quotaAndLimits.ddl.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether DDL request throttling is enabled.False
    - -## `quotaAndLimits.ddl.collectionRate` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of collection-related DDL requests per second.
    • -
  • Setting this item to 10 indicates that Milvus processes no more than 10 collection-related DDL requests per second, including collection creation requests, collection drop requests, collection load requests, and collection release requests.
    • -
  • To use this setting, set quotaAndLimits.ddl.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.ddl.partitionRate` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of partition-related DDL requests per second.
    • -
  • Setting this item to 10 indicates that Milvus processes no more than 10 partition-related requests per second, including partition creation requests, partition drop requests, partition load requests, and partition release requests.
    • -
  • To use this setting, set quotaAndLimits.ddl.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.indexRate.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether index-related request throttling is enabled.False
    - -## `quotaAndLimits.indexRate.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of index-related requests per second.
    • -
  • Setting this item to 10 indicates that Milvus processes no more than 10 partition-related requests per second, including index creation requests and index drop requests.
    • -
  • To use this setting, set quotaAndLimits.indexRate.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.flushRate.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether flush request throttling is enabled.False
    - -## `quotaAndLimits.flush.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of flush requests per second.
    • -
  • Setting this item to 10 indicates that Milvus processes no more than 10 flush requests per second.
    • -
  • To use this setting, set quotaAndLimits.flushRate.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.compaction.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether flush request throttling is enabled.False
    - -## `quotaAndLimits.compaction.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of manual-compaction requests per second.
    • -
  • Setting this item to 10 indicates that Milvus processes no more than 10 manual-compaction requests per second.
    • -
  • To use this setting, set quotaAndLimits.compaction.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.dml.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether DML request throttling is enabled.False
    - -## `quotaAndLimits.dml.insertRate.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Highest data insertion rate per second.
    • -
  • Setting this item to 5 indicates that Milvus only allows data insertion at the rate of 5 MB/s.
    • -
  • To use this setting, set quotaAndLimits.dml.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.dml.insertRate.collection.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Highest data insertion rate per collection per second.
    • -
  • Setting this item to 5 indicates that Milvus only allows data insertion to any collection at the rate of 5 MB/s.
    • -
  • To use this setting, set quotaAndLimits.dml.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.dml.deleteRate.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Highest data deletion rate per second.
    • -
  • Setting this item to 0.1 indicates that Milvus only allows data deletion at the rate of 0.1 MB/s.
    • -
  • To use this setting, set quotaAndLimits.dml.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.dml.deleteRate.collection.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Highest data deletion rate per second.
    • -
  • Setting this item to 0.1 indicates that Milvus only allows data deletion from any collection at the rate of 0.1 MB/s.
    • -
  • To use this setting, set quotaAndLimits.dml.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.dql.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether DQL request throttling is enabled.False
    - -## `quotaAndLimits.dql.searchRate.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of vectors to search per second.
    • -
  • Setting this item to 100 indicates that Milvus only allows searching 100 vectors per second no matter whether these 100 vectors are all in one search or scattered across multiple searches.
    • -
  • To use this setting, set quotaAndLimits.dql.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.dql.searchRate.collection.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of vectors to search per collection per second.
    • -
  • Setting this item to 100 indicates that Milvus only allows searching 100 vectors per second per collection no matter whether these 100 vectors are all in one search or scattered across multiple searches.
    • -
  • To use this setting, set quotaAndLimits.dql.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.dql.queryRate.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of queries per second.
    • -
  • Setting this item to 100 indicates that Milvus only allows 100 queries per second.
    • -
  • To use this setting, set quotaAndLimits.dql.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.dql.queryRate.collection.max` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of queries per collection per second.
    • -
  • Setting this item to 100 indicates that Milvus only allows 100 queries per collection per second.
    • -
  • To use this setting, set quotaAndLimits.dql.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.limitWriting.ttProtection.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether the backpressure based on time tick delay is enabled.False
    - -## `quotaAndLimits.limitWriting.ttProtection.maxTimeTickDelay` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum time tick delay. A time tick delay is the difference between RootCoord TSO and the minimum time tick of all flow graphs on DataNodes and QueryNodes.
    • -
  • Setting this item to 300 indicates that Milvus reduces the DML request rate as the delay increases and drops all DML requests once the delay reaches the set maximum in seconds.
    • -
  • To use this setting, set quotaAndLimits.limitWriting.ttProtection.enabled to true at the same time.
    • -     		300
    - -## `quotaAndLimits.limitWriting.memProtection.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether the backpressure based on memory water level is enabled.False
    - -## `quotaAndLimits.limitWriting.memProtection.dataNodeMemoryLowWaterLevel` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Low memory water level on DataNodes. The memory water level is the ratio between the used memory and total memory on DataNodes.
    • -
  • Setting this item to 0.85 indicates that Milvus reduces the DML request rate as the memory water level on DataNodes reaches the set value.
    • -
  • To use this setting, set quotaAndLimits.limitWriting.memProtection.enabled to true at the same time.
    • -     		0.85
    - -## `quotaAndLimits.limitWriting.memProtection.queryNodeMemoryLowWaterLevel` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Low memory water level on QueryNodes. The memory water level is the ratio between the used memory and total memory on QueryNodes.
    • -
  • Setting this item to 0.85 indicates that Milvus reduces the DML request rate as the memory water level on QueryNodes reaches the set value.
    • -
  • To use this setting, set quotaAndLimits.limitWriting.memProtection.enabled to true at the same time.
    • -     		0.85
    - -## `quotaAndLimits.limitWriting.memProtection.dataNodeMemoryHighWaterLevel` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • High memory water level on DataNodes. The memory water level is the ratio between the used memory and total memory on DataNodes.
    • -
  • Setting this item to 0.95 indicates that Milvus drops all DML requests as the memory water level on DataNodes reaches the set value.
    • -
  • To use this setting, set quotaAndLimits.limitWriting.memProtection.enabled to true at the same time.
    • -     		0.95
    - -## `quotaAndLimits.limitWriting.memProtection.queryNodeMemoryHighWaterLevel` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • High memory water level on QueryNodes. The memory water level is the ratio between the used memory and total memory on QueryNodes.
    • -
  • Setting this item to 0.95 indicates that Milvus drops all DML requests as the memory water level on QueryNodes reaches the set value.
    • -
  • To use this setting, set quotaAndLimits.limitWriting.memProtection.enabled to true at the same time.
    • -     		0.95
    - -## `quotaAndLimits.limitWriting.diskProtection.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether the backpressure based on disk quota is enabled.False
    - -## `quotaAndLimits.limitWriting.diskProtection.diskQuota` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Disk quota allocated to binlog.
    • -
  • Setting this item to 8192 indicates that Milvus drops all DML requests as the size of binlog reaches the set value.
    • -
  • To use this setting, set quotaAndLimits.limitWriting.diskProtection.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.limitWriting.diskProtection.diskQuotaPerCollection` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Disk quota allocated to binlog per collection.
    • -
  • Setting this item to 8192 indicates that Milvus drops all DML requests in a collection as the size of binlog of the collection reaches the set value.
    • -
  • To use this setting, set quotaAndLimits.limitWriting.diskProtection.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.limitWriting.forceDeny` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether to manually configure Milvus to drop all DML requests.False
    - -## `quotaAndLimits.limitReading.queueProtection.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether the backpressure based on the lengths of the search and query queue is enabled.False
    - -## `quotaAndLimits.limitReading.queueProtection.nqInQueueThreshold` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Maximum number of search vectors or queries. Note that a search request containing multiple search vectors are regarded as multiple seaches, while a query is the same as a search request containing only one search vector.
    • -
  • Setting this item to 10000 indicates that Milvus reduces the DQL request rate as the number of searches and queries reaches the set maximum in milliseconds, and the backpressure is resolved when the number decreases below the set value. The reduction rate id determined by quotaAndLimits.limitReading.coolOffSpeed.
    • -
  • To use this setting, set quotaAndLimits.limitReading.queueProtection.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.limitReading.queueProtection.queueLatencyThreshold` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Average latency of the queued searches and queries. Note that a search request containing multiple search vectors are regarded as multiple seaches, while a query is the same as a search request containing only one search vector.
    • -
  • Setting this item to 200 indicates that Milvus reduces the DQL request rate as the average latency reaches the set maximum in milliseconds, and the backpressure is resolved when the number decreases below the set value in milliseconds. The reduction rate id determined by quotaAndLimits.limitReading.coolOffSpeed.
    • -
  • To use this setting, set quotaAndLimits.limitReading.queueProtection.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.limitReading.resultProtection.enabled` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether the backpressure based on the rate of the query results is enabled.False
    - -## `quotaAndLimits.limitReading.resultProtection.maxReadResultRate` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    -
  • Rate of the data returned to the client.
    • -
  • Setting this item to 2 indicates that Milvus reduces the DQL request rate as the data rate reaches the set maximum in MB/s, and the backpressure is resolved when the number decreases below the set value in MB/s. The reduction rate id determined by quotaAndLimits.limitReading.coolOffSpeed.
    • -
  • To use this setting, set quotaAndLimits.limitReading.resultProtection.enabled to true at the same time.
    • -     		∞
    - -## `quotaAndLimits.limitWriting.forceDeny` - - - - - - - - - - - - - - -
    DescriptionDefault Value
    Whether to manually configure Milvus to drop all DQL requests.False
    diff --git a/site/en/reference/sys_config/configure_quotaandlimits.md b/site/en/reference/sys_config/configure_quotaandlimits.md new file mode 100644 index 000000000..aeb3bdadf --- /dev/null +++ b/site/en/reference/sys_config/configure_quotaandlimits.md @@ -0,0 +1,1428 @@ +--- +id: configure_quotaandlimits.md +related_key: configure +group: system_configuration.md +summary: Learn how to configure quotaAndLimits for Milvus. +--- + +# quotaAndLimits-related Configurations + +QuotaConfig, configurations of Milvus quota and limits. + +By default, we enable: + + 1. TT protection; + + 2. Memory protection. + + 3. Disk quota protection. + +You can enable: + + 1. DML throughput limitation; + + 2. DDL, DQL qps/rps limitation; + + 3. DQL Queue length/latency protection; + + 4. DQL result rate protection; + +If necessary, you can also manually force to deny RW requests. + +## `quotaAndLimits.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • `true` to enable quota and limits, `false` to disable.
    		• true
    + + +## `quotaAndLimits.quotaCenterCollectInterval` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • quotaCenterCollectInterval is the time interval that quotaCenter
    • +
  • collects metrics from Proxies, Query cluster and Data cluster.
    • +
  • seconds, (0 ~ 65536)
    		• 3
    + + +## `quotaAndLimits.ddl.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `quotaAndLimits.ddl.collectionRate` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps, default no limit, rate for CreateCollection, DropCollection, LoadCollection, ReleaseCollection
    		• -1
    + + +## `quotaAndLimits.ddl.partitionRate` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps, default no limit, rate for CreatePartition, DropPartition, LoadPartition, ReleasePartition
    		• -1
    + + +## `quotaAndLimits.ddl.db.collectionRate` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps of db level , default no limit, rate for CreateCollection, DropCollection, LoadCollection, ReleaseCollection
    		• -1
    + + +## `quotaAndLimits.ddl.db.partitionRate` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps of db level, default no limit, rate for CreatePartition, DropPartition, LoadPartition, ReleasePartition
    		• -1
    + + +## `quotaAndLimits.indexRate.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `quotaAndLimits.indexRate.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps, default no limit, rate for CreateIndex, DropIndex
    		• -1
    + + +## `quotaAndLimits.indexRate.db.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps of db level, default no limit, rate for CreateIndex, DropIndex
    		• -1
    + + +## `quotaAndLimits.flushRate.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `quotaAndLimits.flushRate.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps, default no limit, rate for flush
    		• -1
    + + +## `quotaAndLimits.flushRate.collection.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps, default no limit, rate for flush at collection level.
    		• -1
    + + +## `quotaAndLimits.flushRate.db.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps of db level, default no limit, rate for flush
    		• -1
    + + +## `quotaAndLimits.compactionRate.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `quotaAndLimits.compactionRate.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps, default no limit, rate for manualCompaction
    		• -1
    + + +## `quotaAndLimits.compactionRate.db.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps of db level, default no limit, rate for manualCompaction
    		• -1
    + + +## `quotaAndLimits.dml.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • dml limit rates, default no limit.
    • +
  • The maximum rate will not be greater than max.
    		• false
    + + +## `quotaAndLimits.dml.insertRate.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.insertRate.db.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.insertRate.collection.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.insertRate.partition.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.upsertRate.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.upsertRate.db.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.upsertRate.collection.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.upsertRate.partition.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.deleteRate.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.deleteRate.db.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.deleteRate.collection.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.deleteRate.partition.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.dml.bulkLoadRate.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit, not support yet. TODO: limit bulkLoad rate
    		• -1
    + + +## `quotaAndLimits.dml.bulkLoadRate.db.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit, not support yet. TODO: limit db bulkLoad rate
    		• -1
    + + +## `quotaAndLimits.dml.bulkLoadRate.collection.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit, not support yet. TODO: limit collection bulkLoad rate
    		• -1
    + + +## `quotaAndLimits.dml.bulkLoadRate.partition.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB/s, default no limit, not support yet. TODO: limit partition bulkLoad rate
    		• -1
    + + +## `quotaAndLimits.dql.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • dql limit rates, default no limit.
    • +
  • The maximum rate will not be greater than max.
    		• false
    + + +## `quotaAndLimits.dql.searchRate.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • vps (vectors per second), default no limit
    		• -1
    + + +## `quotaAndLimits.dql.searchRate.db.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • vps (vectors per second), default no limit
    		• -1
    + + +## `quotaAndLimits.dql.searchRate.collection.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • vps (vectors per second), default no limit
    		• -1
    + + +## `quotaAndLimits.dql.searchRate.partition.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • vps (vectors per second), default no limit
    		• -1
    + + +## `quotaAndLimits.dql.queryRate.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps, default no limit
    		• -1
    + + +## `quotaAndLimits.dql.queryRate.db.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps, default no limit
    		• -1
    + + +## `quotaAndLimits.dql.queryRate.collection.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps, default no limit
    		• -1
    + + +## `quotaAndLimits.dql.queryRate.partition.max` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • qps, default no limit
    		• -1
    + + +## `quotaAndLimits.limits.maxCollectionNum` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    65536
    + + +## `quotaAndLimits.limits.maxCollectionNumPerDB` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    65536
    + + +## `quotaAndLimits.limits.maxInsertSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • maximum size of a single insert request, in bytes, -1 means no limit
    		• -1
    + + +## `quotaAndLimits.limits.maxResourceGroupNumOfQueryNode` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • maximum number of resource groups of query nodes
    		• 1024
    + + +## `quotaAndLimits.limitWriting.forceDeny` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • forceDeny false means dml requests are allowed (except for some
    • +
  • specific conditions, such as memory of nodes to water marker), true means always reject all dml requests.
    		• false
    + + +## `quotaAndLimits.limitWriting.ttProtection.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `quotaAndLimits.limitWriting.ttProtection.maxTimeTickDelay` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • maxTimeTickDelay indicates the backpressure for DML Operations.
    • +
  • DML rates would be reduced according to the ratio of time tick delay to maxTimeTickDelay,
    • +
  • if time tick delay is greater than maxTimeTickDelay, all DML requests would be rejected.
    • +
  • seconds
    		• 300
    + + +## `quotaAndLimits.limitWriting.memProtection.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • When memory usage > memoryHighWaterLevel, all dml requests would be rejected;
    • +
  • When memoryLowWaterLevel < memory usage < memoryHighWaterLevel, reduce the dml rate;
    • +
  • When memory usage < memoryLowWaterLevel, no action.
    		• true
    + + +## `quotaAndLimits.limitWriting.memProtection.dataNodeMemoryLowWaterLevel` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • (0, 1], memoryLowWaterLevel in DataNodes
    		• 0.85
    + + +## `quotaAndLimits.limitWriting.memProtection.dataNodeMemoryHighWaterLevel` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • (0, 1], memoryHighWaterLevel in DataNodes
    		• 0.95
    + + +## `quotaAndLimits.limitWriting.memProtection.queryNodeMemoryLowWaterLevel` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • (0, 1], memoryLowWaterLevel in QueryNodes
    		• 0.85
    + + +## `quotaAndLimits.limitWriting.memProtection.queryNodeMemoryHighWaterLevel` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • (0, 1], memoryHighWaterLevel in QueryNodes
    		• 0.95
    + + +## `quotaAndLimits.limitWriting.growingSegmentsSizeProtection.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • No action will be taken if the growing segments size is less than the low watermark.
    • +
  • When the growing segments size exceeds the low watermark, the dml rate will be reduced,
    • +
  • but the rate will not be lower than minRateRatio * dmlRate.
    		• false
    + + +## `quotaAndLimits.limitWriting.growingSegmentsSizeProtection.minRateRatio` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    0.5
    + + +## `quotaAndLimits.limitWriting.growingSegmentsSizeProtection.lowWaterLevel` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    0.2
    + + +## `quotaAndLimits.limitWriting.growingSegmentsSizeProtection.highWaterLevel` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    0.4
    + + +## `quotaAndLimits.limitWriting.diskProtection.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • When the total file size of object storage is greater than `diskQuota`, all dml requests would be rejected;
    		• true
    + + +## `quotaAndLimits.limitWriting.diskProtection.diskQuota` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB, (0, +inf), default no limit
    		• -1
    + + +## `quotaAndLimits.limitWriting.diskProtection.diskQuotaPerDB` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB, (0, +inf), default no limit
    		• -1
    + + +## `quotaAndLimits.limitWriting.diskProtection.diskQuotaPerCollection` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB, (0, +inf), default no limit
    		• -1
    + + +## `quotaAndLimits.limitWriting.diskProtection.diskQuotaPerPartition` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • MB, (0, +inf), default no limit
    		• -1
    + + +## `quotaAndLimits.limitReading.forceDeny` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • forceDeny false means dql requests are allowed (except for some
    • +
  • specific conditions, such as collection has been dropped), true means always reject all dql requests.
    		• false
    + + +## `quotaAndLimits.limitReading.queueProtection.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `quotaAndLimits.limitReading.queueProtection.nqInQueueThreshold` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • nqInQueueThreshold indicated that the system was under backpressure for Search/Query path.
    • +
  • If NQ in any QueryNode's queue is greater than nqInQueueThreshold, search&query rates would gradually cool off
    • +
  • until the NQ in queue no longer exceeds nqInQueueThreshold. We think of the NQ of query request as 1.
    • +
  • int, default no limit
    		• -1
    + + +## `quotaAndLimits.limitReading.queueProtection.queueLatencyThreshold` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • queueLatencyThreshold indicated that the system was under backpressure for Search/Query path.
    • +
  • If dql latency of queuing is greater than queueLatencyThreshold, search&query rates would gradually cool off
    • +
  • until the latency of queuing no longer exceeds queueLatencyThreshold.
    • +
  • The latency here refers to the averaged latency over a period of time.
    • +
  • milliseconds, default no limit
    		• -1
    + + +## `quotaAndLimits.limitReading.resultProtection.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    false
    + + +## `quotaAndLimits.limitReading.resultProtection.maxReadResultRate` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • maxReadResultRate indicated that the system was under backpressure for Search/Query path.
    • +
  • If dql result rate is greater than maxReadResultRate, search&query rates would gradually cool off
    • +
  • until the read result rate no longer exceeds maxReadResultRate.
    • +
  • MB/s, default no limit
    		• -1
    + + +## `quotaAndLimits.limitReading.resultProtection.maxReadResultRatePerDB` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    -1
    + + +## `quotaAndLimits.limitReading.resultProtection.maxReadResultRatePerCollection` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    -1
    + + +## `quotaAndLimits.limitReading.coolOffSpeed` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • colOffSpeed is the speed of search&query rates cool off.
    • +
  • (0, 1]
    		• 0.9
    + + diff --git a/site/en/reference/sys_config/configure_rocksmq.md b/site/en/reference/sys_config/configure_rocksmq.md index e80ef6f46..16b9d683b 100644 --- a/site/en/reference/sys_config/configure_rocksmq.md +++ b/site/en/reference/sys_config/configure_rocksmq.md @@ -2,17 +2,39 @@ id: configure_rocksmq.md related_key: configure group: system_configuration.md -summary: Learn how to configure RocksMQ for Milvus standalone. -title: RocksMQ-related Configurations +summary: Learn how to configure rocksmq for Milvus. --- -# RocksMQ-related Configurations +# rocksmq-related Configurations -This topic introduces the RocksMQ-related configurations of Milvus. +If you want to enable kafka, needs to comment the pulsar configs -RocksMQ is the underlying engine supporting Milvus standalone's reliable storage and publication/subscription of message streams. It is implemented on the basis of RocksDB. +kafka: + + brokerList: + + saslUsername: + + saslPassword: + + saslMechanisms: + + securityProtocol: + + ssl: + + enabled: false # whether to enable ssl mode + + tlsCert: # path to client's public key (PEM) used for authentication + + tlsKey: # path to client's private key (PEM) used for authentication + + tlsCaCert: # file or directory path to CA certificate(s) for verifying the broker's key + + tlsKeyPassword: # private key passphrase for use with ssl.key.location and set_ssl_cert(), if any + + readTimeout: 10 -Under this section, you can configure message size, retention time and size, etc. ## `rocksmq.path` @@ -27,17 +49,33 @@ Under this section, you can configure message size, retention time and size, etc -
  • Prefix of the key to where Milvus stores data in RocksMQ.
    • -
  • Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
    • -
  • It is recommended to change this parameter before starting Milvus for the first time.
    • -
  • Set an easy-to-identify root key prefix for Milvus if etcd service already exists.
    • - +
  • The path where the message is stored in rocksmq
    • +
  • please adjust in embedded Milvus: /tmp/milvus/rdb_data
    • /var/lib/milvus/rdb_data +## `rocksmq.lrucacheratio` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • rocksdb cache memory ratio
    		• 0.06
    + + ## `rocksmq.rocksmqPageSize` @@ -50,10 +88,8 @@ Under this section, you can configure message size, retention time and size, etc - +
  • 64 MB, 64 * 1024 * 1024 bytes, The size of each page of messages in rocksmq
    • +
    -
  • The maximum size of messages in each page in RocksMQ. Messages in RocksMQ are checked and cleared (when expired) in batch based on this parameters.
    • -
  • Unit: Byte
    • -     		2147483648 67108864
    @@ -71,10 +107,8 @@ Under this section, you can configure message size, retention time and size, etc -
  • The maximum retention time of acked messages in RocksMQ. Acked messages in RocksMQ are retained for the specified period of time and then cleared.
    • -
  • Unit: Minute
    • - - 10080 +
  • 3 days, 3 * 24 * 60 minutes, The retention time of the message in rocksmq.
    • + 4320 @@ -92,14 +126,13 @@ Under this section, you can configure message size, retention time and size, etc -
  • The maximum retention size of acked messages of each topic in RocksMQ. Acked messages in each topic are cleared if their size exceed this parameter.
    • -
  • Unit: MB
    • - +
  • 8 GB, 8 * 1024 MB, The retention size of the message in rocksmq.
    • 8192 + ## `rocksmq.compactionInterval` @@ -112,17 +145,16 @@ Under this section, you can configure message size, retention time and size, etc +
  • 1 day, trigger rocksdb compaction every day to remove deleted data
    • -
  • Time interval to trigger rocksdb compaction to remove deleted data.
    • -
  • Unit: Second
    • -     		86400
    -## `rocksmq.lrucacheratio` - +## `rocksmq.compressionTypes` + +
    @@ -132,9 +164,10 @@ Under this section, you can configure message size, retention time and size, etc - +
  • compaction compression type, only support use 0,7. 0 means not compress, 7 will use zstd. Length of types means num of rocksdb level.
    • +
    Description
    -
  • Rocksdb cache memory ratio.
    • -     		0.06 0,0,7,7,7
    + + diff --git a/site/en/reference/sys_config/configure_rootcoord.md b/site/en/reference/sys_config/configure_rootcoord.md index beee0c14d..49325210a 100644 --- a/site/en/reference/sys_config/configure_rootcoord.md +++ b/site/en/reference/sys_config/configure_rootcoord.md @@ -2,22 +2,16 @@ id: configure_rootcoord.md related_key: configure group: system_configuration.md -summary: Learn how to configure root coordinator of Milvus. -title: Root Coordinator-related Configurations +summary: Learn how to configure rootCoord for Milvus. --- -# Root Coordinator-related Configurations +# rootCoord-related Configurations -This topic introduces the root coordinator-related configurations of Milvus. +Related configuration of rootCoord, used to handle data definition language (DDL) and data control language (DCL) requests -Root coordinator (root coord) handles data definition language (DDL) and data control language (DCL) requests, manages TSO (timestamp Oracle), and publishes time tick messages. - -Under this section, you can configure root coord address, index building threshold, etc. - - -## `rootCoord.address` +## `rootCoord.dmlChannelNum` - +
    @@ -27,18 +21,16 @@ Under this section, you can configure root coord address, index building thresho - +
  • The number of dml channels created at system startup
    • +
    Description
    -
  • TCP/IP address of root coordinator.
    • -
  • Root coordinator monitors all IPv4 addresses if this parameter is set as 0.0.0.0.
    • -     		localhost 16
    -## `rootCoord.port` +## `rootCoord.maxPartitionNum` - +
    @@ -47,16 +39,17 @@ Under this section, you can configure root coord address, index building thresho - - + +
    Description
    TCP port of root coordinator.53100 +
  • Maximum number of partitions in a collection
    		• 4096
    -## `rootCoord.grpc.serverMaxRecvSize` +## `rootCoord.minSegmentSizeToEnableIndex` - +
    @@ -66,18 +59,16 @@ Under this section, you can configure root coord address, index building thresho - +
  • It's a threshold. When the segment size is less than this value, the segment will not be indexed
    • +
    Description
    -
  • The maximum size of each RPC request that the root coord can receive.
    • -
  • Unit: Byte
    • -     		2147483647 1024
    -## `rootCoord.grpc.serverMaxSendSize` +## `rootCoord.enableActiveStandby` - +
    @@ -86,19 +77,16 @@ Under this section, you can configure root coord address, index building thresho - - + +
    Description
    -
  • The maximum size of each respond the root coord can send when receiving an RPC request.
    • -
  • Unit: Byte
    • -     		2147483647false
    -## `rootCoord.grpc.clientMaxRecvSize` +## `rootCoord.maxDatabaseNum` - +
    @@ -108,17 +96,16 @@ Under this section, you can configure root coord address, index building thresho - +
  • Maximum number of database
    • +
    Description
    -
  • The maximum size of each respond that the root coord can receive when sending an RPC request.
    • -
  • Unit: Byte
    • -     		104857600 64
    -## `rootCoord.grpc.clientMaxSendSize` - +## `rootCoord.maxGeneralCapacity` + +
    @@ -128,18 +115,16 @@ Under this section, you can configure root coord address, index building thresho - +
  • upper limit for the sum of of product of partitionNumber and shardNumber
    • +
    Description
    -
  • The maximum size of each RPC request that the root coord can send.
    • -
  • Unit: Byte
    • -     		104857600 65536
    -## `rootCoord.activeStandby.enabled` +## `rootCoord.gracefulStopTimeout` - +
    @@ -149,16 +134,16 @@ Under this section, you can configure root coord address, index building thresho - +
  • seconds. force stop node without graceful stop
    • +
    Description
    - Whether the rootCoord works in active-standby mode. - false 5
    -## `rootCoord.replicas` - +## `rootCoord.ip` + +
    @@ -168,16 +153,16 @@ Under this section, you can configure root coord address, index building thresho - +
  • if not specified, use the first unicastable address
    • +
    Description
    - Number of rootCoord pods. This is required if `rootCoord.activeStandby.enabled` is set to `true`. - 1
    -## `rootCoord.dmlChannelNum` - +## `rootCoord.port` + +
    @@ -186,18 +171,16 @@ Under this section, you can configure root coord address, index building thresho - - + +
    Description
    - The number of DML-Channels to create at the root coord startup. - 25653100
    -## `rootCoord.maxPartitionNum` +## `rootCoord.grpc.serverMaxSendSize` - +
    @@ -206,20 +189,16 @@ Under this section, you can configure root coord address, index building thresho - - + +
    Description
    -
  • The maximum number of partitions in each collection.
    • -
  • New partitions cannot be created if this parameter is set as 0 or 1.
    • -
  • Range: [0, INT64MAX]
    • -     		4096536870912
    -## `rootCoord.minSegmentSizeToEnableIndex` +## `rootCoord.grpc.serverMaxRecvSize` - +
    @@ -228,19 +207,16 @@ Under this section, you can configure root coord address, index building thresho - - + +
    Description
    -
  • The minimum row count of a segment required for creating index.
    • -
  • Segments with smaller size than this parameter will not be indexed, and will be searched with brute force.
    • -     		1024268435456
    -## `rootCoord.importTaskExpiration` +## `rootCoord.grpc.clientMaxSendSize` - +
    @@ -249,19 +225,16 @@ Under this section, you can configure root coord address, index building thresho - - + +
    Description
    -
  • A file import task expires after importTaskExpiration seconds.
    • -
  • Unit: second
    • -
  • You should also change the value of the parameter in the internal/util/paramtable/component_param.go file.
    • -     		900268435456
    -## `rootCoord.importTaskRetention` - +## `rootCoord.grpc.clientMaxRecvSize` + +
    @@ -270,11 +243,10 @@ Under this section, you can configure root coord address, index building thresho - - + + -
    Description
    -
  • Milvus keeps the record of import tasks for at least importTaskRetention seconds.
    • -
  • You should also change the value of the parameter in the internal/util/paramtable/component_param.go file.
    • -     		86400536870912
    \ No newline at end of file + + + diff --git a/site/en/reference/sys_config/configure_tikv.md b/site/en/reference/sys_config/configure_tikv.md new file mode 100644 index 000000000..cf9e450dc --- /dev/null +++ b/site/en/reference/sys_config/configure_tikv.md @@ -0,0 +1,205 @@ +--- +id: configure_tikv.md +related_key: configure +group: system_configuration.md +summary: Learn how to configure tikv for Milvus. +--- + +# tikv-related Configurations + +Related configuration of tikv, used to store Milvus metadata. + +Notice that when TiKV is enabled for metastore, you still need to have etcd for service discovery. + +TiKV is a good option when the metadata size requires better horizontal scalability. + +## `tikv.endpoints` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Note that the default pd port of tikv is 2379, which conflicts with etcd.
    		• 127.0.0.1:2389
    + + +## `tikv.rootPath` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • The root path where data is stored in tikv
    		• by-dev
    + + +## `tikv.metaSubPath` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • metaRootPath = rootPath + '/' + metaSubPath
    		• meta
    + + +## `tikv.kvSubPath` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • kvRootPath = rootPath + '/' + kvSubPath
    		• kv
    + + +## `tikv.requestTimeout` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • ms, tikv request timeout
    		• 10000
    + + +## `tikv.snapshotScanSize` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • batch size of tikv snapshot scan
    		• 256
    + + +## `tikv.ssl.enabled` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • Whether to support TiKV secure connection mode
    		• false
    + + +## `tikv.ssl.tlsCert` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • path to your cert file
    • + + +## `tikv.ssl.tlsKey` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • path to your key file
    • + + +## `tikv.ssl.tlsCACert` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • path to your CACert file
    • + + diff --git a/site/en/reference/sys_config/configure_tls.md b/site/en/reference/sys_config/configure_tls.md new file mode 100644 index 000000000..f8ec2d6bf --- /dev/null +++ b/site/en/reference/sys_config/configure_tls.md @@ -0,0 +1,65 @@ +--- +id: configure_tls.md +related_key: configure +group: system_configuration.md +summary: Learn how to configure tls for Milvus. +--- + +# tls-related Configurations + +Configure the proxy tls enable. + +## `tls.serverPemPath` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    configs/cert/server.pem
    + + +## `tls.serverKeyPath` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    configs/cert/server.key
    + + +## `tls.caPemPath` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    configs/cert/ca.pem
    + + diff --git a/site/en/reference/sys_config/configure_trace.md b/site/en/reference/sys_config/configure_trace.md new file mode 100644 index 000000000..35a1f23c2 --- /dev/null +++ b/site/en/reference/sys_config/configure_trace.md @@ -0,0 +1,108 @@ +--- +id: configure_trace.md +related_key: configure +group: system_configuration.md +summary: Learn how to configure trace for Milvus. +--- + +# trace-related Configurations + + + +## `trace.exporter` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • trace exporter type, default is stdout,
    • +
  • optional values: ['noop','stdout', 'jaeger', 'otlp']
    		• noop
    + + +## `trace.sampleFraction` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • fraction of traceID based sampler,
    • +
  • optional values: [0, 1]
    • +
  • Fractions >= 1 will always sample. Fractions < 0 are treated as zero.
    		• 0
    + + +## `trace.jaeger.url` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • when exporter is jaeger should set the jaeger's URL
    • + + +## `trace.otlp.endpoint` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    +
  • example: "127.0.0.1:4318"
    • + + +## `trace.otlp.secure` + + + + + + + + + + + + + + +
    DescriptionDefault Value
    true
    + + diff --git a/site/en/reference/sys_config/system_configuration.md b/site/en/reference/sys_config/system_configuration.md index 305e147db..bd7505675 100644 --- a/site/en/reference/sys_config/system_configuration.md +++ b/site/en/reference/sys_config/system_configuration.md @@ -3,7 +3,6 @@ id: system_configuration.md related_key: configure group: system_configuration.md summary: Learn about the system configuration of Milvus. -title: Milvus System Configurations Checklist --- # Milvus System Configurations Checklist @@ -18,229 +17,223 @@ In current release, all parameters take effect only after being configured at th ## Sections -For the convenience of maintenance, Milvus classifies its configurations into 17 sections based on its components, dependencies, and general usage. +For the convenience of maintenance, Milvus classifies its configurations into %s sections based on its components, dependencies, and general usage. ### `etcd` -etcd is the metadata engine supporting Milvus' metadata storage and access. - -Under this section, you can configure etcd endpoints, relevant key prefixes, etc. +Related configuration of etcd, used to store Milvus metadata & service discovery. See [etcd-related Configurations](configure_etcd.md) for detailed description for each parameter under this section. +### `metastore` + + + +See [metastore-related Configurations](configure_metastore.md) for detailed description for each parameter under this section. + +### `tikv` + +Related configuration of tikv, used to store Milvus metadata. + +Notice that when TiKV is enabled for metastore, you still need to have etcd for service discovery. + +TiKV is a good option when the metadata size requires better horizontal scalability. + +See [tikv-related Configurations](configure_tikv.md) for detailed description for each parameter under this section. + +### `localStorage` + + + +See [localStorage-related Configurations](configure_localstorage.md) for detailed description for each parameter under this section. + ### `minio` -Milvus supports MinIO and Amazon S3 as the storage engine for data persistence of insert log files and index files. Whereas MinIO is the de facto standard for S3 compatibility, you can configure S3 parameters directly under MinIO section. +Related configuration of MinIO/S3/GCS or any other service supports S3 API, which is responsible for data persistence for Milvus. -Under this section, you can configure MinIO or S3 address, relevant access keys, etc. +We refer to the storage service as MinIO/S3 in the following description for simplicity. -See [MinIO-related Configurations](configure_minio.md) for detailed description for each parameter under this section. +See [minio-related Configurations](configure_minio.md) for detailed description for each parameter under this section. -### `pulsar` +### `mq` + +Milvus supports four MQ: rocksmq(based on RockDB), natsmq(embedded nats-server), Pulsar and Kafka. + +You can change your mq by setting mq.type field. -Pulsar is the underlying engine supporting Milvus cluster's reliable storage and publication/subscription of message streams. +If you don't set mq.type field as default, there is a note about enabling priority if we config multiple mq in this file. -Under this section, you can configure Pulsar address, the message size, etc. +1. standalone(local) mode: rocksmq(default) > natsmq > Pulsar > Kafka -See [Pulsar-related Configurations](configure_pulsar.md) for detailed description for each parameter under this section. +2. cluster mode: Pulsar(default) > Kafka (rocksmq and natsmq is unsupported in cluster mode) + +See [mq-related Configurations](configure_mq.md) for detailed description for each parameter under this section. + +### `pulsar` + +Related configuration of pulsar, used to manage Milvus logs of recent mutation operations, output streaming log, and provide log publish-subscribe services. + +See [pulsar-related Configurations](configure_pulsar.md) for detailed description for each parameter under this section. ### `rocksmq` -RocksMQ is the underlying engine supporting Milvus standalone's reliable storage and publication/subscription of message streams. It is implemented on the basis of RocksDB. +If you want to enable kafka, needs to comment the pulsar configs -Under this section, you can configure message size, retention time and size, etc. +kafka: -See [RocksMQ-related Configurations](configure_rocksmq.md) for detailed description for each parameter under this section. + brokerList: -### `nats` + saslUsername: -NATS is a message-oriented middleware that allows data exchange between applications and services, segmented in the form of messages. Milvus uses NATS as a underlying engine for reliable storage and pub/sub of message streams. You can use it as an alternative to RocksMQ. + saslPassword: -Under this section, you can configure NATS server, monitoring properties, and rention time and size, etc. + saslMechanisms: -See [NATS-related Configurations](configure_nats.md) for detailed description for each parameter under this section. + securityProtocol: -### `kafka` + ssl: -Apache Kafka is an open-source distributed event streaming platform for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. It serves as an alternative to RocksMQ and Pulsar for reliable storage and publication/subscription of message streams. + enabled: false # whether to enable ssl mode -See [Kafka-related Configurations](configure_kafka.md) for detailed description for each parameter under this section. + tlsCert: # path to client's public key (PEM) used for authentication -### `rootCoord` + tlsKey: # path to client's private key (PEM) used for authentication -Root coordinator (root coord) handles data definition language (DDL) and data control language (DCL) requests, manages TSO (timestamp Oracle), and publishes time tick messages. + tlsCaCert: # file or directory path to CA certificate(s) for verifying the broker's key -Under this section, you can configure root coord address, index building threshold, etc. + tlsKeyPassword: # private key passphrase for use with ssl.key.location and set_ssl_cert(), if any -See [Root Coordinator-related Configurations](configure_rootcoord.md) for detailed description for each parameter under this section. + readTimeout: 10 -### `proxy` -Proxy is the access layer of the system and endpoint to users. It validates client requests and reduces the returned results. -Under this section, you can configure proxy port, system limits, etc. +See [rocksmq-related Configurations](configure_rocksmq.md) for detailed description for each parameter under this section. -See [Proxy-related Configurations](configure_proxy.md) for detailed description for each parameter under this section. +### `natsmq` -### `queryCoord` +natsmq configuration. + +more detail: https://docs.nats.io/running-a-nats-service/configuration + +See [natsmq-related Configurations](configure_natsmq.md) for detailed description for each parameter under this section. + +### `rootCoord` + +Related configuration of rootCoord, used to handle data definition language (DDL) and data control language (DCL) requests + +See [rootCoord-related Configurations](configure_rootcoord.md) for detailed description for each parameter under this section. + +### `proxy` + +Related configuration of proxy, used to validate client requests and reduce the returned results. + +See [proxy-related Configurations](configure_proxy.md) for detailed description for each parameter under this section. -Query coordinator (query coord) manages topology and load-balancing of the query nodes, and handoff operation from growing segments to sealed segments. +### `queryCoord` -Under this section, you can configure query coord address, auto handoff, auto load-balancing, etc. +Related configuration of queryCoord, used to manage topology and load balancing for the query nodes, and handoff from growing segments to sealed segments. -See [Query coordinator-related Configurations](configure_querycoord.md) for detailed description for each parameter under this section. +See [queryCoord-related Configurations](configure_querycoord.md) for detailed description for each parameter under this section. ### `queryNode` -Query node performs hybrid search of vector and scalar data on both incremental and historical data. +Related configuration of queryNode, used to run hybrid search between vector and scalar data. + +See [queryNode-related Configurations](configure_querynode.md) for detailed description for each parameter under this section. + +### `indexCoord` -Under this section, you can configure query node port, graceful time, etc. -See [Query Node-related Configurations](configure_querynode.md) for detailed description for each parameter under this section. + +See [indexCoord-related Configurations](configure_indexcoord.md) for detailed description for each parameter under this section. ### `indexNode` -Index node builds indexes for vectors. -Under this section, you can configure index node port, etc. -See [Index Node-related Configurations](configure_indexnode.md) for detailed description for each parameter under this section. +See [indexNode-related Configurations](configure_indexnode.md) for detailed description for each parameter under this section. ### `dataCoord` -Data coordinator (data coord) manages the topology of data nodes, maintains metadata, and triggers flush, compact, and other background data operations. -Under this section, you can configure data coord address, segment settings, compaction, garbage collection, etc. -See [Data Coordinator-related Configurations](configure_datacoord.md) for detailed description for each parameter under this section. +See [dataCoord-related Configurations](configure_datacoord.md) for detailed description for each parameter under this section. ### `dataNode` -Data node retrieves incremental log data by subscribing to the log broker, processes mutation requests, and packs log data into log snapshots and stores them in the object storage. -Under this section, you can configure data node port, etc. -See [Data Node-related Configurations](configure_datanode.md) for detailed description for each parameter under this section. +See [dataNode-related Configurations](configure_datanode.md) for detailed description for each parameter under this section. -### `localStorage` - -Milvus stores the vector data in local storage during search or query to avoid repetitive access to MinIO or S3 service. +### `log` -Under this section, you can enable local storage, and configure the path, etc. +Configures the system log output. -See [Local Storage-related Configurations](configure_localstorage.md) for detailed description for each parameter under this section. +See [log-related Configurations](configure_log.md) for detailed description for each parameter under this section. -### `log` +### `grpc` -Using Milvus generates a collection of logs. By default, Milvus uses logs to record information at debug or even higher level for standard output (stdout) and standard error (stderr). -Under this section, you can configure the system log output. -See [Log-related Configurations](configure_log.md) for detailed description for each parameter under this section. +See [grpc-related Configurations](configure_grpc.md) for detailed description for each parameter under this section. -### `msgChannel` +### `tls` -Under this section, you can configure the message channel name prefixes and component subscription name prefixes. +Configure the proxy tls enable. -See [Message Channel-related Configurations](configure_messagechannel.md) for detailed description for each parameter under this section. +See [tls-related Configurations](configure_tls.md) for detailed description for each parameter under this section. ### `common` -Under this section, you can configure the default names of partition and index, and the Time Travel (data retention) span of Milvus. - -See [Common Configurations](configure_common.md) for detailed description for each parameter under this section. - -### `knowhere` - -[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus. - -Under this section, you can configure the default SIMD instruction set type of the system. - -See [Knowhere-related Configurations](configure_knowhere.md) for detailed description for each parameter under this section. - -## Frequently used parameters - -Below list some frequently used parameters categorized in accordance with the purposes of modification. - -### Performance tuning - -The following parameters control the system behaviors that influence the performance of index creation and vector similarity search. - - - -### Data and metadata retention - -The following parameters control the retention of data and metadata. - - - - -### Administration - -The following parameters control the log output and object storage access. - - - -### Quotas and limits - - - -## What's next - -- Learn how to [configure Milvus](configure-docker.md) before installation. - -- Learn more about the installation of Milvus: - - [Install Milvus Standalone](install_standalone-docker.md) \ No newline at end of file + + +See [common-related Configurations](configure_common.md) for detailed description for each parameter under this section. + +### `quotaAndLimits` + +QuotaConfig, configurations of Milvus quota and limits. + +By default, we enable: + + 1. TT protection; + + 2. Memory protection. + + 3. Disk quota protection. + +You can enable: + + 1. DML throughput limitation; + + 2. DDL, DQL qps/rps limitation; + + 3. DQL Queue length/latency protection; + + 4. DQL result rate protection; + +If necessary, you can also manually force to deny RW requests. + +See [quotaAndLimits-related Configurations](configure_quotaandlimits.md) for detailed description for each parameter under this section. + +### `trace` + + + +See [trace-related Configurations](configure_trace.md) for detailed description for each parameter under this section. + +### `gpu` + +#when using GPU indexing, Milvus will utilize a memory pool to avoid frequent memory allocation and deallocation. + +#here, you can set the size of the memory occupied by the memory pool, with the unit being MB. + +#note that there is a possibility of Milvus crashing when the actual memory demand exceeds the value set by maxMemSize. + +#if initMemSize and MaxMemSize both set zero, + +#milvus will automatically initialize half of the available GPU memory, + +#maxMemSize will the whole available GPU memory. + +See [gpu-related Configurations](configure_gpu.md) for detailed description for each parameter under this section. +