Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][standalone] Milvus memory usage is large than before causing Milvus OOM #38213

Closed
1 task done
wangting0128 opened this issue Dec 4, 2024 · 9 comments
Closed
1 task done

[Bug]: [benchmark][standalone] Milvus memory usage is large than before causing Milvus OOM #38213

wangting0128 opened this issue Dec 4, 2024 · 9 comments
Assignees
@czs007
Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@wangting0128
Copy link
Contributor

wangting0128 commented Dec 4, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master-20241204-86363920-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc124
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: multi-vector-corn-kjs2s
test case name: test_hybrid_search_locust_shard16_float_dql_ivf_sq8_standalone

server:

NAME                                                              READY   STATUS        RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-corn-kjs2s-32-etcd-0                                 1/1     Running       0                8h      10.104.27.228   4am-node31   <none>           <none>
multi-vector-corn-kjs2s-32-milvus-standalone-c5754f54f-t429k      1/1     Running       3 (99m ago)      8h      10.104.34.244   4am-node37   <none>           <none>
multi-vector-corn-kjs2s-32-minio-84f8ddd47c-bmbs9                 1/1     Running       0                8h      10.104.21.87    4am-node24   <none>           <none>

image

image

client steps:

  1. vector fields build HNSW index and concurrent hybrid_search
  2. vector fields rebuild IVF_FLAT index and concurrent hybrid_search
  3. vector fields rebuild IVF_SQ8 index and concurrent hybrid_search <- milvus standalone OOM when concurrent hybrid_search

normal running

image: 2.4-20240415-1e4f5ee9-amd64
index IVF_SQ8 memory usage is less then 35G
image

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `shard_num=16, float_vector DQL`
            verify concurrent DQL scenario which has 4 float_vector fields(IVF_SQ8) and 60 scalar fields

        :test steps:
            1. create collection with fields:
                'float_vector': 32768dim,
                'float_vector_1': 32768dim,
                'float_vector_2': 32768dim,
                'float_vector_3': 32768dim,
                all scalar fields: varchar max_length=10, array max_capacity=7
            2. build indexes:
                IVF_SQ8: 'float_vector', 'float_vector_1', 'float_vector_2', 'float_vector_3'
                default_scalar_index: 'int64_1'
                INVERTED: 'id', 'bool_3'
            3. insert 100k data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 1
            7. concurrent request: concurrent_number=[1, 20]
                - hybrid_search

Milvus Log

No response

Anything else?

test result:
concurrent_number=1

[2024-12-04 10:32:26,281 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-12-04 10:32:26,281 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-12-04 10:32:26,281 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-12-04 10:32:26,281 -  INFO - fouram]: grpc     hybrid_search                                                                   5430     2(0.04%) |    481     307   60791    370 |    1.51        0.00 (stats.py:789)
[2024-12-04 10:32:26,281 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-12-04 10:32:26,281 -  INFO - fouram]:          Aggregated                                                                      5430     2(0.04%) |    481     307   60791    370 |    1.51        0.00 (stats.py:789)
[2024-12-04 10:32:26,282 -  INFO - fouram]:  (stats.py:790)
[2024-12-04 10:32:26,284 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': '',
            'deploy_mode': '',
            'config_name': '',
            'config': {},
            'host': 'multi-vector-corn-kjs2s-32-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_shard16_float_dql_ivf_sq8_standalone',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 32768,
                                                    'max_length': 10,
                                                    'scalars_index': {'int8_1': {}, 'id': {'index_type': 'INVERTED'}, 'bool_3': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 1024},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 1024},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 1024},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'array_int8_1': {'params': {'max_capacity': 7}},
                                                                       'array_int16_1': {'params': {'max_capacity': 7}},
                                                                       'array_int32_1': {'params': {'max_capacity': 7}},
                                                                       'array_int64_1': {'params': {'max_capacity': 7}},
                                                                       'array_double_1': {'params': {'max_capacity': 7}},
                                                                       'array_float_1': {'params': {'max_capacity': 7}},
                                                                       'array_varchar_1': {'params': {'max_capacity': 7}},
                                                                       'array_bool_1': {'params': {'max_capacity': 7}},
                                                                       'array_int8_2': {'params': {'max_capacity': 7}},
                                                                       'array_int16_2': {'params': {'max_capacity': 7}},
                                                                       'array_int32_2': {'params': {'max_capacity': 7}},
                                                                       'array_int64_2': {'params': {'max_capacity': 7}},
                                                                       'array_double_2': {'params': {'max_capacity': 7}},
                                                                       'array_float_2': {'params': {'max_capacity': 7}},
                                                                       'array_varchar_2': {'params': {'max_capacity': 7}},
                                                                       'array_bool_2': {'params': {'max_capacity': 7}},
                                                                       'array_int8_3': {'params': {'max_capacity': 7}},
                                                                       'array_int16_3': {'params': {'max_capacity': 7}},
                                                                       'array_int32_3': {'params': {'max_capacity': 7}},
                                                                       'array_int64_3': {'params': {'max_capacity': 7}},
                                                                       'array_double_3': {'params': {'max_capacity': 7}},
                                                                       'array_float_3': {'params': {'max_capacity': 7}},
                                                                       'array_varchar_3': {'params': {'max_capacity': 7}},
                                                                       'array_bool_3': {'params': {'max_capacity': 7}}},
                                                    'dataset_name': 'local',
                                                    'dataset_size': 100000,
                                                    'ni_per': 100},
                                 'collection_params': {'other_fields': ['float_vector_1', 'float_vector_2', 'float_vector_3', 'int8_1', 'int16_1', 'int32_1',
                                                                        'int64_1', 'double_1', 'float_1', 'varchar_1', 'bool_1', 'json_1', 'array_int8_1',
                                                                        'array_int16_1', 'array_int32_1', 'array_int64_1', 'array_double_1', 'array_float_1',
                                                                        'array_varchar_1', 'array_bool_1', 'int8_2', 'int16_2', 'int32_2', 'int64_2',
                                                                        'double_2', 'float_2', 'varchar_2', 'bool_2', 'json_2', 'array_int8_2', 'array_int16_2',
                                                                        'array_int32_2', 'array_int64_2', 'array_double_2', 'array_float_2', 'array_varchar_2',
                                                                        'array_bool_2', 'int8_3', 'int16_3', 'int32_3', 'int64_3', 'double_3', 'float_3',
                                                                        'varchar_3', 'bool_3', 'json_3', 'array_int8_3', 'array_int16_3', 'array_int32_3',
                                                                        'array_int64_3', 'array_double_3', 'array_float_3', 'array_varchar_3', 'array_bool_3',
                                                                        'varchar_tail_1', 'varchar_tail_2', 'varchar_tail_3', 'varchar_tail_4',
                                                                        'varchar_tail_5', 'varchar_tail_6', 'varchar_tail_7', 'varchar_tail_8'],
                                                       'shards_num': 16},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 1, 'during_time': '1h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 1,
                                                                  'reqs': [{'search_param': {'nprobe': 32},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': 'id > 10000',
                                                                            'top_k': 10},
                                                                           {'search_param': {'nprobe': 64},
                                                                            'anns_field': 'float_vector_1',
                                                                            'expr': 'int8_1 > 64',
                                                                            'top_k': 50},
                                                                           {'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector_2',
                                                                            'expr': 'array_length(array_int8_2) == 7',
                                                                            'top_k': 1000},
                                                                           {'search_param': {'nprobe': 256},
                                                                            'anns_field': 'float_vector_3',
                                                                            'expr': 'bool_3 == True',
                                                                            'top_k': 16384}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 60,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}}]},
            'run_id': 2024120440574486,
            'datetime': '2024-12-04 09:20:57.032005',
            'client_version': '2.5.0'},
 'result': {'test_result': {'index': {'RT': 143.6691,
                                      'float_vector_1': {'RT': 125.2365},
                                      'float_vector_2': {'RT': 122.6238},
                                      'float_vector_3': {'RT': 128.169},
                                      'int8_1': {'RT': 49.9804},
                                      'id': {'RT': 50.2209},
                                      'bool_3': {'RT': 48.5985}},
                            'load': {'RT': 17.4512},
                            'Locust': {'Aggregated': {'Requests': 5430,
                                                      'Fails': 2,
                                                      'RPS': 1.51,
                                                      'fail_s': 0.0,
                                                      'RT_max': 60791.61,
                                                      'RT_avg': 481.64,
                                                      'TP50': 370.0,
                                                      'TP99': 1200.0},
                                       'hybrid_search': {'Requests': 5430,
                                                         'Fails': 2,
                                                         'RPS': 1.51,
                                                         'fail_s': 0.0,
                                                         'RT_max': 60791.61,
                                                         'RT_avg': 481.64,
                                                         'TP50': 370.0,
                                                         'TP99': 1200.0}}}}}

concurrent_number=20

[2024-12-04 11:32:27,314 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-12-04 11:32:27,315 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-12-04 11:32:27,315 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-12-04 11:32:27,315 -  INFO - fouram]: grpc     hybrid_search                                                                  10413     0(0.00%) |   6400    2026   11017   6700 |    2.89        0.00 (stats.py:789)
[2024-12-04 11:32:27,315 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-12-04 11:32:27,315 -  INFO - fouram]:          Aggregated                                                                     10413     0(0.00%) |   6400    2026   11017   6700 |    2.89        0.00 (stats.py:789)
[2024-12-04 11:32:27,315 -  INFO - fouram]:  (stats.py:790)
[2024-12-04 11:32:27,317 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': '',
            'deploy_mode': '',
            'config_name': '',
            'config': {},
            'host': 'multi-vector-corn-kjs2s-32-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_shard16_float_dql_ivf_sq8_standalone',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 32768,
                                                    'max_length': 10,
                                                    'scalars_index': {'int8_1': {}, 'id': {'index_type': 'INVERTED'}, 'bool_3': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 1024},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 1024},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 1024},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'array_int8_1': {'params': {'max_capacity': 7}},
                                                                       'array_int16_1': {'params': {'max_capacity': 7}},
                                                                       'array_int32_1': {'params': {'max_capacity': 7}},
                                                                       'array_int64_1': {'params': {'max_capacity': 7}},
                                                                       'array_double_1': {'params': {'max_capacity': 7}},
                                                                       'array_float_1': {'params': {'max_capacity': 7}},
                                                                       'array_varchar_1': {'params': {'max_capacity': 7}},
                                                                       'array_bool_1': {'params': {'max_capacity': 7}},
                                                                       'array_int8_2': {'params': {'max_capacity': 7}},
                                                                       'array_int16_2': {'params': {'max_capacity': 7}},
                                                                       'array_int32_2': {'params': {'max_capacity': 7}},
                                                                       'array_int64_2': {'params': {'max_capacity': 7}},
                                                                       'array_double_2': {'params': {'max_capacity': 7}},
                                                                       'array_float_2': {'params': {'max_capacity': 7}},
                                                                       'array_varchar_2': {'params': {'max_capacity': 7}},
                                                                       'array_bool_2': {'params': {'max_capacity': 7}},
                                                                       'array_int8_3': {'params': {'max_capacity': 7}},
                                                                       'array_int16_3': {'params': {'max_capacity': 7}},
                                                                       'array_int32_3': {'params': {'max_capacity': 7}},
                                                                       'array_int64_3': {'params': {'max_capacity': 7}},
                                                                       'array_double_3': {'params': {'max_capacity': 7}},
                                                                       'array_float_3': {'params': {'max_capacity': 7}},
                                                                       'array_varchar_3': {'params': {'max_capacity': 7}},
                                                                       'array_bool_3': {'params': {'max_capacity': 7}}},
                                                    'dataset_name': 'local',
                                                    'dataset_size': 100000,
                                                    'ni_per': 100},
                                 'collection_params': {'other_fields': ['float_vector_1', 'float_vector_2', 'float_vector_3', 'int8_1', 'int16_1', 'int32_1',
                                                                        'int64_1', 'double_1', 'float_1', 'varchar_1', 'bool_1', 'json_1', 'array_int8_1',
                                                                        'array_int16_1', 'array_int32_1', 'array_int64_1', 'array_double_1', 'array_float_1',
                                                                        'array_varchar_1', 'array_bool_1', 'int8_2', 'int16_2', 'int32_2', 'int64_2',
                                                                        'double_2', 'float_2', 'varchar_2', 'bool_2', 'json_2', 'array_int8_2', 'array_int16_2',
                                                                        'array_int32_2', 'array_int64_2', 'array_double_2', 'array_float_2', 'array_varchar_2',
                                                                        'array_bool_2', 'int8_3', 'int16_3', 'int32_3', 'int64_3', 'double_3', 'float_3',
                                                                        'varchar_3', 'bool_3', 'json_3', 'array_int8_3', 'array_int16_3', 'array_int32_3',
                                                                        'array_int64_3', 'array_double_3', 'array_float_3', 'array_varchar_3', 'array_bool_3',
                                                                        'varchar_tail_1', 'varchar_tail_2', 'varchar_tail_3', 'varchar_tail_4',
                                                                        'varchar_tail_5', 'varchar_tail_6', 'varchar_tail_7', 'varchar_tail_8'],
                                                       'shards_num': 16},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20, 'during_time': '1h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 1,
                                                                  'reqs': [{'search_param': {'nprobe': 32},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': 'id > 10000',
                                                                            'top_k': 10},
                                                                           {'search_param': {'nprobe': 64},
                                                                            'anns_field': 'float_vector_1',
                                                                            'expr': 'int8_1 > 64',
                                                                            'top_k': 50},
                                                                           {'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector_2',
                                                                            'expr': 'array_length(array_int8_2) == 7',
                                                                            'top_k': 1000},
                                                                           {'search_param': {'nprobe': 256},
                                                                            'anns_field': 'float_vector_3',
                                                                            'expr': 'bool_3 == True',
                                                                            'top_k': 16384}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 60,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}}]},
            'run_id': 2024120440574486,
            'datetime': '2024-12-04 09:20:57.032005',
            'client_version': '2.5.0'},
 'result': {'test_result': {'index': {'RT': 143.6691,
                                      'float_vector_1': {'RT': 125.2365},
                                      'float_vector_2': {'RT': 122.6238},
                                      'float_vector_3': {'RT': 128.169},
                                      'int8_1': {'RT': 49.9804},
                                      'id': {'RT': 50.2209},
                                      'bool_3': {'RT': 48.5985}},
                            'load': {'RT': 17.4512},
                            'Locust': {'Aggregated': {'Requests': 10413,
                                                      'Fails': 0,
                                                      'RPS': 2.89,
                                                      'fail_s': 0.0,
                                                      'RT_max': 11017.32,
                                                      'RT_avg': 6400.6,
                                                      'TP50': 6700.0,
                                                      'TP99': 8300.0},
                                       'hybrid_search': {'Requests': 10413,
                                                         'Fails': 0,
                                                         'RPS': 2.89,
                                                         'fail_s': 0.0,
                                                         'RT_max': 11017.32,
                                                         'RT_avg': 6400.6,
                                                         'TP50': 6700.0,
                                                         'TP99': 8300.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Dec 4, 2024
@wangting0128 wangting0128 added this to the 2.5.0 milestone Dec 4, 2024
@czs007 czs007 assigned czs007 and unassigned yanliang567 Dec 4, 2024
@wangting0128
Copy link
Contributor Author

wangting0128 commented Dec 4, 2024 •
edited
Loading

same case，but shard_num=1, memory usage is normal

test case name: test_hybrid_search_locust_shard1_float_dql_ivf_sq8_standalone
image: master-20241204-86363920-amd64
server:

NAME                                                              READY   STATUS        RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-corn-kjs2s-31-etcd-0                                 1/1     Running       0                7h38m   10.104.26.51    4am-node32   <none>           <none>
multi-vector-corn-kjs2s-31-milvus-standalone-6cfc5b9658-2gkhn     1/1     Running       2 (7h36m ago)    7h38m   10.104.17.52    4am-node23   <none>           <none>
multi-vector-corn-kjs2s-31-minio-5cb6c9db4b-f42jm                 1/1     Running       0                7h38m   10.104.17.51    4am-node23   <none>           <none>

image

@wangting0128
Copy link
Contributor Author

wangting0128 commented Dec 4, 2024 •
edited
Loading

enabled queryNode.mmap.scalarField=true

test case name: test_hybrid_search_locust_shard16_float_dql_ivf_sq8_standalone
image: master-20241203-e09f4318-amd64

server:

NAME                                                              READY   STATUS        RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-concurrent-t5llj-2-etcd-0                                 1/1     Running       0                3h21m   10.104.32.221   4am-node39   <none>           <none>
fouramf-concurrent-t5llj-2-milvus-standalone-6f4f946656-bwmtc     1/1     Running       1 (102m ago)     3h21m   10.104.19.128   4am-node28   <none>           <none>
fouramf-concurrent-t5llj-2-minio-5ddf844c57-7jcvl                 1/1     Running       0                3h21m   10.104.32.222   4am-node39   <none>           <none>

milvus OOM
截屏2024-12-04 20 23 58

@wangting0128
Copy link
Contributor Author

wangting0128 commented Dec 4, 2024 •
edited
Loading

enabled all mmap

截屏2024-12-04 20 26 05

test case name: test_hybrid_search_locust_shard16_float_dql_ivf_sq8_standalone
image: master-20241203-9e008685-amd64

server:

NAME                                                       READY   STATUS             RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-9tzpn-79-1562-etcd-0                               1/1     Running            0                3h30m   10.104.25.55    4am-node30   <none>           <none>
fouramf-9tzpn-79-1562-milvus-standalone-66798bd986-7m6bk   1/1     Running            1 (3h29m ago)    3h30m   10.104.17.126   4am-node23   <none>           <none>
fouramf-9tzpn-79-1562-minio-7f47f6f844-49fqw               1/1     Running            0                3h30m   10.104.33.106   4am-node36   <none>           <none>

not OOM, but memory usage is very high
截屏2024-12-04 20 27 32

@wangting0128
Copy link
Contributor Author

March master image: master-20240329-88d426f39-amd64

test case name: test_hybrid_search_locust_shard16_float_dql_ivf_sq8_standalone

server:

NAME                                                              READY   STATUS        RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-txlfm-8-7142-etcd-0                                       1/1     Running       0                3h17m   10.104.15.218   4am-node20   <none>           <none>
fouramf-txlfm-8-7142-milvus-standalone-7df6c74465-9r94n           1/1     Running       0                3h17m   10.104.21.65    4am-node24   <none>           <none>
fouramf-txlfm-8-7142-minio-7b5bcb4547-96psp                       1/1     Running       0                3h17m   10.104.15.217   4am-node20   <none>           <none>

not OOM, and low memory usage
截屏2024-12-04 20 30 22

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 4, 2024
@wangting0128
Copy link
Contributor Author

2.4 branch: 2.4-20241204-febed0ab-amd64

test case name: test_hybrid_search_locust_shard16_float_dql_ivf_sq8_standalone

server:

NAME                                                              READY   STATUS             RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-qgvc5-79-3398-etcd-0                                      1/1     Running            0                 3h8m    10.104.33.47    4am-node36   <none>           <none>
fouramf-qgvc5-79-3398-milvus-standalone-8cc86d7b7-ngtgh           1/1     Running            1 (3h8m ago)      3h8m    10.104.17.172   4am-node23   <none>           <none>
fouramf-qgvc5-79-3398-minio-5c9c64487b-gn426                      1/1     Running            0                 3h8m    10.104.15.236   4am-node20   <none>           <none>
截屏2024-12-05 10 44 55

@xiaofan-luan
Copy link
Collaborator

I doubt this is a delete issue.
@XuanYang-cn please help on checking

@wangting0128
Copy link
Contributor Author

I doubt this is a delete issue. @XuanYang-cn please help on checking

There is no delete in this scenario, only concurrent hybrid_search

@czs007 is helping with this

@czs007
Copy link
Collaborator

czs007 commented Dec 16, 2024

related to #38492

@czs007 czs007 mentioned this issue Dec 16, 2024
Merged
@wangting0128
Copy link
Contributor Author

verification passed

argo task：fouramf-xwrqz-rerun
test case name：test_hybrid_search_locust_shard16_float_dql_ivf_sq8_standalone
image：master-20241218-6ee94d00-amd64

截屏2024-12-18 17 15 33

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@czs007 czs007

Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
4 participants
@wangting0128 @czs007 @yanliang567 @xiaofan-luan