Skip to content

Commit

Permalink
Merge pull request opensearch-project#3472 from kolchfa-aws/perf-fix
Browse files Browse the repository at this point in the history
Fix typo and formatting in performance blog
  • Loading branch information
kolchfa-aws authored Nov 27, 2024
2 parents 6ae2085 + bb2e9d2 commit ac195c5
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions _posts/2024-11-26-opensearch-performance-2.17.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Our commitment to enhancing OpenSearch's performance remains unwavering, and thi

OpenSearch 2.17 offers a remarkable **6x performance boost** compared to OpenSearch 1.3, enhancing key operations like text queries, terms aggregations, range queries, date histograms, and sorting. Additionally, the improvements in semantic vector search now allow for highly configurable settings, enabling you to balance response time, accuracy, and cost according to your needs. These advancements are a testament to the dedicated community whose contributions and collaboration propel OpenSearch forward.

The first section focuses on key query operations, including text queries, 297s aggregations, range queries, date histograms, and sorting. These improvements were evaluated using the [OpenSearch Big5 workload](https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/big5), which represents common use cases in both search and analytics applications. The benchmarks provide a repeatable framework for measuring real-world performance enhancements. The next section reports on vector search improvements. Finally, we present our roadmap for 2025, where you'll see that we're making qualitative improvements in many areas, in addition to important incremental changes. We are improving query speed by processing data in real time. We are building a query planner that uses resources more efficiently. We are speeding up intra-cluster communications. And we're adding efficient join operations to query domain-specific language (DSL), Piped Processing Language (PPL), and SQL. To follow our work in more detail, and to contribute comments or code, please participate on the [OpenSearch forum](https://forum.opensearch.org/) as well as directly in our GitHub repos.
The first section focuses on key query operations, including text queries, terms aggregations, range queries, date histograms, and sorting. These improvements were evaluated using the [OpenSearch Big5 workload](https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/big5), which represents common use cases in both search and analytics applications. The benchmarks provide a repeatable framework for measuring real-world performance enhancements. The next section reports on vector search improvements. Finally, we present our roadmap for 2025, where you'll see that we're making qualitative improvements in many areas, in addition to important incremental changes. We are improving query speed by processing data in real time. We are building a query planner that uses resources more efficiently. We are speeding up intra-cluster communications. And we're adding efficient join operations to query domain-specific language (DSL), Piped Processing Language (PPL), and SQL. To follow our work in more detail, and to contribute comments or code, please participate on the [OpenSearch forum](https://forum.opensearch.org/) as well as directly in our GitHub repos.

<style>

Expand Down Expand Up @@ -275,11 +275,11 @@ In 2025, we will push the boundaries of OpenSearch's query engine with several k

In 2025, we will continue to invest in the following key initiatives aimed at performance improvements and cost savings:

* **Index build acceleration with GPUs and SIMD:** k-NN performance can be enhanced by using libraries with GPU support. Because vector distance calculations are compute-heavy, GPUs can speed up computations and reduce index build and search query times.
* **Autotuning k-NN indexes:** OpenSearch's vector database offers a toolkit of algorithms tailored for diverse workloads. In 2025, our goal is to enhance the out-of-the-box experience by autotuning hyperparameters and settings based on access patterns and hardware resources.
* **Cold-warm tiering:** In version 2.18, we added support for enabling vector search on remote snapshots. We will continue focusing on decoupling index read/write operations to extend vector indexes to different storage systems in order to reduce storage and compute costs.
* **Memory footprint reduction:** We will continue to aggressively reduce the memory footprint of vector indexes. One of our goals is to support the ability to partially load HNSW indexes into native engines. This complements our disk-optimized search and helps further reduce the operating costs of OpenSearch clusters.
* **Reduced disk storage using derived source:** Currently, vector data is stored both in a doc-values-like format and in the stored `_source` field. The stored `_source` field can contribute more than 60% of the overall vector storage requirement. We plan to create a custom stored field format that will inject the vector fields into the source from the doc-values-like format, creating a derived source field. In addition to storage savings, this approach will improve indexing throughput, reduce shard size, and even accelerate search.
* **Index build acceleration with GPUs and SIMD**: k-NN performance can be enhanced by using libraries with GPU support. Because vector distance calculations are compute-heavy, GPUs can speed up computations and reduce index build and search query times.
* **Autotuning k-NN indexes**: OpenSearch's vector database offers a toolkit of algorithms tailored for diverse workloads. In 2025, our goal is to enhance the out-of-the-box experience by autotuning hyperparameters and settings based on access patterns and hardware resources.
* **Cold-warm tiering**: In version 2.18, we added support for enabling vector search on remote snapshots. We will continue focusing on decoupling index read/write operations to extend vector indexes to different storage systems in order to reduce storage and compute costs.
* **Memory footprint reduction**: We will continue to aggressively reduce the memory footprint of vector indexes. One of our goals is to support the ability to partially load HNSW indexes into native engines. This complements our disk-optimized search and helps further reduce the operating costs of OpenSearch clusters.
* **Reduced disk storage using derived source**: Currently, vector data is stored both in a doc-values-like format and in the stored `_source` field. The stored `_source` field can contribute more than 60% of the overall vector storage requirement. We plan to create a custom stored field format that will inject the vector fields into the source from the doc-values-like format, creating a derived source field. In addition to storage savings, this approach will improve indexing throughput, reduce shard size, and even accelerate search.

### Neural search

Expand Down

0 comments on commit ac195c5

Please sign in to comment.