From 0bef689dac65c36864fb1c1e6a705ffc02d8f20c Mon Sep 17 00:00:00 2001 From: Arpit Bhayani Date: Sun, 15 Dec 2024 00:57:31 +0530 Subject: [PATCH] Editorial changes --- .../blog/dicedb-a-reactive-database.md | 80 +++++++++++++++++++ .../src/content/blog/dicedb-is-open-source.md | 25 +++--- 2 files changed, 91 insertions(+), 14 deletions(-) create mode 100644 docs/src/content/blog/dicedb-a-reactive-database.md diff --git a/docs/src/content/blog/dicedb-a-reactive-database.md b/docs/src/content/blog/dicedb-a-reactive-database.md new file mode 100644 index 000000000..8aa2f990c --- /dev/null +++ b/docs/src/content/blog/dicedb-a-reactive-database.md @@ -0,0 +1,80 @@ +--- +title: "DiceDB, a reactive database" +description: "" +published_at: 2024-11-12 +author: arpit +--- + +In traditional databases, we query data when it's needed; a process that is simple but can become highly inefficient when large number of expensive and redundant queries are frequently fired on the database. Let's understand this by building a leaderboard. + +Imagine a traditional database that maintains a leaderboard for a competitive game or application. Here, an asynchronous flow updates the leaderboard data continuously in the database, reflecting new scores or rankings as they come in. + +Now consider the users: thousands of clients query this leaderboard every 10 seconds to get the latest rankings. Every query requires the database to compute the current leaderboard standings based on the updated data and then respond to the client with the latest results. Upon receiving the result, each client proceeds to render the leaderboard in its application, ensuring users see the current standings. + +TODO: image. + +In this setup, several issues arise: + +1. Redundant Computation: Since thousands of clients are querying the database every few seconds, the database repeatedly computes the leaderboard, even if there has been little or no change in scores. This results in substantial redundant processing, significantly taxing the database resources. + +2. High Load and Latency: The high frequency and volume of queries introduce a considerable load on the database, leading to latency. The database must allocate resources to process each query individually, slowing down response times as the number of clients scales. + +3. Increased Infrastructure Costs: To handle the high query volume, the system may require powerful hardware and extensive scaling strategies, leading to increased operational costs. + +This traditional query model is inefficient for high-read, rapidly updating data structures like leaderboards, where many clients want real-time information. This setup incurs significant resource overhead and diminishes the overall system performance, especially under heavy load. + +This is where DiceDB, a reactive database chimes in ... + +## What is a reactive database? + +The concept of a reactive database is simple, a reactive database pushes updated query results to clients as soon as the underlying data changes. DiceDB, upon detecting the change in data, automatically re-evaluates the relevant queries and immediately streams the updated result sets to all clients who have subscribed to it. + +TODO Image. + +This real-time reactivity ensures that clients always receive the latest data without the need to send repeated queries, dramatically reducing network load and latency while providing an efficient, seamless experience for data-driven applications. + +### Reactive Databases are not CDC + +One common comparison that is draw to reactive data is of CDC. Dice DB is not same as CDC. CDC is about capturing the delta of changes in data, but in DiceDB the output of the Query is streamed and it is not just a notification about the change. This makes DiceDB a truly real-time reactive database. + +## DiceDB: A Reactive Database + +**DiceDB** exemplifies the reactive database model. Designed as a Redis-compatible, in-memory, multi-threaded database, it focuses on real-time reactivity and efficiency, enabling it to handle modern application demands for both speed and scale. In DiceDB, clients can set up **query subscriptions** for specific data keys or patterns. When a value tied to a subscribed query changes, the updated result set is **pushed directly to the subscribed clients**. This push model eliminates the need for clients to continuously poll for updates. + +### How Query Subscriptions Work in DiceDB + +DiceDB's query subscription model is designed around the concept of **data-driven reactivity**: + +1. **Subscription Creation**: Clients initiate a subscription by registering a query or specific data key on DiceDB. For example, if an application wants to monitor changes to a leaderboard score, it subscribes to the relevant leaderboard key. + +2. **Data Monitoring**: DiceDB continuously monitors data for any changes affecting the subscribed queries. Unlike traditional databases, where monitoring is external and client-driven, DiceDB handles this process internally and in real time. + +3. **Automatic Query Execution on Data Changes**: When DiceDB detects a change in a subscribed key, it evaluates the relevant query automatically. This step is only triggered by a change, meaning no computational resources are used to check for updates unless necessary. + +4. **Result Streaming to Subscribed Clients**: After query evaluation, DiceDB streams the updated result set to each client that subscribed to the query. This eliminates the need for each client to independently query the database, thus reducing network traffic, CPU usage, and latency. + +### Real-World Application: Leaderboards + +To illustrate the power of DiceDB’s reactive architecture, consider the use case of a leaderboard in a gaming application: + +- In a traditional setup, if there are \( n \) users, each user might query the database every 10 seconds to see if the leaderboard has changed. This would result in \( n \) redundant queries every 10 seconds, all asking for the same information. + +- With DiceDB, each client can **subscribe** to the leaderboard key. As players’ scores change, DiceDB will automatically evaluate the leaderboard query only when a score update occurs. The updated leaderboard is then **pushed to all subscribed clients**. This approach drastically reduces the number of database operations, as DiceDB processes the query once per update instead of once per client, per interval. + +For instance, if the leaderboard updates once every 10 seconds, DiceDB would process the query once, and stream the result to all clients simultaneously. This efficiency means DiceDB can scale to support significantly larger user bases without experiencing the query-based bottlenecks seen in traditional databases. + +### Efficiency Gains and Cost Reduction + +The reactivity model provided by DiceDB offers several key benefits: + +- **Reduced Query Load**: DiceDB eliminates the need for repeated, redundant queries by moving from a client-polling approach to a server-push model. This is especially beneficial in high-read applications where many clients request the same information. + +- **Lower Latency**: With data being pushed as soon as it changes, clients receive updates faster, leading to lower latency and a more seamless user experience. + +- **Resource Savings**: By processing a query once and distributing the result to all subscribers, DiceDB significantly reduces CPU, memory, and network usage. This approach can lead to cost savings by reducing the need for scaling infrastructure to handle repetitive loads. + +### Conclusion: DiceDB and the Future of Reactive Databases + +DiceDB exemplifies a shift towards a data-reactive paradigm, where the database is not merely a repository for queries but an active participant in maintaining data synchronization with minimal client intervention. Through query subscriptions and real-time data streaming, DiceDB provides a model that can efficiently handle the demands of modern applications, enabling real-time, highly reactive user experiences at a reduced cost. + +This transition to reactivity represents a substantial evolution in database technology. As applications increasingly require real-time interactivity and scalable performance, systems like DiceDB are poised to play a critical role in powering the next generation of real-time applications. diff --git a/docs/src/content/blog/dicedb-is-open-source.md b/docs/src/content/blog/dicedb-is-open-source.md index ee915642f..3cc6ee763 100644 --- a/docs/src/content/blog/dicedb-is-open-source.md +++ b/docs/src/content/blog/dicedb-is-open-source.md @@ -2,7 +2,7 @@ title: "DiceDB is now open source" description: "Since the inception of DiceDB, our vision has always been rooted in building a high-performance caching backbone for modern architecture. Today, we are thrilled to announce that DiceDB -is now open source under the GNU Affero General Public License (AGPL) 3.0. This marks a significant milestone in our journey - one that reflects both our technical clarity and our commitment to the open-source ethos." +is now open source under the GNU Affero General Public License (AGPL) 3.0. This marks a significant milestone in our journey - one that reflects both our technical clarity and our commitment to the open source ethos." published_at: 2024-12-14 author: arpit --- @@ -10,16 +10,13 @@ author: arpit Since the inception of DiceDB, [our](dicedb.io/team) vision has always been rooted in building a high-performance caching backbone for modern architecture. Today, we are thrilled to announce that DiceDB is now open source under the [GNU Affero General Public License (AGPL) 3.0](https://github.com/dicedb/dice?tab=AGPL-3.0-1-ov-file#readme). -This marks a significant milestone in our journey - one that reflects both our technical clarity and our commitment to the open-source ethos. +This marks a significant milestone in our journey - one that reflects both our technical clarity and our commitment to the open source ethos. ## Why we switched In the early days of DiceDB, we explored a [Business Source License (BSL)](https://en.wikipedia.org/wiki/Business_Source_License) to balance monetization and community engagement. However, as our product evolved, -so did our understanding of its core mission. - -DiceDB isn't just another in-memory database; it's the foundation for the next generation -of applications. By streamlining our +so did our understanding of its core mission. By streamlining our technical and business priorities, we realized that the AGPL 3.0 license better aligns with where we are headed — an open, collaborative ecosystem for the future of caching infrastructure. @@ -30,7 +27,7 @@ to use, inspect, and improve. By adopting this license, we're doubling down on accessibility and transparency. We believe that great infrastructure shouldn't be proprietary; it should be a shared foundation for the ecosystem to build upon. -The AGPL license removes ambiguity around our open-source intent, signaling +The AGPL license removes ambiguity around our open source intent, signaling to developers and organizations alike that DiceDB is here to stay as a truly open technology. This clarity encourages collaboration, enabling developers and engineers to confidently build, adapt, and contribute back to the project. @@ -38,7 +35,7 @@ Whether it's improving availability, or performance, or even building advanced c every contribution will strengthen DiceDB for everyone. Additionally, the AGPL is designed to ensure fairness. It prevents -the “freeloading” problem — where companies take open-source software, modify it, +the "freeloading" problem — where companies take open source software, modify it, and deploy it as a proprietary service without contributing back. With the AGPL, any modifications made while running DiceDB as a service must also be shared under the same license. This keeps the playing @@ -56,14 +53,14 @@ together. Every line of code written to build DiceDB is open for inspection, lea This license switch isn't just a symbolic move; it's a strategic decision. Our roadmap includes multi-tenancy, horizontal scalability, infrastructure abstraction, reactivity, observability, persistence, fault tolerance, seamless cluster management, and a -robust control plane with high observability. +robust control plane. We are also enhancing query capabilities and data structure support to meet -the demands of AI-first applications—an approach poised to become the standard +the demands of AI-first applications - an approach poised to become the standard in the coming decade. Furthermore, lightning-fast replica creation will reinforce DiceDB's ability to handle production-grade workloads with exceptional reliability and resumability. -By going fully open-source under AGPL 3.0, we're inviting the +By going fully open source under AGPL 3.0, we're inviting the community to join us in building the caching backbone that powers the next generation of applications. Together, we can push the boundaries of what's possible. @@ -73,9 +70,9 @@ let's build the future of real-time data together. ## Things you will find interesting -As you explore DiceDB, you’ll uncover several intriguing facets of our journey. +As you explore DiceDB, you'll uncover several intriguing facets of our journey. Dive into our [community hub](/community) to learn more about our vibrant community and browse through notes from our weekly calls. -If performance excites you, don’t miss our [benchmarks](/benchmarks) showcasing how DiceDB achieves 10x better throughput +If performance excites you, don't miss our [benchmarks](/benchmarks) showcasing how DiceDB achieves 10x better throughput by leveraging the full potential of the underlying hardware. -At our core, we’re just a [group of passionate engineers](/team) pursuing an ambitious vision; join us and help shape and build the caching backbone of modern architecture. +At our core, we're just a [group of passionate engineers](/team) pursuing an ambitious vision; join us and help shape and build the caching backbone of modern architecture.