This repository offers a modeling technique for designing scalable systems. Every new system should be designed with scalability in mind — scalability is essential for success. With Scalable Modeling, you can embed scalability in your system design from the start. Influenced by Clean Architecture, Event-Driven Architecture (EDA), Domain-Driven Design (DDD), EventStorming, and CQRS, this approach remains flexible and doesn't rigidly adhere to any single methodology. That is why this event-centric architectural approach has different name: CEQS - Command-Event-Query Separation.
Justification for the red arrows in sections: Queries & Time Travel.
Webinar recording: Software engineering for the future: Fast, scalable and built to last (November 26, 2024).
- Scalable Modeling – An Event-centric Approach
- Scalable Modeling – The Upfront Modeling Technique
- License For Using the Pictures
There are three opportunities and three challenges in scalability and understanding them takes you far already. Combining the opportunities & challenges with an upfront modeling technique provides a solid foundations for modeling scalable systems.
- Decomposition - scale by splitting different things
- Duplication - scale by cloning
- Partition - scale by splitting similar things
Immutability (of messages/events) plays key role in each aspect.
Software is ultimately a model — a conceptual solution that, while invisible, solves real-world challenges. In software engineering, three aspects are critical:
- Understanding: WHY the software is needed (understanding the purpose and the problem it should address).
- Designing: WHAT is the conceptual model for the solution.
- Developing: HOW the solution is implemented.
Scalable Modeling serves as an opinionated bridge from WHY to HOW, with a primary focus on the WHAT.
Iteration is naturally much cheaper when it is done on the conceptual model rather than on the implementation level.
"There is nothing so useless as doing efficiently that which should not be done at all."
Peter Drucker
Scalable Modeling is a method for shifting left in the software
engineering process. It helps crystallize the 'why' by focusing on the 'what,' allowing the creation of a
result that serves as an opinionated bridge to the 'how.'
Domain knowledge should be the starting point in building the conceptual model:
"It's developer (mis)understanding that's released in production, not the experts' knowledge."
Alberto Brandolini
Without a proper understanding of the domain, it's impossible to implement a conceptual model that accurately reflects it. Effective collaboration with domain experts is essential to bridge this gap. Events play a central role in uncovering domain insights and fostering a shared language. The immutability of events contributes significantly to scalability, particularly in event-driven architectures. Thus, we adopt an event-centric approach.
More about the method in chapter: The Upfront Modeling Technique.
- Deduplication - as exactly-once delivery is impossible in distributed systems
- Tailoring Consistency - as strong consistency is the wrong default
- Time Travel - as distribution causes eventual consistency
"A complex system that works is invariably found to have evolved from a simple system that worked." John Gall
Design flaws in the simple system tend to compound and lead to exponentially increasing complexity in the complex system.
"It's developer (mis)understanding that's released in production, not the experts' knowledge." Alberto Brandolini
Without a proper understanding of the domain, it's impossible to implement a conceptual model that accurately reflects it. Effective collaboration with domain experts is essential to bridge this gap.
"Doing the wrong thing right is not nearly as good as doing the right thing wrong." Russel L. Ackoff
Without a proper understanding of the domain, it's easy to prioritize technical correctness over solving the actual problem.
"Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations." Melvin E. Conway
Conceptual models derived from the domain often lead to software structures that should mirror the communication patterns within the organization.
"Civilization advances by extending the number of important operations which we can perform without thinking of them." Alfred North Whitehead
When we can focus solely on implementing conceptual models from the domain without worrying about the technical details we can abstract away, we achieve the highest velocity.
Building a scalable foundation allows businesses to adapt to increasing demands and capitalize on opportunities without being limited by technical constraints. As these influential leaders have noted, scalability is not an afterthought; it is integral to achieve sustainable success.
Without a scalable system from day one, even the best ideas can be slowed down by bottlenecks and inefficiencies as they grow. Designing with scalability in mind ensures that as demand increases, systems can grow exponentially to meet those demands without sacrificing performance or creating technical debt.
"The faster you scale, the more wealth you create." Reid Hoffman (Co-founder of LinkedIn)
"In order to win, you must be able to scale exponentially." Marc Andreessen (Founder of Andreessen Horowitz)
"The most scalable businesses in the world are software businesses." Bill Gates (Founder of Microsoft)
Success is not just about growing fast — it's about building the right infrastructure from the outset, so that growth becomes an advantage, not a challenge.
In Event Centrism, everything we experience is either an event or a trace of one. Reality is not made up of static objects but a continuous flow of occurrences, where everything, from a falling leaf to a mountain, is part of an ongoing process. Even seemingly permanent things are temporary outcomes of past events, always subject to change.
Our lives are driven by events — every thought, action, and emotion is triggered by something, and memories are a collection of past events that define who we are. Traces of past events — like an old building or a weathered book — remind us of what once occurred, continuing to shape the present.
Events can also be potential, waiting to happen, or hidden, unfolding beyond our perception, like biological processes or distant cosmic phenomena. Every event is part of a cause-and-effect continuum, influencing future occurrences. Time, in this view, is meaningful only as the medium through which events unfold.
Even the self is an ongoing event, constantly shaped by experiences and interactions. In Event Centrism, everything in life is fluid, dynamic, and interconnected, emphasizing that our world and our identities are in constant motion, driven by the events we experience.
Event Centrism is not an absolute truth but rather one way to understand the world — a very useful way to model systems.
Event-Driven Architecture (EDA) is a design pattern where systems react to events in near real-time. Components communicate by producing, detecting, and responding to events, enabling asynchronous processing and loose coupling between services, which allows for more scalable and resilient systems.
CQRS is a pattern that separates the responsibilities of updating data (commands) and reading data (queries). By dividing these operations, CQRS improves performance, scalability, and security, allowing for more efficient handling of complex, high-demand systems.
Scalable Modeling does not go into purism in CQRS - in Scalable Modeling queries can (when well justified) also query command models for improved consistency where it does not jeopardise the scalability. Commands can also return simple data like sequence number of the produced events.
Universal Scalability Law, Wikipedia
Limiting contention is the key for higher scalability.
In this section we are in the context of:
- Event-Driven Architecture (EDA)
- Command Query Separation (CQS)
- Command Query Responsibility Segregation (CQRS)
The diagram below illustrates a novel software architecture concept called CEQS — Command-Event-Query Separation. It is an enhancement of the traditional CQS (Command Query Separation) pattern highlighting the importance of events in scalable systems.
In scalable systems, events should be treated as first-class citizens, as their immutability is fundamental to achieving scalability. Their asynchronous nature enables loose coupling between services, while their immutability allows events to be queried or streamed as an accurate record of what has occurred within the system.
Immutability ensures that events are append-only, allowing distributed systems to replicate and process data consistently across services without conflicts.
Commands, Events and Queries each serve as distinct interfaces to a service:
- Commands initiate state changes, resulting in the creation of Events.
- Events represent the immutable outcomes of those state changes, facilitating asynchronous communication and providing a historical log.
- Queries retrieve current or historical data without causing side effects.
This clear separation of concerns ensures that systems remain scalable, decoupled, and maintainable.
Additionally, a Bounded Context defines the boundaries within which a specific domain model applies, promoting clarity and enforcing the separation of responsibilities across the system. Each Bounded Context encapsulates its own Commands, Events, and Queries, ensuring that interfaces remain consistent and reducing the risk of cross-domain coupling.
- Clean Domain Logic
- Separating commands and queries results in clearer, more focused domain logic. Commands handle state changes, while queries handle data retrieval, ensuring that business logic remains concise and purpose-driven.
- Separation of Concerns
- Commands are responsible for writing data (changing state), while queries are responsible for reading data. This distinction enhances maintainability by providing cleaner code and reducing the risk of unintended side effects.
- Loose Coupling
- Within Services: Commands and queries are decoupled through immutable, private events, allowing each to evolve independently without interference.
- Between Services: Public events decouple microservices, enabling them to operate and scale independently, reducing dependencies between service boundaries.
- Resiliency
- The system is more resilient because failures in one component (e.g. command processing) do not cascade to others (e.g. queries). This design ensures fault isolation and minimizes downtime.
- Near-Real-Time Integrations
- Events enable near-real-time communication between services, allowing for fast and seamless integration. This improves responsiveness to changes in the system and facilitates use cases like live updates.
- Low Latency at Any Scale
- By separating read and write responsibilities, the system can optimize availability and scalability. For example:
- High Availability: Read-heavy workloads can be handled by scaling replicas.
- Scalability: This architecture supports all three dimensions of scalability, ensuring low latency even under high demand.
- By separating read and write responsibilities, the system can optimize availability and scalability. For example:
- Events
- Ubiquitous Language
- Commands & State
- Queries
- Policies
- Hotspots & Descriptions
- Consistency Boundaries
Ubiquitous language is more than connections and stickies in the picture above. At best, it is a shared conceptual framework that allows both technical and non-technical stakeholders to collaborate effectively, ensuring that all parties have a common understanding of the domain and its logic. It helps avoid ambiguity, enabling precise communication and design decisions, fostering a deep connection between the code and the business model it represents.
This language allows everyone involved to speak the same language, preventing misunderstandings and ensuring that the system's design remains consistent with the business goals and domain requirements.
I understand that querying the command model (red arrow in the picture above) is a topic that often sparks strong opinions. In my view, the command model serves as a special type of model that validates commands, and while it is not designed for querying, there are cases where it can be read from if necessary. For example, in in-memory command models Akka-style, where the model is clustered, a valid use case might involve querying the state of an entity immediately after its creation or update. Since the entity is already in memory, fetching it quickly from there can be justified. However, querying the command model should be avoided wherever possible, as there are many ways this can lead to misuse, such as introducing performance bottlenecks or inconsistencies. For this reason, querying the command model requires strong justification.
Interestingly, even if this is done as a last step in the Scalable Modeling, this often requires business-driven decisions. For instance, while financial transactions demand strict consistency, less critical processes like reporting can tolerate delays.
Dynamic Consistency Boundaries (DCB) is an emerging term and as we leave defining the consistency boundaries as a last step we make Scalable Modeling compatible with DCBs.
Since exactly-once delivery is impossible in distributed systems, we use effectively-once or idempotent processing to ensure duplicate messages don’t affect the outcome.
In distributed systems, defining consistency boundaries is crucial for balancing performance and correctness. These boundaries determine where strong consistency is enforced and where eventual consistency is acceptable. Interestingly, tailoring these boundaries often requires business-driven decisions. For instance, while financial transactions demand strict consistency, less critical processes like reporting can tolerate delays.
By explicitly defining consistency boundaries, you ensure the system aligns with business needs while optimizing scalability. This approach prevents over-engineering and allows the system to scale efficiently without unnecessary constraints.
When using CQRS with eventually consistent read models, the system's read models may reflect different points in time due to the delay in propagating updates. This allows for a form of "time travel," where users can observe data at various stages of consistency. As new events are processed, the read model gradually "catches up" with the latest state. However, during this period, the system might expose outdated or future-looking states, effectively letting users or systems "travel" between these states.
This is particularly significant when querying or presenting data that spans multiple sources or services across consistency boundaries. For example, one source might reflect an older version of the data, while another presents the most recent state.
As a result, combining data from these sources may yield conflicting or incoherent views of the overall system. This inconsistency poses a challenge when trying to form a unified or accurate perspective of the current state. To mitigate these issues, designers need to carefully consider the timing and aggregation of data to avoid misleading or inconsistent results, especially in cases where decisions are being made based on these read models.
The following persons have had a lot of influence on what this repository describes:
Creator of Event Storming, a collaborative workshop technique used to explore and model complex business processes through domain events. His approach helps teams rapidly gain insights into business domains by focusing on key events that drive processes.
Known for pioneering Domain-Driven Design (DDD), a software development philosophy that emphasizes aligning the software model closely with the business domain. His work focuses on creating a shared understanding between technical teams and domain experts to ensure the software reflects real-world complexity.
Renowned for developing and promoting Command Query Responsibility Segregation (CQRS) and Event Sourcing. His work centers on separating read and write operations in systems, improving scalability, and using event sourcing to maintain the history of all changes in a system, offering resilience and insights into past system states.
Author of Clean Architecture and a key figure in the promotion of software craftsmanship. His principles focus on building flexible, maintainable, and scalable systems by adhering to the separation of concerns and reducing dependencies between different layers of the system. Martin advocates for architecture that allows software to evolve over time, ensuring it remains easy to understand, extend, and refactor, even as requirements change. His work is centered on creating systems that prioritize independence from frameworks, databases, and UI, ensuring longevity and adaptability in software design.
The trigger that led to the creation of this page came from a LinkedIn post by Allard Buijze.
The post was about aggregates and their necessity. I studied the post and ended up watching Sara Pellegrini's and Milan
Savic's talk: The Aggregate is dead. Long live the Aggregate!.
After overcoming my cognitive dissonance, I needed to try out the new ideas by drawing them with
Excalidraw, which led to new ideas that I have collected in this repository.
Thanks for triggering the ideas!
See License.
<script src="https://giscus.app/client.js" data-repo="roikonen/scalablemodeling" data-repo-id="R_kgDOM8SonQ" data-category="General" data-category-id="DIC_kwDOM8Sonc4Cj6Rl" data-mapping="pathname" data-strict="0" data-reactions-enabled="1" data-emit-metadata="0" data-input-position="bottom" data-theme="light" data-lang="en" crossorigin="anonymous" async> </script>