ArmoniK Load Balancer

AEP/aep-00004.md

@@ -0,0 +1,122 @@
+# AEP 4: ArmoniK Load Balancer
+
+|                   |ArmoniK Enhancement Proposal|
+---:                |:---
+**AEP**             | 4
+**Title**           | ArmoniK Load Balancer
+**Author**          | Florian Lemaitre <<[email protected]>>
+**Status**          | Draft
+**Type**            | Standard
+**Creation Date**   | 2024-04-12
+
+# Abstract
+
+This AEP proposes an ArmoniK native Load Balancer, which can be used to scale ArmoniK clusters.
+
+# Motivation
+
+ArmoniK is built around a centralized database, queue and object storage, and that makes it hard to scale. While it is fairly easy to have a scalable (distributed) object, it is much more complex to setup a scalable database or queue.
+
+# Rationale
+
+A simple alternative is to have multiple isolated ArmoniK clusters, with their own database, queue, and object storage. To make such a design transparent and easy to work with, an ArmoniK Load Balancer would be needed.
+
+Such a Load Balancer would understand the ArmoniK concepts and forwards user requests based on ArmoniK data like the session ID.
+
+# Specification
+
+## solution overview
+
+An ArmoniK Load Balancer would be an external service that would serve ArmoniK API in gRPC and redirect them to a cluster.
+Upon Session creation, a single cluster is assigned to the session, chosen by the implementation. Afterwards, all requests targeting the session will be redirected to its assigned cluster.
+This binding enables to easily keep track of the tasks and data within a session, even if more tasks and data are created from the workers.
+
+No additional API is required, and few adjustments are required infrastructure wise. The ingress in front of the actual cluster should accept authentication header from the load balancer itself, or the load balancer being an authenticated user of the cluster that is able to impersonate any user of the cluster.
+
+The Load Balancer could scale to many servers without issues, especially if they do not need a database.
+
+### Database-less
+
+The simplest design would be to compute deterministcally the cluster where to send the session by computing a hash of the session ID.
+While this would enable infinite scaling in theory, it comes with a group of issues not that easy to solve.
+
+When the session is created, the cluster must be chosen before the session ID is known. A solution to this problem would be to add an RPC to create a session with a specific session ID and let the Load Balancer generate this ID before choosing a cluster.
+
+Some RPCs do not have the session ID in the request message which would make it hard to get the right cluster. A possible aolution to this is to request the data on all clusters, and cache which result is bound to which session. When the client request the creation of a result, the link between the result and its session could also be recorded in this cache. Apart from Load Balancer booting, this would limit the number of time where the load balancer need to request all clusters. Another solution would be to add the session ID to all requests in the API.
+
+If the number of clusters change over time, how one could compute to which cluster is bound a session ID is unclear. Moving sessions between clusters to rebalance the distributed hash table does not seem possible. Of course, this problem does not exist if the clusters are static.
+
+### with database
+
+To work around the issues related to absence of a database, we can add a database to the load balancers to record the links between the sessions and the clusters. In particular, there is no need to know the session ID beforehand when creating the session.
+
+Having a database to store the link between sessions and clusters also enable smart decisions about which cluster to choose for new sessions. For instance, the Load Balancer could choose the cluster with the least load. It could even allow the user to choose the cluster themselves using some request metadata.
+
+It would also enable dynamic clusters as no rebalancing of the clusters would be needed.
+
+The downside to the database approach is the loss of the stateless property of the Load Balancer. Because the LB is baked by a database, it will also be limited in scaling by said database.
+
+## Implementation
+
+The implementation of such a Load Balancer would consist in a gRPC server and client. The server would take ArmoniK requests from the user, and forward them to the right cluster using ArmoniK API. It would forward the response back to the user.
+
+In particular, it is transparent from both ends, client and control plane.
+
+No specific API is required.
+
+## Remarks
+
+### On-demand clusters
+
+An extension of the Load Balancer could be to provision clusters on-the-fly. When a session is created, the Load Balancer could deploy a fresh cluster that will be destroyed when the session is deleted.
+
+The design of such a solution is more complex and should be considered later on.
+
+### Authentication
+
+There are many ways to handle the authentication in the presence of load balancing.
+Whatever solution is chosen, the TLS connexion must be terminated at the load balancer to let it interpret the queries.
+
+One possibility would be to perform authentication and authorization at the Load Balancer level, and forward the request to the cluster only if the user is permitted to do the action.
+Authentication would be disabled on the clusters, or at least would allow the load balancer to perform any action.
+This would greatly complexify the Load Balancer itself.
+
+Next possibility would be to let the load balancer impersonate users to the cluster using the standard impersonification faeture of the authentication system.
+This would force te load balancer to have a dedicated user on the clusters with the permission to impersonnate any user.
+As it is currently not possible to chain impersonifications, it would also forbid end-users to impersonate themselves, even if they have the permission to do it.
+
+Last possibility would be make the load balancer forward the identity headers to the clusters, in the same way the nginx ingress transmit the identity to the control plane currently.
+Extra configuration at the cluster ingress would be required to allow the load balacner to set those sensitive headers.
+This last solution is the preferred one as it is both simple to implement, secure, and featureful.
+
+### Partitions
+
+As a single session cannot span multiple clusters, all the partitions requested by the session must exist on the target cluster.
+The load-balancer would be able to select a cluster that support all the partitions requested, if any.
+It should avoid selecting a cluster that does not support a partition, and if no cluster support all the partitions, the load balancer would raise an error.
+
+As an example, consider a cluster A with partitions p1 and p2, and a cluster B with partitions p2 and p3.
+If a session need to target p2, it can go on either A or B.
+If a session targets p1 and p2, it can only go on A.
+If a session targets p3, it can only on B.
+If a session targets p1 and p3, it cannot go anywhere, and the load balancer raise an error.
+If a session targets p4, the load balancer raise an error.
+
+### Monitoring
+
+The presence of load balancers would make the monitoring harder because of the listing accross clusters.
+
+The recommended way would be to enforce listing tasks and results on sessions, and adapt the GUI to match that requirement.
+To have a global overview, it would be necessary to add some session-wide statistics like the number of tasks completed and in error to enable the view at the session level, without the need to query all the clusters.
+The aggregation could be performed at regular interval on all opened sessions to reduce the quantity of sessions to query, as a closed session cannot have more tasks added.
+
+### Cluster field
+
+An alternative to the session id field being used to load balance the request, we could add a cluster field to all the rpc, and use that to load balance the requests.
+It could also be use for the monitoring instead of imposing a filter on the session.
+
+This change would be more intrusive API-wise.
+
+# Copyright
+
+This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.