Merge pull request #1 from M6Web/chore/add_sources

chore: add sources

.gitignore

@@ -0,0 +1 @@
+__pycache__

Pipfile

@@ -0,0 +1,14 @@
+[[source]]
+name = "pypi"
+url = "https://pypi.org/simple"
+verify_ssl = true
+
+[dev-packages]
+
+[packages]
+boto3 = "*"
+ruamel-yaml = "*"
+prometheus-client = "*"
+
+[requires]
+python_version = "3"

README.md

@@ -1,2 +1,185 @@
-# hsdo
-HAProxy Service Discovery Operator
+# haproxy-service-discovery-orchestrator
+Orchestrate Service Discovery for HAProxy.
+
+We are currently using HSDO to load-balance our VOD traffic, between CDNs and our origins.
+
+CDNs --> NLB -> HAProxy (with HSDO) --> origins
+
+Using an NLB is optional if HAProxy instances are running in a public VPC subnet.
+
+You'll need to take care of updating DNS records if using public HAProxy endpoints (failures, Spot reclaims, etc.)
+
+
+We have tested this platform with tens of HAProxy instances, reaching up to 200Gbps traffic.
+
+We are using Spot instances to run HAProxy with HSDO, using multiple AZs, but by optimizing traffic through AZ (because inter-AZ traffic is extremely expensive).
+
+We are currently running it along with HAProxy 2.2.
+
+- ASG or Consul is listing available servers
+- HAProxy SDO Server gets servers from ASG, sort them, and save them in DynamoDB
+- HAproxy SDO Clients get sorted servers from DynamoDB and send configuration to HAProxy Runtime API
+- All HAProxy instances have the same configuration
+
+![HSDO Simple](doc/hsdo-simple.png)
+
+We have one server, and as much clients as haproxy load balancers.
+
+## Why HSDO
+
+AWS load balancers don't allow algorithms different from round robin.
+
+HSDO allows to use HAProxy in front of one or multiple AutoScalingGroups on AWS.
+
+HSDO implements ordered backend servers lists to use functionalities like consistent hashing, which makes it possible to use all the power of HAProxy, but on AWS.
+
+By design, HSDO is able to run several HAProxy instances, to load balance from ten to hundreds of backend servers and separate traffic depending of AvailabilityZone.
+It is reliable and fault tolerant, as each HAProxy server updates its configuration asynchronously from a DynamoDB table.
+
+We wanted a very simple and efficient implementation for HSDO, which we didn't find in Consul.
+
+## Prerequisities
+
+This project is using pipenv. If you don't have it, please see [here](https://github.com/pypa/pipenv#installation).
+
+You need to have `AWS_PROFILE`, `AWS_DEFAULT_REGION` setted and to be authenticated to access your DynamoDB table.
+
+## Usage
+
+In project directory:
+
+```sh
+pipenv install
+pipenv shell
+python3 src/main.py --[client|server] (--[debug]) (--[help])
+```
+
+## Configuration
+
+Parameters can be defined through a config file or environment variables.
+Environment variables will overwrite `conf/env.yaml`.
+
+### Server Only
+
+Configuration that is specific to HSDO Server.
+
+`SERVER_ASG_NAMES`: List of ASG names where to find target servers (EC2 instances for which HAProxy will load balance traffic). May be a list, separated with comma. If `aws` mode enabled. Default to ` `.
+
+`SERVER_CONSUL_API_URL`: Consul address where to find your target servers. If `consul` mode enabled. Default to ` `.
+
+`SERVER_CONSUL_SERVICE_NAME`: Consul service name where to find your target servers. May be a list, separated with comma. If `consul` mode enabled. Default to ` `.
+
+`SERVER_HAPROXY_BACKEND_SERVER_MIN_WEIGHT`: Minimum weight of a newly added backend server. Default to `1`.
+
+`SERVER_HAPROXY_BACKEND_SERVER_MAX_WEIGHT`: Maximum weight of a backend server. Default to `10`.
+
+`SERVER_HAPROXY_BACKEND_SERVER_INCREASE_WEIGHT`: Defines the level of increase in the weight of the newly added servers. Every 'SERVER_HAPROXY_BACKEND_SERVER_INCREASE_WEIGHT_INTERVAL', the weight of a new server will be increased by this value. Default to `1`.
+
+`SERVER_HAPROXY_BACKEND_SERVER_INCREASE_WEIGHT_INTERVAL`: In seconds, time between each weight increasing. For example, if we want a new server to have its target weight 5mns after it has been added to the backend, going from weight 1 to 10, we would use interval 30: 30s interval, 10 times between 1 and 10: 300secs. Default to `30`.
+
+`SERVER_MODE`: Can be `aws` or `consul`. Default to ` `. `consul` is higly experimental, it probably doesn't work. Only `aws` mode is prod ready.
+
+### Client Only
+
+Configuration that is specific to each HSDO Client, next to HAProxy.
+
+`CLIENT_HAPROXY_SOCKET_PATH`: HAProxy socket to use [Runtime API](https://cbonte.github.io/haproxy-dconv/2.2/management.html#9.3). Default to `/var/run/haproxy/admin.sock`.
+
+`CLIENT_HAPROXY_BACKEND_NAME`: HAProxy default backend name. Default to ` `.
+
+`CLIENT_HAPROXY_BACKEND_BASE_NAME`: HAProxy default backend base name for server template. Default to ` `.
+
+`CLIENT_HAPROXY_BACKEND_SERVER_PORT`: Port of target servers. Default to `80`.
+
+HAProxy default backend configuration can be seen in `haproxy.cfg`:
+```
+backend {{ CLIENT_HAPROXY_BACKEND_NAME }}
+server-template  {{ CLIENT_HAPROXY_BACKEND_BASE_NAME }} 1-{{ HAPROXY_BACKEND_SERVERS_LIST_SIZE }} 127.0.0.2:{{ CLIENT_HAPROXY_BACKEND_SERVER_PORT }} check disabled
+```
+
+For example, with :
+```
+backend http-back
+server-template mywebapp 1-10 127.0.0.2:80 check disabled
+```
+You will have this kind of statistic page : 
+
+![](doc/backend-servers-list.png)
+
+### Both
+
+`INTERVAL`: Interval between each loop for client/server. Default to `1`.
+
+`HAPROXY_BACKEND_SERVERS_LIST_SIZE`: As max range describe [here](https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#4-server-template). Default to `10`.
+
+`DEBUG`: To enable debug log. Default to `false`.
+
+`DYNAMODB_TABLE_NAME`: Name of dynamodb table. Default to ` `.
+
+`AWS_DEFAULT_REGION`: default region needed for dynamodb access. Default to ` `.
+
+`EXPORTER_PORT`: port for prometheus exporter. Default to `6789`
+
+## Dedicated ASG Configuration (AWS Only)
+
+HSDO Client can be configured to follow specific ASGs that are present in `SERVER_ASG_NAMES`.
+
+For example, if `SERVER_ASG_NAMES` contains `ASG1,ASG2,ASG3`, `CLIENT_ASG_NAMES` may follow `ASG2`. 
+
+This is usefull if you want to split traffic per AZ.
+
+![Dedicated ASG Implementation Example](doc/HSDO_AZ_Limiter.png)
+
+This is possible if you enable `CLIENT_DEDICATED_ASG`.
+
+If the target's ASG name is in `CLIENT_ASG_NAMES`, then the target is put in default HAProxy backend.
+
+If the target's ASG name is not in `CLIENT_ASG_NAMES`, then the target is put in fallback HAProxy backend.
+
+If needed, ASG name in `CLIENT_ASG_NAMES` can alse be added in fallback HAProxy backend with `CLIENT_ALL_SERVERS_IN_FALLBACK_BACKEND` enabled.
+
+Fallback from default HAProxy backend to fallback HAProxy backend are not handled by HSDO Client.
+
+### Client only
+
+`CLIENT_DEDICATED_ASG`: HSDO Client will use `CLIENT_ASG_NAMES` to configure default HAProxy backend, and put the other ones in fallbackend HAProxy backend. Default to `false`.
+
+`CLIENT_ASG_NAMES`: List of ASG that HSDO Client will use in default HAProxy backend. May be a list, separated with comma. Needed with `CLIENT_DEDICATED_ASG`. Default to ` `.
+
+`CLIENT_HAPROXY_FALLBACK_BACKEND_NAME`: HAProxy fallback backend name. Needed with `CLIENT_DEDICATED_ASG`. Default to ` `.
+
+`CLIENT_HAPROXY_FALLBACK_BACKEND_BASE_NAME`: HAProxy fallback backend base name for server template. Needed with `CLIENT_DEDICATED_ASG`. Default to ` `.
+
+`CLIENT_ALL_SERVERS_IN_FALLBACK_BACKEND`: to put also all default HAProxy backend servers in the fallback HAProxy backend. Default to `false`.
+
+## DynamoDB
+
+What dynamodb table should look like (terraform code):
+
+```
+resource "aws_dynamodb_table" "haproxy_service_discovery_orchestrator_table" {
+  name           = "haproxy-service-discovery-orchestrator"
+  billing_mode   = "PROVISIONED"
+  read_capacity  = 20
+  write_capacity = 20
+  hash_key       = "BackendServerID"
+
+  attribute {
+    name = "BackendServerID"
+    type = "N"
+  }
+
+  tags = {
+    name         = "haproxy-service-discovery-orchestrator"
+  }
+}
+```
+
+## Tests
+
+From root directory
+
+```sh
+pipenv shell
+python3 -m unittest
+```

conf/env.yaml

@@ -0,0 +1,34 @@
+###
+# Server Only
+###
+SERVER_CONSUL_API_URL: ""
+SERVER_CONSUL_SERVICE_NAME: ""
+SERVER_ASG_NAMES: ""
+SERVER_HAPROXY_BACKEND_SERVER_MIN_WEIGHT: 1
+SERVER_HAPROXY_BACKEND_SERVER_MAX_WEIGHT: 10
+SERVER_HAPROXY_BACKEND_SERVER_INCREASE_WEIGHT: 1
+SERVER_HAPROXY_BACKEND_SERVER_INCREASE_WEIGHT_INTERVAL: 30
+SERVER_MODE: ""
+
+###
+# Client Only
+###
+CLIENT_HAPROXY_SOCKET_PATH: "/var/run/haproxy/admin.sock"
+CLIENT_HAPROXY_BACKEND_NAME: ""
+CLIENT_HAPROXY_BACKEND_BASE_NAME: ""
+CLIENT_HAPROXY_BACKEND_SERVER_PORT: 80
+CLIENT_DEDICATED_ASG: "false"
+CLIENT_ASG_NAMES: ""
+CLIENT_HAPROXY_FALLBACK_BACKEND_NAME: ""
+CLIENT_HAPROXY_FALLBACK_BACKEND_BASE_NAME: ""
+CLIENT_ALL_SERVERS_IN_FALLBACK_BACKEND: "false"
+
+###
+# Both
+###
+INTERVAL: 1
+HAPROXY_BACKEND_SERVERS_LIST_SIZE: 10
+DEBUG: "false"
+DYNAMODB_TABLE_NAME: ""
+AWS_DEFAULT_REGION: "eu-west-1"
+EXPORTER_PORT: 9000

doc/HSDO_AZ_Limiter.drawio.xml

@@ -0,0 +1 @@
+<mxfile host="app.diagrams.net" modified="2020-11-25T13:42:25.936Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36" etag="o_1fXtvywVQeRKy2Ftkw" version="13.9.3" type="device"><diagram id="mgokIonY5ivCf-a4258m" name="Page-1">7V1fl6I4Fv80PloHEgL4qFb39EPPTJ2t2bMz+zIHJSrblHEQu8r+9BuUICRRghKDtvVgSQgB7v3dv7mJPTh++/glCVaLX0mI4x6wwo8efO4BACwb0H9Zy3bf4kJ/3zBPonDfZB8aXqMfOG+08tZNFOJ1pWNKSJxGq2rjlCyXeJpW2oIkIe/VbjMSV++6CuZYaHidBrHY+p8oTBf7Vh94h/YvOJov2J1td7A/8xawzvmbrBdBSN5LTfBTD44TQtL9t7ePMY4z4jG67K/7fORs8WAJXqYqF6TP34av1r/Ib7+vnq1wOArf/+31Uf5w34N4k7/x8L/0GG/673id9uEkf/h0yyiSkM0yxNmgdg+O3hdRil9XwTQ7+04xQNsW6Vucn/6OkzSi1BzG0XxJ2yYkTckbPREHExyPSBLiZExiktBzS7Kkg4zyB6IX4o+jr2oXBKTIw+QNp8mWdmEXMJrnoAOD/Pj9wELHz9sWJfYB1hjksJkXYx8oS7/kxG1AaB8IhD5BWauesus0Id8wo10PQHfq48msOMPQCrLLSBL9IMs0OMaWlGSjz6I4Lg04Qp/Hn8ctcQR6T6jCE2hLeOJAkSfaWIJgDfaDW8R+H3rdA7/9U4O/D5HdOfQ7LbOEo99sNgPTqYxVoTtxkXsJq1rRRz7HEQcqcsS2dbHErWcJHYV6PpmqoO7EKmucxmQT1rOnBZqhql6RILiwu1dBsCdSJ6S+W35IknRB5mQZxJ8OraMqpA99vpIMWTtK/Q+n6TZ3RINNSqp0pLRKtn9m11P85Id/5cPtDp4/Kkfb/IgXD3+K5eIx8ZGDrIJj2Sud5helANkkU1wPrDRI5jg90W/vrIsASHAcpNH36oPI+Jlf+kIi+ogFcKBbFTWbh8T+BfKrOFQUj3E+UAZGgPIRpSWc0KO/SmcOKMkOtmXI3Bq4WIRXjy6kB13I5VS5oHI044tpwocmqsUA0KNhCpvMEGCpIWCYJMG21G2VdVg3uI/Lxdxcf2B5p/rTL/snaBeOthE4PvQd39G7V30n5jJ++zoSQNcoWNbgnzqKDqqry0G1HbOC6JUl0VKUxIocHsTyiCSuqRSkwyzP2iuSGLu2z1FGq/yKkPWYxsF6HU33jXmXQzxfDgU5Ecd2iLAnE/GB68HAbVnEkaKIQ0uOyGYi3tQKQb9qVaB12gp51sn+mqwQEpRED7gxpeYojL7Tr/Ps65fhS0I+tuzMJGEnWAu9d6n7mCzX0TrFu7f7EqwX0XJ+/NoZ2fWbFmBx/9mQfQc4HluW65abqtdOiid8ff6dDjKOo91di6eaSJ50f0PWfIk65OAfBtifSS1ckfhqQYGCAW9ZIDCuQz0BRkcx80CHTnQ4TvfQAQy7ug0s7H1aSpai7KilBLChpXQNWEpGw0aW8mEXr6f5bEeYvDSv+SSTly8vtOEVJ/RlW6WycnzdsvfhGI/ggDhLduNUFq14B6gshgo3TmVRY3SAyq5ZX+mstKBXyQsWaW+1vCDnCpWdJqlzNGzbOfIVnSN0BE0XZgpdu4pBBDls7R/s4qw47zW58LSX5XDzgVx/TV6WmeliKfxVc3FPjdDftdBAFf1mQgPB1fcahgbeNUDrC5Yx97s1mEZleFxkGvlkpKNaR6LPMJqtDyjrhUJLNDSMdRNmbaiGSgLjkJ4o5zDsag4jz3IcEhgtz8Qxc6Z/Ju4ifLHHLAnx83YZvJHndoNhM/ILPNPyC8Xq0teAMhVYuxJbSk1K4p26pObAiqN1ptWjJf04ygZKnrR3ouI0R3aZAXlTkFcqTik5qX4WSxjfojDcaRAZc6tapQVuuSzsKBIXSOQWlHDLR7q4JWa7uFQSFctAhWdWsAx3aazlLJpvkozhRZLMSheUlvNFNtpmmUZv2dlgFd0vpwvnnsmljNPeVTktpqhEQyvLeAvMYIbrkni+3uKIpC2RDkkox9ouLejgCvZtXlMeKecQ4yqnZqAj8V5b3iqU1ZSLiew//njZEe2fDV5nHrosoS31yL5m6yaq+FAXwgSvox/BZDdehoo8PqCDo1EPPedp8RJ+Pg3dkXvSaOZrzvJBe8VKrzKyTkjGUVHuW0/QhRVWsnL9C6HmPbn2oPxXuYlbHY7MZmusByhipTsX1ljfMF5lajwJpt8ySz6jH8PVqmIWUiqMWd/h6y/3q9d5C46QxN+6qgV3fKl0/oTlro/C+5oVdrJ1X3t1v14FS6bsqfz+PXx5+bs8ZVg+L8Dt2pWFtjBpgmTrt5jHVVlUp2314s+9gEuypq4DK7gQMqIa76T0un6ehGXq1Gq0W9enfRtCw7XX6BZn8W4HYczt/pkR9lhXqYyWZmtG7hItg6Mu3qPMu5vlbH17wKPGfD2bK4YKjzpvM/BwOogOWSr5USLbIdBAB3QPNXdXvSlR3eYLC927K98UVWAHqCzmzG+dyqLO6ACZxfVmp1KWw66mLClxeQybz1m6RmYOOrTy6txl1j3lmLV+SkI1uG2nLFOyJYbcS2g62Y0QJZt1+OOcD34DkJZKndn7XnN9GNtR9CE15qSGpTmMSY2w6Qd0zhQcz/eerOOCw8tjS4LTty1O8K8hOqyy5iE6Bg0ONCs6fYAGT+WaF7+KQ5urk1A2QA5XCDvgBmpLcgpn96qSY2ZlvcmVX+dMcrUoJ8rrZVw9+0o5HnoCXMTA2xhDC8b6oMGCMcndQOVqx1OznK2JEhBE6WJBaR6w10uvYfwrByb7tF77LpbgYQ3O9LCEkRykJketIQ4+ENemZ7LPvXUYcX0bCBnLa2NO3OzugbkLAklNWg4Kuklxq1QJ5mwBc1decOCJ5XZ3g7lzC2KM6Ec9WO0DD1RyfFU/7nxteXrcq2tOsaTvblBsBI169hnXhsYO2G6x5O9uENgBPapu8/X4mX3gntR3t+wBiDOKD+SaQK4unYs45FaXRTr8Sg516FoCdPm0kG7oitN6dwPdNs2+4bRQHzinIXjuBHZfLOXia7c1I9AXZ8fuBoEdUJ7qyNVl9uFp5PLbwSkj1xeQyzu/mpE7eCQ4VRAIgCICgR7zDazBk1OaefW4LdQU16o0HddRnLZqC422JU6WPuB4CRw1zfDYwj7OZ0/xCENdO2a3WRT3AF1LoDuyKculoAMCUs6NtyVDXTncplHTbYCuRQAp/xiQLiMq/HT22VGvZCjFoLfxTxQh+S6fxy26f6q/nhon27qROUsjaNakDsWd0c8NoCVDXTmAtq0bmYA0AiBNThwSuH5uHCsZSjGO1a0O0ZFNkjWrwxuZiGwTzZ5h4+4e/aGI5lVn/FBI0yKYGnR2xbjfyKSmETRrMu5ebVSsjGZhKMR7vYbQXKPLdaH5RiY6jaBZk6fh14bbymgWhkKK66zqAUQPE5JtRHHongSrxa8kzDbL/fR/</diagram></mxfile>

src/client/client.py

@@ -0,0 +1,55 @@
+##
+# Imports
+##
+import threading, sys, time
+from client.haproxy import HAProxy
+from common.logger import Logger
+from common.dynamodb import DynamoDB
+from common.configuration import Configuration
+from common.server_model import ServerModel
+from common.prometheus import Prometheus
+from common.dynamodb import DynamoDB
+##
+# Handle client mode
+##
+class Client(threading.Thread):
+    ##
+    # Initialization
+    ##
+    def __init__(self):
+        threading.Thread.__init__(self)
+        self.logger = Logger("HSDO.client.client")
+        self.haproxy = HAProxy()
+        self.dynamodb = DynamoDB()
+
+    def run(self):
+        self.dynamodb.checkTableReady()
+        self.haproxy.checkSockConf()
+        self.haproxy.checkBackendConf()
+        oldDynamodbServers = []
+        dynamodbServers = []
+        while True:
+            oldDynamodbServers = dynamodbServers
+            dynamodbServers = self.dynamodb.listServers()
+
+            # If dynamodb server is removed then remove metric
+            for oldServer in oldDynamodbServers:
+                if not self.isServerInList(oldServer, dynamodbServers):
+                    self.logger.info("Removed : " + oldServer.toString())
+                    Prometheus().removeMetric(oldServer)
+            # If dynamodb server is added then display metric
+            for dServer in dynamodbServers:
+                if not self.isServerInList(dServer, oldDynamodbServers) and dServer.backendServerStatus != "disabled":
+                    self.logger.info("Added : " + dServer.toString())
+                    Prometheus().serverWeightMetric(dServer)
+
+            for server in dynamodbServers:
+                self.haproxy.setServer(server)
+            time.sleep(Configuration().get("INTERVAL"))
+
+    def isServerInList(self, server, serverList):
+        match = False
+        for s in serverList:
+            if s.IPAddress == server.IPAddress:
+                match = True
+        return match