This repository has been archived by the owner on Mar 31, 2023. It is now read-only.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Documentation] Node Metadata Manager design #209
base: master
Are you sure you want to change the base?
[Documentation] Node Metadata Manager design #209
Changes from all commits
7fc8661File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
MAC Manager Design Specification
Overview
MAC manager is an Alcor microservice which manages the lifecycle of all MAC addresses. Its responsibilities include but not limited to, allocating a unique mac address for a given port, releasing the mac address when not needed, and onboarding a region-specific mac ranges. It interacts with Subnet Manager and Port Manager to allocate mac addresses for gateway ports and customer ports.
Service Requirements
Allocate virtual MAC address for all the tenants/vpcs, e.g., (VPC 1, Port 1) → MAC1
Release (port, MAC) association when a port is released
Guarantee uniqueness of MAC address in region scope. Each region has seperate Alcor controller.
Algorithm with random, but less conflict possibility
Reserved MAC address by IANA should be avoided [1]
Fall back mechanism if there is a conflict among MAC addresses
Allow onboarding a list of MAC ranges for a region
Delete a list of MAC ranges for a region.
APIs to activate and deactivate a MAC address. MAC address has 3 states such as "allocated & activated", "allocated & deactivated", and "free (= not allocated yet)"
Working together with other services including subnet manager and port manager.
Concurrency control mechanism should be provided to process multiple concurrent MAC allocation requests.
REST APIs
API Snapshot
Verify MAC State
GET
/macs/{mac}, /v4/macs/{mac}
MAC state [sample]
Create MAC State
POST
/macs, /v4/macs
MAC state [sample]
Activate MAC Address
PUT
/macs/{mac}, /v4/macs/{mac}
MAC state [sample]
Deactivate MAC Address
PUT
/macs/{mac}, /v4/macs/{mac}
MAC state [sample]
Release MAC State
DELETE
/macs/{mac}, /v4/macs/{mac}
MAC state [sample]
Get MAC Range
GET
/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}
MAC range state [sample]
List MAC Ranges
GET
/macs/ranges, /v4/macs/ranges
MAC range states [sample]
Create MAC Range
POST
/macs/ranges, /v4/macs/ranges
MAC range state [sample]
Update MAC Range
PUT
/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}
MAC range state [sample]
Delete MAC Range
DELETE
/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}
deleted MAC range id [sample]
Verify/Get MAC State
GET
/macs/{mac}, /v4/macs/{mac}
MAC state [sample]
Create MAC State
POST
/macs, /v4/macs
MAC state [sample]
Create MAC State From Range
POST
/macs/range/{rangeid}, /v4/macs/range/{rangeid}
MAC state [sample]
Activate MAC Address
PUT
/macs/{mac}, /v4/macs/{mac}
MAC state [MacState_Put1]
Deactivate MAC Address
PUT
/macs/{mac}, /v4/macs/{mac}
MAC state [sample]
Release MAC State
DELETE
/macs/{mac}, /v4/macs/{mac}
MAC state [sample]
Get MAC Range
GET
/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}
MAC range state [sample]
List MAC Ranges
GET
/macs/ranges, /v4/macs/ranges
MAC range states [sample]
Create MAC Range
POST
/macs/ranges, /v4/macs/ranges
MAC range state [sample]
Update MAC Range
PUT
/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}
MAC range state [sample]
Delete MAC Range
DELETE
/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}
deleted MAC range id [sample]
API Specification
Method:
GETRequest:
/macs/{mac}, /v4/macs/{mac}Request Parameter:
@PathVariable String macResponse: mac state
Normal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/macs/00-14-2A-3B-47-37 Response: { "mac_state": { "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50", "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038", "port_id": "f37810eb-7f83-45fa-a4d4-1b31e75399df", "mac": "00-14-2A-3B-47-37", “state” “Active” } }(2) Allocate MAC Address for Port(2) Allocate MAC AddressMethod:
POSTRequest:
"/macs", "/v4/macs"Request Parameter:
Response:
mac stateNormal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example
Request: http://127.0.0.1:8080/macs Body: { "mac_state": { "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50", "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038", "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df" "mac": “null”, “state”: “null” } } Response: { "mac_state": { "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50", "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038", "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df", "mac": "00-14-2A-3B-47-37", “state” “Active” } }(3) Allocate MAC Address From MAC Range
Method:
POSTRequest:
"/macs/range/{rangeid}", "/v4/macs/{rangeid}"Request Parameter:
@PathVariable String rangeidResponse:
mac stateNormal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example
Request: http://127.0.0.1:8080/macs Body: { "mac_state": { "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50", "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038", "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df" "mac": “null”, “state”: “null” } } Response: { "mac_state": { "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50", "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038", "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df", "mac": "00-14-2A-3B-47-37", “state” “Active” } }(3) Activate MAC Address(4) Activate MAC AddressMethod:
PUTRequest:
/macs/{mac}", "/v4/macs/{mac}Request Parameter:
@PathVariable String macResponse:
mac stateNormal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/macs/00-14-2A-3B-47-37 Body: { "mac_state": { "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50", "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038", "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df" "mac": “00-14-2A-3B-47-37”, “state”: “Active” } } Response: { "mac_state": { "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50", "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038", "port_id": "f37810eb-7f83-45fa-a4d4-1b31e75399df", "mac": "00-14-2A-3B-47-37", “state”: “Active” } }(4) Deactivate MAC Address(5) Deactivate MAC AddressMethod:
PUTRequest:
/macs/{mac}", "/v4/macs/{mac}Request Parameter:
@PathVariable String macResponse:
mac stateNormal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/macs/00-14-2A-3B-47-37 Body: { "mac_state": { "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50", "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038", "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df" "mac": “00-14-2A-3B-47-37”, “state”: “Inactive” } } Response: { "mac_state": { "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50", "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038", "port_id": "f37810eb-7f83-45fa-a4d4-1b31e75399df", "mac": "00-14-2A-3B-47-37", “state”: “Inactive” } }(5) Delete/Release MAC State By MAC Address(6) Delete/Release MAC State By MAC AddressMethod:
DELETERequest:
/macs/{mac}", "/v4/macs/{mac}Request Parameter:
@PathVariable String macResponse: deleted MAC address
Normal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/mac/{00-14-2A-3B-47-37} Response: { "mac": "00-14-2A-3B-47-37" }(6) Get MAC Range by MAC Range Id(7) Get MAC Range by MAC Range IdMethod:
GETRequest:
/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}Request Parameter:
@PathVariable String rangeidResponse: mac range state
Normal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/macs/ranges/range0 Response: { "mac_range": { “range_id": "range0”, "from": “00-14-2A-00-00-00”, "to": “00-14-2A-FF-FF-FF”, "state": "Active" } }(7) List All MAC Ranges(8) List All MAC RangesMethod:
GETRequest:
/macs/ranges, /v4/macs/rangesRequest Parameter:
Response: mac range state list
Normal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/macs/ranges Response: {"mac_ranges": [ {"range_id":"range0", "from":"00-AA-BB-AA-AA-00", "to":"00-AA-BB-BB-BB-00", "state":"Inactive"}, {"range_id":"range1", "from":"00-AA-BB-00-00-00", "to":"00-AA-BB-FF-FF-FF", "state":"Active"} ]}(8) Onboard MAC Range(9) Onboard MAC RangeMethod:
POSTRequest:
/macs/rangesRequest Parameter:
Response:
mac range stateNormal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example
Request: http://127.0.0.1:8080/macs/ranges Body: { "mac_range": { “range_id": "range1”, "from": “00-14-2A-3B-47-00”, "to": “00-14-2A-3B-47-FF”, "state": "Active" } } Response: { “mac_range”: { “id": "range1”, "from": “00-14-2A-3B-47-00”, "to": “00-14-2A-3B-47-FF”, "state": "Active" } }(9) Update MAC Range(10) Update MAC RangeMethod:
PUTRequest:
/macs/ranges/rangeidRequest Parameter:
Response:
mac range stateNormal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example
Request: http://127.0.0.1:8080/macs/ranges/range1 Body: { "mac_range": { “range_id": "range1”, "from": “00-14-2A-3B-47-11”, "to": “00-14-2A-3B-47-FF”, "state": "Inactive" } } Response: { “mac_range”: { “range_id": "range1”, "from": “00-14-2A-3B-47-11”, "to": “00-14-2A-3B-47-FF”, "state": "Inactive" } }(10) Remove MAC Range by Range Id(11) Remove MAC Range by Range IdMethod:
DELETERequest:
/macs/ranges/range1Request Parameter:
Response:
mac range idNormal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/macs/ranges/range1 Response: { "range_id": "range1" }Database Data Schema
MAC Address State
One MAC address falls into one of three states:
Active
MAC address is allocated to a port AND it is in use. This is default.
Inactive
MAC address is allocated to a port AND it is NOT in use.
Free
MAC address is NOT allocated to a port yet.
MAC Range
MAC State
Figure 1. MAC data schema and management
Concurrency Handling
To handle concurrent MAC allocation requests, the following three things should be considered:
(1) Performance
MAC allocation time per request should be reasonable even if there are multiple simultaneous requests. For this purpose, this system performs MAC creation and MAC allocation processes in parallel using MAC pool as shown in [Figure 1]. In other words, it creates MAC addresses and stores them in MAC pool in advance. When there are MAC allocation requests, MAC pool provides pre-generated MAC addresses. During request processing, MAC creation keeps generating new MAC addresses.
(2) Uniqueness
Allocated MAC addresses should be unique in spite of concurrent MAC allocation requests. MAC manager achieves this goal by checking in two ways. First, it checks MAC address uniqueness when it creates a new MAC address. Second, it confirms the uniqueness when it allocates a MAC to a port. Additionally, MAC allocation is stored in a hash table whose key is MAC address. Therefore, the system can detect duplicated MAC addresses.
(3) Fallback
In case the uniqueness check for MAC address fails, it allocates new MAC addresses to deactivated ports first. If ports are active, it reallocates new MAC addresses to them until the conflict is resolved.
Comparison with OpenStack Neutron
Neutron creates a random MAC address when it creates port. There are 3 options to create a MAC address.
(1) Non DVR
When network is configured as non-distributed virtual router by default, Neutron uses 6 octets of base MAC address for VIFs. The first 3 octets of the base MAC address will remain unchanged. If the 4th octet is not 00, it will also be used. The other octets will be randomly generated [2].
(2) Distributed virtual router (enable_distributed_routing=True)
Every unique DVR those are instantiated by Neutron uses unique base MAC address, which is called ‘dvr_base_mac’. Similar to non DVR, the first 3 octets are unchanged. If the 4th octet is not 00, it is unchanged. The other octets are randomly generated. The ‘dvr_base_mac’ must be different from ‘base_mac’ to avoid mixing them up with MACs allocated for tenant ports [2].
(3) Static MAC address assigned by admin
When it creates new port, admin can assign MAC address along with other port information using the following option. Only admin can use this option because updating MAC can cause IP address loss [3].
References
[1] https://www.iana.org/assignments/ethernet-numbers/ethernet-numbers.xhtml
[2] https://wiki.openstack.org/wiki/Neutron/DVR_L2_Agent#get_dvr_mac_address_by_host_.28hostname.29
[3] https://specs.openstack.org/openstack/neutron-specs/specs/kilo/mac_addr_update.html
Node Manager - Meta Data Management
Overview
The node manager is a control plane component responsible of managing and configurating physical servers/nodes/hosts inside a data center. It maintains the detailed configuration for each node and collect its health state continuously in a configurable interval. The node health states could be used collectively in many user scenarios, for example, determine if a deployment of network resource (for example, port) is successful, or if a load balancing backend is reachable.
User Cases
Port manager talks to node manager to pull node details from a node id (UUID).
Service Requirements
Basic Requirements
Manage mapping from node id to node info (including IP, mac, veth)
Populating the node mapping in the controller launch time (through a configuration file) during new region or data center buildout
Expose a CURD API to register, update, or delete a node.
Define a health model, data schema and APIs for node health
Working together with other services including port manager.
Require HA and high scalability as 100,000+ nodes could upload health data continuously.
Advanced Requirements
Node info extensible to more node resources including FPGA etc.
Fall back mechanism when a node state is stall (e.g. proactively pull instead of waiting for agent to push)
REST APIs
API Snapshot
Get Node Info.
GET
/nodes/{nodeid}, /v4/nodes/{nodeid}
Node Information [sample]
List All Nodes
GET
/nodes, /v4//nodes
All Nodes [sample]
Create Node
POST
/nodes, /v4/nodes
Node Information [sample]
Create Nodes Bulk
POST
/nodes/upload, /v4/nodes/upload
Tottal Number of Created Nodes [sample]
Update Node
PUT
/nodes/{nodeid}, /v4/nodes/{nodeid}
Node Information [sample]
Delete Node
DELETE
/nodes/{nodeid}, /v4/nodes/{nodeid}
Node Id [NodeInfo_Delete]
API Specification
(1) Get Node Info By Node Id
Method:
GETRequest:
/nodes/{nodeid}, /v4/nodes/{nodeid}Request Parameter:
@PathVariable String nodeidResponse: node information
Normal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/nodes/f37810eb-7f83-45fa-a4d4-1b31e75399df Response: { "node": { "node_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df", "name": "node1", "ip": "10.0.0.2", "mac": "00-14-2A-3B-47-37", "veth": "veth0", "GRPC": "9090", "status": "running” } }(2) Get All Nodes
Method:
GETRequest:
/nodes/", "/v4/nodes/Request Parameter:
Response:
list of all nodesNormal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/nodes/f37810eb-7f83-45fa-a4d4-1b31e75399df Response: {[{ "node": { "node_id": "f37810eb-7f83-45fa-a4d4-1b31e75399df", "name": "node1", "ip": "10.0.0.3", "mac": "00-14-2A-3B-47-37", "veth": "veth1", "GRPC": "9090", "status": “stopped” } }, {[{ "node": { "node_id": " c57510eb-bf73-45fa-b2d6-3ccce75defee", "name": "node2", "ip": "10.0.0.5", "mac": "00-14-2A-3B-AC-AC", "veth": "veth5", "GRPC": "9090", "status": “stopped” } }] }(3) Create Node
Method:
POSTRequest:
"/nodes", "/v4/nodes"Request Parameter:
Response:
node informationNormal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example
Request: http://127.0.0.1:8080/macs Body: { "node": { "node_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df", "name": "node1", "ip": "10.0.0.2", "mac": "00-14-2A-3B-47-37", "veth": "veth0", "GRPC": "9090", "status": “” } } Response: { "node": { "node_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df", "name": "node1", "ip": "10.0.0.2", "mac": "00-14-2A-3B-47-37", "veth": "veth0", "GRPC": "9090", "status": “stopped” } }(4) Create Nodes From File Upload
Method:
POSTRequest:
"/nodes/upload", "/v4/nodes/upload"Request Parameter:
@RequestParam MultipartFile fileResponse:
total number of created nodesNormal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example
Request: http://127.0.0.1:8080/nodes/upload RequestParameters: { [file="@/mnt/alcor/config/machine.json"] } Response: { "total nodes": 100 }(5) Update Node By Node Id
Method:
PUTRequest:
/nodes/{nodeid}", "/v4/nodes/{nodeid}Request Parameter:
@PathVariable String nodeidResponse:
updated node informationNormal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/macs/00-14-2A-3B-47-37 Body: http://127.0.0.1:8080/nodes/f37810eb-7f83-45fa-a4d4-1b31e75399df Body: { "node": { "node_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df", "name": "node1", "ip": "10.0.0.3", "mac": "00-14-2A-3B-47-37", "veth": "veth1", "GRPC": "9090", "status": “stopped” } } Response: { "node": { "node_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df", "name": "node1", "ip": "10.0.0.3", "mac": "00-14-2A-3B-47-37", "veth": "veth1", "GRPC": "9090", "status": “stopped” } }(6) Delete Node
Method:
DELETERequest:
/nodes/{nodeid}", "/v4/nodes/{nodeid}Request Parameter:
@PathVariable String nodeidResponse:
deleted node idNormal response codes: 200
Error response codes: 400, 412, 500
Example
Request: http://127.0.0.1:8080/nodes/f37810eb-7f83-45fa-a4d4-1b31e75399df Response: { "node": " f37810eb-7f83-45fa-a4d4-1b31e75399df" }Database Data Schema
Node Info
Node info. contains meta data information of nodes.
node_id
name
ip
mac
veth
gRPC_server_port
f37810eb
node1
10.0.0.2
AA-BB-CC-11-22-33
veth0
9090
System Structure
Figure 1. System structure of node meta data manager
References
[1] #151
[2] https://kubernetes.io/docs/concepts/architecture/nodes/#api-object
[3] http://manpages.ubuntu.com/manpages/xenial/man1/top.1.html