-
Notifications
You must be signed in to change notification settings - Fork 59
Infrastructure Specification
The purpose of the Team project is to provide a mechanism to team multiple NICs (ports) into one logical one (teamdev) at L2 layer. The process is called "channel bonding", "Ethernet bonding", "channel teaming", "link aggregation", etc. This is already implemented in the Linux kernel by the bonding driver. The main thing to realize is that the Team project is not trying to replicate or mimic the bonding driver. What it does is solve the same problem using a different approach. Therefore, for example, the way Team is configured differs dramatically from the way bonding is. Team has many advantages over Bonding. These will be described later in this text.
An example setup might look like this. Team softdev Linux driver instance is netdev called team0. It has two ports: eth0 and eth1. team0 has an assigned IP address X. Note that eth0 and eth1 do not have an IP assigned. It would not make sense because as a part of team0, the Team softdev Linux driver collects all received traffic and "changes" it so that it appears to be coming from team0. More info about this later in the text.
The Team project consists of several parts. Some of them are essential, some of them are optional.
A very important part is the Team kernel driver (referred to as "driver" in the text). The driver is part of the Linux Kernel (since version 3.3). Although it is designed to be very slim, it has the key role in the project. Its motivation is to implement all things "which should be done fast", mainly transmit and receive packet (skbs) flows. The motto is: "If something can be done in userspace, do it in userspace". To avoid confusion I want to emphasize the fact that skbs do not leave kernel space, they are not copied to userspace. It would make no sense to do so and also it would dramatically slow down the skb processing. Team softdev Linux driver also provides Team Netlink API. This interface is implemented using Generic Netlink and provides an ability for userspace to set up or change the driver behaviour and to get events (like port state changes, etc) from the driver. The driver on its own has no logic. It does not decide the behaviour on its own. That is the job of the userpace application.
Another very important part is the Libteam lib (referred to as "lib" in the text). This lib is a part of the Libteam project. This lib uses libnl and its primary purpose is to do userspace wrapping of Team Netlink communication. The user does not have to know anything about the Netlink API. The user only calls the provided function and the lib takes care of constructing and parsing Netlink messages. All messages coming from the driver are cached so when the user requests some data, there is no need to send and receive any messages.
Also, the lib wraps-up RT Netlink messages (such as newlink, dellink etc.). These are exported to the user to provide a possibility to create or delete a Team softdev Linux driver instance, to add or remove a port etc., to get or set hardware addresses of network interfaces, etc.
Note that it is not absolutely necessary to use this lib. An application can do the implementation of Team Netlink API client side on its own. But that is not recommended. An application which wants to communicate with the Team softdev Linux driver should use this lib.
teamd stands for "Team daemon". teamd is an application which is a part of the Libteam project and it uses Libteam lib. It runs as a daemon and one instance of teamd works with one instance of Team softdev Linux driver (one team netdev, for example team0). The purpose is to implement various logic of Team's behaviour. From the most basic ones such as round-robin, to more complex such as active-backup and load-balancing. The logic is implemented in teamd parts called "runners". More about runners can be found later in the text.
teamd tries to be universal to provide various kinds of behaviour logic (which can be extended by custom runners). Note that teamd is an optional part. Users can write their own application to use the lib and implement the logic.
To download Libteam repository (containing Libteam lib and teamd) do:
$ git clone git://github.com/jpirko/libteam.git
This section describes Team Netlink API. It's probably better to know how driver and lib communicate with each other before they are described. There are two types of messages. Port list and option list.
Port list message take place only for the driver to lib direction (data are read-only). Schema of the message looks like this:
-
port item
- interface index (unsigned 32bit value)
- changed flag (boolean) - tells if any of this port status value changed
- removed flag (boolean) - tells if this port was removed so userspace can react to that properly
- link up (boolean) - link status taken directly from ethtool
- speed (unsigned 32bit value) - speed in Mbps taken directly from ethtool
- duplex (unsigned 8bit value (0 or 1)) - duplex taken directly from ethtool
-
port item
- ....
-
port item
- ....
Option list message takes place in both directions. In the driver to lib direction, driver data options are exposed and transferred into userspace. In the lib to driver direction the message is used to inform the driver what option values should be changed and how. Schema of the message looks like this:
-
option item
- name (string) - name of the option
- changed flag (boolean) - tells if any of this option value changed
- removed flag (boolean) - tells if this option was removed so userspace can react to that properly
- type (unsigned 8bit value) - determines the type of the following data field. Can be either unsigned 32bit value, string, boolean or binary data.
- data (dynamic type) - actual option value of a type determined by type field.
- port interface index (optional, unsigned 32bit value) - in case the option is "per-port" this field is set to appropriate port interface index. More about per-port option will be stated later in the text.
- array index (optional, unsigned 32bit value) - in case the option is "array" this field is set to appropriate array index. More about array option will be stated later in the text.
-
option item
- ....
-
option item
- ....
There are the following cases in which messages are sent:
- lib requests the port list - In this case the message contains all present ports.
- lib requests the option list - In this case the message contains all present options.
- lib requests option value change - Message contains one or more option items for those options whose value should be changed in the driver.
- driver port status change event - Driver is notified by net core that its port device changed status. In this case the message contains only the port which changed or was removed.
- driver option value change event - Happens either asynchronously or as a reaction to point 3. - In this case the message contains only the options whose value changed or the options which were removed.
In case of point 4. and 5. Netlink multicast facility is used so multiple userspace apps can get the message. Applications can therefore monitor possible changes other applications are performing.
teamnl uses libteam and provides a wrapper for Team device Netlink communication.
For a detailed list and descriptions of teamnl command line parameters, please see the appropriate manpage (man 8 teamnl).
The driver allows creating its instance using RTNL. So one can easily create or delete a team instance using the ip utility like this:
# ip link add name team0 type team
# ip link delete team0
A random hardware address is generated for a newly created device unless the user explicitly specifies it.
Also, the ip utility can be used to add or remove ports like this:
# ip link set eth0 master team0
# ip link set eth0 nomasterThe Team softdev Linux driver uses the netdevice notification facility to catch events happening on ports in order to be able to perform necessary actions in reaction to that. Events such as port link change, port disappearance and so on. When an instance of Team is created (call it for example team0) it looks like any other network device. So one can perform any action on it as if it was an ordinary network device (assign an IP addresses, use it in iptables rules, etc.). The difference is that it is not by itself able to receive or transmit skbs. It uses other network devices (typically ones representing real NICs) to do that.
Notice what happens when net core asks a Team softdev Linux driver instance (called teamdev in the text) to transmit a skb. In an ordinary driver for a real NIC, skb is pushed into HW and HW does the transmit. In the Team softdev Linux driver, the chosen port-selection algorithm (see Team modes later in the text) selects one port best suited for the skb transmit at a time. This port netdevice is then asked to transmit skb. This process is transparent for net core and for port netdevice as well.
On the receive side, the Team softdev Linux driver uses rx_handler hook in net core rx procedure to intercept incoming skbs. The same hook is used by bonding, bridging, macvlan and openvswitch. Using this hook, the Team softdev Linux driver changes the skb so it looks as if it came from the teamdev (the one who owns the originating port netdevice). Net core code executed after that thinks that the skb actually came from the teamdev.
The pressure is put on achieving the best possible performance. Therefore fast path (skb transmit and receive paths) are lockless (only using RCU). Also, data are put into memory to achieve maximum locality and minimum pointer dereferences.
An option infrastructure is necessary for modular and transparent work with driver options. It separates the code so the actual option holder does not have to know anything about Netlink and Team Netlink core does not have to know anything about option values. The main part is option object. This object contains the name of the option, its type, the function for getting an option value (getter), the option for setting an option value (setter), etc. This object is registered to option facility. The facility then creates option object instance with a reference to the originating option object. This instance is after that being used by Team Netlink core. In Team Netlink messages, each option instance has own option item.
When a Netlink message of option list type with a request for an option value to be set arrives, Netlink core parses the message and searches through all existing instances. The desired instance is found and Netlink core calls the setter function of option object to change the option value.
Also when option list is requested by lib, Netlink core composes the message by iterating over all option instances calling their option object getter functions and using those values.
There are two specialties which extend options. The first one is the per-port option. In this case a per-port flag needs to be set for the option object. During the registration, the option facility creates multiple option instances, one per each port in teamdev. These options are used for example for enabling and disabling ports (e.g. enable option).
The second one is the array option. This is similar to the per-port option, only the number of instances is set to option object before registration. Each option instance is indexed as if it was in an array.
Modes are implemented as separate modules. Team core exposes mode string option by which userspace application can set desired mode by its name. Modes implement handlers which are called from Team core. These handlers defines the behaviour:
- init - Handler called after mode is selected. Typical place for allocating memory and registering mode-specific options.
- exit - Called after mode is deselected
- receive - Called from receive hook. This handler allows mode to look at incoming skbs and to change them eventually.
- transmit - Called from Team core transmit function. This is the place where transmit port selection takes place.
- port_enter - Called whenever a port is added into teamdev.
- port_leave - Called whenever a port is removed from teamdev.
- port_change_mac - Called when hardware address change is detected.
There are five modes defined:
- broadcast - Basic mode in which all packets are sent via all available ports.
- roundrobin - Basic mode with very simple transmit port-selecting algorithm based on looping around the port list. This is the only mode able to run on its own without userspace interactions.
- random - Basic mode similar to the previous one. Transmit port is selected randomly for each outgoing skb.
- activebackup - In this mode, only one port is active at a time and able to perform transmit and receive of skb. The rest of the ports are backup ports. Mode exposes activeport option through which userspace application can specify the active port.
- loadbalance - A more complex mode used for example for LACP and userspace controlled transmit and receive load balancing. LACP protocol is part of the 802.3ad standard and is very common for smart switches. Hashing is used to identify similar skbs. Hash length is 8 bits so there are 256 variants of skbs (multiple different skbs can fall into one hash). The hash computation mechanism is BPF (Berkeley Packet Filter) based. This mode exposes the following options:
- bpf_hash_func - A binary option containing the BPF function doing the computation of the hash from skb. Userspace can assemble and setup a function which computes a hash from desired parts of skb. For example source and destination IP address, hardware addresses, etc.
- lb_hash_stats and lb_port_stats - read-only array options that expose internal skb transmit and receive counters for each of 256 hashes and each port. A userspace application can use these counters to distinguish the load of ports and possibly rebalance hashes to different ports.
- lb_stats_refresh_interval - this option is used to tell the driver how often it should refresh previously described stats options. If stats of any port or hash changes, it is communicated into userspace (similar to a port status change event).
- lb_tx_hash_to_port_mapping - an array option used for hash-to-port mapping. Allows a userspace application to tell the driver which port should be used for transmitting skb which belongs to certain hash.
- lb_tx_method - a string option of two possible values. The hash value tells the driver to use computed hash directly to get transmit port. The hash_to_port_mapping value turns on hash-to-port mapping and takes previous array option values into account in the process of selecting a port to transmit the skb.
If users need another mode, because the modes listed above do not cover a special cases that they need, then they can easily implement their own mode. The mode API is very well defined and can be easily enhanced to suit other needs.
This is the major part of the project. If possible, it is very much preferred to implement features in teamd rather than in Team softdev Linux driver. teamd can be looked at as the "puppeteer" whereas teamdev is its "puppet".
For detailed list and descriptions of teamd command line parameters, please see the appropriate manpage (man 8 teamd).
teamd takes care of teamdev (Team softdev Linux driver instance) creation and deletion. teamdev is created during teamd start and it's destroyed before teamd terminates. The name of teamdev (for example team0) is specified in the config file. More about config will be said later in the text.
The core of teamd take care of the following:
- Config parsing.
- Daemon forking and related things like PID creation, signals etc. This might not be needed for systems running systemd but teamd should also be able to run on other systems as well.
- Team softdev Linux driver instance creation and deletion.
- Libteam lib communication initialization.
- Providing main-loop infrastructure for registering file descriptor or time oriented watchers and handlers.
- Initialization of event machine.
- Port addition and removal event handling.
- Initialization of link-watchers.
- Initialization of runner.
- Initialization of D-Bus interface.
- Adding ports to teamdev.
teamd is designed to be a single-process, single-thread application. Currently, there is really no need to do it another way. This results in much simpler code. All the runtime work is done in main-loop. So whatever runner or link-watcher any other component needs to do on their own (like periodic work or socket data receive) they must use main-loop infrastructure to do so.
Link-watchers serve for link monitoring purposes. Depending on the particular type they use different methods to find out if a port is capable of data transfers. In other words "if the link is up".
There are currently the following types:
- ethtool - Uses Libteam lib to get port ethtool state changes.
- arp_ping - ARP requests are sent through a port. If an ARP reply is received, the link is considered to be up. Target IP address, interval and other options can be setup in teamd config.
- nsna_ping - Similar to the previous, only it uses the IPv6 Neighbour Solicitation and Neighbour Advertisement mechanism. This is an alternative to arp_ping and becomes handy in pure-IPv6 environments.
Either one link-watch is set for all ports or each port can have its own link-watch. This allows users more complex setups. For example, port eth0 can use ethtool link-watch and eth1 can use arp_ping.
User can also specify multiple link-watchers used at the same time. In that case, link is up if any of the link-watchers reports the link up.
Runners determine the behaviour of the Team device. They operate using the kernel Team mode they want. Runners watch for port link state changes (propagated by the selected link-watch) and react to that. They may implement other functionality as well.
The following runners can be used (Team softdev Linux driver modes are stated in parenthesis):
- broadcast (broadcast) - Does almost nothing because it only says to put teamdev into broadcast mode.
- roundrobin (roundrobin) - Does almost nothing because it only says to put teamdev into roundrobin mode.
- random (random) - Does almost nothing because it only says to put teamdev into roundrobin mode.
- activebackup (broadcast) - Watches for link changes and selects active port to be used for data transfers. Each port can be configured to have its priority and to be "sticky" or not. Being "sticky" here means to not be de-activated even if a port with a better priority gains its link.
- loadbalance (loadbalance) - To do passive load balancing, runner only setups BPF hash function which will determine port for skb transmit. To do active load balancing, runner moves hashes among available ports trying to reach a perfect balance. lb_hash_stats array option is used to get statistics. lb_tx_hash_to_port_mapping array option is used to map hashes to TX port.
- lacp (loadbalance) - Implements 802.3ad LACP protocol. Can use TX port selection which is used either in hash or loadbalance runner.
Teamd provides possibility to be controlled via control API. This is implemented by D-Bus and Unix socket.
Unix domain socket API is enabled upon teamd start by default. If "-D" option is passed to teamd, D-Bus API is acrivated and D-Bus service org.libteam.teamd.[teamdevname] is created.
Following methods are supported:
- ConfigDump() - Returns the whole teamd JSON config.
- ConfigDumpActual() - Returns the whole teamd JSON config. This time only present ports are in the JSON document, non-present ones are filtered out.
- StateDump() - Returns the state of teamd and all its subparts in JSON.
- PortAdd(String port_devname) - Adds port to teamdev.
- PortRemove(String port_devname) - Removes port from teamdev.
- PortConfigUpdate(String port_devname, String port_config) - Updates config for a given port. Port config is sub-config of teamd JSON config for one port.
- PortConfigDump(String port_devname) - Dumps config for a given port. Port config is sub-config of teamd JSON config for one port.
- StateItemValueGet(String state_item_path) - Finds state item in state tree and returns its value.
- StateItemValueSet(String state_item_path, String value) - Finds state item in state tree and sets its value by value parameter.
There is a plan to extend this interface with more methods in the future. Also there is an idea to extend the API so it would be able to implement a runner as an outside application communicating with teamd over control API.
teamdctl provides a wrapper for control API using it to monitor and control teamd runtime. It wraps the API methods 1:1.
For a detailed list and descriptions of teamdctl command line parameters, please see the appropriate manpage (man 8 teamdctl).
teamd is configured using JSON config string. This can be passed to teamd either on the command line or in a file. JSON format was chosen because it's easy to specify (and parse) hierarchic configurations using it.
Example teamd config (teamd1.conf):
{
"device": "team0",
"hwaddr": "10:22:33:44:55:66",
"runner": {"name": "activebackup"},
"link_watch": {
"name": "nsna_ping",
"interval": 200,
"missed_max": 15,
"target_host": "fe80::210:18ff:feaa:bbcc"
},
"ports": {
"eth0": {
"prio": -10,
"sticky": true
},
"eth1": {
"prio": 100,
"link_watch": {"name": "ethtool"}
},
"eth2": {
"link_watch": {
"name": "arp_ping",
"interval": 100,
"missed_max": 30,
"source_host": "192.168.23.2",
"target_host": "192.168.23.1"
}
},
"eth3": {}
}
}
The config is pretty much self-explanatory. Only link_watch sections might look confusing. So in this example, the default link-watch is nsna_ping, setup to send NAs (Neighbour Advertisements) every 200 milliseconds. Maximum number of missed replies is 15. If more replies are missed, link is then considered as down. Ports eth1 and eth2 specify their own link-watchers.
Another example of teamd config (teamd2.conf):
{
"device": "team0",
"runner": {
"name": "lacp",
"active": true,
"fast_rate": true,
"tx_hash": ["eth", "ipv4", "ipv6"]
},
"link_watch": {"name": "ethtool"},
"ports": {"eth1": {}, "eth2": {}}
}
In this example the tx_hash section is worth mentioning. It specifies what parts of skb should be used to compute the hash.
For a detailed list and descriptions of available config options, please see the appropriate manpage (man 8 teamd.conf).
# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 52:54:00:b2:a7:f1 brd ff:ff:ff:ff:ff:ff
3: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:07:e9:11:22:33 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 52:54:00:3d:c7:6d brd ff:ff:ff:ff:ff:ff
5: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 52:54:00:73:15:c2 brd ff:ff:ff:ff:ff:ff
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:b2:a7:f1 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.182/24 brd 192.168.122.255 scope global eth0
inet6 fe80::5054:ff:feb2:a7f1/64 scope link
valid_lft forever preferred_lft forever
3: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:07:e9:11:22:33 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:3d:c7:6d brd ff:ff:ff:ff:ff:ff
5: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:73:15:c2 brd ff:ff:ff:ff:ff:ff
# teamd -f teamd2.conf -d
# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 52:54:00:b2:a7:f1 brd ff:ff:ff:ff:ff:ff
3: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:07:e9:11:22:33 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP mode DEFAULT qlen 1000
link/ether fa:29:06:7a:81:b8 brd ff:ff:ff:ff:ff:ff
5: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP mode DEFAULT qlen 1000
link/ether fa:29:06:7a:81:b8 brd ff:ff:ff:ff:ff:ff
6: team0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT
link/ether fa:29:06:7a:81:b8 brd ff:ff:ff:ff:ff:ff
# tail /var/log/messages
May 24 10:56:02 test1 kernel: [ 36.351762] team0: Port device eth1 added
May 24 10:56:02 test1 kernel: [ 36.357303] 8139cp 0000:00:08.0: eth2: link up, 100Mbps, full-duplex, lpa 0x05E1
May 24 10:56:02 test1 kernel: [ 36.359975] team0: Port device eth2 added
May 24 10:56:02 test1 teamd[883]: 0.1 sucessfully started.
May 24 10:56:03 test1 teamd[883]: eth1: Using link_watch "ethtool".
May 24 10:56:03 test1 teamd[883]: eth1: Changed port state: "disabled" -> "expired"
May 24 10:56:03 test1 teamd[883]: eth2: Using link_watch "ethtool".
May 24 10:56:03 test1 teamd[883]: eth2: Changed port state: "disabled" -> "expired"
May 24 10:56:03 test1 teamd[883]: eth1: Changed port state: "expired" -> "current"
May 24 10:56:03 test1 teamd[883]: eth2: Changed port state: "expired" -> "current"
# ip addr add 192.168.24.2/24 dev team0
# ping 192.168.24.1 -c 10 -i0.1
PING 192.168.24.1 (192.168.24.1) 56(84) bytes of data.
64 bytes from 192.168.24.1: icmp_req=1 ttl=64 time=0.511 ms
64 bytes from 192.168.24.1: icmp_req=2 ttl=64 time=0.862 ms
64 bytes from 192.168.24.1: icmp_req=3 ttl=64 time=0.927 ms
64 bytes from 192.168.24.1: icmp_req=4 ttl=64 time=0.917 ms
64 bytes from 192.168.24.1: icmp_req=5 ttl=64 time=0.966 ms
64 bytes from 192.168.24.1: icmp_req=6 ttl=64 time=0.829 ms
64 bytes from 192.168.24.1: icmp_req=7 ttl=64 time=0.916 ms
64 bytes from 192.168.24.1: icmp_req=8 ttl=64 time=0.861 ms
64 bytes from 192.168.24.1: icmp_req=9 ttl=64 time=0.866 ms
64 bytes from 192.168.24.1: icmp_req=10 ttl=64 time=0.884 ms
--- 192.168.24.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 910ms
rtt min/avg/max/mdev = 0.511/0.853/0.966/0.126 ms
# teamd -f teamd2.conf -k
# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 52:54:00:b2:a7:f1 brd ff:ff:ff:ff:ff:ff
3: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether 00:07:e9:11:22:33 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 1000
link/ether 52:54:00:3d:c7:6d brd ff:ff:ff:ff:ff:ff
5: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 1000
link/ether 52:54:00:73:15:c2 brd ff:ff:ff:ff:ff:ff