Announcing the TSN Evaluation Framework for Linux OS

[edersondisouza]

Hi everyone,

While developing TSN features - or when using them - it's really common to measure things: Did a packet arrive on time? How long can a packet take to arrive? How long does it spend on the TX path? Or RX path?

We all have been facing these questions, so it's natural that we have some scripts here and there to try to answer those questions. Now I'd like to announce a project that we've been working lately, that aims to help collect common measurements. The TSN Evaluation Framework for Linux OS collects data regarding the latency a packet experience from being sent until is received on another machine, providing not only the end-to-end latency, but also a breakdown into TX/RX and across the Linux network stack (for
AF_PACKET sockets).

An interesting characteristic of TSN environment is that several steps need to be taken to set it up: tuning the network interface, enabling Qdiscs, setting synchronisation up, etc. The framework should help with all of that:
it provides a JSON configuration file that can be used to describe the environment to be set, and it take care of doing all of that.

After data collection, one needs to make sense of it: the framework also provides some analysis tools, that will compute overall statics of the experiments, provide charts to help visualise the data, as well as tools
to compare different runs - so one can check gains or losses of using a different Qdisc or configuration.

Some of the output that can be generated:

Receive Latency Statistics

+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
| Payload(bytes)   | TransmissionInterval(us)   |   Mean(us) |   Stdev(us) |   Min(us) |   Max(us) |   Range(us) |     CV |
+==================+============================+============+=============+===========+===========+=============+========+
| 48               | 125                        |     12.506 |       0.244 |     7.402 |    34.318 |      26.916 |  1.952 |
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
| 48               | 250                        |     12.563 |       0.216 |     7.707 |    28.448 |      20.741 |  1.719 |
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
| 48               | 500                        |     12.637 |       0.209 |     7.472 |    27.829 |      20.357 |  1.651 |
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
[clip]
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
| 1500             | 1000                       |     27.469 |       0.191 |    26.974 |    52.950 |      25.976 |  0.695 |
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
| ALL              | ALL                        |     17.650 |       5.819 |     7.402 |    57.029 |      49.627 | 32.971 |
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+

Some of the charts:

Currently, it supports AF_PACKET and AF_XDP evaluation - but AF_XDP support is limited to end-to-end analysis. It's also designed to be "NIC agnostic". It has been tested in two (i210 and stmmac), but it should be fairly easy
to add support for new ones.

The framework is structured as Python applications that perform the experiment phase (environment setup and data collection) and the analysis phase. We tried to make the experiment application leaner, so it can run on low power devices (although "perf" is necessary to perform the latency breakdown). The analysis part uses Pandas and Numpy to help process the data, but can run on a different machine.

Also, it's nice to note that the applications used to send and receive the experiment data can also be used outside of the framework for quick tests and experiments.

Future work includes breakdown of latency for AF_XDP evaluation, more detailed breakdown for different Qdiscs, live analysis support and more platforms (patches welcome =D). Naturally, we're also open to other ideas about how to improve it.

It's now temporarily hosted on my Github account (https://github.com/edersondisouza/linux-tsn-eval), so you can evaluate it.

As we believe this project can be interesting to the AVNU Github community, we'd like to contribute it here, where further development would take place. So I'd like to ask about your interest in hosting it under
https://github.com/AVnu, just like we do with libavtp and the TSN Documentation Project for Linux. If there's interest, I'd like to ask the maintainers to create a new repository under https://github.com/AVnu, to where I could move the project.

[tingyawang]

Hi everyone,

While developing TSN features - or when using them - it's really common to measure things: Did a packet arrive on time? How long can a packet take to arrive? How long does it spend on the TX path? Or RX path?

We all have been facing these questions, so it's natural that we have some scripts here and there to try to answer those questions. Now I'd like to announce a project that we've been working lately, that aims to help collect common measurements. The TSN Evaluation Framework for Linux OS collects data regarding the latency a packet experience from being sent until is received on another machine, providing not only the end-to-end latency, but also a breakdown into TX/RX and across the Linux network stack (for
AF_PACKET sockets).

An interesting characteristic of TSN environment is that several steps need to be taken to set it up: tuning the network interface, enabling Qdiscs, setting synchronisation up, etc. The framework should help with all of that:
it provides a JSON configuration file that can be used to describe the environment to be set, and it take care of doing all of that.

After data collection, one needs to make sense of it: the framework also provides some analysis tools, that will compute overall statics of the experiments, provide charts to help visualise the data, as well as tools
to compare different runs - so one can check gains or losses of using a different Qdisc or configuration.

Some of the output that can be generated:

Receive Latency Statistics

+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
| Payload(bytes)   | TransmissionInterval(us)   |   Mean(us) |   Stdev(us) |   Min(us) |   Max(us) |   Range(us) |     CV |
+==================+============================+============+=============+===========+===========+=============+========+
| 48               | 125                        |     12.506 |       0.244 |     7.402 |    34.318 |      26.916 |  1.952 |
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
| 48               | 250                        |     12.563 |       0.216 |     7.707 |    28.448 |      20.741 |  1.719 |
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
| 48               | 500                        |     12.637 |       0.209 |     7.472 |    27.829 |      20.357 |  1.651 |
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
[clip]
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
| 1500             | 1000                       |     27.469 |       0.191 |    26.974 |    52.950 |      25.976 |  0.695 |
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+
| ALL              | ALL                        |     17.650 |       5.819 |     7.402 |    57.029 |      49.627 | 32.971 |
+------------------+----------------------------+------------+-------------+-----------+-----------+-------------+--------+

Some of the charts:

Currently, it supports AF_PACKET and AF_XDP evaluation - but AF_XDP support is limited to end-to-end analysis. It's also designed to be "NIC agnostic". It has been tested in two (i210 and stmmac), but it should be fairly easy
to add support for new ones.

The framework is structured as Python applications that perform the experiment phase (environment setup and data collection) and the analysis phase. We tried to make the experiment application leaner, so it can run on low power devices (although "perf" is necessary to perform the latency breakdown). The analysis part uses Pandas and Numpy to help process the data, but can run on a different machine.

Also, it's nice to note that the applications used to send and receive the experiment data can also be used outside of the framework for quick tests and experiments.

Future work includes breakdown of latency for AF_XDP evaluation, more detailed breakdown for different Qdiscs, live analysis support and more platforms (patches welcome =D). Naturally, we're also open to other ideas about how to improve it.

It's now temporarily hosted on my Github account (https://github.com/edersondisouza/linux-tsn-eval), so you can evaluate it.

As we believe this project can be interesting to the AVNU Github community, we'd like to contribute it here, where further development would take place. So I'd like to ask about your interest in hosting it under
https://github.com/AVnu, just like we do with libavtp and the TSN Documentation Project for Linux. If there's interest, I'd like to ask the maintainers to create a new repository under https://github.com/AVnu, to where I could move the project.

Hello,
I am interested in this topic. I would like ask one question that it can be run ubuntun VM for simulation TSN / AVB environment without TSN/AVB-NIC support ?