Benchmarking perfSONAR hardware

This page list a few ways one can benchmark perfSONAR hardware to see how they perform under different testing load: high throughput, high packet rate and possibly others. While we don't want to list specific hardware recommendations, we provide here some commands that one can run to see how good a machine is behaving with regards to some specific tests. And then discuss what conclusions we can draw from numbers.

In time, we might provide scripts to automate this benchmarking process.

Testing conditions

The best is to have 2 perfSONAR devices connected back to back, so to take any network element out of the equation. Unless stated otherwise, the perfSONAR stack is the default one and the kernel tuning parameters are the one provided by the perfsonar-toolkit-sysctl package.

Packet rate

To test if the hardware can generate high packet rate, latency tests with high frequency is a useful benchmark (the -i parameter is giving the interval between packets, in seconds). pScheduler is using owping to perform those tests.

Commands examples

pscheduler task latency -p 1008 -i 0.00001 -c 100000
pscheduler task latency -p 1008 -i 0.00001 -c 300000
pscheduler task latency -p 1008 -i 0.00001 -c 600000

Verify the number of packets actually sent and the number of packets received (or lost), monitoring CPU and link usage at the time the test is running will also provide valuable information. Then, compare the number of packets with what was specified in the task run:

• if the same number of packets requested is being sent and received: all ok
• if some packets are not generated (the number of packets sent is smaller than the requested count -c):
  • if the CPU usage is high, then the hardware is probably limited by a CPU bottleneck,
  • if the CPU usage is low, then the hardware is probably limited by a NIC or a bus bottleneck,
• if some packets lost (and the network/cable is good and not saturated):
  • if the CPU usage is high, then the receiving CPU is the bottleneck,
  • if the CPU usage is low, then the receiving NIC or bus is the bottleneck.

Throughput

To test for CPU (core) bottleneck, a throughput test with high-speed network cards (100 Gbps at the moment) and an iperf3 run is usually all that it takes. iperf3 is single-threaded, so only very high-end CPU cores can generate enough packets to fill a 100 Gbps link.

To test for NIC/network bottleneck, run a throughput test with high-speed network cards and the multi-threaded iperf2 or ethr test. These tools will generate packets from different CPU cores and can then hand-over packets to the NIC at a higher rate.