Table of Contents
- Introduction
- Architecture
- Getting Started with the First Test
- General Tests and System Tuning
- Scalability vs. Latency
The Lightstreamer Load Test Toolkit (LLTT) has been developed to help performing load tests on Lightstreamer Server. It complements, rather than replacing, any third-party software that supports Lightstreamer. In particular, the LLTT is made up of an Adapter Simulator and a Client Simulator. This means it is quite useful in the preliminary phases of a project, when no Adapters and Clients have been developed yet, but it is necessary to do a capacity planning of the system based on different load scenarios. When the system has been fully developed (including Lightstreamer Adapters and Clients), it is possible to use other off-the-shelf products to load test the Server while targeting the real Adapters and simulating the real Clients.
This user's guide describes the architecture of the LLTT and provides the instructions on how to configure, deploy, and run it.
Binary executables and Java source code are provided for both the Adapter Simulator and the Client Simulator.
-
The binary executables are available in a zip file under the release section.
-
To compile the source code, a Maven pom.xml is available.
Session: a Lightstreamer session that simulates a user. Every session subscribes to one or more items and receives updates from Lightstreamer Server.
Client Simulator: a standalone application that creates a configured number of sessions in streaming mode.
Adapter Simulator: a Lightstreamer Data Adapter that generates items and updates based on its configuration.
There are several possibilities when choosing how many Client Simulators to use and where to deploy them. In the sections below we will go through some of them.
In the simplest case, there are two machines involved in the test: one machine hosting a Client Simulator and one machine hosting the Lightstreamer Server (whose process includes the Adapter Simulator). In this case, Client Simulator 1, running on Machine 2, will create K sessions. The value of K depends on the hardware characteristics of Machine 1 and the heaviness of the subscribed data.
If Machine 1 does not scale enough to simulate the required number of users, more machines can be added hosting more Client Simulators, as shown below.
The LLTT includes a facility to accurately measure and statistically analyze latencies for the pushed data.
Activating latency reporting on a Client Simulator makes it much less scalable. So it is a good practice to generate most of the load on a number of Client Simulators with latency reporting turned off. Then you can use a further Client Simulator with latency reporting turned on, simulating a much smaller number of clients. This will be a good sample of the global latency distribution.
For the latency reporting system to show reliable results the machine hosting the Lightstreamer Server and the machine hosting the Client Simulator with latency reporting on should have their clocks synchronized. You may use NTP to synchronize the clocks of those machines: http://www.ntp.org/ .
The Client Simulator with latency reporting on can also be deployed directly on the same machine of the Server (provided that the Client Simulator process does not result too much resource intensive) thus avoiding the complexity of clocks synchronization and removing any network latencies. This scenario leads to the architecture below:
M is usually a small number, to avoid stealing CPU cycles from the Lightstreamer Server, which is the subject under observation. If Client Simulator 4 is deployed on a dedicated machine, then M could be larger.
NOTE: If testing a “broadcasting” scenario (that is, a few items subscribed to by all clients) the delays may not be randomly distributed among the clients; in other words, some clients may always receive updates before other clients do, so some clients may experience latencies constantly lower than average, while other clients may experience latencies constantly higher. This must be taken into account when analyzing the latency statistics and when determining a suitable value for M.
When you need to simulate very high numbers of users (hundreds of thousands or millions) and each machine dedicated to Client Simulators needs to go beyond 64K TCP connections, you have to configure Lightstreamer Server to listen on multiple TCP ports and to set “serverPorts” parameter (in the XML configuration file of the Client Simulator) to the number of configured server ports.
For example if the Server listens on ports 80, 81 and 82 (NB ports numbers have to be adjacent), then you have to set “port” parameter to 80 (the base port) and “serverPorts” to 3 (because the server listens on three ports starting from port number 80) in the Client Simulator configuration file.
NOTE: In order to be able to open 50,000 or more client ports, you may need to increase the default port range of the underlying operating system, as explained below.
- Linux:
- Find the current port range: sysctl net.ipv4.ip_local_port_range
- Set a new port range: sudo sysctl -w net.ipv4.ip_local_port_range="1024 64000"
- You may need to edit “/etc/sysctl.conf” file to make changes to /proc filesystem permanently. For example, append the following to your “/etc/sysctl.conf” file: net.ipv4.ip_local_port_range = 1024 65535
-
Windows:
- Set the following registry key to
64000:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort
- Set the following registry key to
64000:
On Linux, the limit on the open files also has to be set higher than
the expected number of connections.
The launch script, “start_client.sh”, tries to raise the “soft”
limit up to the “hard” limit granted to the user. Check if you have
to raise the user's “hard” limit as well.
Let's set up the system for a simple test. As for the instructions below, we will refer to a simple deployment on two machines: Machine 1 hosting Lightstreamer Server and Machine 2 hosting an instance of the Client Simulator.
Configuration of the Adapter Simulator and the Client Simulator is as follows:
- 100 sessions are created by the Client Simulator
- 500 items are generated by the Adapter Simulator
- 5 items are subscribed to by each session, taken from the 500 available items
- for each item, the Adapter Simulator generates one update every 3 seconds
- each update is made up of 5 fields
- each field has a payload of 10 bytes
Make sure the latest version of the chosen operating system is being used, with all the patches and updates applied.
Install the latest available release of Java SE Development Kit on both the machines. If you need to use several gigabytes of memory, you need to install the 64-bit version of the JDK.
Install Lightstreamer Server v. 7.1 (or newer) on the Server machine, configure it following the GETTING_STARTED.TXT instructions contained in the package, and verify that it works correctly, by running one of the preinstalled demos.
NOTE: To fully leverage the LLTT features, you need to configure the Lightstreamer server with Enterprise Edition, since with Community Edition the test could be compromised by the limitations of the edition.
Open lightstreamer_conf.xml (under the “conf” folder) and make sure you have the following settings:
- <max_buffer_size> should be set to 10 (this will avoid buffering too much data in case a Client Simulator should saturate its own CPU)
- <disable_session_mbeans> (directly under the <JMX> block) should be left unset, or set to Y
- <delta_delivery> should be set to N (this turns off the algorithm that compares subsequent updates on the same item in order to extract and send only the delta)
- <reuse_pump_buffers> should be set to Y (unless you are simulating a very large number of sessions with a very low update rate)
- <sendbuf> should be set to 5000 (this will reduce the latencies, especially on the local loop)
The above settings are suitable for the test scenario. If your production scenario calls for different values for some of these settings, you may evaluate the related impact on performances.
Open lightstreamer_log_conf.xml (under the “conf” folder) and make sure you have the following settings:
-
The “LSProducer” appender should be changed as follows (to avoid flooding the Monitor Console channel with useless activity messages):
<appender name="LSProducer" class="com.lightstreamer.logback.ProducerAppender"> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>warn</level> </filter> </appender> -
The “LSConsole” appender should be changed as follows (to avoid flooding the Console shell with useless activity messages):
<appender name="LSConsole" class="ch.qos.logback.core.ConsoleAppender"> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>warn</level> </filter> <encoder> <pattern>%d{"dd.MMM.yy HH:mm:ss,SSS"} <%5.5(%p%marker)> %m%n</pattern> </encoder> </appender> -
The “LightstreamerMonitorText” and “LightstreamerMonitorTAB” categories should be changed as follows (to log the monitoring info more often and disable the tabular version of the same):
<logger name="LightstreamerMonitorText" level="TRACE"> <appender-ref ref="LSDailyRolling" /> </logger> <logger name="LightstreamerMonitorTAB" level="ERROR"> <appender-ref ref="LSDailyRolling" /> </logger> -
The “LightstreamerLogger.pump” category should be changed as follows (to avoid flooding the log with possible loops of notifications of lost updates):
<logger name="LightstreamerLogger.pump" level="WARN" />
Install the Adapter Simulator: get the LLTT package under the release section; then extract from the zipped package the “AdapterSimulator” folder and copy it in the “adapters” folder of the Lightstreamer Server installation.
To install the Client Simulator, get the LLTT package under the release section; then extract from the zipped package the “ClientSimulator” folder and copy it to a folder of choice.
Edit the launch script file (“start_client.sh” for Linux, or “start_client.bat” for Windows) in order to set the correct location of the JVM to use.
Edit the configuration.xml file and specify the address of Machine 1 as the “host” parameter.
The default configuration of the Adapter Simulator and the Client Simulator is as follows:
- 100 sessions are created by the Client Simulator
- 500 items are generated by the Adapter Simulator
- 5 items are subscribed to by each session, taken from the 500 available items
- for each item, the Adapter Simulator generates one update every 3 seconds
- each update is made up of 5 fields
- each field has a payload of 10 bytes
Launch Lightstreamer Server on Machine 1.
Launch the Client Simulator on Machine 2.
While the test is running, it is a good idea to monitor the systems and application resources. In particular, the following observations are recommended:
- Monitor the CPU usage of both the Server and the Clients.
- Open the Monitoring Dashboard of Lightstreamer Server in your browser (by default, it is available via the “/dashboard/” URI).
- Check the Lightstreamer Server log, where the Monitor info is reported periodically and any errors will appear.
- Check the Lightstreamer Server console output, where any error related to heap memory exhaustion will be reported.
- Check the Client Simulator logs, where information on test progression and any error will be reported.
- Check any network bottlenecks and other issues by using the ifconfig and netstat –statistics commands (on Linux machines). Look for lost and retransmitted packets.
You can configure the LLTT to reflect a broad range of simulation scenarios. You may increase the total number of items, change the update frequency, change the update size, change the number of items subscribed to by each session, run the test with TLS, enforce http streaming instead of WebSocket, etc.
Refer to the inline comments within adapters.xml (for Adapter Simulator) and configuration.xml (for Client Simulator) to know the meaning of the available parameters.
Both the Client Simulator and the Adapter Simulator contain their own log4j configuration file, which can be edited: log_conf.xml for the Client; adapter_log_conf.xml for the Adapter.
In case you need to simulate very low frequency scenarios, where no updates should be sent to a session for several seconds or even minutes, you should take into considerations the keep-alive messages. When no data is flowing on a stream connection, Lightstreamer Server sends small keep-alive messages to keep the connection open. The frequency of such keep-alive messages is configured in lightstreamer_conf.xml, via the <default_keepalive_millis> element. The default configuration is 5 seconds.
In case of a test with TLS, make sure that the client process is granted enough memory, by manually changing the JVM_PROPERTIES in the start_client script. The memory requirements related with the handling of TLS is on the order of a few tens of KB.
In case the data generation activity requested to the Adapter Simulator is quite intense, it is a good idea to perform a preliminary test where the Adapter Simulator is launched as a stand-alone process. This way, it is possible to measure how much CPU and run it will steal to Lightstreamer Server during the actual test. The “optional_standalone_launcher.sh” and “optional_standalone_launcher.bat” script files are supplied for this purpose (you may need to edit them).
In Lightstreamer Server installation, edit LS.sh (under the “bin/unix-like” folder) or LS.bat (under the “bin\windows” folder) as follows:
- If needed, tune the Java heap configuration (min and max memory) by editing the JAVA_OPTS parameter. You may need to allocate much more memory for more intense tests.
- If needed, tune the JVM garbage collector by adding specific fields to the JAVA_OPTS parameter. Again, for a basic test the default garbage collector should be good.
- On Linux and UNIX, increase the file descriptor limit, possibly setting it to unlimited.
Open lightstreamer_conf.xml (under the “conf” folder) and ensure that all the configuration settings under the “LOAD CONFIGURATION” section are left to their defaults. This ensures, in particular, that the pools devoted to internal CPU-bound tasks (i.e. the “events pool” and the “pump pool”) contain one thread per each available core.
Using a Gigabit Ethernet or better is recommended for the tests.
To allow very high numbers of incoming connections, you may need some further tuning of the operating system. For Linux, below is a tentative list of possible settings. You should put such settings in “/etc/sysctl.conf” then run sysctl -p to apply them. There should be no need to reboot.
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_mem = 786432 1048576 26777216
net.ipv4.tcp_max_tw_buckets = 360000
net.core.netdev_max_backlog = 2500
net.ipv4.netfilter.ip_conntrack_max = 1048576
fs.file-max = 1048576
vm.min_free_kbytes = 65536
net.ipv4.ip_local_port_range = 1024 65535
For instance, on Linux, if the nf_conntrack module is loaded, it may introduce a limit on the connections, which can be enlarged through the net.ipv4.netfilter.ip_conntrack_max setting. In fact, we observed such a limit to be set with a default of about 260000 connections when deploying the Server through a Docker container.
To turn on latency reporting on a Client Simulator instance, you need to
- Set <param name="injectTimestamps"> to true in the adapters.xml file of the Adapter Simulator within Lightstreamer Server installation.
- Set priority value to INFO for the "com.lightstreamer.load_test.reports.latency_reporting" category in the log_conf.xml file of the Client Simulator, before launching the client.
Load testing is a complex discipline, as many variables usually tend to affect the results. Once some variables have been fixed, others can be tuned to optimized the performance.
After the following variables have been fixed:
- Platform and hardware configurations
- Application scenario (that is, concurrent sessions, number of items, event frequencies, etc.)
more variables can be tuned that affect a very important trade-off: scalability vs. latency.
Usually, the higher is the acceptable latency for the event dispatching, the higher is the scalability in terms of concurrent sessions.
The main parameters affecting that trade-off are the following:
-
Configuration of the max_delay_millis element in lightstreamer_conf.xml. It sets the longest delay that the Server is allowed to apply to outgoing updates in order to collect more updates in the same TCP packet. Even small values (5 or 10 milliseconds) can provide great benefits in terms of scalability with respect to the scenario where 0 is used. Being able to forge larger TCP packets provide benefits to both network usage and CPU usage.
-
Configuration of the garbage collection algorithm of the Java Virtual Machine. The Java garbage collection strategy has a direct impact over the real-time grade of the system and induced latencies. In order to reduce the maximum latencies, a garbage collector that provides continuous collection (rather than periodically suspending the application) should be chosen (again, as part of a trade-off with scalability).
Tuning the garbage collection requires specific skills, because several collection algorithms with different specific parameters are provide by the JVM. Please look for the Garbage Collection Tuning Guide for your Java installation for an overview of the available algorithms. If a "pauseless collector" is available, then this is obviously the best choice. If not, you may try the following settings: "-server -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -XX:G1ReservePercent=20 -XX:+AlwaysPreTouch" In case of a test with TLS, the heap limit may need to be higher.
To configure the garbage collector, edit LS.sh or LS.bat, and modify JAVA_OPTS.