Tips to Optimize the Performance of JMeter and Test Script

7 min readOct 26, 2020

JMeter is a 100% pure Java application and obviously, it runs on a JVM. So it is really important that you know how to fine-tune the JVM before doing anything.

I used JMeter to do some performance testing for Transactions Per Second 200 (TPS) and Payload size of 50 KB. For this, I used a basic JMeter script where I did not use any sort of listener. For this test, I used an ordinary jmeter-docker inside the ECS cluster. The capacity of the container was 2 vCPU and 4 GB memory(which is equivalent to t2.medium AWS EC2 instance). Initially, I was able to run this load for more than 30 minutes continuously. Further challenges came up when the TPS was bumped up to 500 since JMeter could not handle this volume in this setup. For example, when I execute this load, the docker container throws ‘OutOfMemoryError: Container killed due to memory usage’.

Serious configuration changes were required to scale up JMeter to a level where it could handle the desired load. I had two options left:

Increase the capacity of the docker container
Improve the performance of JMeter and the running environment

Things I’m going to talk about here are a set of general tips which you can improve the performance of any JMeter Test and the running environment.

Use Java JDK NOT JRE

As per the Java Garbage Collection handbook:

The java.lang.OutOfMemoryError: GC overhead limit exceeded error is the JVM’s way of signaling that your application spends too much time doing garbage collection with too little result.

By default the JVM is configured to throw Out Of Memory errors if it spends more than 98% of the total time doing GC and when after the GC only less than 2% of the heap is recovered.

Here is the difference:

JRE: Java Runtime Environment. It is basically the Java Virtual Machine where your Java programs run on. It also includes browser plugins for Applet execution.
JDK: It’s the full-featured Software Development Kit for Java, including JRE, and the compilers and tools (like JavaDoc, and Java Debugger) to create and compile programs.

The reason to go with a JDK is that it contains a server optimized compiler that is better at optimizing Java code execution, thus JMeter Execution.

Increase JVM Heap Space

The default heap size allocated to JMeter is 512 MB. It’s always good to have 1024 MB or more to avoid the java ‘Out of Memory‘ error which is usually thrown when test plans are running at higher concurrency levels.

-Xms<size>-Xmx<size>

These parameters specify the initial Java Heap size as well as the maximum Heap size.

The Xmx value for a machine that generates the load for your tests can be calculated as follows: Total system memory — (memory used by the OS + memory used by the JVM + any other scripts required to run on the machine). If you have a dedicated machine for testing, to avoid reallocation of the Xms during testing runtime, set the Xms = Xmx from the beginning.

-Xmn<size>

This parameter specifies the size of the young generation heap space. Applications with emphasis on performance tend to use -Xmn to size the young generation, because it combines the use of -XX:MaxNewSize and -XX:NewSize and almost always explicitly sets -XX:PermSize and -XX:MaxPermSize to the same value. One important rule to follow though is to keep the young generation less than 50% of the heap space.

Further Tweak JVM Settings

WARNING: danger zone. Tweaking the JVM is mostly not required and can lead to problems. However, if you really want to squeeze the last bit of performance out of the JVM, you can use the following JVM settings (at your own risk):

-server: This switches JVM into “server” mode with runtime parameters optimization, as explained above it enables more aggressive compiler optimizations,
-XX:+UseConcMarkSweepGC: Enables concurrent Mark Sweep GC which can improve CPU usage on multi-core CPUs,(to see more on GC)
-XX:+DisableExplicitGC: Prevents explicit GC Calls. Unfortunately (and in spite of the disclaimers in the documentation), many developers decide they know better than the JVM and call the Garbage Collector programmatically to collect memory. I’m aware of many instances where the -XX:+DisableExplicitGC improved the production environment and zero instances where there were any negative side effects.

Tune the operating system for optimal performance

These settings are generic settings to utilize the full port range and to reduce the TCP fin timeout for better socket reuse, and set a good limit on the number of files handles permitted to the process.

Edit /etc/sysctl.conf and append

# Increase system IP port limits 
net.ipv4.ip_local_port_range = 1024 65535  # Decrease TIME_WAIT seconds 
net.ipv4.tcp_fin_timeout = 60  # Increase system file descriptor limit 
fs.file-max = 2097152  # Reuse TIME_WAIT sockets faster 
net.ipv4.tcp_tw_reuse = 1  # Controls the use of TCP syncookies 
net.ipv4.tcp_syncookies = 0  # The maximum number of "backlogged sockets"" 
net.core.somaxconn = 1024

Edit /etc/security/limits.conf and append

* soft nofile 65535* hard nofile 65535

Upgrade Java and JMeter

Make sure to use the most recent versions of:

Java: use the latest 64 bits Java Virtual Machine, as it allows to use of more than 1536 MB or RAM as Xmx,
JMeter: latest JMeter versions usually contain many fixes and improvements,
JMeter Plugins: make sure to use the latest JMeter Plugins,
Operating System: upgrade your operating system to the latest version.

Generally speaking, avoid using old software versions otherwise you may encounter bugs that have been fixed in the latest versions.

JMeter Non-Gui Mode

You can execute a JMeter test from the command line without launching the UI (which consumes a lot of memory):

jmeter -n -t scenario.jmx

-n: runs JMeter in non-GUI mode,
-t: specifies the path to source .jmx script to run.

Disable Listeners

Listeners receive Sample Results and do some processing with it, this takes resources (memory, CPU) so during Load Testing, a very simple rule is to remove all listeners! , you will ask How can I use my results then? , the answer is, you will do it after the run. So NEVER EVER have one of these listeners in your Test Plan during a Load Test :

View Results in Table => OutOfMemory guarantee in GUI Mode
View Results in Tree => OutOfMemory guarantee in GUI Mode
Graph Results => Performance issues
Assertion Results => OutOfMemory guarantee
Comparison Assertion Visualizer
Distribution Graph (alpha) => Performance issues
Graph Results => Performance issues

Tune JMeter SaveService

JMeter’s SaveService manages the data which is written into the JTL result file. You can vastly improve the throughput by reducing the amount of data being written into the JTL files by editing the jmeter.properties file.

The properties below are pretty well optimized and can be used as default settings:

# true when field should be saved; false otherwise# assertion_results_failure_message only affects CSV output
#jmeter.save.saveservice.assertion_results_failure_message=true
# legitimate values: none, first, all
#jmeter.save.saveservice.assertion_results=none
#jmeter.save.saveservice.data_type=true
#jmeter.save.saveservice.label=true
#jmeter.save.saveservice.response_code=true
# response_data is not currently supported for CSV output
#jmeter.save.saveservice.response_data=false# Save ResponseData for failed samples
#jmeter.save.saveservice.response_data.on_error=false
#jmeter.save.saveservice.response_message=true
#jmeter.save.saveservice.successful=true
#jmeter.save.saveservice.thread_name=true
#jmeter.save.saveservice.time=true
#jmeter.save.saveservice.subresults=true
#jmeter.save.saveservice.assertions=true
#jmeter.save.saveservice.latency=true# Only available with HttpClient4
#jmeter.save.saveservice.connect_time=true
#jmeter.save.saveservice.samplerData=false
#jmeter.save.saveservice.responseHeaders=false
#jmeter.save.saveservice.requestHeaders=false
#jmeter.save.saveservice.encoding=false
#jmeter.save.saveservice.bytes=true# Only available with HttpClient4
#jmeter.save.saveservice.sent_bytes=true
#jmeter.save.saveservice.url=false
#jmeter.save.saveservice.filename=false
#jmeter.save.saveservice.hostname=false
#jmeter.save.saveservice.thread_counts=true
#jmeter.save.saveservice.sample_count=false
#jmeter.save.saveservice.idle_time=true

Distributed Tests

At some point, you won’t be able to further improve JMeter’s memory usage if you need to run a massive amount of concurrent users. You won’t be able to run 10.000 concurrent users from a single machine, that’s simply not working (and not simply because of CPU and memory usage, but also network usage).

Distributed JMeter testing is tricky and requires a lot of knowledge. Always make sure to configure each JMeter instance running on each computer with the same SaveService settings, same JVM settings etc. You will have to set up each load generator manually and edit each configuration file.

Without scaling out the capacity of the container I applied the above configurations into my Jmeter-docker image and was able to reach 1000 TPS without any issue. Following is the statistic comparison after applying the changes.

References

You can find an example of how to use my JMeter-docker image here.

You can find a variety of articles and tutorials on JVM Performance Optimization. Below are a few of them I’ve found useful.

Select your JDK version → Tools → Tools Reference/JDK Tool Specifications → Java

Note: There are other steps that you can improve in your test script. But again it depends on your use case. Please be careful when applying those in your test script.