BlazeMeter Blog How to Run JMeter Tests with TeamCity for Continuous Integration

JMeterWe recently covered how to run JMeter in a Continuous Integration (CI) environment with Jenkins and with Bamboo. Today's article covers the process of running a JMeter test using TeamCity. Continuous Integration and Continuous Delivery are vital for developing and producing complex applications, technology and automation help to move faster and more efficiently. Being an important part of the software development lifecycle, load testing needs to be included into the Continuous Integration process, bringing confidence to software performance and capacity. TeamCity is a build and continuous integration server from JetBrains. While it is a commercial product, it can be used for free if you don't need more than 20 build configurations and 3 build agents. Free license is granted to open-source projects as well. TeamCity was released 10 years ago and now it's being used by thousands of software development companies. It has awesome features like running a build prior to VCS commit, integrated code coverage reports and code inspections, the capability of running multiple builds and/or tests on different platforms, and more. Configuring TeamCity for Running JMeter Tests TeamCity is capable of running a JMeter test out of the box, however, you will have to analyze the results manually. This doesn't make a lot of sense given the goal to have load tests running in an unattended and automated manner. There is a JMeter plugin for TeamCity, which has 2 main functions: Performance Statistics - a collection of performance metrics and visualizing them as charts on build results pagePerfMon - visualizing a build agent and application under test server health- A collection of the main OS health metrics from a build agent for the test duration- A collection of the main OS health metrics from a remote machine JMeter Plugin for TeamCity Installation The JMeter plugin for TeamCity can be installed in a few easy steps: Download the latest version of the plugin from https://teamcity.jetbrains.com/repository/download/TeamCityPluginsByJetBrains_JMeterPlugin_Build/.lastSuccessful/jmeter.zipDrop "jmeter.zip" file to the TeamCity plugins folder - by default it is:C:\ProgramData\JetBrains\TeamCity\plugins - for MS Windows Family.BuildServer/plugins/ directory under the home folder of the user which is running the TeamCity server - for Linux, Unix and derivativesRestart TeamCity If everything goes as expected, the JMeter plugin should be listed among others under TeamCity -> Administration -> Plugins page: Running a JMeter Test with TeamCity Configuring the JMeter Build Step TeamCity supports a wide range of build steps including, but not limited to: - Running an arbitrary Command Line (either a custom shell script or an executable one with parameters)- Invoking an Apache Ant task- Running an Apache Maven project Usually it is better to use your main project build system. You may know that a JMeter test can be executed via the aforementioned methods, if not - you're welcome to check out Five Ways To Launch a JMeter Test without Using the JMeter GUI. If your application is being built using Maven, go for Maven, and if it uses Ant, run the JMeter test using JMeter Ant Task, etc. In this article, I'll stick to the command-line non-GUI mode, as it is the most common approach of running a JMeter test. In order to trigger a JMeter test from TeamCity, you need to add a Build Step which will call JMeter and provide your test script as a parameter. The minimal working configuration will look like: - Runner Type: Command Line- Run: Executable with parameters- Command executable: path to the JMeter startup script, i.e. C:\jmeter\bin\jmeter.bat- Command parameters: standard JMeter arguments like you run the test from the command line, i.e. -n -t c:\jmeter\test.jmx -l test.jtl- You can also specify an arbitrary "Step Name" especially if you plan to use the Remote PerfMon feature. Important! The JMeter TeamCity plugin expects the results file to be in a specific non-default format, the most important requirements being: The metrics need to be separated by TAB charactersThe first line needs to be column headers The relevant JMeter properties are: jmeter.save.saveservice.default_delimiter=\tjmeter.save.saveservice.print_field_names=true So in order to be able to use the JMeter TeamCity plugin, add the above 2 lines to the user.properties file (it is located under the "bin" folder of your JMeter installation). Configuring the JMeter TeamCity Plugin Build Feature The next step is to "point" the JMeter TeamCity plugin to the .jtl results file so you could see performance statistics along with the build results. In order to do so, go to the Build Features page in TeamCity and add the "Performance Metrics Calculation" feature. The minimal working configuration should look like: Now you should be ready to run your test. You should see an extra tab on the build results page - "Performance Statistics". This is where you will be able to see the performance metrics and trends. Remember that you should have at least 2 builds in order to see the charts. It is also possible to set reference values to check the metrics against. This way you will be able to conditionally fail the build in the case of exceeding a threshold, such as if the average response time is above the SLA or the non-200 HTTP response codes number is higher than 1% (these are just a few examples). Monitoring the Application Under Test Health Metrics It is also possible to plot some health metrics from the server where your application lives in the TeamCity build results page: Download the PerfMon Server Agent and install it on the server your applications runs onMake sure that TeamCity is able to connect the server using a default port of 4444Configure the TeamCity JMeter plugin to collect metrics from the remote server. To do so, add the "Remote Performance Monitoring" Build Feature. Replace "localhost" with hostname or the IP address of the machine where your application is deployed. With the "Remote Performance Monitoring" build feature enabled, you will be able to see the application under the test server health metrics, along with main performance KPIs on the build results page. See How to Monitor Your Server Health & Performance During a JMeter Load Test article for more information on the application under test health monitoring concept and comprehensive installation and usage instructions. Now you should be ready to add JMeter test to your TeamCity build process. If anything remains unclear, feel free to ask in the comments sections below. BlazeMeter users can integrate their cloud tests with local TeamCity deployments using the BlazeMeter plugin To TeamCity: BlazeMeter Plugin To TeamCityLoad Testing with TeamCity and BlazeMeterContinuous IntegrationTeamCityJMeterApache JMeteropen sourceLoad testingperformance testingauthor_info: Dmitri Tikhanski is a Contributing Writer to the BlazeMeterauthor_name: Dmitri Tikhanski