jmeter-load-testing

Overview

Apache JMeter is the long-running JVM-native load-testing tool. Tests are XML .jmx files authored in JMeter's GUI; CI runs them headless via the jmeter CLI (jmeter-getstarted).

The official guidance is explicit:

"GUI mode should only be used for creating the test script, CLI mode (NON GUI) must be used for load testing." (jmeter-getstarted)

This skill covers the CLI / CI side. Authoring is GUI-driven and out of scope here - see the JMeter user manual for the Thread Group / HTTP Sampler / Assertion authoring flow.

When to use

• The project already has .jmx test plans.
• The team is on the JVM and prefers JMeter's mature ecosystem (BlazeMeter, Taurus, plugin manager, distributed mode).
• A specific protocol is needed that JMeter has first-class support for: JDBC, JMS, MQTT, FTP, LDAP, SOAP - k6's plugin landscape is weaker for niche protocols.

If the team is starting fresh, evaluate k6-load-testing (developer-friendly JS) or the Gatling (JVM DSL) and Locust (Python) references in load-testing-overview before adopting JMeter - XML authoring has a steep learning curve.

Install

JMeter requires Java 8 or higher with JAVA_HOME set. Download the latest release from jmeter.apache.org and extract - there is no installer (jmeter-getstarted).

For Docker-based CI, official images are at apache/jmeter. Pin to a specific tag rather than latest.

Running

Canonical CLI invocation

Per jmeter-getstarted:

jmeter -n -t test.jmx -l results.jtl

Flag	Purpose
-n	Non-GUI mode. Required for load tests.
-t	Path to the .jmx test plan.
-l	Output file for raw sample results (JTL - CSV-shaped).

With HTML dashboard

Per jmeter-getstarted:

jmeter -n -t test.jmx -l results.jtl -e -o report_folder

Flag	Purpose
-e	Generate the HTML dashboard report automatically.
-o	Output folder (must be empty or non-existent).

The dashboard shows percentiles, throughput, error rates, response- time graphs, and per-sampler breakdowns - the canonical JMeter output for human review.

Other useful flags

Flag	Purpose
-J<property>=<value>	Override a JMeter property at the JVM level.
-G<property>=<value>	Override a property in distributed (remote) mode.
-Jjmeter.save.saveservice.output_format=csv	Force CSV JTL (vs. XML).
-q <props-file>	Additional properties file.
-r	Start the test on remote slaves (distributed mode).
-X	Exit JMeter when the test finishes.

For multi-environment runs, parameterize the test plan with ${__P(api.base.url, default)} and override at the CLI:

jmeter -n -t orders.jmx -l results.jtl \
  -Japi.base.url=https://staging.example.com \
  -Japi.token=$API_TOKEN

Parsing results

The JTL file is the structured artifact for CI gating. Default JTL columns (CSV): timeStamp, elapsed, label, responseCode, responseMessage, threadName, dataType, success, failureMessage, bytes, sentBytes, grpThreads, allThreads, URL, Latency, IdleTime, Connect.

Quick-and-dirty pass/fail with awk:

# Count errors
awk -F',' 'NR>1 && $8=="false"' results.jtl | wc -l

# Compute mean response time
awk -F',' 'NR>1 { sum+=$2; n++ } END { print sum/n }' results.jtl

For richer parsing, use the JMeter HTML report's statistics.json under the report folder - it contains the percentiles per sampler in JSON form.

CI integration

# .github/workflows/jmeter.yml
name: load-test

on:
  pull_request:
    paths: ['tests/load/**.jmx']
  schedule:
    - cron: '0 4 * * *'

jobs:
  jmeter:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - name: Run JMeter via Docker
        env:
          API_TOKEN: ${{ secrets.STAGING_API_TOKEN }}
        run: |
          docker run --rm \
            -v "$PWD:/work" \
            -w /work \
            apache/jmeter \
              -n -t tests/load/orders.jmx \
              -l results.jtl \
              -e -o report \
              -Japi.token=$API_TOKEN \
              -Japi.base.url=https://staging.example.com

      - name: Pass/fail gate
        run: |
          ERRORS=$(awk -F',' 'NR>1 && $8=="false"' results.jtl | wc -l)
          if [ "$ERRORS" -gt 10 ]; then
            echo "::error::Got $ERRORS errors (>10 threshold)"
            exit 1
          fi

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: jmeter-report
          path: |
            report/
            results.jtl
          retention-days: 14

The Docker invocation pattern keeps the CI runner clean - no Java / JMeter install on the runner.

Anti-patterns

Anti-pattern	Why it fails	Fix
Running tests in GUI mode "just for this one quick run"	GUI runner adds overhead; metrics are skewed; per jmeter-getstarted this is explicitly forbidden for load tests.	Always -n. GUI is for authoring only.
Hard-coding URLs / tokens in the .jmx XML	Test plan binds to one environment.	Parameterize: ${__P(api.base.url, ...)}; override with -J.
Saving JTL in XML format	XML JTL files are 5-10x larger than CSV; slow to parse.	Force CSV with -Jjmeter.save.saveservice.output_format=csv.
Listeners enabled in CI runs	UI listeners (View Results Tree, Aggregate Report) consume RAM proportional to sample count; OOMs at scale.	Disable all listeners in .jmx; emit JTL only; generate HTML report post-run via -e -o.
One mega-test-plan with 100 samplers	Failure attribution is hard; runtime dominated by one slow endpoint.	Split into per-domain .jmx files; run as separate CI jobs.
Threshold gates only on error rate	A 30-second response that succeeds passes the rate gate but breaks UX.	Pair error-rate with percentile gates parsed from statistics.json.

Limitations

• XML authoring. .jmx files are not human-friendly; reviews of test-plan changes are awkward.
• Memory hungry. JMeter's per-sample memory overhead is high vs. k6 / Gatling.
• Single-machine VU limits. ~1000 threads per machine before JVM contention; for higher loads, use distributed mode (master + remote slaves) or move to a tool with a smaller per-VU footprint.
• Plugin ecosystem fragmentation. Many JMeter plugins are abandoned; vet plugin maintenance status before depending on one.

References

• jmeter-getstarted - canonical CLI flags, GUI-vs-CLI guidance, dashboard generation.
• Apache JMeter user manual - https://jmeter.apache.org/usermanual/
• k6-load-testing and the Gatling / Locust references in load-testing-overview - alternatives by stack.
• perf-budget-gate - downstream gate aggregating multiple runner verdicts.