Stress Testing with Load Impact’s k6

Test-driven development ensures our project satisfies some requirements set by our product owner. “Why some requirements?”, you might ask — it is because TDD often emphasizes on the functional requirements. Non-functional requirements — like stability of a system — will need to be tested separately, often using external tools separate from the project’s framework. In this post I’ll explore on how to the stability of iur project using Load Impact’s k6 stress tester.

k6 is a website stress tester, which (according to my understanding) is a glorified concurrent while-true loop where each loops curl‘d our website and check whether the HTTP response code is 200 OK or not. It might also check the contents of the HTTP response, similar to unit tests we’re all familiar with.

k6 is distributed as a docker image — which is basically a VM but lighter and GUI-less, akin to a sandboxed environment. This allows k6 to has its own network subnet (e.g. subnet of docker0 network interface) and simulate webpage access from IPs other than localhost.

k6 scripts are written in JavaScript, which is frankly, pretty straightforward. This is an example stress test which visits our project’s homepage.

import { check } from "k6";
import http from "k6/http";

export default function() {
	let res = http.request(
		"GET",
		"http://172.17.0.1:3000/",
		{},
		{}
	);

	check(res, {
		"is status 200": (r) => r.status === 200
	});
};

That script will stress test our webpage on http://172.17.0.1:3000/ and check whether the HTTP response code is 200 or not. The server’s IP (172.17.0.1) is obtained from my docker’s virtual network interface docker0, which can be displayed using ifconfig.

If we’re going to stress test pages behind a login mechanism, we should attach additional request headers than contains the cookie. The cookie can be viewed by accessing the browser’s dev tools (F12) and then check the request headers. The following script will fetch our Project Details page, while also check the HTML for "Project Details" substring:

import { check } from "k6";
import http from "k6/http";
import parseHTML from "k6/html";

export default function() {
	let res = http.request(
		"GET",
		"http://172.17.0.1:3000/projects/40",
		{},
		{
			headers: {
				"Cookie": "_phrogress_session=K2Y4a2o4alhaSzZxRUxIb1V3azBpajlUNjlMbG10dlZ6MFl0MGJOVnZTbFJQbDgwcGNxbTB6WE5yazg0cUVKSGxCUHg4OU91U3ltTkZGNU0wNmxzWlFXTVJqRHpCdHhjVCtCRGVsR3hGMFlFZkZmR0NvSG84ZERVZEpmVUdoVzgtLXVCb1VpVlVtT3VHMFZVbDJSbWpTcWc9PQ%3D%3D--527e9f33e2e60601b08117e20984ad5d53a69bd7; path=/; HttpOnly"
		  	}
		}
	);

	check(res.body, {
		"contains Project Details": (r) => r.indexOf("Project Details") != -1
	});

	check(res, {
		"is status 200": (r) => r.status === 200
	});
};

Let’s save that script as main_page.js and project_detail.js. To run the test we simply run this command:

docker run -i loadimpact/k6 run --vus 50 --duration 60s - < main_page.js

There are some notable parameters for that command:

  • --vus denotes the number of Virtual Users (VUs). This will affect how the number of requests sent per second to the target.
  • --duration denotes the duration of the test.

So, the output of that command will be:

$ docker run -i loadimpact/k6 run --vus 50 --duration 60s - < main_page.js


          /\      |‾‾|  /‾‾/  /‾/   
     /\  /  \     |  |_/  /  / /   
    /  \/    \    |      |  /  ‾‾\  
   /          \   |  |‾\  \ | (_) | 
  / __________ \  |__|  \__\ \___/  Welcome to k6 v0.14.0!

  execution: local
     output: -
     script: - (js)

   duration: 1m0s, iterations: 0
        vus: 50, max: 50

    web ui: http://0.0.0.0:6565/

[running   ] 900ms / 1m0s
[running   ] 1.9s / 1m0s
[running   ] 2.9s / 1m0s
[running   ] 3.9s / 1m0s
...
[done      ] 1m0s / 1m0s

    ✗ 96.61% - is status 200

    checks................: 99.87%
    data_received.........: 4.6 MB (76 kB/s)
    data_sent.............: 416 kB (6.9 kB/s)
    http_req_blocked......: avg=0s max=0s med=0s min=0s p90=0s p95=0s
    http_req_connecting...: avg=0s max=0s med=0s min=0s p90=0s p95=0s
    http_req_duration.....: avg=135.16ms max=1.32s med=130.19ms min=62.28ms p90=169.36ms p95=184.84ms
    http_req_receiving....: avg=4.65ms max=37.44ms med=2.85ms min=52.07µs p90=11.46ms p95=15.65ms
    http_req_sending......: avg=27.61µs max=2.65ms med=24.68µs min=11.19µs p90=33.31µs p95=39.4µs
    http_req_waiting......: avg=130.47ms max=1.3s med=125.7ms min=59.7ms p90=163.91ms p95=180.63ms
    http_reqs.............: 1486 (24.766666666666666/s)
    iterations............: 1486 (24.766666666666666/s)
    vus...................: 50
    vus_max...............: 50

k6 reports 96.61% successful request. While that seems to be good, websites are supposed to have 99.99%-ish uptime for way more than 50 users. Upping the number of VUs to 200 drops the uptime even further:

$ docker run -i loadimpact/k6 run --vus 200 --duration 60s - < main_page.js


          /\      |‾‾|  /‾‾/  /‾/   
     /\  /  \     |  |_/  /  / /   
    /  \/    \    |      |  /  ‾‾\  
   /          \   |  |‾\  \ | (_) | 
  / __________ \  |__|  \__\ \___/  Welcome to k6 v0.14.0!

  execution: local
     output: -
     script: - (js)

   duration: 1m0s, iterations: 0
        vus: 200, max: 200

    web ui: http://0.0.0.0:6565/

[running   ] 900ms / 1m0s
[running   ] 1.9s / 1m0s
[running   ] 2.9s / 1m0s
[running   ] 3.9s / 1m0s
...
[done      ] 1m0s / 1m0s

    ✗ 89.07% - is status 200

    checks................: 99.51%
    data_received.........: 4.8 MB (80 kB/s)
    data_sent.............: 460 kB (7.7 kB/s)
    http_req_blocked......: avg=45.69µs max=63.01ms med=0s min=924.64µs p90=0s p95=0s
    http_req_connecting...: avg=45.32µs max=62.84ms med=0s min=885.12µs p90=0s p95=0s
    http_req_duration.....: avg=376.4ms max=1m0s med=120.56ms min=50.19ms p90=153.71ms p95=165.92ms
    http_req_receiving....: avg=4.46ms max=35.62ms med=2.7ms min=0s p90=10.81ms p95=14.45ms
    http_req_sending......: avg=38.77µs max=17.03ms med=25.52µs min=10.05µs p90=38.71µs p95=46.88µs
    http_req_waiting......: avg=371.9ms max=1m0s med=115.94ms min=47.06ms p90=149.44ms p95=161.32ms
    http_reqs.............: 1646 (27.433333333333334/s)
    iterations............: 1646 (27.433333333333334/s)
    vus...................: 200
    vus_max...............: 200

The same thing happened with project details:

$ docker run -i loadimpact/k6 run --vus 50 --duration 60s - < project_details.js


          /\      |‾‾|  /‾‾/  /‾/   
     /\  /  \     |  |_/  /  / /   
    /  \/    \    |      |  /  ‾‾\  
   /          \   |  |‾\  \ | (_) | 
  / __________ \  |__|  \__\ \___/  Welcome to k6 v0.14.0!

  execution: local
     output: -
     script: - (js)

   duration: 1m0s, iterations: 0
        vus: 50, max: 50

    web ui: http://0.0.0.0:6565/

[running   ] 900ms / 1m0s
[running   ] 1.9s / 1m0s
[running   ] 2.9s / 1m0s
[running   ] 3.9s / 1m0s
...
[done      ] 1m0s / 1m0s

    ✗ 93.96% - contains Project Details
    ✗ 93.96% - is status 200

    checks................: 100.00%
    data_received.........: 8.4 MB (140 kB/s)
    data_sent.............: 356 kB (5.9 kB/s)
    http_req_blocked......: avg=3.2µs max=847.42µs med=0s min=0s p90=0s p95=0s
    http_req_connecting...: avg=2.81µs max=784.33µs med=0s min=0s p90=0s p95=0s
    http_req_duration.....: avg=384.13ms max=662.18ms med=372.41ms min=171.99ms p90=480.18ms p95=520.83ms
    http_req_receiving....: avg=4.09ms max=41.97ms med=2.14ms min=88.77µs p90=11.21ms p95=13.93ms
    http_req_sending......: avg=31.68µs max=92.62µs med=30.29µs min=19.79µs p90=39.42µs p95=44.87µs
    http_req_waiting......: avg=380.01ms max=642.57ms med=365.98ms min=170.27ms p90=475.22ms p95=512.34ms
    http_reqs.............: 778 (12.966666666666667/s)
    iterations............: 778 (12.966666666666667/s)
    vus...................: 50
    vus_max...............: 50
$ docker run -i loadimpact/k6 run --vus 200 --duration 60s - < project_details.js


          /\      |‾‾|  /‾‾/  /‾/   
     /\  /  \     |  |_/  /  / /   
    /  \/    \    |      |  /  ‾‾\  
   /          \   |  |‾\  \ | (_) | 
  / __________ \  |__|  \__\ \___/  Welcome to k6 v0.14.0!

  execution: local
     output: -
     script: - (js)

   duration: 1m0s, iterations: 0
        vus: 200, max: 200

    web ui: http://0.0.0.0:6565/

[running   ] 900ms / 1m0s
[running   ] 1.9s / 1m0s
[running   ] 2.9s / 1m0s
[running   ] 3.9s / 1m0s
...
[done      ] 1m0s / 1m0s

    ✗ 79.42% - contains Project Details
    ✗ 79.42% - is status 200

    checks................: 100.00%
    data_received.........: 8.3 MB (139 kB/s)
    data_sent.............: 353 kB (5.9 kB/s)
    http_req_blocked......: avg=2.62µs max=763.04µs med=0s min=0s p90=0s p95=0s
    http_req_connecting...: avg=2.12µs max=695.96µs med=0s min=0s p90=0s p95=0s
    http_req_duration.....: avg=386.55ms max=678.75ms med=369.96ms min=249.01ms p90=483.5ms p95=516.05ms
    http_req_receiving....: avg=4.21ms max=45.1ms med=2.5ms min=100.14µs p90=10.58ms p95=14.08ms
    http_req_sending......: avg=32.1µs max=248.55µs med=30.02µs min=20.09µs p90=41.56µs p95=47.1µs
    http_req_waiting......: avg=382.3ms max=677.65ms med=365.78ms min=246.53ms p90=478.7ms p95=512.87ms
    http_reqs.............: 772 (12.866666666666667/s)
    iterations............: 772 (12.866666666666667/s)
    vus...................: 200
    vus_max...............: 200

This is rather concerning as our partner, Peentar, has around 60 employees — which is pretty close to 50. So, our system will can only reliably serve 57 people if all the employees access our system at the same time.

When this article was written, we still can’t determine the source of this problem — it can be our code, the Rails framework, the database server, my weak laptop’s inadequate performance, or even the docker’s virtual network interface.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s