Profile Photo

Jamie Skipworth


Technology Generalist | Software & Data


Load-Testing with Locust on Google Cloud

Everyone’s obsessed with scale.

It seems that even if you’re selling funny dog-themed Christmas socks to grannies online you need Google-scale. Building everything on Kubernetes seems to have become standard practice everywhere. You know, just in case granny socks catch on and become the next big thing.

But let’s pretend you really need ‘Google-scale’ for whatever you’re building. If it gets popular or it’s an actual requirement, then it’s going to have to handle a significant chunk of your users making a ton of concurrent requests.

Recently I helped stand-up a cloud-based REST API server for a national financial institution. These APIs would be consumed via a mobile app, and being customer-facing and highly visible it needed to be reliable and scalable.

So we deployed to Google Kubernetes Engine (GKE) within Google Cloud Platform (GCP), using a ton of other GCP-ecosystem tools and services.

Super stuff, and check out all the buzzwords I used! But how do you test that your application can actually scale?

Locust

Well, there’s a nifty little framework out there called Locust. It’s an open-source, Python-based distributed load-testing framework that runs on GKE.

Locust logo

In English, what that means is that it can simulate a butt-load of concurrent users making requests at the same time. And because Locust behaviour is defined as regular Python code, it’s hugely configurable, and there’s no need to write thousands of lines of YAML or XML.

It’s configurability is a huge bonus; what I originally wrote as a quick one-off gradually transformed into something that could dynamically generate requests with different POST payloads, parameters, and headers, and even work with Kafka.

Demo: Locust On GKE

In this blog post I’m going to demonstrate how you’d set up Locust to load-test a pretend API. I’ll be using the following GCP components:

The next sections will describe how to set all of this up and get it all running (hopefully).

First, A Few Things…

The sample applications and configuration files I’ll use below can be found on GitHub here, if you want to have a sneaky peak before continuing.

So, first we’ll need to first install some required software and tools. Namely:

Clicking on the links above should take you to the documentation and installation instructions for your platform. I won’t spoon-feed you too much, you’re adults.

Once those are installed a few Google Cloud SDK components are needed:

  • gcloud Beta Commands
  • kubectl
  • gcloud app Python Extensions
  • gcloud app Python Extensions (Extra Libraries)
  • Google Container Registry’s Docker credential helper

These can be summoned as if by magic like so:

$ gcloud components install \
kubectl \
app-engine-python \
app-engine-python-extras \
docker-credential-gcr \
beta

A Google account is required to do anything useful on Google Cloud Platform, so don’t forget to register for one. Once one is created, authenticate the Google Cloud SDK:

$ gcloud auth login

This will open a browser tab and ask you to perform a gentle OAuth waltz. Follow the prompts until the You are now logged in as [[email protected]] message pops-up in the terminal. Once authenticated, create a new project in the Console, and tell the SDK to use it.

$ gcloud config set project <PROJECT_ID>

A Pretend REST API Application

To simulate a REST API server I’ve created a dummy Flask application that has two endpoints:

  • / - returns the string, Hello, world!
  • /datetime - returns the current date and time.

It doesn’t have to be fancy for this demo. And it’s so small it fits nicely into my similarly-sized brain:

#!/usr/bin/env python
from google.appengine.ext import vendor
vendor.add('lib') # Consult GAE docs for an explanation of this mysterious gubbins.

from flask import Flask
from datetime import datetime
from flask import jsonify

app = Flask(__name__)

@app.route('/')
def index():
    return "Hello, World!"

@app.route('/datetime')
def get_datetime():
    data = { 'datetime': datetime.now().strftime('%Y-%m-%d %H:%M:%S') }
    return jsonify(data)

if __name__ == '__main__':
    app.run(debug=True)

Like I said, nothing fancy at all.

Now, because this Flask application will run on Google App Engine Standard (GAE), we have to use Python 2 (NOTE: at the time of writing, anyway). So let’s get the code, and setup our Python 2 environment.

$ git clone https://github.com/scrollocks/locust-loadtesting.git
$ cd locust-loadtesting
$ virtualenv -p python2 .venv2 
$ source .venv2/bin/activate
$ cd locust-loadtesting/flask-app

Because we’re running on GAE we have to package any 3rd-party libraries into the application directory. Easily done using pip’s -t option:

$ pip install -r requirements.txt -t lib

That should be all we need to get this running locally, so lets give it a try:

$ dev_appserver.py dummy_app.yaml

You should see something like this if it’s running properly:

$ dev_appserver.py dummy_app.yaml
INFO     2019-02-15 23:36:37,249 devappserver2.py:278] Skipping SDK update check.
INFO     2019-02-15 23:36:37,303 api_server.py:275] Starting API server at: http://localhost:34221
INFO     2019-02-15 23:36:37,310 dispatcher.py:256] Starting module "dummy" running at: http://localhost:8080
INFO     2019-02-15 23:36:37,310 admin_server.py:150] Starting admin server at: http://localhost:8000
INFO     2019-02-15 23:36:51,532 instance.py:294] Instance PID: 3643

In a browser, navigate to http://localhost:8080/ and you should also see a Hello, World! message.

Hello, world

The endpoint we’ll be hitting with Locust is http://localhost:8080/datetime. It should return the datetime as JSON.

Hello, world

Now we can deploy the dummy application to Google App Engine:

$ gcloud app deploy dummy_app.yaml

Almost too easy. Once deployed the following command can be run to navigate directly to the deployed app:

$ gcloud app browse

And that’s our dummy app done and deployed. Next!

Locust on Kubernetes

We’re about half-way there. Now let’s set up a Kubernetes cluster and deploy Locust to it. Navigate back to the cloned repository locust-loadtesting and set up a Python 3 environment:

$ cd ..
$ deactivate # Leave the Python 2 virtualenv
$ virtualenv -p python3 .venv3 # Set up a new Python 3 virtualenv
$ source .venv3/bin/activate # Switch to the Python 3 environment
$ pip install -r requirements.txt # Install libraries
$ cd locust/docker/locust # Navigate to where the Locust code is

The Locust code itself is very simple. We’re only defining 2 classes:

  • DummyLoadTester - an HttpLocust class - this allows us to make HTTP requests
  • DummyTaskSet - a TaskSet class - defines our HTTP calls

The code itself looks like this:

import os
import time
import logging
from locust import HttpLocust, TaskSet, task

logger = logging.getLogger(__name__)
logger.setLevel(os.getenv("LOCUST_LOG_LEVEL", "INFO").upper())

class DummyTaskSet(TaskSet):

    @task
    def get_dummy_datetime(self):
        time_start = time.time()
        response = self.client.get("/datetime")
        time_end = time.time()
        logger.info("Response - URL: {url}. Status code: {status}. "
                    "Latency: {duration}".format(url=response.url,
                                                 status=response.status_code,
                                                 duration=round(time_end - time_start, 3)))


class DummyLoadTester(HttpLocust):
    host = os.getenv("LOCUST_TARGET_HOST", "localhost:8080")
    task_set = DummyTaskSet
    min_wait = 5000
    max_wait = 15000

I won’t explain too much about the ins-and-outs of it here, so head over to the Locust documentation to learn more. The code will pick-up the host for hammering from the LOCUST_TARGET_HOST environment variable. This is defined in a Kubernetes deployment config (covered briefly later).

Running Locust Locally

So, as it is, everything should ‘just work’. You can run Locust locally against the dummy Flask app we created earlier to see how it operates.

Run the Flask App

In a separate terminal, navigate back to the flask-app directory and run it:

$ cd ../../../flask-app
$ dev_appserver.py dummy_app.yaml

Run Locust

You can then execute the following from the command-line to run locust with a web-UI at http://localhost:8089:

$ locust -f dummy_loadtest.py
[2019-02-16 12:29:00,303] jamie-desktop/INFO/locust.main: Starting web monitor at *:8089
[2019-02-16 12:29:00,303] jamie-desktop/INFO/locust.main: Starting Locust 0.9.0

Once you’ve input the number of clients and the spawn rate, you’ll see things start to happen. This is from the ‘charts’ page:

Locust running

Cool! We’ve got Locust running locally and hitting the /datetime end-point of our Flask app!

Docker Image

For us to be able to run Locust on Kubernetes we’ll need to bundle all the Locust logic up into a Docker image, and push it to Google Container Registry (GCR).

Execute the following and replace the repository path below (asia.gcr.io…) to an appropriate one for your region and project.

$ cd ../locust/docker/
$ docker build -f Dockerfile .
$ docker tag <HASH> asia.gcr.io/<PROJECT_ID>/locust
$ docker push asia.gcr.io/<PROJECT_ID>/locust

It’s worth mentioning here that the Dockerfile specifies the run.sh script in the Locust directory as the container entry-point. It starts Locust in either master, worker, or standalone modes.

Because Locust is a distributed framework, it needs a master node to coordinate all the worker nodes. When Kubernetes deploys Locust, it will create one master node and several worker ones as defined in the deployment.yaml file.

Anyway, now that’s done, we can start configuring Kubernetes to run it.

Kubernetes Configuration

Kubernetes is a whole massive subject area on it’s own, so I won’t delve into the ins and outs of it. There’s documentation here.

Let’s create a small GKE cluster of 2 nodes. The default is 3 but I’m using 2 here because 3 blew my CPU resource quota (I’m using the free tier). Set your compute/zone to whatever is sensible for where you are.

$ gcloud config set compute/zone australia-southeast1
$ gcloud container clusters create locust-gke --num-nodes=2
kubeconfig entry generated for locust-gke.
NAME        LOCATION              MASTER_VERSION  MASTER_IP     MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
locust-gke  australia-southeast1  1.11.6-gke.2    xx.xxx.x.xxx  n1-standard-1  1.11.6-gke.2  6          RUNNING

The number of nodes specified by --num-nodes (actual VMs) is different to what GKE reports as NUM_NODES (node components). This is documented here.

Next, connect to the cluster we just created:

$ gcloud beta container clusters get-credentials locust-gke \
--region australia-southeast1 \
--project <PROJECT_ID>

Now we can apply our deployment configuration (I.E. get the thing running on GKE).

An important thing to mention here is that the configuration defines 2 deployments; one for the Locust master, and one for the Locust worker. It also defines the LOCUST_TARGET_HOST environment variable that Locust will use as the host to target.

Once deployed, you should see something like this:

$ kubectl apply -f deployment.yaml 
deployment.apps "locust-master-deployment" created
deployment.apps "locust-worker-deployment" created
$ kubectl get pods
NAME                                        READY     STATUS    RESTARTS   AGE
locust-master-deployment-844cc549bc-ws8b6   1/1       Running   0          3m
locust-worker-deployment-7dbdbd88c8-65js7   1/1       Running   0          3m
locust-worker-deployment-7dbdbd88c8-7dxcn   1/1       Running   0          3m
locust-worker-deployment-7dbdbd88c8-rlkvp   1/1       Running   0          3m

Kubernetes has created 1 master and 3 worker pods. In the console, under ‘Workloads’ you should see the two deployments.

Locust GKE

Now we need to create a few services to allow communication. Locust requires ports 5557 and 5558 for communications between master and worker, and we need to be able to access the web UI on port 8089.

$ kubectl expose pod locust-master-deployment-844cc549bc-ws8b6 \
--type NodePort \
--port 5558 \
--target-port 5558 \
--name locust-5558
service "locust-5558" exposed
$ kubectl expose pod locust-master-deployment-844cc549bc-ws8b6 \
--type NodePort \
--port 5557 \
--target-port 5557 \
--name locust-5557
service "locust-5557" exposed
$ kubectl expose pod locust-master-deployment-844cc549bc-ws8b6 \
--type LoadBalancer \
--port 80 \
--target-port 8089 \
--name locust-web
service "locust-web" exposed

After a minute or two we’ll be able to see our exposed Locust services:

$ kubectl get services
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
kubernetes    ClusterIP      xx.xx.xxx.x     <none>          443/TCP          3m
locust-5557   NodePort       xx.xx.xxx.xx    <none>          5557:32384/TCP   3m
locust-5558   NodePort       xx.xx.xxx.xx    <none>          5558:32486/TCP   4m
locust-web    LoadBalancer   xx.xx.xxx.xxx   1.2.3.4         80:30948/TCP     3m

We can now connect to the Locust web UI at the external IP listed to run tests.

Locust GKE charts

That’s it! It’s a simplistic example, but next time you’re writing a web service (or site) and you want to know how well it’ll handle load, give Locust a go.