Profiling golang microservices for high throughput on Kubernetes/Openshift clusters

Profiling golang microservices for high throughput on kubernetes/openshift clusters

I have developed a simple golang microservice that stores a JSON payload from a medical wearable photoplethysmograph device (measuring blood pressure, spo2, heart rate and resting heart rate using low-intensity infra-red light). The device connects to a Bluetooth gateway, which posts the JSON payload to a set URL (my microservice).

So I decided to see if my microservice can handle throughput, basically, I wanted to check when it breaks, check the requests per second and see if I can make any improvements. The first thing I needed was to capture the payload from the device (gateway) and store it in a JSON file. I also needed to produce a high load so I cloned this repo (https://github.com/cmpxchg16/gobench), made a slight modification (added content-type as application/JSON) and compiled it.

The next part was to include the necessary libraries to profile the microservice. Golang has a powerful profiler called pprof (net/http/pprof). I included it in and recompiled my microservice and started the service. 

A note before you go ahead and do profiling on any Kubernetes/OpenShift cluster, first run it past your DevOps/IT team. I had the privilege of bringing a cluster down in our AWS  hosted OpenShift installation, the logging agent (fluentd) pushed tons of data flooding elasticsearch (disk pressure – basically no space left on our devices).

Profiling golang microservices for high throughput on kubernetes/openshift clusters

So moving on …

Launch the profiler (I launched this locally). I also disabled calls to the database, the first thing I want is to see if the microservice can perform “as is” and then we can look at database read/writes.

Profiling golang microservices for high throughput on kubernetes/openshift clusters

I was feeling adventurous and wanted to see what happens by setting a million requests using 1000 concurrent users each creating 1000 post requests (from my captured JSON payload).

Profiling golang microservices for high throughput on kubernetes/openshift clusters

Not too bad for the first run 21276 requests/sec lets see if we can make improvements.

Using the profiler’s interactive mode I ran ‘top’

Profiling golang microservices for high throughput on kubernetes/openshift clusters

You can also get a graphical view (with the command ‘web’)

Profiling golang microservices for high throughput on kubernetes/openshift clusters

This doesn’t help me much but I did notice in the profiler output ‘encoding/json.Indent’ was about 5.74% of the cumulative reading. The next output showed more revealing details using the command ‘list VitalSignsHandler’, I have a breakdown of time spent in each part of the code.

Profiling golang microservices for high throughput on kubernetes/openshift clusters

Immediately I could see most of the time was spent in the logging (Info) and also the JSON.UnmarshalIndent of the payload. In production, we shouldn’t need all the info logging (only enable error logging). So that’s the first modification I did. The second was to look at the unmarshal functionality. Actually, I could do the response with a simple string and so I removed it from using a struct to string.

I was amazed at the results

From 21276 requests/sec to 62500 requests/sec that’s nearly a 300 percent improvement

I also looked at the heap profile (using the command with the flag -alloc-objects) to give me more specific details.

So with the profiler, gobench, some simple probing and small code changes I was able to improve the overall performance and memory usage of the microservice. I have more confidence in knowing that my microservice will scale and handle high throughput.

Some valuable lessons :

  • fmt.Sprintf in the logger uses precious cpu cycles – avoid unnecessary logging
  • Avoid unmarshalling of JSON to a struct. If not needed use simple string (JSON) responses
  • String concat is also heavy – use bytes.Buffer + buffer.WriteString/buffer.WriteByte
  • For small structs use the value rather than a pointer to the struct. For larger structs, a pointer would be ‘cheaper’

I have just looked at the tip of the iceberg, there are lots of other improvements I can make with more profiling (goroutines, block, and using benchmarking) but for me, the (80-20) rule applies.

Of course, we need to enable the database and try again. This gives us a good foundation or base. Any regression is now due to the database integration, and that’s another spider’s web.

Above all, I had loads of fun. Golang is awesome 🙂


GET STARTED WITH DEVOPS


How can organizations help development teams maximize value delivery

Author
Devinder Sharma
Product Owner | Ammeon

Creating an application template from an existing application

Ammeon | September 10, 2021

In this blog post we’ll be looking at how to take that running application and create an application template from it.  This will allow the whole application to form a simple repeatable deployment based on a few given parameters. 

READ MORE

Ammeon awarded Container Platform Specialist status by Red Hat

Ammeon | August 27, 2021

Red Hat have awarded us with Container Platform Specialist status. This has been awarded to us for our consistent high standards of Red Hat OpenShift delivery, as well as our specialist expertise and experience that we bring to projects. Ammeon has become one of Red Hat’s leading professional service partners across Europe and our work with OpenShift has been a major reason for this award. We design, build, deploy and manage OpenShift models for our customers across a range of …

READ MORE

How Can Flow Efficiency Improve Productivity

Ammeon | July 4, 2021

Flow efficiency examines the two basic components that make up your lead time: work and wait time.

READ MORE