Profiling golang microservices for high throughput on Kubernetes/Openshift clusters

Profiling golang microservices for high throughput on kubernetes/openshift clusters

I have developed a simple golang microservice that stores a JSON payload from a medical wearable photoplethysmograph device (measuring blood pressure, spo2, heart rate and resting heart rate using low-intensity infra-red light). The device connects to a Bluetooth gateway, which posts the JSON payload to a set URL (my microservice).

So I decided to see if my microservice can handle throughput, basically, I wanted to check when it breaks, check the requests per second and see if I can make any improvements. The first thing I needed was to capture the payload from the device (gateway) and store it in a JSON file. I also needed to produce a high load so I cloned this repo (https://github.com/cmpxchg16/gobench), made a slight modification (added content-type as application/JSON) and compiled it.

The next part was to include the necessary libraries to profile the microservice. Golang has a powerful profiler called pprof (net/http/pprof). I included it in and recompiled my microservice and started the service. 

A note before you go ahead and do profiling on any Kubernetes/OpenShift cluster, first run it past your DevOps/IT team. I had the privilege of bringing a cluster down in our AWS  hosted OpenShift installation, the logging agent (fluentd) pushed tons of data flooding elasticsearch (disk pressure – basically no space left on our devices).

Profiling golang microservices for high throughput on kubernetes/openshift clusters

So moving on …

Launch the profiler (I launched this locally). I also disabled calls to the database, the first thing I want is to see if the microservice can perform “as is” and then we can look at database read/writes.

Profiling golang microservices for high throughput on kubernetes/openshift clusters

I was feeling adventurous and wanted to see what happens by setting a million requests using 1000 concurrent users each creating 1000 post requests (from my captured JSON payload).

Profiling golang microservices for high throughput on kubernetes/openshift clusters

Not too bad for the first run 21276 requests/sec lets see if we can make improvements.

Using the profiler’s interactive mode I ran ‘top’

Profiling golang microservices for high throughput on kubernetes/openshift clusters

You can also get a graphical view (with the command ‘web’)

Profiling golang microservices for high throughput on kubernetes/openshift clusters

This doesn’t help me much but I did notice in the profiler output ‘encoding/json.Indent’ was about 5.74% of the cumulative reading. The next output showed more revealing details using the command ‘list VitalSignsHandler’, I have a breakdown of time spent in each part of the code.

Profiling golang microservices for high throughput on kubernetes/openshift clusters

Immediately I could see most of the time was spent in the logging (Info) and also the JSON.UnmarshalIndent of the payload. In production, we shouldn’t need all the info logging (only enable error logging). So that’s the first modification I did. The second was to look at the unmarshal functionality. Actually, I could do the response with a simple string and so I removed it from using a struct to string.

I was amazed at the results

From 21276 requests/sec to 62500 requests/sec that’s nearly a 300 percent improvement

I also looked at the heap profile (using the command with the flag -alloc-objects) to give me more specific details.

So with the profiler, gobench, some simple probing and small code changes I was able to improve the overall performance and memory usage of the microservice. I have more confidence in knowing that my microservice will scale and handle high throughput.

Some valuable lessons :

  • fmt.Sprintf in the logger uses precious cpu cycles – avoid unnecessary logging
  • Avoid unmarshalling of JSON to a struct. If not needed use simple string (JSON) responses
  • String concat is also heavy – use bytes.Buffer + buffer.WriteString/buffer.WriteByte
  • For small structs use the value rather than a pointer to the struct. For larger structs, a pointer would be ‘cheaper’

I have just looked at the tip of the iceberg, there are lots of other improvements I can make with more profiling (goroutines, block, and using benchmarking) but for me, the (80-20) rule applies.

Of course, we need to enable the database and try again. This gives us a good foundation or base. Any regression is now due to the database integration, and that’s another spider’s web.

Above all, I had loads of fun. Golang is awesome 🙂


GET STARTED WITH DEVOPS


How can organizations help development teams maximize value delivery

Author
Devinder Sharma
Product Owner | Ammeon

How Can Flow Efficiency Improve Productivity

Ammeon | July 4, 2021

Flow efficiency examines the two basic components that make up your lead time: work and wait time.

READ MORE

Case Study – DevOps Delivers for Service Providers

Ammeon | June 29, 2021

A global leader in consulting, deal advisory, tax, law and technology services engaged Ammeon to assist one of their internal research and development teams. Ammeon were entrusted to future-proof their business, drive efficiency and value to their business through implementing Agile and DevOps.

READ MORE

How can organizations help development teams maximize value delivery

Ammeon | June 17, 2021

A Product Owner is a value maximizer who maximizes the value of the development team’s work. Read more to know the 2 important areas we believe should be addressed in any effort to maximize the value of the team’s output

READ MORE