I have developed a simple golang microservice that stores a JSON payload from a medical wearable photoplethysmograph device (measuring blood pressure, spo2, heart rate and resting heart rate using low-intensity infra-red light). The device connects to a Bluetooth gateway, which posts the JSON payload to a set URL (my microservice).
So I decided to see if my microservice can handle throughput, basically, I wanted to check when it breaks, check the requests per second and see if I can make any improvements. The first thing I needed was to capture the payload from the device (gateway) and store it in a JSON file. I also needed to produce a high load so I cloned this repo (https://github.com/cmpxchg16/gobench), made a slight modification (added content-type as application/JSON) and compiled it.
The next part was to include the necessary libraries to profile the microservice. Golang has a powerful profiler called pprof (net/http/pprof). I included it in and recompiled my microservice and started the service.
A note before you go ahead and do profiling on any Kubernetes/OpenShift cluster, first run it past your DevOps/IT team. I had the privilege of bringing a cluster down in our AWS hosted OpenShift installation, the logging agent (fluentd) pushed tons of data flooding elasticsearch (disk pressure – basically no space left on our devices).
So moving on …
Launch the profiler (I launched this locally). I also disabled calls to the database, the first thing I want is to see if the microservice can perform “as is” and then we can look at database read/writes.
I was feeling adventurous and wanted to see what happens by setting a million requests using 1000 concurrent users each creating 1000 post requests (from my captured JSON payload).
Not too bad for the first run 21276 requests/sec lets see if we can make improvements.
Using the profiler’s interactive mode I ran ‘top’
You can also get a graphical view (with the command ‘web’)
This doesn’t help me much but I did notice in the profiler output ‘encoding/json.Indent’ was about 5.74% of the cumulative reading. The next output showed more revealing details using the command ‘list VitalSignsHandler’, I have a breakdown of time spent in each part of the code.
Immediately I could see most of the time was spent in the logging (Info) and also the JSON.UnmarshalIndent of the payload. In production, we shouldn’t need all the info logging (only enable error logging). So that’s the first modification I did. The second was to look at the unmarshal functionality. Actually, I could do the response with a simple string and so I removed it from using a struct to string.
I was amazed at the results
From 21276 requests/sec to 62500 requests/sec that’s nearly a 300 percent improvement.
I also looked at the heap profile (using the command with the flag -alloc-objects) to give me more specific details.
So with the profiler, gobench, some simple probing and small code changes I was able to improve the overall performance and memory usage of the microservice. I have more confidence in knowing that my microservice will scale and handle high throughput.
Some valuable lessons :
- fmt.Sprintf in the logger uses precious cpu cycles – avoid unnecessary logging
- Avoid unmarshalling of JSON to a struct. If not needed use simple string (JSON) responses
- String concat is also heavy – use bytes.Buffer + buffer.WriteString/buffer.WriteByte
- For small structs use the value rather than a pointer to the struct. For larger structs, a pointer would be ‘cheaper’
I have just looked at the tip of the iceberg, there are lots of other improvements I can make with more profiling (goroutines, block, and using benchmarking) but for me, the (80-20) rule applies.
Of course, we need to enable the database and try again. This gives us a good foundation or base. Any regression is now due to the database integration, and that’s another spider’s web.
Above all, I had loads of fun. Golang is awesome 🙂
Technical Architect | Ammeon