Recently I needed to profile a Go application running inside a Kubernetes pod using net/http/pprof. I got stuck for a while trying to figure out how to copy the profile file from a pod but there is an easier way.
net/http/pprof – A Short Intro
First, a little about profiling in Go. net/http/pprof is a library for profiling live Go applications and exposing the profiling data via HTTP. If you want to profile an application, it needs to be instrumented before profiling. Here are some articles that describe that process:
Once you have instrumented your application, you just need to be able to access it from outside of a Kubernetes cluster.
Kubernetes and Pprof
The easiest way to get at the application in the pod is to use port forwarding with kubectl.
kubectl port-forward pod-123ab -n a-namespace 6060
The HTTP endpoint will now be available as a local port.
You can now generate the file for the CPU profile with curl and pipe the data to a file (7200 seconds is two hours):
curl "http://127.0.0.1:6060/debug/pprof/profile?seconds=7200" > cpu.pprof
It is also possible to send the data directly to the pprof tool. The pprof tool (not the same thing as the net/http/prof library) is a tool for generating a pdf or svg analysis from the profile data.
To save the pprof data with the pprof tool you can use the interactive mode:
go tool pprof http://localhost:8282/debug/pprof/profile
And per default the generated data will be saved as a tar file in the pprof subdirectory in your home directory. Exit interactive mode by typing exit.
Congratulations, you now have the raw profile from your application from inside a Kubernetes pod!
Some bonus information on the pprof tool
To generate an analysis, you will need the binary file for your Go application.
Here is how to pipe the profile data in directly:
go tool pprof --pdf your-binary-file http://localhost:8282/debug/pprof/profile > profile.pdf
go tool pprof --svg your-binary-file http://localhost:8282/debug/pprof/profile > profile.svg
You can do a lot more with net/http/pprof and the pprof tool.
Memory profile for in use space:
go tool pprof --pdf your-binary-file http://localhost:8282/debug/pprof/heap > in-use-space.pdf
Memory profile for allocated objects:
1. Generate the data:
go tool pprof http://localhost:8282/debug/pprof/heap
2. Exit interactive mode by typing exit.
3. Analyse the data:
go tool pprof -alloc_objects --svg your-binary-file /home/username/pprof/pprof.localhost:6060.alloc_objects.alloc_space.003.pb.gz > alloc-objects.svg
There are also switches for in use object counts (-inuse_objects) and allocated space (-alloc_space).
The pprof tool has an interactive mode that has lots of nifty functions like topn. Read more about that on the official Go blog.
danlimerick 