Continuous profiling in production: always-on sampling profiles and what they answer

What it is

The Parca documentation defines continuous profiling as taking profiles (CPU, memory, I/O and others) of programs in a systematic way, then collecting, storing and querying them over time. The stored unit is a profile series identified by profile type and key/value labels and queried with a label-selector language. Parca explains why this can run permanently: it uses sampling profiling, which records a stack trace at intervals rather than instrumenting every call, and therefore has little enough overhead to be always on in production. Grafana Pyroscope describes itself the same way, as a continuous profiling aggregation system that correlates profiles with metrics, logs and traces.

Profiles come from three kinds of sources: runtime endpoints (Go's net/http/pprof registers handlers under /debug/pprof/ that serve CPU profiles for a requested number of seconds, heap, goroutine, block and mutex profiles), SDKs that push profiles from inside the process, and eBPF agents that profile every process on a host without changes to the application.

Why it matters

A one-off profile answers "where is time spent now". A continuous store answers the questions that come up in operations: what changed between the version deployed yesterday and today, which function costs the most CPU summed across all instances, why memory grows over a week, and what the process was doing at 03:12 when latency rose. Those questions cannot be answered by taking a profile after the fact.

How to apply

• Start with CPU and allocation profiles for the services that dominate spend or latency; add mutex and block profiles when contention is suspected (Go serves them only after runtime.SetBlockProfileRate or runtime.SetMutexProfileFraction has been called).
• Label profiles with service, version and instance so two versions can be compared as a difference of two flame graphs.
• Bind runtime profiling endpoints to localhost or a private port; the Go example serves them on localhost:6060, and a scraper reaches them through the private network.
• Keep raw profile retention short and rely on the aggregated series for trends.
• Verify the overhead on a staging copy under realistic load before enabling fleet-wide, and record the measurement with the rollout.
• Link from a latency alert's time window to the profile of the same window.

Pitfalls

Profiling endpoints exposed publicly leak code structure and allow anyone to trigger expensive profile collection. Stripped binaries or missing frame pointers produce stacks without names. Sampling profilers under-represent short-lived processes and single rare events; they show where time goes on average, not why one request was slow, which is a question for tracing.