autopgo

Autopgo is collection of components that create an automation pipeline for profile guided optimization, specifically for Go applications.

It consists of 3 main components:

Scraper - Used to periodically sample Go applications that expose pprof endpoints via the net/http/pprof package and forward them to the server component.
Server - Used to upload and download pprof profiles from the scraper or via the command-line.
Worker - Used to merge multiple profiles for the same application into a single base profile for use with the go build command.

See the section below for more information on the individual components

Components

This section outlines the responsibilities of each component and how their configuration/operation is managed.

Scraper

The scraper is responsible for performing regular samples of profiles across multiple applications. Typically, it will run alongside your applications. Depending on the configuration, it will periodically sample your applications for profiles and forward them to the server component. The scraper can handle sampling many different instances of your application concurrently.

Command

To run the scraper, use the following command:

autopgo scrape

Configuration

The scrape command accepts a single argument that is contextual depending on the mode specified via the --mode flag. The mode flag accepts file, kube, consul & nomad as values, defaulting to file.

The scrape command also accepts some command-line flags that may also be set via environment variables. They are described in the table below:

Flag	Environment Variable	Default	Description
`--log-level`, `-l`	`AUTOPGO_LOG_LEVEL`	`info`	Controls the verbosity of log output, valid values are `debug`, `info`, `warn` & `error`
`--api-url`, `-u`	`AUTOPGO_API_URL`	`http://localhost:8080`	The base URL of the profile server where scraped profiles will be sent
`--port`, `-p`	`AUTOPGO_PORT`	`8080`	Specifies the port to use for HTTP traffic
`--sample-size`, `-s`	`AUTOPGO_SAMPLE_SIZE`	None	Specifies the maximum number of targets to profile concurrently
`--app`, `-a`	`AUTOPGO_APP`	None	Specifies the the application name that profiles will be uploaded for
`--frequency`, `-f`	`AUTOPGO_FREQUENCY`	`60s`	Specifies the interval between profiling runs
`--duration`, `-d`	`AUTOPGO_DURATION`	`30s`	Specifies the amount of time a target will be profiled for
`--mode`, `-m`	`AUTOPGO_MODE`	`file`	What mode to run the scraper in (file, kube, nomad, consul)

File Mode

When the mode flag is set to file, the path to a JSON-encoded configuration file is expected as the argument. This file should be a JSON array of objects describing where the pprof endpoints are exposed.

[
  {
    // The scheme, host & port combination of the target.
    "address": "http://localhost:5000",
    // The path to the pprof profile endpoint, defaults to /debug/pprof/profile.
    "path": "/debug/pprof/profile"
  }
]

When using file mode, the file can be updated without restarting the scraper using a SIGHUP signal.

Kube Mode

When the mode flag is set to kube, the first argument becomes an optional path to a kubeconfig file. Scraping targets are then queried directly from the Kubernetes API. To run using an "in-cluster" configuration with the appropriate RBAC & service account, you can ignore the first argument.

To make your applications discoverable you must set the autopgo.scrape and autopgo.app labels & the autopgo.port annotation at the pod level. The table below describes each label/annotation supported by the scraper.

Key	Type	Example	Required	Description
`autopgo.scrape`	Label	`autopgo.scrape: "true"`	Yes	Informs the scraper that this is a scrape target.
`autopgo.scrape.app`	Label	`autopgo.app: "hello-world"`	Yes	Informs the scraper which application the profile belongs to.
`autopgo.scrape.port`	Annotation	`autopgo.port: "8080"`	Yes	Allows for specifying the port the application serves pprof endpoints on.
`autopgo.scrape.path`	Annotation	`autopgo.path: "/debug/pprof/profile"`	No	Allows for specifying the path to the pprof endpoint, defaults to /debug/pprof/profile.
`autopgo.scrape.scheme`	Annotation	`autopgo.scheme: "http"`	No	Informs the scraper whether the endpoint uses HTTP or HTTPS, defaults to HTTP.

It's important to note that in kube mode an individual scraper per-application is still required. This is done to keep sampling code simple and maximising uptime for profiling your applications individually.

Below is an example of a Kubernetes deployment that appropriately sets all labels & annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      name: example-app
      labels:
        app: example-app
        autopgo.scrape: "true"
        autopgo.scrape.app: "example-app"
      annotations:
        autopgo.scrape.path: "/debug/pprof/profile"
        autopgo.scrape.port: "8080"
        autopgo.scrape.scheme: "http"
    spec:
      containers:
        - name: example-app
          image: my-image

Nomad Mode

When running the scraper in nomad mode the first argument usually reserved for a configuration file is no longer required. If running the scraper against Nomad's services API you instead need to set the typical environment variables for any nomad client, such as NOMAD_ADDR etc. One key difference between this mode and kube mode is that the autopgo.port tag is not required as it can be obtained from the service itself.

The scraper will then source targets from Nomad's services API, searching for any services with appropriate tags added to their specification. The table below describes these tags and provides examples:

Tag	Example	Required	Description
`autopgo.scrape`	`autopgo.scrape=true`	Yes	Informs the scraper that this is a scrape target.
`autopgo.scrape.app`	`autopgo.app=hello-world`	Yes	Informs the scraper which application the profile belongs to.
`autopgo.scrape.path`	`autopgo.path=/debug/pprof/profile`	No	Allows for specifying the path to the pprof endpoint, defaults to /debug/pprof/profile.
`autopgo.scrape.scheme`	`autopgo.scheme=http`	No	Informs the scraper whether the endpoint uses HTTP or HTTPS, defaults to HTTP.

As in all other operating modes, a single scraper instance is required per application you wish to scrape. Below is an example of a Nomad job specification that contains a service with all usable tags:

job "example-app    {
  type = "service"
  group "example-app" {
    count = 1

    network {
      port "http" { to = 8080 }
    }

    task "example-app" {
      driver = "docker"

      config {
        image = "my-image:latest"
        ports = ["http"]
      }

      service {
        name = "example-app"
        port = "http"
        tags = [
          "autopgo.scrape=true",
          "autopgo.scrape.app=example-app",
          "autopgo.scrape.path=/debug/pprof/profile",
          "autopgo.scrape.scheme=http"
        ]
      }
    }
  }
}

Consul Mode

When running the scraper in consul mode the first argument usually reserved for a configuration file is no longer required. If running the scraper against the Consul service catalogue you instead need to set the typical environment variables for any consul client, such as CONSUL_ADDR etc. One key difference between this mode and kube mode is that the autopgo.port tag is not required as it can be obtained from the service itself.

The scraper will then source targets from the Consul service catalogue, searching for any services with appropriate tags added to their entry. The table below describes these tags and provides examples:

Tag	Example	Required	Description
`autopgo.scrape`	`autopgo.scrape=true`	Yes	Informs the scraper that this is a scrape target.
`autopgo.scrape.app`	`autopgo.app=hello-world`	Yes	Informs the scraper which application the profile belongs to.
`autopgo.scrape.path`	`autopgo.path=/debug/pprof/profile`	No	Allows for specifying the path to the pprof endpoint, defaults to /debug/pprof/profile.
`autopgo.scrape.scheme`	`autopgo.scheme=http`	No	Informs the scraper whether the endpoint uses HTTP or HTTPS, defaults to HTTP.

As in all other operating modes, a single scraper instance is required per application you wish to scrape.

Sampling

The sampling behaviour of the scraper is fairly simple. At the interval defined by the --frequency flag, a number of randomly selected targets (up to the maximum defined in the --sample-size flag) have their profiling endpoint called for the duration defined in the --duration flag.

These profiles are taken concurrently and streamed to the upstream profile server, whose base URL is defined via the --api-url flag.

Server

The server component runs as an HTTP server and handles inbound profiles from the scraper. Upon receiving a profile its validity is checked before being stored within the configured blob storage provider. Once stored, an event is published onto the configured event bus to notify the worker component that a profile is ready to be merged.

The server is also where merged profiles can be downloaded for use with the go build command.

Command

To run the server, use the following command:

autopgo server

Configuration

The server command accepts a number of command-line flags that may also be set via environment variables. They are described in the table below:

Flag	Environment Variable	Default	Description
`--log-level`, `-l`	`AUTOPGO_LOG_LEVEL`	`info`	Controls the verbosity of log output, valid values are `debug`, `info`, `warn` & `error`
`--event-writer-url`	`AUTOPGO_EVENT_WRITER_URL`	None	Specifies the event bus to use for publishing profile events. See the documentation on URLs for more details
`--blob-store-url`	`AUTOPGO_BLOB_STORE_URL`	None	Specifies the blob storage provider to use for reading & writing profiles. See the documentation on URLs for more details
`--port`, `-p`	`AUTOPGO_PORT`	`8080`	Specifies the port to use for HTTP traffic

Worker

The worker is responsible for handling events published by the server component that indicate new profiles have been uploaded. When it is notified of a new profile, it merges it with any existing profile for the application specified in the event payload. It reads and writes profile data to a configured blob storage provider.

Once merged, the worker publishes its own event to indicate a successful merge. When the worker receives the notification indicating a successful merge, the uploaded profile is deleted from blob storage.

Command

To run the worker, use the following command:

autopgo worker

Configuration

The worker command accepts a number of command-line flags that may also be set via environment variables. They are described in the table below:

Flag	Environment Variable	Default	Description
`--log-level`, `-l`	`AUTOPGO_LOG_LEVEL`	`info`	Controls the verbosity of log output, valid values are `debug`, `info`, `warn` & `error`
`--event-writer-url`	`AUTOPGO_EVENT_WRITER_URL`	None	Specifies the event bus to use for publishing profile events. See the documentation on URLs for more details
`--event-reader-url`	`AUTOPGO_EVENT_READER_URL`	None	Specifies the event bus to use for consuming profile events. See the documentation on URLs for more details
`--blob-store-url`	`AUTOPGO_BLOB_STORE_URL`	None	Specifies the blob storage provider to use for reading & writing profiles. See the documentation on URLs for more details
`--port`, `-p`	`AUTOPGO_PORT`	`8080`	Specifies the port to use for HTTP traffic
`--prune`	`AUTOPGO_PRUNE`	None	Specifies the location of the configuration file for profile pruning

Pruning

The worker supports pruning pprof profiles via a configuration file allowing multiple regular expressions per application to determine frames to drop and keep. This may be useful for managing the size of profiles as typically profiles will increase in size over time and eliminating code paths from profiles you are not interested in optimising can help keep those profile sizes low.

Larger profiles will typically cause longer build times, so pruning is something to consider as profiles grow.

To utilise pruning, provide the --prune flag with a location to a JSON file that describes the regular expressions to use. An example configuration is below:

[
  {
    // The name of the application whose profile you want to prune, this should match a value provided to a scraper via
    // the --app flag or AUTOPGO_APP environment variable.
    "app": "example",
    // The pruning rules applied when profiles for the application are merged
    "rules": [
      {
        // Remove all nodes below the node matching the "drop" regular expression.
        "drop": "github.com\/example\/.*",
        // Optionally, keep any nodes that would otherwise be dropped matching the "keep" regular expression.
        "keep": "github.com\/example\/example-dependency"
      }
    ]
  }
]

It is recommended to take a backup of your profile using the download command prior to applying a new pruning rule for the first time.

For specific details on how pruning works, see the implementation documentation.

Events

The server and worker components communicate via events published to and read from an event bus. This section describes the event structure and possible payloads. All events are JSON-encoded.

Envelope

Every event is wrapped with an envelope containing metadata about the event and its payload. Below is an example:

{
  // A UUID unique to the event.
  "id": "e9e23fa8-9d46-4c66-9190-2f64eacd73a2",
  // When the event was published.
  "timestamp": "2024-10-28T12:16:34.964Z",
  // The type of event, denotes the payload structure.
  "type": "profile.merged",
  // The payload contents.
  "payload": {}
}

Payloads

This section describes individual events published by autopgo components.

profile.uploaded

Event that indicates a new profile has been uploaded and is staged for merging.

{
  // The name of the application the profile is for.
  "app": "example-app",
  // The location of the profile in blob storage.
  "profileKey": "example-app/staging/1730075435311"
}

profile.merged

Event that indicates an uploaded profile has been merged with the base profile and can be deleted.

{
  // The name of the application the profile is for.
  "app": "example-app",
  // The location of the profile in blob storage.
  "profileKey": "example-app/staging/1730075435311",
  // The location of the base profile.
  "mergedKey": "example-app/default.pgo"
}

profile.deleted

Event that indicates the merged and any pending profiles have been deleted for an application.

{
  // The name of the application that has been deleted.
  "app": "example-app"
}

URL Configuration

The server and worker components both utilise URLs for configuring access to both blob storage and event buses. Currently, these URLs support:

Blob Storage:
- S3 (or any compatible S3 storage like Minio)
- Google Cloud Storage
- Azure Blob Storage
Event Buses:
- AWS SNS/SQS
- Azure Service Bus
- GCP Pub Sub
- Apache Kafka
- NATS

Where supported, messages published to event buses will use ordering/partition keys using the application name for the profile. Autopgo uses gocloud.dev to provide general access to a variety of cloud providers. Please see their documentation for pubsub and blob to determine the URL string and additional environment variables required to configure the server & worker components.

Utilities

This section outlines additional commands used to work with the main autopgo components.

Upload

The CLI provides an upload command that can be used to manually upload a pprof profile to the server to be merged with any existing profiles for that application.

Command

To upload a profile, use the following command, specifying the profile location as the only argument:

autopgo upload profile.pprof

Configuration

The upload command also accepts some command-line flags that may also be set via environment variables. They are described in the table below:

Flag	Environment Variable	Default	Description
`--log-level`, `-l`	`AUTOPGO_LOG_LEVEL`	`info`	Controls the verbosity of log output, valid values are `debug`, `info`, `warn` & `error`
`--api-url`, `-u`	`AUTOPGO_API_URL`	`http://localhost:8080`	The base URL of the profile server where the specified profile will be sent
`--app`, `-a`	`AUTOPGO_APP`	None	The name of the application the profile belongs to.

Download

The CLI provides a download command that can be used to download a merged profile from the server.

Command

To download a profile, use the following command, specifying the application name as the only argument:

autopgo download hello-world

Configuration

The upload command also accepts some command-line flags that may also be set via environment variables. They are described in the table below:

Flag	Environment Variable	Default	Description
`--log-level`, `-l`	`AUTOPGO_LOG_LEVEL`	`info`	Controls the verbosity of log output, valid values are `debug`, `info`, `warn` & `error`
`--api-url`, `-u`	`AUTOPGO_API_URL`	`http://localhost:8080`	The base URL of the profile server where the specified profile will be sent
`--output`, `-o`	`AUTOPGO_OUTPUT`	`default.pgo`	The location on the local file system to store the downloaded profile.

List

The CLI provides a list command that can be used to list all merged profiles managed by the server.

Command

To list profiles, use the following command:

autopgo list

Configuration

The list command also accepts some command-line flags that may also be set via environment variables. They are described in the table below:

Flag	Environment Variable	Default	Description
`--log-level`, `-l`	`AUTOPGO_LOG_LEVEL`	`info`	Controls the verbosity of log output, valid values are `debug`, `info`, `warn` & `error`
`--api-url`, `-u`	`AUTOPGO_API_URL`	`http://localhost:8080`	The base URL of the profile server where the specified profile will be sent

Delete

The CLI provides a delete command that can be used to delete a merged profile from the server.

Command

To download a profile, use the following command, specifying the application name as the only argument:

autopgo delete hello-world

Configuration

The delete command also accepts some command-line flags that may also be set via environment variables. They are described in the table below:

Flag	Environment Variable	Default	Description
`--log-level`, `-l`	`AUTOPGO_LOG_LEVEL`	`info`	Controls the verbosity of log output, valid values are `debug`, `info`, `warn` & `error`
`--api-url`, `-u`	`AUTOPGO_API_URL`	`http://localhost:8080`	The base URL of the profile server where the specified profile will be sent

Clean

The CLI provides a clean command that can be used to remove profiles larger than a specified size or that have not been updated for a specified duration.

Command

To perform a clean, use the following command, specifying either the --older-than or --larger-than flags (or both).

autopgo clean --older-than 24h --larger-than 10240

Configuration

The clean command accepts command-line flags that may also be set via environment variables. They are described in the table below:

Flag	Environment Variable	Default	Description
`--log-level`, `-l`	`AUTOPGO_LOG_LEVEL`	`info`	Controls the verbosity of log output, valid values are `debug`, `info`, `warn` & `error`
`--api-url`, `-u`	`AUTOPGO_API_URL`	`http://localhost:8080`	The base URL of the profile server where the specified profile will be sent
`--older-than`, `-d`	`AUTOPGO_OLDER_THAN`	None	How long a profile must not have been updated for to be eligible for cleaning
`--larger-than`, `-s`	`AUTOPGO_LARGER_THAN`	None	The minimum size (in bytes) a profile must be to be eligible for cleaning

Operations

This section contains information for use by those running the various autopgo components.

Health & Readiness

Each component exposes both health and readiness endpoints at the /api/health and /api/ready paths respectively. These can be used by your operator of choice to handle health and readiness checks. The health endpoint's response contains specifics on the individual dependencies of that component and its own health.

Below is an example health check response from the server component:

{
  "status": "healthy",
  "dependencies": [
    {
      "name": "blob",
      "status": "healthy"
    },
    {
      "name": "event-reader",
      "status": "healthy"
    },
    {
      "name": "event-writer",
      "status": "healthy"
    }
  ]
}

Health endpoints will return a 503 status code if one or more of the individual components are in an unhealthy state and a message field will be present with the error message received when checking that dependency's health.

Files

README.md

Latest commit

History

README.md

File metadata and controls

autopgo

Components

Scraper

Command

Configuration

File Mode

Kube Mode

Nomad Mode

Consul Mode

Sampling

Server

Command

Configuration

Worker

Command

Configuration

Pruning

Events

Envelope

Payloads

profile.uploaded

profile.merged

profile.deleted

URL Configuration

Utilities

Upload

Command

Configuration

Download

Command

Configuration

List

Command

Configuration

Delete

Command

Configuration

Clean

Command

Configuration

Operations

Health & Readiness