Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add pprof and debug docs #72

Merged
merged 2 commits into from
Oct 30, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 27 additions & 3 deletions cmd/finch-daemon/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ import (
"syscall"
"time"

// #nosec
// register HTTP handler for /debug/pprof on the DefaultServeMux.
_ "net/http/pprof"

"github.com/containerd/containerd"
"github.com/containerd/nerdctl/pkg/api/types"
"github.com/containerd/nerdctl/pkg/config"
Expand Down Expand Up @@ -47,9 +51,10 @@ const (
)

type DaemonOptions struct {
debug bool
socketAddr string
socketOwner int
debug bool
socketAddr string
socketOwner int
debugAddress string
}

var options = new(DaemonOptions)
Expand All @@ -65,6 +70,7 @@ func main() {
rootCmd.Flags().StringVar(&options.socketAddr, "socket-addr", defaultFinchAddr, "server listening Unix socket address")
rootCmd.Flags().BoolVar(&options.debug, "debug", false, "turn on debug log level")
rootCmd.Flags().IntVar(&options.socketOwner, "socket-owner", -1, "Uid and Gid of the server socket")
rootCmd.Flags().StringVar(&options.debugAddress, "debug-addr", "", "")
if err := rootCmd.Execute(); err != nil {
log.Printf("got error: %v", err)
log.Fatal(err)
Expand Down Expand Up @@ -99,6 +105,24 @@ func run(options *DaemonOptions) error {
if err := os.Chown(options.socketAddr, options.socketOwner, options.socketOwner); err != nil {
return fmt.Errorf("failed to chown the finch-daemon socket: %w", err)
}

if options.debugAddress != "" {
logger.Infof("Serving debugging endpoint on %q", options.debugAddress)
go func() {
debugListener, err := net.Listen("tcp", options.debugAddress)
if err != nil {
logger.Fatal(err)
}
debugServer := &http.Server{
Handler: http.DefaultServeMux,
ReadHeaderTimeout: 5 * time.Second,
}
if err := debugServer.Serve(debugListener); err != nil && !errors.Is(err, http.ErrServerClosed) {
pendo324 marked this conversation as resolved.
Show resolved Hide resolved
logger.Fatal(err)
}
}()
}

server := &http.Server{
Handler: r,
ReadHeaderTimeout: 5 * time.Minute,
Expand Down
40 changes: 40 additions & 0 deletions docs/debug.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Debugging the finch daemon

This document outlines where to find/access logs and how to configure profiling tools for finch daemon.

## Logs

Logs are the first place to check when you suspect a problem with finch-daemon. If `finch-daemon` was started via `systemd` then you can obtain logs using `journalctl`:

```shell
sudo journalctl -u finch
```

> **Note**
> The command above assumes that you have used the unit file definition [finch.service](../finch.service) we have provided. If you have created your own unit file for `finch-daemon` and replace `finch-daemon` with the one you have made. Amazon Linux distributions of Finch also use the name `finch` for the finch-daemon service.

If you have started `finch-daemon` manually, logs will either be emitted to stderr/stdout.

## CPU Profiling

We can use Golangs `pprof` tool to profile the snapshotter. To enable profiling you must set the `--debug-addr` CLI parameter when invoking `finch-daemon`:
pendo324 marked this conversation as resolved.
Show resolved Hide resolved

```shell
./finch-daemon --debug-addr localhost:6060
```

> **Note**
pendo324 marked this conversation as resolved.
Show resolved Hide resolved
> Similarly to adding the command line option for a local run of finch-daemon, any systemd service file can also be modified to include the `--debug-addr` option.


Once you have configured the debug address you can send a `GET` to the `/debug/pprof/profile` endpoint to receive a CPU profile of the snapshotter. You can specify an optional argument `seconds` to limit the results to a certain time span:

```shell
curl http://localhost:6060/debug/pprof/profile?seconds=40 > out.pprof
```

You can use the `pprof` tool provided by the Go CLI to visualize the data within a web browser:

```shell
go tool pprof -http=:8080 out.pprof
```
Loading