hystrix-go

Hystrix is a great project from Netflix.

Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.

I think the Hystrix patterns of programmer-defined fallbacks and adaptive health monitoring are good for any distributed system. Go routines and channels are great concurrency primitives, but don't directly help our application stay available during failures.

hystrix-go aims to allow Go programmers to easily build applications with similar execution semantics of the Java-based Hystrix library.

For more about how Hystrix works, refer to the Java Hystrix wiki

For API documentation, refer to GoDoc

How to use

import "github.com/wallstreetcn/hystrix-go/hystrix"

Execute code as a Hystrix command

Define your application logic which relies on external systems, passing your function to hystrix.Go. When that system is healthy this will be the only thing which executes.

hystrix.Go("my_command", func() error {
	// talk to other services
	return nil
}, nil)

Defining fallback behavior

If you want code to execute during a service outage, pass in a second function to hystrix.Go. Ideally, the logic here will allow your application to gracefully handle external services being unavailable.

This triggers when your code returns an error, or whenever it is unable to complete based on a variety of health checks.

hystrix.Go("my_command", func() error {
	// talk to other services
	return nil
}, func(err error) error {
	// do this when services are down
	return nil
})

Waiting for output

Calling hystrix.Go is like launching a goroutine, except you receive a channel of errors you can choose to monitor.

output := make(chan bool, 1)
errors := hystrix.Go("my_command", func() error {
	// talk to other services
	output <- true
	return nil
}, nil)

select {
case out := <-output:
	// success
case err := <-errors:
	// failure
}

Synchronous API

Since calling a command and immediately waiting for it to finish is a common pattern, a synchronous API is available with the hystrix.Do function which returns a single error.

err := hystrix.Do("my_command", func() error {
	// talk to other services
	return nil
}, nil)

Configure settings

During application boot, you can call hystrix.ConfigureCommand() to tweak the settings for each command.

hystrix.ConfigureCommand("my_command", hystrix.CommandConfig{
	Timeout:               1000,
	MaxConcurrentRequests: 100,
	ErrorPercentThreshold: 25,
})

You can also use hystrix.Configure() which accepts a map[string]CommandConfig.

Enable dashboard metrics

In your main.go, register the event stream HTTP handler on a port and launch it in a goroutine. Once you configure turbine for your Hystrix Dashboard to start streaming events, your commands will automatically begin appearing.

hystrixStreamHandler := hystrix.NewStreamHandler()
hystrixStreamHandler.Start()
go http.ListenAndServe(net.JoinHostPort("", "81"), hystrixStreamHandler)

Send circuit metrics to Statsd

c, err := plugins.InitializeStatsdCollector(&plugins.StatsdCollectorConfig{
	StatsdAddr: "localhost:8125",
	Prefix:     "myapp.hystrix",
})
if err != nil {
	log.Fatalf("could not initialize statsd client: %v", err)
}

metricCollector.Registry.Register(c.NewStatsdCollector)

Aggregator for multiple streams

Use Turbine to aggregate multiple instances' metric to one. Turbine supports multiple kinds of service discovery [Consul, Etcd, Eureka] to discover hystrix streams.

FAQ

What happens if my run function panics? Does hystrix-go trigger the fallback?

No. hystrix-go does not use recover() so panics will kill the process like normal.

Build and Test

• Install vagrant and VirtualBox
• Clone the hystrix-go repository
• Inside the hystrix-go directory, run vagrant up, then vagrant ssh
• cd /go/src/github.com/wallstreetcn/hystrix-go
• go test ./...