The Netflix stack, using Spring Boot - Part 2: Hystrix

Netflix has always been a proud contributor to the open source world. It's fascinating to see how each of their libraries facilitate a lot of tasks and can help create your development in a tremendous way.

In this series of blogposts - The Netflix stack, using Spring Boot - I'll be going over some of the libraries which Netflix has created and how to incorporate them in your spring applications. As always, it'll be more of a hands-on experience, as this blogpost will basically just be an overview of what you can find in the accompanying repository

Hystrix

Last week we showed how we can leverage the capabilities of Eureka to make our microservices discoverable. This week, we'll be looking at totally different, but extremely useful Library called Hystrix.

From the Netflix Hystrix Github repo:

Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.

In short: Hystrix is a properly written circuit breaker.

Circuit Breaker?

In the field of electronics, a circuit breaker is an automatically operated electrical switch designed to protect an electrical circuit from damage caused by overcurrent/overload or short circuit.

In the field of software development, the purpose of a circuit breaker isn't that much different. In theory, a circuit breaker is designed to automatically detect failures to access remote (or local) services and provide fallback mechanisms where needed.

Hystrix By Example

As I do in most of my blogposts regarding libraries or technologies, I try to incorporate them in a small project, to show you exactly how to set up, configure and use it.
I extended my repository to not only contain Eureka, but also Hystrix Examples.

As you'll see, we've added a new microservice: the api-service, which will be the main entry point for our application. The api-service will find and locate the necessary microservices using our Eureka-server and will perform calls on them. Hystrix will come into play when a microservice appears to be down, falling back to other methods and saving the state as an open circuit, so future calls know that this microservice is unreachable.

The configuration

I won't post the entire configuration of our application, as it would just bloat this blogpost with unnecessary code. If you'd like to see what an entire application in Spring Boot looks like, just head over to the repository to check it out.

build.gradle

compile 'org.springframework.cloud:spring-cloud-starter-hystrix'

Enabling Hystrix

If Hystrix is the only circuit breaker on the classpath, you can enable it by simply adding the next annotation to your main application class or a configuration class.

@EnableCircuitBreaker

If it's not the only one however, you can add

@EnableHystrix

And that's it. Hystrix is now enabled, but not really doing an awful lot. Let's change the logic of our application to use the behaviour of hystrix as a circuit breaker. Please note that it'll enable hystrix-javanica. Which is actually a wrapper around native Hystrix. Hystrix javanica has the benefits of giving us annotational support for Hystrix, using aspects. We won't cover all of the configurations you can do, because Hystrix Javanica has decent documentation.

Making methods circuit-aware

The default way to use hystrix, would be to annotate a method with ==@HystrixCommand==. In this annotation, you can define a method that can be called when the annotated method fails (read: throws an exception).

@HystrixCommand(fallbackMethod = "statusNotFound")
public InstanceStatus notificationsStatus() {
    return discoveryClient.getNextServerFromEureka("notification-service", false)
            .getStatus();
}

public InstanceStatus statusNotFound() {
    return InstanceStatus.DOWN;
}

Upon calling the method, Hystrix will determine whether the circuit is open or closed. If it's closed, it will will not execute the function, but instead route the flow to the fallback.

Asynchronous Calls?

@HystrixCommand also provides the ability to work with asynchronous requests.
Fallbacks to asynchronous commands can be both

• asynchronous
• sychronous

Note however that you can never have an asynchronous fallback to a synchronous HystrixCommand.

You have the ability to both use implementations of AsyncResult or Observable, which is provided by RxJava.

    @HystrixCommand(groupKey = "notification-service", fallbackMethod = "statusPageDown")
    public Observable<String> statusPageUrl() {
        return new ObservableResult<String>() {
            @Override
            public String invoke() {
                return discoveryClient.getNextServerFromEureka(SERVICE_NAME, false)
                        .getStatusPageUrl();
            }
        };
    }

    public String statusPageDown() {
        LOG.debug("Seems like our notifications-service is down: ");
        return "notification stuff down";
    }

Configuring our commands

You can configure your command keys, threadpools and more properties. We won't go into detail, as this is explained very well in the documentation and optional.

Monitoring Hystrix

Having an entire system in place that monitors our circuits certainly is nice, but monitoring is nothing without a visual representation where you can quickly grasp if something is wrong.

Hystrix can be monitored in a few ways, which we'll briefly discuss, as they all speak for themselves once you know where to look.

Spring Actuator - Hystrix Health Endpoint

If you enabled Hystrix in your microservice, Spring Actuator will automatically add the Hystrix Health to your application's health endpoint. (In our case, when running the api-service, it's configured to be found at http://localhost:9000/health

{ 
    [removed for brevity]
    hystrix: {
        status: "UP"
    }
}

The Hystrix Dashboard

When you added Hystrix-javanica, the application also provides us with an extra endpoint: an http-stream sending out all of the events concerning hystrix. You can find this endpoint by navigating to http://localhost:9000/hystrix.stream .

The good folks at Netflix created a dashboard on top of this endpoint. Simply add the dependency to your build path.

compile 'org.springframework.cloud:spring-cloud-starter-hystrix-dashboard'

and start your application with

@EnableHystrixDashboard

The dashboard can be found at http://localhost:9000/hystrix.

Below you can see two examples of a running application. One has a healthy hystrix circuit, the other one is open, resulting in a bypass of the failing call.

Closed

Open

Multiple Hystrix Endpoints

If you have multiple Hystrix endpoints, it can become a bit difficult to monitor the health of each and every single application. In one of my next blogposts, I'll be showing you how you can use Netflix Turbine to aggregate the server-sent events that are being emitted by the Hystrix streaming endpoint.

The Github Repository

As we said before, this is not just an ordinary blogpost. It's more of a guide on how to set up your environment to quickly start working with the discussed technology. That's why we're always making sure we have an accompanying github repository available, so people can easily see how it works and have a working example at hand.

The repository that's accompanying this blogpost is a bit different. Each time I'm releasing an new part of this series of blogposts, the repository will have a new branch, which will contain the new technology that will be discussed. In the end, I'll hope to come up with a nice example of how all the technologies can work together.

The repository can be found here.