Getting started with Akka Cluster

In a previous part of this series about Akka we introduced the core abstraction provided by Akka: actors. Now we want to take a look at how these can be used in a cluster, i.e. in a distributed system.

Cluster Membership

The akka-cluster module provides a very simple cluster membership service which lays the foundations for all higher-level cluster features of Akka: A cluster is made up from collaborating actor systems called member nodes. It is important to understand that cluster membership happens at the level of actor systems, not individual actors. Nevertheless it can be used to build high-level cluster features which apply to individual actors, as we will see later.

It does not matter whether the member nodes reside on the same host or on different ones, yet in a typical production setting one would most probably spread the member nodes across multiple hosts to get scalability and resilience. Of course it is also possible to make use of container technologies like Docker or leverage resource managers like Apache Mesos .

In a nutshell, nodes can join an existing cluster and existing member nodes can leave deliberately or by failure. The following picture from the Akka Cluster documentation shows all the possible states a member node can have:

As Akka is distributed by default, all we have to do in order to make use of the cluster features is using the ClusterActorRefProvider instead of the default one. Hence we have to add the following configuration settings:

akka {
  actor {
    provider = "akka.cluster.ClusterActorRefProvider"
  }
}

In order to join a cluster, the joining actor system must have the same name like all the other member nodes:

val system = ActorSystem("some-system")

Last but not least, it is necessary to know the address of at least one of the existing member nodes. If these are known before starting the joining actor system, we can use seed nodes, either statically configured or provided via system properties.

akka {
  cluster {
    seed-nodes = ["akka.tcp://some-system@192.168.99.100:2552"]
  }
}

In a completely dynamic environment, though, it is necessary to employ a coordination service like etcd and actually join by using the Akka Cluster API – ConstructR is an open-source library that largely helps with this task.

Cluster Events

The membership service provided by akka-cluster allows us to monitor state transitions of member nodes – and some other information – by registering an actor as listener for ClusterDomainEvents.

Out of these the most interesting are MemberUp and MemberRemoved, both of type MemberEvent as well as UnreachableMember and ReachableMember, both of type ReachabilityEvent.

Here’s an example keeping track of running member nodes where running means either of the two states Joining or Up:

1object ClusterView {
2 
3  case object GetMemberNodes
4 
5  final val Name = "cluster-view"
6 
7  def props: Props = Props(new ClusterView)
8}
9 
10class ClusterView extends Actor with ActorLogging {
11  import ClusterEvent._
12  import ClusterView._
13 
14  private var members = Set.empty[Address]
15 
16  Cluster(context.system).subscribe(self, InitialStateAsEvents, classOf[MemberEvent])
17 
18  override def receive = {
19    case GetMemberNodes =>
20      sender() ! members
21 
22    case MemberJoined(member) =>
23      log.info("Member joined: {}", member.address)
24      members += member.address
25 
26    case MemberUp(member) =>
27      log.info("Member up: {}", member.address)
28      members += member.address
29 
30    case MemberRemoved(member, _) =>
31      log.info("Member removed: {}", member.address)
32      members -= member.address
33  }
34}

In addition to what is demonstrated in this example, the Member class also holds other useful information, e.g. the – possibly empty – set of cluster roles which can be assigned to member nodes via configuration.

Cluster-aware Routers

How can the membership service be used to provide cluster features which apply to individual actors? One example are cluster-aware routers, which are also part of the akka-cluster module.

In akka-actor a router is sort of a proxy that forwards messages to a certain set of actors called routees. Routers come in two flavors: either as pool routers which create routees as child actors, or as group routers which use existing routees. Cluster-aware routers simply make it possible to create or use routees on remote member nodes.

Building upon the publish-subscribe example from the previous part we are now going to extend the PubSubMediator to work in a distributed setting, too. This can easily be achieved by using a cluster-aware BroadcastGroup router connecting the PubSubMediator peers running on each member node and using that to forward each valid Publish command to all peers.

Here are the most interesting parts of the extended PubSubMediator:

1object PubSubMediator {
2 
3  ...
4 
5  private[pubsub] def peersGroup = ClusterRouterGroup(
6    BroadcastGroup(Nil),
7    ClusterRouterGroupSettings(
8      totalInstances = Int.MaxValue,
9      routeesPaths = List(s"/user/$Name"),
10      allowLocalRoutees = false,
11      useRole = None
12    )
13  )
14}
15 
16class PubSubMediator extends Actor {
17  import PubSubMediator._
18 
19  private val peers = createPeers()
20 
21  ...
22 
23  override def receive = {
24    case publish @ Publish(topic, message) =>
25      subscribers(topic).foreach(_ ! message)
26      peers ! publish
27      sender() ! Published(publish)
28 
29    ...
30  }
31 
32  protected def createPeers(): ActorRef = context.actorOf(peersGroup.props(), "peers")
33}

As you can see, we create the peers actor by using a ClusterRouterGroup which is initialized with an empty BroadcastGroup and configured to look up actors on remote member nodes only under the path s”/user/$Name”. The full code for the PubSubMediator can be accessed on GitHub .

Conclusion

We introduced the cluster membership service provided by the akka-cluster module and showed how the ClusterDomainEvents can be used to keep track of the member nodes and their status. This is the groundwork for any other cluster feature in Akka. We also showed cluster-aware routers as one application of cluster membership service.

Stay tuned for follow-up posts covering higher-level cluster features and more. As always, questions and feedback are highly appreciated.