Chapter 7: Remoting
The power of Akka is hopefully apparent to you by now, but really, we've only just begun to scratch the surface. We've already highlighted the fact that Akka brings great flexibility and resiliency for problems involving concurrency, but so far we've only focused on running Akka at the single machine level. Bringing additional machines into the picture is where the capabilities of Akka really show their strongest side.
Akka's support for remoting is where the strength of the message passing approach is perhaps best highlighted. This, and the fundamental focus on making everything in Akka support remote actors as easily as local actors, makes a dramatic difference in your ability to take a single system application and transform it into a distributed system without extreme contortions. That alone is far from typical in usual enterprise frameworks that most of us have been subjected to over the years.
This ability to transparently interact between actors that may be local or remote without having to change the underlying code, is referred to as location transparency, as mentioned in the first chapter. Building your applications to take advantage of this can be driven almost entirely from a configuration-based approach. In this chapter we'll look at how to get started with Akka's remoting module and how to go about getting to the point where location transparency is just an assumed feature of your system.
Setting up remoting
We've used the most basic of the libraries that Akka provides. In order to get started with Akka remoting, you'll need to add the following to your build definition:
libraryDependencies += "com.typesafe.akka" %% "akka-remote" % "2.1.0"
If you're running in the sbt console, don't forget to run reload so that it reloads the updated build configuration. The next time you try to compile anything, sbt will see that there is a new dependency and attempt to retrieve it.
The other key addition is to your Akka configuration. These additions will enable remote access from other Akka instances. It's important to note that the akka.remote.netty.hostname setting needs to reference a hostname that any remote Akka instances will be able to resolve. In many of the Akka tutorials you'll see online, this field is set to localhost or 127.0.0.1, which causes needless frustration when some hapless developer is trying to learn how to use Akka remoting and can't figure out why two instances can't talk to each other. Make sure that if you use a hostname here (and there's no reason not to — I'm using IP addresses for convenience) it is globally addressable by all Akka instances.
The port number specified can also vary, as needed. In particular, if you want to try running multiple Akka instances on a single machine, they will need to use unique port numbers to avoid errors. Here I use 2552, but if I were to run another instance of an Akka remoting-based application on the same machine, I would need to use a different port.
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
transport = "akka.remote.netty.NettyRemoteTransport"
netty {
hostname = "192.168.5.5"
port = 2552
}
}
} Looking up and creating actors remotely
Now that we have the configuration in place, actually using remote actors is incredibly simple. Sending a message to a remote actor requires simply knowing the actor's address. This code looks up an actor called remoteActor that lives on the host with IP 192.168.5.5 using port 2552, with an actor system named remoteActorSystem:
val remoteActor = context
.actorFor("akka://[email protected]:2552/user/remoteActor")
remoteActor ! SomeMessage("hello!")
Similarly, you can create an actor running on a remote node with just a bit more work:
import akka.actor.{ Props, Deploy, AddressFromURIString }
import akka.remote.RemoteScope
val remoteAddress =
AddressFromURIString("akka://[email protected]:2552/user/remoteActor")
val newRemoteActor = context.actorOf(Props[MyRemoteActor]
.withDeploy(Deploy(scope = RemoteScope(remoteAddress))))
There is an additional requirement here, though, which is that the class MyRemoteActor that's being instantiated here needs to be available to the runtime in both the local and remote JVM instances.
For convenience, you can also map out the remote addresses in the configuration. This would be the equivalent configuration for the previous example:
akka.actor.deployment {
/remoteActor {
remote = "akka://[email protected]:2552"
}
} With this configuration in place, the previous code is simplified:
val newRemoteActor = context.actorOf(Props[MyRemoteActor],
name = "remoteActor")
You should notice that this looks just like code we've used previously to start up actors. In other words, we're creating a remote actor without the code having to take the location into account. This is truely location transparency.
Remoting with routers
Routers are also able to take advantage of remoting successfully, as you might imagine. This is done via some additional configuration directives that I didn't cover in the Routing chapter. A simple example showing how remoting and routers can be combined would be a broadcaster that allows for broadcasting messages to a set of remote actors.
On the target remote nodes, you might define the receiving actors as follows:
import akka.actor._
class ReceiverActor extends Actor {
def receive = {
case msg => println(msg)
}
}
Then, to send a message to each of these nodes concurrently, you would start up your router, to which you can then send a message, which will appear on each of the remotes:
val broadcaster = context.actorOf(Props[BroadcastReceivingActor]
.withRouter(FromConfig()), name = "broadcaster")
broadcaster ! "hello, remote nodes!"
The configuration that tells this router to use a BroadcastRouter with remote nodes is very simple:
akka.actor.deployment {
/broadcaster {
router = broadcast
nr-of-instances = 3
target {
nodes = [
"akka://[email protected]:2552",
"akka://[email protected]:2552",
"akka://[email protected]:2552"
]
}
}
} Scatter-gather across a set of remote workers
Returning to the earlier example application, let's add some handy functionality to our system that grabs pages we bookmark. I'm going to use a very simplified approach that's not really ideal, but that will at least get the idea across, and then show you how you can use remoting effectively here to help ensure you actually retrieve the page successfully. First, here's the new Crawler actor that I'll be using to grab the pages:
import akka.actor.Actor
import io.Source
object Crawler {
case class RetrievePage(bookmark: Bookmark)
case class Page(bookmark: Bookmark, content: String)
}
class Crawler extends Actor {
import Crawler.{RetrievePage, Page}
def receive = {
case RetrievePage(bookmark) =>
val content = Source.fromURL(bookmark.url).getLines().mkString(" ")
sender ! Page(bookmark, content)
}
}
There's not much to this. It just receives a request to retrieve a page using the URL in a given bookmark and then sends the response back to the original requestor. There are a few different ways we could hook this into our existing application, but keeping with the theme of simplicity, I'll just hook it in to the code that stores the bookmark.
class BookmarkStore(database: Database[Bookmark, UUID]) extends Actor {
import BookmarkStore.{GetBookmark, AddBookmark}
import Crawler.{RetrievePage, Page}
val crawler = context.actorOf(Props[Crawler])
def receive = {
case AddBookmark(title, url) =>
val bookmark = Bookmark(title, url)
database.find(bookmark) match {
case Some(found) => sender ! None
case None =>
database.create(UUID.randomUUID, bookmark)
sender ! Some(bookmark)
crawler ! RetrievePage(bookmark)
}
case GetBookmark(uuid) =>
sender ! database.read(uuid)
case Page(bookmark, pageContent) =>
database.find(bookmark).map {
found =>
database.update(found._1, bookmark.copy(content = Some(pageContent)))
}
}
} The main items to note here are where I send the RetrievePage message to the crawler and then, later, when we get the Page response back, we go look the bookmark back up from the database and add the freshly retreived contents to it. This is all well and good, but imagine that you had a system that was not very reliable at pulling these page contents, and you wanted a higher likelihood of actually getting the page contents successfully. My approach would be to use a ScatterGatherFirstCompletedRouter combined with a handful of remote nodes to farm out the request handling. A few little changes will show how this might work:
import akka.actor.{Props, Actor}
import java.util.UUID
import akka.routing.ScatterGatherFirstCompletedRouter
import scala.concurrent.duration._
import akka.pattern.{ask, pipe}
import akka.util.Timeout
class BookmarkStore(database: Database[Bookmark, UUID], crawlerNodes: Seq[String]) extends Actor {
import BookmarkStore.{GetBookmark, AddBookmark}
import Crawler.{RetrievePage, Page}
val crawlers = crawlerNodes.map(context.actorFor)
val crawlerRouter =
context.actorOf(Props.empty.withRouter(ScatterGatherFirstCompletedRouter(routees = crawlers,
within = 30 seconds)))
def receive = {
case AddBookmark(title, url) =>
val bookmark = Bookmark(title, url)
database.find(bookmark) match {
case Some(found) => sender ! None
case None =>
database.create(UUID.randomUUID, bookmark)
sender ! Some(bookmark)
import context.dispatcher
implicit val timeout = Timeout(30 seconds)
(crawlerRouter ? RetrievePage(bookmark)) pipeTo self
}
case GetBookmark(uuid) =>
sender ! database.read(uuid)
case Page(bookmark, pageContent) =>
database.find(bookmark).map {
found =>
database.update(found._1, bookmark.copy(content = Some(pageContent)))
}
}
} There are a few new things here that we'll cover in more detail in the next chapter on futures, but the idea here is that the scatter-gather approach is sending the request to each of the actors we've defined, which is determined by the crawlerNodes parameter. I'm assuming this will be passed in as a Seq of remote actor system addresses, for example: "akka://[email protected]:2553/user/crawler". The router will send the request to each node, waiting up to 30 seconds for a response. The first result it gets back is then sent to the BookmarkStore actor itself to process, just as it did earlier.
Gotchas and troubleshooting
The two most difficult things about working with remote actors are understanding when to use them and troubleshooting things when they aren't working. The first of these is not easy to answer, but understanding the characteristics of the problem you're trying to solve is a good place to start. You should ask yourself whether the system you're building either requires more resources than a single system can provide, if it necessarily requires being spread across multiple systems, or if there is some other design constraint that makes interaction between multiple actor systems a requirement. Perhaps the important thing to keep in mind is that you still need to focus on developing clear protocols for communicating between your actors. If you've designed a reasonably robust actor hierarchy, with appropriate supervisors and topologies for distributing work, nothing should need to change.
The point I want you to come away with is that working with remote actors should not generally be any different from working with local actors. You should be assuming your actors may fail, behave unexpectedly (did you really cover all the corner cases of that data parsing actor you wrote?), or whatever, no matter whether it is designed to run locally or remotely. Sure there will be differences, particularly if you are doing things that are implicitly designed around system-locality -- for example, accessing local files could fall into this bucket. But those are differences you are hopefully taking into account regardless of whether you're using actors or not. But with remoting and the ease of taking a local actor and transforming it into a remote one, Akka gives you one more reason to start designing with actors.
When things do go wrong while you're using remote actors, keep in mind that they are a network-based resource. So troubleshooting primarily falls into the same set of strategies you would use for troubleshooting any other network based resource. If your traffic appears to simply not be appearing on your remote system, start with the network layer and work up from there. Check any firewall rules; make sure any host names you use are actually resolving correctly and that TCP ports are not being used by other resources or services. And, of course, don't forget to check that things are actually plugged in!
There are also configuration options that Akka provides to help you debug the messages being sent between your remote systems. These go hand-in-hand with the same steps mentioned in chapter three. To be specific, you need to make sure your actors include logging functionality, by adding the akka.actor.ActorLogging trait to your actors and by using the akka.event.LoggingReceive wrapper around the receive block in your actor. The configuration directives are:
akka {
remote {
log-received-messages = on
log-sent-messages = on
}
}
You can also get a ton of debugging information by listening to Akka's EventBus, which is used internally by Akka to distribute internal messages. You can listen to either the remote server events, remote client events, or both, by listening for one of the event types RemoteServerLifeCycleEvent, RemoteClientLifeCycleEvent, or RemoteLifeCycleEvent. Each of these is actually part of a hierarchy of events you can listen for, but these top level events are often what you want when you're just debugging. To find information about the additional event types, see the documentation. Listening to this event bus is simply a matter of registering an actor and subscribing to the particular event class type you want to listen for (this is called a classifier in the documentation), as this example shows:
import akka.remote.RemoteLifeCycleEvent
val eventStreamListener = system.actorOf(Props(new Actor {
def receive = {
case msg => println(msg)
}
}))
system.eventStream.subscribe(eventStreamListener, classOf[RemoteLifeCycleEvent])
Wrap-up
Truthfully, we've only scratched the surface, both with remoting and Akka in general. But now you've seen how to put these tools into use and you should be able to take them from these simple examples to start building real applications. It's important to remember to use the resources you have available to you, so be sure to continue experimenting, reading and exploring with Akka. The best resource at your disposal as real experience.