Background: I am using Scala, Akka and Kafka, and do not want a JVM per service. This is a brief debate about Akka in the world of Kafka Topologies…
A post about the difficult choice between actors and Kafka Processor Topologies.
On the one hand, Akka is cool - single threaded stateful units of work which scale as you add CPU’s to your tin.
On the other hand, Kafka topologies are cool, they scale over data centers without the micro-service devOps investment.
Service vs microservices
Microservices - lets define them as a service which can be rewritten by a new team of 3 in less than a week. They are cool because if you make them reactive event driven then the spec is the events in and out.
You can scale your teams as each can work on small units of functionality.
But, having 800 microservices deployed is monstrous, when some of them will do as little as confirming a username and password are correct.
I’m not convinced by containerised micro servervices. Staff attrition and brain drain could result in prod outages simply because of the complexity of explaining the run time environment - even with your devOps information radiator investments.
So, Kafka has a slightly different way to think about it. Don’t throw out the JVM, its still a great and robust runtime for JVM languages. Instead, chain your services together as single threaded stateful units within the JVM. Then, for free, they give you scaling, disaster recovery and so on.
But, Akka comes with its own threading model, and typeless Actors (wrote this in Aug 2017). But, Kafka comes with its own thread pool and a simple close down exception catcher which is, well, simple.
So, is there a role for Akka in a Kafka topology? I want there to be…Akka is a general framework, Kafka Topologies is a single technology.
Trying to use Actors in a processor topology
Design your software with a boostrap which creates all of your topology - any sink topics can delegate off to Akka. Any that are simply transformations with no asynch blocking should not be Actors. You should not have the simple Kafka Topology threads sending messages to the Akka threads and then blocking waiting for a result. On the other hand…
How about IO bound Async transformers?
Lets go back to the login processor.
- Input: Request for authentication of credentials
- Processor: Lets run a cassandra query which uses a future.
- On success you send out Logged on.
- On failure you send out Bad credentials.
This processor is IO bound, so you have a remote Cassandra query, which means your processor is already blocking - ie awaiting the result.
Does the processor Topology work as well as the Actor model?
To do this in Kafka topologies the simplest would be to do a query and block on the result (either because of the query API, or in your own code)
Given we write Actors, or use Futures, the query could run within Akka actors. But if it does it will have to ‘ask’ and wait on the future reply.
What about using someone elses library
There are Akka streaming libs - but they are older than the Topology API, and I’m not convinced they will keep up. Especially with the Lagom push, taking Akka out to compete with Restful MicroServices.
A mutant hybrid
I could use Akka, bootstrap the Topology prior to the Akka system.
Any failure in the Kafka topology could shut down the Akka system.
Any processor which is synchronous could just call Objects methods/lamdas.
Any processor which is asynchronous could message the Actors, and wait on a reply - using a Future.
This feels pretty horrible compared to using Actors for everything, but it does work. Its actually the same as would be done in a Rest API gateway - every async call uses a webserver thread while its waiting (actually not true if the API Gateway is written with Akka, but lets think Tomcat or something.)
Don’t throw away the Actor model just yet. Accept Topologies will run with their thread pool and Actors will run in the Akka execution context. Think about synchronous stateless codeblocks, and keep them out of Actors.
Set up the exception handler for the Kafka topology, and use that to close down the Actor System.
Any asynch or statefull processor should be an actor, and given that a processor topology is running multiple services in a JVM runtime then the Actor model will help you scale the asynch logic for the services as it will do all threading for you.
Containers can now be used, but they wrap topologies, containing ActorSystems as well as Kafka topologies.
One last advantage
If you design this right, then a really heavily used service (Kafka processor) could be easily removed from the topology and run in its own service.
Just make sure you have a multi-project project with each async service (Kafka processor) able to be released as its own jar (with main and boostrap), or be bundled with the topology.
Another advantage of coding it as a standalone service, and also an embeddable processor is you can have test harnesses for each service without having to build and stress the entire topology.
Which brings us back to a team of 3 rewriting in it in a week - the test harness becomes the contract.