Riak Core on Partisan on Elixir Tutorial: Introduction
This tutorial covers the practical aspects of building a distributed in memory key value store in Elixir using Riak Core, in this case a fork of it that uses Partisan as it's underlying distribution runtime instead of the default Distributed Erlang runtime.
This same tutorial can be used without change with standard Riak Core, I may add a chapter to set it up without partisan in the future, but it won't make much difference in terms of development.
What is Riak Core?
(From the Partisan website)
Riak Core is a distributed programming framework written in Erlang and based on the Amazon Dynamo system that influenced the design of the distributed database Riak, Apache Cassandra, and the distributed actor framework Akka.
In Riak Core, a distributed hash table is used to partition a hash space across a cluster of nodes. These virtual nodes—the division of the hash space into N partitions—are claimed by a node in the cluster, and the resulting ownership is stored in a data structure known as the ring that is periodically gossiped to all nodes in the cluster.
Requests for a given key are routed to a node in the cluster based on the current partitioning of virtual nodes to cluster nodes in the ring structure using consistent hashing, which minimizes the impact of reshuffling when nodes join and leave the cluster. Background processes are used for cluster maintenance; ownership handoff, (transferring virtual node ownership) metadata anti-entropy (an internal KVS for configuration metadata) and ring gossip (information about the cluster’s virtual node to node mapping).
What is Partisan?
(From the Partisan website)
Partisan is the design of an alternative runtime system for improved scalability and reduced latency in actor applications.
Partisan provides:
Better scalability by leveraging different network topologies for communication
Reduced latency through efficient parallel message scheduling for actor-to-actor communication
Partisan is provided as a user library in Erlang and achieves up to an order of magnitude increase in the number of nodes the system can scale to through runtime overlay selection, up to a 34.96x increase in throughput, and up to a 13.4x reduction in latency over Distributed Erlang.
Why Partisan?
I think it's an interesting project, and providing an easy way to set up a useful example on top of it may help more people learn about it, take advantage of it and maybe contribute to it.
Check the Scaling Riak Core section on the Partisan page for more information about the benefits of using Partisan.
I should mention that this fork is not the official one, so consider this if you want to put something based on this stack in production.
How to Follow this Tutorial
The tutorial provides the commands and code you need to add and change, so you can follow it by copying and pasting it in the right place.
It also provides a repository with one tag for each step, the tag corresponding to each step will be linked near the code changes.
Following the tutorial it's recommended that you don't change the project's name unless you want to make lot of renames and maybe get some errors if you forgot one. Of course for a real project based on this code you can rename it, usually you have to find and replace all mentions of 'civile', 'Civile' and 'civiledb'.
Here's the link to the project: https://gitlab.com/marianoguerra/civiledb/
Here's the link to the project tags: https://gitlab.com/marianoguerra/civiledb/-/tags
To follow the tutorial from the example repository you will need to have git installed, check the Git Website for instructions on how to install it.
You can clone the repository with the following command:
To change to another tag (replace start-1 with the tag you want to change to):
To go back to the last change:
Resources
This tutorial is based on many resources around the web, here's an incomplete list of them, some of them provide more details, theory and explanation, some are in Erlang, some don't run any more, but you can get something extra out of each of those.
Next post: Riak Core on Partisan on Elixir Tutorial: Setup.