Mariano Guerra's Log (Posts about erlang)

Riak Core Tutorial Part 8: HTTP API

Mariano Guerra — Wed, 05 Sep 2018 19:18:14 GMT

The content of this chapter is in the 06-http-api branch.

https://gitlab.com/marianoguerra/tanodb/tree/06-http-api

How it Works

We are adding a simple HTTP API to our system, it will run on all nodes and allow us to interact with it from the outside.

We will use the Cowboy 2.0 HTTP Server.

Implementing it

We need to add cowboy as a dependency on rebar.config and tanodb.app.src.

Then in tanodb_app.erl we need to start the HTTP API:

setup_http_api() ->
  Dispatch = cowboy_router:compile([{'_', [{"/kv/:bucket/:key", tanodb_h_kv, []}]}]),

  HttpPort = application:get_env(tanodb, http_port, 8080),
  HttpAcceptors = application:get_env(tanodb, http_acceptors, 100),
  HttpMaxConnections = application:get_env(tanodb, http_max_connections, infinity),

  lager:info("Starting HTTP API at port ~p", [HttpPort]),

  {ok, _} = cowboy:start_clear(tanodb_http_listener,
    [{port, HttpPort},
     {num_acceptors, HttpAcceptors},
     {max_connections, HttpMaxConnections}],
    #{env => #{dispatch => Dispatch}}),

  ok.

We get the configuration from the environment, which is set by cuttlefish.

The API handler module tanodb_h_kv's main code:

init(ReqIn=#{method := <<"GET">>}, State) ->
    {Bucket, Key} = bindings(ReqIn),
    reply(ReqIn, State, tanodb:get(Bucket, Key));
init(ReqIn=#{method := <<"POST">>}, State) ->
    {Bucket, Key} = bindings(ReqIn),
    {ok, Value, Req1} = read_all_body(ReqIn),
    reply(Req1, State, tanodb:put(Bucket, Key, Value));
init(ReqIn=#{method := <<"DELETE">>}, State) ->
    {Bucket, Key} = bindings(ReqIn),
    reply(ReqIn, State, tanodb:delete(Bucket, Key)).

Trying it

Single Node

We are going to first test it on a single node.

Get key k1 in bucket b1, which doesn't exist yet:

curl -X GET  http://localhost:8098/kv/b1/k1

{
  "replies": [
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb@127.0.0.1",
      "ok": false,
      "partition": "1301649895747835411525156804137939564381064921088"
    },
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb@127.0.0.1",
      "ok": false,
      "partition": "1278813932664540053428224228626747642198940975104"
    },
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb@127.0.0.1",
      "ok": false,
      "partition": "1255977969581244695331291653115555720016817029120"
    }
  ]
}

Put key k1 in bucket b1 with content hi there:

curl -X POST  http://localhost:8098/kv/b1/k1 -d 'hi there'

{
  "replies": [
    {
      "node": "tanodb@127.0.0.1",
      "ok": true,
      "partition": "1301649895747835411525156804137939564381064921088"
    },
    {
      "node": "tanodb@127.0.0.1",
      "ok": true,
      "partition": "1278813932664540053428224228626747642198940975104"
    },
    {
      "node": "tanodb@127.0.0.1",
      "ok": true,
      "partition": "1255977969581244695331291653115555720016817029120"
    }
  ]
}

Get key k1 in bucket b1, which now exists:

curl -X GET  http://localhost:8098/kv/b1/k1

{
  "replies": [
    {
      "bucket": "b1",
      "key": "k1",
      "node": "tanodb@127.0.0.1",
      "ok": true,
      "partition": "1301649895747835411525156804137939564381064921088",
      "value": "hi there"
    },
    {
      "bucket": "b1",
      "key": "k1",
      "node": "tanodb@127.0.0.1",
      "ok": true,
      "partition": "1278813932664540053428224228626747642198940975104",
      "value": "hi there"
    },
    {
      "bucket": "b1",
      "key": "k1",
      "node": "tanodb@127.0.0.1",
      "ok": true,
      "partition": "1255977969581244695331291653115555720016817029120",
      "value": "hi there"
    }
  ]
}

Delete key k1 in bucket b1:

curl -X DELETE  http://localhost:8098/kv/b1/k1

{
  "replies": [
    {
      "node": "tanodb@127.0.0.1",
      "ok": true,
      "partition": "1301649895747835411525156804137939564381064921088"
    },
    {
      "node": "tanodb@127.0.0.1",
      "ok": true,
      "partition": "1278813932664540053428224228626747642198940975104"
    },
    {
      "node": "tanodb@127.0.0.1",
      "ok": true,
      "partition": "1255977969581244695331291653115555720016817029120"
    }
  ]
}

Get key k1 in bucket b1, which shouldn't exist anymore:

curl -X GET  http://localhost:8098/kv/b1/k1

{
  "replies": [
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb@127.0.0.1",
      "ok": false,
      "partition": "1301649895747835411525156804137939564381064921088"
    },
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb@127.0.0.1",
      "ok": false,
      "partition": "1278813932664540053428224228626747642198940975104"
    },
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb@127.0.0.1",
      "ok": false,
      "partition": "1255977969581244695331291653115555720016817029120"
    }
  ]
}

Cluster

We are going to test it on a cluster now, notice that the port changes, we are sending each request to a different node.

You can see each node's port on the logs at startup:

[info] Starting HTTP API at port 8198

Get key k1 in bucket b1, which doesn't exist yet:

curl -X GET  http://localhost:8198/kv/b1/k1

Notice the node name on the partition field, it may change for you depending on the state of handoff or how vnodes were distributed.

{
  "replies": [
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb2@127.0.0.1",
      "ok": false,
      "partition": "1301649895747835411525156804137939564381064921088"
    },
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb1@127.0.0.1",
      "ok": false,
      "partition": "1278813932664540053428224228626747642198940975104"
    },
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb1@127.0.0.1",
      "ok": false,
      "partition": "1255977969581244695331291653115555720016817029120"
    }
  ]
}

Put key k1 in bucket b1 with content hi there:

curl -X POST  http://localhost:8298/kv/b1/k1 -d 'hi there'

{
  "replies": [
    {
      "node": "tanodb1@127.0.0.1",
      "ok": true,
      "partition": "1255977969581244695331291653115555720016817029120"
    },
    {
      "node": "tanodb1@127.0.0.1",
      "ok": true,
      "partition": "1278813932664540053428224228626747642198940975104"
    },
    {
      "node": "tanodb2@127.0.0.1",
      "ok": true,
      "partition": "1301649895747835411525156804137939564381064921088"
    }
  ]
}

Get key k1 in bucket b1, which now exists:

curl -X GET  http://localhost:8398/kv/b1/k1

{
  "replies": [
    {
      "bucket": "b1",
      "key": "k1",
      "node": "tanodb1@127.0.0.1",
      "ok": true,
      "partition": "1278813932664540053428224228626747642198940975104",
      "value": "hi there"
    },
    {
      "bucket": "b1",
      "key": "k1",
      "node": "tanodb1@127.0.0.1",
      "ok": true,
      "partition": "1255977969581244695331291653115555720016817029120",
      "value": "hi there"
    },
    {
      "bucket": "b1",
      "key": "k1",
      "node": "tanodb2@127.0.0.1",
      "ok": true,
      "partition": "1301649895747835411525156804137939564381064921088",
      "value": "hi there"
    }
  ]
}

Delete key k1 in bucket b1:

curl -X DELETE  http://localhost:8198/kv/b1/k1

{
  "replies": [
    {
      "node": "tanodb2@127.0.0.1",
      "ok": true,
      "partition": "1301649895747835411525156804137939564381064921088"
    },
    {
      "node": "tanodb1@127.0.0.1",
      "ok": true,
      "partition": "1278813932664540053428224228626747642198940975104"
    },
    {
      "node": "tanodb1@127.0.0.1",
      "ok": true,
      "partition": "1255977969581244695331291653115555720016817029120"
    }
  ]
}

Get key k1 in bucket b1, which shouldn't exist anymore:

curl -X GET  http://localhost:8298/kv/b1/k1

{
  "replies": [
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb1@127.0.0.1",
      "ok": false,
      "partition": "1278813932664540053428224228626747642198940975104"
    },
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb1@127.0.0.1",
      "ok": false,
      "partition": "1255977969581244695331291653115555720016817029120"
    },
    {
      "bucket": "b1",
      "error": "not_found",
      "key": "k1",
      "node": "tanodb2@127.0.0.1",
      "ok": false,
      "partition": "1301649895747835411525156804137939564381064921088"
    }
  ]
}

Riak Core Tutorial Part 7: Coverage Calls

Mariano Guerra — Thu, 30 Aug 2018 11:25:52 GMT

The content of this chapter is in the `05-coverage` branch.

https://gitlab.com/marianoguerra/tanodb/tree/05-coverage

How it Works

Since bucket and key are hashed together to decide to which vnode a request will go it means that the keys for a given bucket may be distributed in multiple vnodes, and in case you are running in a cluster this means your keys are distributed in multiple physical nodes.

This means that to list all the keys from a bucket we have to ask all the vnodes for the keys on a given bucket and then put the responses together and return the set of all responses.

For this Riak Core provides something called coverage calls, which are a way to handle this process of running a command on all vnodes and gathering the responses.

In this chapter we are going to implement the tanodb:keys(Bucket) function using coverage calls.

In this case we call tanodb_coverage_fsm:start({keys, Bucket}, Timeout), which is a new module, it implements a behavior called riak_core_coverage_fsm, short for riak_core_coverage finite state machine, it implements some predefined callbacks that are called on different states of a finite state machine.

The start function calls tanodb_coverage_fsm_sup:start_fsm([ReqId, self(), Request, Timeout]) which starts a supervisor for this new process.

When we start the fsm with a command {keys, Bucket} and a timeout in milliseconds, it starts a supervisor that starts the finite state machine process, it first calls the init function which initializes the state of the process and returns some information to riak_core so it knows what kind of coverage call we want to do, then riak_core calls the handle_coverage function on each vnode and with each response it calls process_result in our process.

When all the results are received or if an error happens (such as a timeout) it will call the finish callback there we send the results to the calling process which is waiting for it.

The handle_coverage implementation is really simple, it uses the ets:match/2 function to match against all the entries with the given bucket and returns the key from the matched results.

You can read more about ets match specs in the match spec chapter on the Erlang documentation.

Implementing it

Code in tanodb.erl is really simple:

keys(Bucket, Opts) ->
    Timeout = maps:get(timeout, Opts, ?TIMEOUT),
    tanodb_coverage_fsm:start({keys, Bucket}, Timeout).

In tanodb_vnode.erl we need to implement the handle_coverage callback:

handle_coverage({keys, Bucket}, _KeySpaces, {_, RefId, _},
                State=#state{kv_state=KvState}) ->
    {Keys, KvState1} = tanodb_kv_ets:keys(KvState, Bucket),
    {reply, {RefId, Keys}, State#state{kv_state=KvState1}};

We add two new modules:

tanodb_coverage_fsm: The FSM implementation for the coverage call.
tanodb_coverage_fsm_sup: The supervisor for the FSM processes.

We also add the tanodb_coverage_fsm_sup to the tanodb_sup supervisor tree.

Trying it

Nums = lists:seq(1, 10).
Buckets = lists:map(fun (N) -> list_to_binary("bucket-" ++ integer_to_list(N)) end,
Nums).
Keys = lists:map(fun (N) -> list_to_binary("key-" ++ integer_to_list(N)) end, Nums).

GenValue = fun (Bucket, Key) -> [{bucket, Bucket}, {key, Key}] end.

lists:foreach(fun (Bucket) ->
        lists:foreach(fun (Key) ->
                Val = GenValue(Bucket, Key),
                tanodb:put(Bucket, Key, Val)
        end, Keys)
end, Buckets).

{ok, Items} = tanodb:keys(<<"bucket-1">>).
[{Partition, Node, Keys} || {Partition, Node, Keys} <- Items, Keys =/= []].

[{296867520082839655260123481645494988367611297792,
  'tanodb@127.0.0.1', [<<"key-10">>]},
 {365375409332725729550921208179070754913983135744,
  'tanodb@127.0.0.1', [<<"key-4">>]},
 {137015778499772148581595453067151533092743675904,
  'tanodb@127.0.0.1', [<<"key-8">>]},
 {707914855582156101004909840846949587645842325504,
  'tanodb@127.0.0.1', [<<"key-9">>]},
 {45671926166590716193865151022383844364247891968,
  'tanodb@127.0.0.1', [<<"key-2">>]},
 {753586781748746817198774991869333432010090217472,
  'tanodb@127.0.0.1', [<<"key-9">>]},
 {274031556999544297163190906134303066185487351808,
  'tanodb@127.0.0.1', [<<"key-10">>]},
 {822094670998632891489572718402909198556462055424,
  'tanodb@127.0.0.1', [<<"key-5">>]},
 {319703483166135013357056057156686910549735243776,
  'tanodb@127.0.0.1', [<<"key-4">>,<<"key-10">>]},
 {342539446249430371453988632667878832731859189760,
  'tanodb@127.0.0.1', [<<"key-4">>]},
 {68507889249886074290797726533575766546371837952,
  'tanodb@127.0.0.1', [<<"key-2">>]},
 {799258707915337533392640142891717276374338109440,
  'tanodb@127.0.0.1', [<<"key-5">>]},
 {91343852333181432387730302044767688728495783936,
  'tanodb@127.0.0.1', [<<"key-2">>]},
 {730750818665451459101842416358141509827966271488,
  'tanodb@127.0.0.1', [<<"key-9">>]},
 {159851741583067506678528028578343455274867621888,
  'tanodb@127.0.0.1', [<<"key-8">>]},
 {182687704666362864775460604089535377456991567872,
  'tanodb@127.0.0.1', [<<"key-8">>]},
 {844930634081928249586505293914101120738586001408,
  'tanodb@127.0.0.1', [<<"key-5">>]},
 {867766597165223607683437869425293042920709947392,
  'tanodb@127.0.0.1', [<<"key-3">>]},
 {890602560248518965780370444936484965102833893376,
  'tanodb@127.0.0.1', [<<"key-3">>]},
 {1050454301831586472458898473514828420377701515264,
  'tanodb@127.0.0.1', [<<"key-6">>]},
 {913438523331814323877303020447676887284957839360,
  'tanodb@127.0.0.1', [<<"key-3">>]},
 {1118962191081472546749696200048404186924073353216,
  'tanodb@127.0.0.1', [<<"key-7">>,<<"key-1">>]},
 {1164634117248063262943561351070788031288321245184,
  'tanodb@127.0.0.1', [<<"key-7">>]},
 {1027618338748291114361965898003636498195577569280,
  'tanodb@127.0.0.1', [<<"key-"...>>]},
 {1096126227998177188652763624537212264741949407232,
  'tanodb@127.0.0.1', [<<...>>]},
 {1073290264914881830555831049026020342559825461248,
  'tanodb@127.0.0.1', [...]},
 {1141798154164767904846628775559596109106197299200,
  'tanodb@127.0.0.1',...}]

Riak Core Tutorial Part 6: Handoff

Mariano Guerra — Thu, 16 Aug 2018 17:54:51 GMT

The content of this chapter is in the 04-handoff branch.

https://gitlab.com/marianoguerra/tanodb/tree/04-handoff

How it Works

With quorum requests we are halfway in our way to tolerate failures in cluster nodes, our values are written to more than one vnode but if a node dies and another takes his work or if we add a new node and the vnodes must be rebalanced we need to handle handoff.

The reasons to start a handoff are:

A ring update event for a ring that all other nodes have already seen.
A secondary vnode is idle for a period of time and the primary, original owner of the partition is up again.

When this happens riak_core will inform the vnode that handoff is starting, calling handoff_starting, if it returns false it's cancelled, if it returns true it calls is_empty, that must return false to inform that the vnode has something to handoff (it's not empty) or true to inform that the vnode is empty, in our case we ask for the first element of the ets table and if it's the special value '$end_of_table' we know it's empty, if it returns true the handoff is considered finished, if false then a call is done to handle_handoff_command passing as first parameter an opaque structure that contains two fields we are insterested in, foldfun and acc0, they can be unpacked with a macro like this:

handle_handoff_command(?FOLD_REQ{foldfun=Fun, acc0=Acc0}, _Sender, State) ->

The FOLD_REQ macro is defined in the riak_core_vnode.hrl header file which we include.

This function must iterate through all the keys it stores and for each of them call foldfun with the key as first argument, the value as second argument and the latest acc0 value as third.

The result of the function call is the new Acc0 you must pass to the next call to foldfun, the last Acc0 must be returned by the handle_handoff_command.

For each call to Fun(Key, Entry, AccIn0) riak_core will send it to the new vnode, to do that it must encode the data before sending, it does this by calling encode_handoff_item(Key, Value), where you must encode the data before sending it.

When the value is received by the new vnode it must decode it and do something with it, this is done by the function handle_handoff_data, where we decode the received data and do the appropriate thing with it.

When we sent all the key/values handoff_finished will be called and then delete so we cleanup the data on the old vnode .

You can decide to handle other commands sent to the vnode while the handoff is running, you can choose to do one of the followings:

Handle it in the current vnode
Forward it to the vnode we are handing off
Drop it

What to do depends on the design of you app, all of them have tradeoffs.

The signature of all the responses is:

-callback handle_handoff_command(Request::term(), Sender::sender(), ModState::term()) ->
{reply, Reply::term(), NewModState::term()} |
{noreply, NewModState::term()} |
{async, Work::function(), From::sender(), NewModState::term()} |
{forward, NewModState::term()} |
{drop, NewModState::term()} |
{stop, Reason::term(), NewModState::term()}.

A diagram of the flow is as follows:

+-----------+      +----------+        +----------+
|           | true |          | false  |          |
| Starting  +------> is_empty +--------> fold_req |
|           |      |          |        |          |
+-----+-----+      +----+-----+        +----+-----+
      |                 |                   |
      | false           | true              | ok
      |                 |                   |
+-----v-----+           |              +----v-----+     +--------+
|           |           |              |          |     |        |
| Cancelled |           +--------------> finished +-----> delete |
|           |                          |          |     |        |
+-----------+                          +----------+     +--------+

Implementing it

We need to add logic to all the empty callbacks related to handoff:

handle_handoff_command(?FOLD_REQ{foldfun=FoldFun, acc0=Acc0}, _Sender,
                       State=#state{partition=Partition, kv_state=KvState}) ->
    lager:info("fold req ~p", [Partition]),
    KvFoldFun = fun ({Key, Val}, AccIn) ->
                        lager:info("fold fun ~p: ~p", [Key, Val]),
                        FoldFun(Key, Val, AccIn)
                end,
    {AccFinal, KvState1} = tanodb_kv_ets:foldl(KvFoldFun, Acc0, KvState),
    {reply, AccFinal, State#state{kv_state=KvState1}};

handle_handoff_command(Message, _Sender, State) ->
    lager:warning("handoff command ~p, ignoring", [Message]),
    {noreply, State}.

handoff_starting(TargetNode, State=#state{partition=Partition}) ->
    lager:info("handoff starting ~p: ~p", [Partition, TargetNode]),
    {true, State}.

handoff_cancelled(State=#state{partition=Partition}) ->
    lager:info("handoff cancelled ~p", [Partition]),
    {ok, State}.

handoff_finished(TargetNode, State=#state{partition=Partition}) ->
    lager:info("handoff finished ~p: ~p", [Partition, TargetNode]),
    {ok, State}.

handle_handoff_data(BinData, State=#state{kv_state=KvState}) ->
    TermData = binary_to_term(BinData),
    lager:info("handoff data received ~p", [TermData]),
    {{Bucket, Key}, Value} = TermData,
    {ok, KvState1} = tanodb_kv_ets:put(KvState, Bucket, Key, Value),
    {reply, ok, State#state{kv_state=KvState1}}.

encode_handoff_item(Key, Value) ->
    term_to_binary({Key, Value}).

is_empty(State=#state{kv_state=KvState, partition=Partition}) ->
    {IsEmpty, KvState1} = tanodb_kv_ets:is_empty(KvState),
    lager:info("is_empty ~p: ~p", [Partition, IsEmpty]),
    {IsEmpty, State#state{kv_state=KvState1}}.

delete(State=#state{kv_state=KvState, partition=Partition}) ->
    lager:info("delete ~p", [Partition]),
    {ok, KvState1} = tanodb_kv_ets:dispose(KvState),
    {ok, KvState2} = tanodb_kv_ets:delete(KvState1),
    {ok, State#state{kv_state=KvState2}}.

Trying it

To test it we will first start a devrel node, put some values and then join two other nodes and see on the console the handoff happening.

To make sure the nodes don't know about each other in case you played with clustering already we will start by removing the devrel builds:

rm -rf _build/dev*

And build the nodes again:

make devrel

Now we will start the first node and connect to its console:

make dev1-console

We generate a list of some numbers:

(tanodb1@127.0.0.1)1> Nums = lists:seq(1, 10).

[1,2,3,4,5,6,7,8,9,10]

And with it create some bucket names:

(tanodb1@127.0.0.1)2>
Buckets = lists:map(fun (N) ->
    list_to_binary("bucket-" ++ integer_to_list(N))
end, Nums).

[<<"bucket-1">>,<<"bucket-2">>,<<"bucket-3">>,
 <<"bucket-4">>,<<"bucket-5">>,<<"bucket-6">>,<<"bucket-7">>,
 <<"bucket-8">>,<<"bucket-9">>,<<"bucket-10">>]

And some key names:

(tanodb1@127.0.0.1)3>
Keys = lists:map(fun (N) ->
    list_to_binary("key-" ++ integer_to_list(N))
end, Nums).

[<<"key-1">>,<<"key-2">>,<<"key-3">>,<<"key-4">>,
 <<"key-5">>,<<"key-6">>,<<"key-7">>,<<"key-8">>,<<"key-9">>,
 <<"key-10">>]

We create a function to generate a value from a bucket and a key:

(tanodb1@127.0.0.1)4>
GenValue = fun (Bucket, Key) -> [{bucket, Bucket}, {key, Key}] end.

And then put some values to the buckets and keys we created:

(tanodb1@127.0.0.1)5>
lists:foreach(fun (Bucket) ->
    lists:foreach(fun (Key) ->
        Val = GenValue(Bucket, Key),
        tanodb:put(Bucket, Key, Val)
    end, Keys)
end, Buckets).

ok

Now that we have some data let's start the other two nodes:

make dev2-console

In yet another shell:

make dev3-console

This part should remind you of the first chapter:

make devrel-join

Success: staged join request for 'tanodb2@127.0.0.1' to 'tanodb1@127.0.0.1'
Success: staged join request for 'tanodb3@127.0.0.1' to 'tanodb1@127.0.0.1'

make devrel-cluster-plan

=============================== Staged Changes =========================
Action         Details(s)
------------------------------------------------------------------------
join           'tanodb2@127.0.0.1'
join           'tanodb3@127.0.0.1'
------------------------------------------------------------------------


NOTE: Applying these changes will result in 1 cluster transition

########################################################################
                         After cluster transition 1/1
########################################################################

================================= Membership ===========================
Status     Ring    Pending    Node
------------------------------------------------------------------------
valid     100.0%     34.4%    'tanodb1@127.0.0.1'
valid       0.0%     32.8%    'tanodb2@127.0.0.1'
valid       0.0%     32.8%    'tanodb3@127.0.0.1'
------------------------------------------------------------------------
Valid:3 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

WARNING: Not all replicas will be on distinct nodes

Transfers resulting from cluster changes: 42
  21 transfers from 'tanodb1@127.0.0.1' to 'tanodb3@127.0.0.1'
  21 transfers from 'tanodb1@127.0.0.1' to 'tanodb2@127.0.0.1'

make devrel-cluster-commit

Cluster changes committed

On the consoles from the nodes you should see some logs like the following, I will just paste some as example.

On the sending side:

00:17:24.240 [info] Starting ownership transfer of tanodb_vnode from
'tanodb1@127.0.0.1' 1118962191081472546749696200048404186924073353216 to
'tanodb2@127.0.0.1' 1118962191081472546749696200048404186924073353216

00:17:24.240 [info] fold req 1118962191081472546749696200048404186924073353216
00:17:24.240 [info] fold fun {<<"bucket-1">>,<<"key-1">>}:
    [{bucket,<<"bucket-1">>},{key,<<"key-1">>}]

...

00:17:24.241 [info] fold fun {<<"bucket-7">>,<<"key-8">>}:
    [{bucket,<<"bucket-7">>},{key,<<"key-8">>}]

00:17:24.281 [info] ownership transfer of tanodb_vnode from
'tanodb1@127.0.0.1' 1118962191081472546749696200048404186924073353216 to
'tanodb2@127.0.0.1' 1118962191081472546749696200048404186924073353216
    completed: sent 575.00 B bytes in 7 of 7 objects in 0.04 seconds
    (13.67 KB/second)

00:17:24.280 [info] handoff finished
    1141798154164767904846628775559596109106197299200:
    {1141798154164767904846628775559596109106197299200,
        'tanodb3@127.0.0.1'}

00:17:24.285 [info] delete
    1141798154164767904846628775559596109106197299200

On the receiving side:

00:13:59.641 [info] handoff starting
    1050454301831586472458898473514828420377701515264:
    {hinted,{1050454301831586472458898473514828420377701515264,
        'tanodb1@127.0.0.1'}}

00:13:59.641 [info] is_empty
    182687704666362864775460604089535377456991567872: true

00:14:34.259 [info] Receiving handoff data for partition
    tanodb_vnode:68507889249886074290797726533575766546371837952 from
    {"127.0.0.1",47440}

00:14:34.296 [info] handoff data received
    {{<<"bucket-8">>,<<"key-1">>},
        [{bucket,<<"bucket-8">>},{key,<<"key-1">>}]}

...

00:14:34.297 [info] handoff data received
    {{<<"bucket-3">>,<<"key-7">>},
        [{bucket,<<"bucket-3">>},{key,<<"key-7">>}]}

00:14:34.298 [info] Handoff receiver for partition
    68507889249886074290797726533575766546371837952 exited after
    processing 5 objects from {"127.0.0.1",47440}

Riak Core Tutorial Part 5: Quorum Requests

Mariano Guerra — Sat, 11 Aug 2018 20:25:13 GMT

The content of this chapter is in the 03-quorum branch.

https://gitlab.com/marianoguerra/tanodb/tree/03-quorum

How it Works

Quorum requests allows sending a command to more than one vnode and wait until a number of responses are received before considering the request succesful.

To implement this we need a process that distributed the requests to the first N vnodes in the preference list and waits for at least W response to arrive before returning to the requester.

We use a gen_fsm to implement this process, which looks like this:

+------+    +---------+    +---------+    +---------+              +------+
|      |    |         |    |         |    |         |remaining = 0 |      |
| Init +--->| Prepare +--->| Execute +--->| Waiting +------------->| Stop |
|      |    |         |    |         |    |         |              |      |
+------+    +---------+    +---------+    +-------+-+              +------+
                                              ^   | |
                                              |   | |        +---------+
                                              +---+ +------->|         |
                                                             | Timeout |
                                      remaining > 0  timeout |         |
                                                             +---------+

Implementing it

To implement it we need to change the code in tanodb.erl instantiate a FSM to handle the request instead of sending the command directly to one vnode.

get(Bucket, Key, Opts) ->
    K = {Bucket, Key},
    Params = K,
    run_quorum(get, K, Params, Opts).

put(Bucket, Key, Value, Opts) ->
    K = {Bucket, Key},
    Params = {Bucket, Key, Value},
    run_quorum(put, K, Params, Opts).

delete(Bucket, Key, Opts) ->
    K = {Bucket, Key},
    Params = K,
    run_quorum(delete, K, Params, Opts).

We are going to generalize that logic in a function called run_quorum, where we can pass options for N, W and Timeout to play with different values:

run_quorum(Action, K, Params, Opts) ->
    N = maps:get(n, Opts, ?N),
    W = maps:get(w, Opts, ?W),
    Timeout = maps:get(timeout, Opts, ?TIMEOUT),
    ReqId = make_ref(),
    tanodb_write_fsm:run(Action, K, Params, N, W, self(), ReqId),
    wait_for_reqid(ReqId, Timeout).

wait_for_reqid(ReqId, Timeout) ->
    receive
        {ReqId, Val} -> Val
    after
        Timeout -> {error, timeout}
    end.

To wait for the right answer we need to generate a unique identifier for each request and send it with the request itself. The identifier will come back in the message sent by the FSM once the request finishes.

If too much time passed waiting for the response we consider it an error and return before receiving it.

wait_for_reqid(ReqId, Timeout) ->
        receive
                {ReqId, {error, Reason}} -> {error, Reason};
                {ReqId, Val} -> Val
        after
                Timeout -> {error, timeout}
        end.

There are two new files:

tanodb_write_fsm.erl: The FSM logic
tanodb_write_fsm_sup.erl: The supervisor for the FSMs

Finally we need to add tanodb_write_fsm_sup to our top level supervisor in tanodb_sup.

Trying it

To test it we are going to run some calls to the API and observe that now the response contains more than one response:

(tanodb@127.0.0.1)1> B1 = b1.
(tanodb@127.0.0.1)2> K1 = k1.
(tanodb@127.0.0.1)3> V1 = v1.

First let's try to get a key that doesn't exist:

(tanodb@127.0.0.1)4> tanodb:get(B1, K1).

{ok,[{[1073290264914881830555831049026020342559825461248,
           'tanodb@127.0.0.1'],
          {not_found,{b1,k1}}},

         {[1050454301831586472458898473514828420377701515264,
           'tanodb@127.0.0.1'],
          {not_found,{b1,k1}}},

         {[1096126227998177188652763624537212264741949407232,
           'tanodb@127.0.0.1'],
          {not_found,{b1,k1}}}]}

Let's do the same call but passing options, we want to run the command in 5 vnodes and wait for the response of the 5, the request should finish under a second:

(tanodb@127.0.0.1)5> tanodb:get(k1, v1, #{n => 5, w => 5, timeout => 1000}).

{ok,[{[456719261665907161938651510223838443642478919680,
           'tanodb@127.0.0.1'],
          {not_found,{k1,v1}}},

         {[433883298582611803841718934712646521460354973696,
           'tanodb@127.0.0.1'],
          {not_found,{k1,v1}}},

         {[411047335499316445744786359201454599278231027712,
           'tanodb@127.0.0.1'],
          {not_found,{k1,v1}}},

         {[388211372416021087647853783690262677096107081728,
           'tanodb@127.0.0.1'],
          {not_found,{k1,v1}}},

         {[365375409332725729550921208179070754913983135744,
           'tanodb@127.0.0.1'],
          {not_found,{k1,v1}}}]}

Let's try deleting a key that doesn't exist:

(tanodb@127.0.0.1)6> tanodb:delete(B1, K1).

{ok,[{[1050454301831586472458898473514828420377701515264,
           'tanodb@127.0.0.1'],
          ok},

         {[1073290264914881830555831049026020342559825461248,
           'tanodb@127.0.0.1'],
          ok},

         {[1096126227998177188652763624537212264741949407232,
           'tanodb@127.0.0.1'],
          ok}]}

Let's put a value:

(tanodb@127.0.0.1)7> tanodb:put(B1, K1, V1).

{ok,[{[1096126227998177188652763624537212264741949407232,
           'tanodb@127.0.0.1'],
          ok},

         {[1073290264914881830555831049026020342559825461248,
           'tanodb@127.0.0.1'],
          ok},

         {[1050454301831586472458898473514828420377701515264,
           'tanodb@127.0.0.1'],
          ok}]}

Now let's get the value:

(tanodb@127.0.0.1)8> tanodb:get(B1, K1).

{ok,[{[1096126227998177188652763624537212264741949407232,
           'tanodb@127.0.0.1'],
          {found,{{b1,k1},v1}}},

         {[1050454301831586472458898473514828420377701515264,
           'tanodb@127.0.0.1'],
          {found,{{b1,k1},v1}}},

         {[1073290264914881830555831049026020342559825461248,
           'tanodb@127.0.0.1'],
          {found,{{b1,k1},v1}}}]}

Let's delete it:

(tanodb@127.0.0.1)9> tanodb:delete(B1, K1).

{ok,[{[1073290264914881830555831049026020342559825461248,
           'tanodb@127.0.0.1'],
          ok},

         {[1096126227998177188652763624537212264741949407232,
           'tanodb@127.0.0.1'],
          ok},

         {[1050454301831586472458898473514828420377701515264,
           'tanodb@127.0.0.1'],
          ok}]}

And try to get it back:

(tanodb@127.0.0.1)10> tanodb:get(B1, K1).

{ok,[{[1073290264914881830555831049026020342559825461248,
           'tanodb@127.0.0.1'],
          {not_found,{b1,k1}}},

         {[1096126227998177188652763624537212264741949407232,
           'tanodb@127.0.0.1'],
          {not_found,{b1,k1}}},

         {[1050454301831586472458898473514828420377701515264,
           'tanodb@127.0.0.1'],
          {not_found,{b1,k1}}}]}

Riak Core Tutorial Part 4: First Commands

Mariano Guerra — Mon, 30 Jul 2018 08:41:45 GMT

The content of this chapter is in the 02-commands branch.

https://gitlab.com/marianoguerra/tanodb/tree/02-commands

This is part of a series, see the previous one at Riak Core Tutorial Part 3: Ping Command

Implementing Get, Put and Delete

For our first commands we will copy the general structure of the ping command.

We will start by adding three new functions to the tanodb.erl file:

get(Bucket, Key) ->
    ReqId = make_ref(),
    send_to_one(Bucket, Key, {get, ReqId, {Bucket, Key}}).

put(Bucket, Key, Value) ->
    ReqId = make_ref(),
    send_to_one(Bucket, Key, {put, ReqId, {Bucket, Key, Value}}).

delete(Bucket, Key) ->
    ReqId = make_ref(),
    send_to_one(Bucket, Key, {delete, ReqId, {Bucket, Key}}).

And generalizing the code used by ping to send a command to one vnode:

send_to_one(Bucket, Key, Cmd) ->
    DocIdx = riak_core_util:chash_key({Bucket, Key}),
    PrefList = riak_core_apl:get_primary_apl(DocIdx, 1, tanodb),
    [{IndexNode, _Type}] = PrefList,
    riak_core_vnode_master:sync_spawn_command(IndexNode, Cmd, tanodb_vnode_master).

In tanodb_vnode.erl we will need to first create an instance of the key-value store per vnode at initialization and keep a reference to its state in the vnode state record:

-record(state, {partition, kv_state}).

init([Partition]) ->
    {ok, KvState} = tanodb_kv_ets:new(#{partition => Partition}),
    {ok, #state { partition=Partition, kv_state=KvState }}.

We then need to add three new clauses to the handle_command callback to handle our two new commands, which translate almost directly to calls in the kv module:

handle_command({put, ReqId, {Bucket, Key, Value}}, _Sender,
               State=#state{kv_state=KvState, partition=Partition}) ->
    Location = [Partition, node()],
    {Res, KvState1} = tanodb_kv_ets:put(KvState, Bucket, Key, Value),
    {reply, {ReqId, {Location, Res}}, State#state{kv_state=KvState1}};

handle_command({get, ReqId, {Bucket, Key}}, _Sender,
               State=#state{kv_state=KvState, partition=Partition}) ->
    Location = [Partition, node()],
    {Res, KvState1} = tanodb_kv_ets:get(KvState, Bucket, Key),
    {reply, {ReqId, {Location, Res}}, State#state{kv_state=KvState1}};

handle_command({delete, ReqId, {Bucket, Key}}, _Sender,
               State=#state{kv_state=KvState, partition=Partition}) ->
    Location = [Partition, node()],
    {Res, KvState1} = tanodb_kv_ets:delete(KvState, Bucket, Key),
    {reply, {ReqId, {Location, Res}}, State#state{kv_state=KvState1}};

Trying it

First let's try to get a key that doesn't exist:

(tanodb@127.0.0.1)1> B1 = b1.
(tanodb@127.0.0.1)2> K1 = k1.
(tanodb@127.0.0.1)3> V1 = v1.

(tanodb@127.0.0.1)4> tanodb:get(B1, K1).

{Ref, {[1050454301831586472458898473514828420377701515264,
        'tanodb@127.0.0.1'],
  {not_found,{b1,k1}}}}

The structure of the response is:

{UniqueRequestReference, {[PartitionId, NodeId], CommandResponse}}.

Let's try deleting a key that doesn't exist:

(tanodb@127.0.0.1)5> tanodb:delete(B1, K1).

{Ref, {[1050454301831586472458898473514828420377701515264,
        'tanodb@127.0.0.1'],
  ok}}

Let's put a value:

(tanodb@127.0.0.1)6> tanodb:put(B1, K1, V1).

{Ref, {[1050454301831586472458898473514828420377701515264,
        'tanodb@127.0.0.1'],
  ok}}

Now let's get the value:

(tanodb@127.0.0.1)7> tanodb:get(B1, K1).

{Ref, {[1050454301831586472458898473514828420377701515264,
        'tanodb@127.0.0.1'],
  {found,{{b1,k1},v1}}}}

Let's delete it:

(tanodb@127.0.0.1)8> tanodb:delete(B1, K1).

{Ref, {[1050454301831586472458898473514828420377701515264,
        'tanodb@127.0.0.1'],
  ok}}

And try to get it back:

(tanodb@127.0.0.1)9> tanodb:get(B1, K1).

{Ref, {[1050454301831586472458898473514828420377701515264,
        'tanodb@127.0.0.1'],
  {not_found,{b1,k1}}}}

Riak Core Tutorial Part 3: Ping Command

Mariano Guerra — Tue, 24 Jul 2018 14:35:57 GMT

The content of this chapter is in the `01-template` branch.

https://gitlab.com/marianoguerra/tanodb/tree/01-template

This is part of a series, see the previous one at Riak Core Tutorial Part 2: Starting

How it Works

Let's see how ping works under the covers.

Its entry point and public API is the tanodb module, that means we have to look into tanodb.erl:

-module(tanodb).
-export([ping/0]).
-ignore_xref([ping/0]).

%% @doc Pings a random vnode to make sure communication is functional
ping() ->
    % argument to chash_key has to be a two item tuple, since it comes from riak
    % and the full key has a bucket, we use a contant in the bucket position
    % and a timestamp as key so we hit different vnodes on each call
    DocIdx = riak_core_util:chash_key({<<"ping">>, term_to_binary(os:timestamp())}),
    % ask for 1 vnode index to send this request to, change N to get more
    % vnodes, for example for replication
    N = 1,
    PrefList = riak_core_apl:get_primary_apl(DocIdx, N, tanodb),
    [{IndexNode, _Type}] = PrefList,
    riak_core_vnode_master:sync_spawn_command(IndexNode, ping, tanodb_vnode_master).

DocIdx = riak_core_util:chash_key({<<"ping">>, term_to_binary(os:timestamp())}),

The line above hashes a key to decide to which vnode the call should go, a riak_core app has a fixed number of vnodes that are distributed across all the instances of your app's physical nodes, vnodes move from instance to instance when the number of instances change to balance the load and provide fault tolerance and scalability.

The call above will allow us to ask for vnodes that can handle that hashed key, let's run it in the app console to see what it does:

(tanodb@127.0.0.1)1> DocIdx = riak_core_util:chash_key({<<"ping">>, term_to_binary(os:timestamp())}).

<<126,9,218,77,97,108,38,92,0,155,160,26,161,3,200,87,134,213,167,168>>

We seem to get a binary back, in the next line we ask for a list of vnodes that can handle that hashed key:

PrefList = riak_core_apl:get_primary_apl(DocIdx, N, tanodb),

Let's run it to see what it does:

(tanodb@127.0.0.1)2> PrefList = riak_core_apl:get_primary_apl(DocIdx, 1, tanodb).

[{{730750818665451459101842416358141509827966271488, 'tanodb@127.0.0.1'},
     primary}]

We get a list with one tuple that has 3 items, a long number, something that looks like a host and an atom, let's try changing the number 1:

(tanodb@127.0.0.1)3> PrefList2 = riak_core_apl:get_primary_apl(DocIdx, 2, tanodb).

[{{730750818665451459101842416358141509827966271488,
   'tanodb@127.0.0.1'}, primary},
 {{753586781748746817198774991869333432010090217472,
   'tanodb@127.0.0.1'}, primary}]

Now we get two tuples, the first one is the same, so what this does is to return the number of vnodes that can handle the request from the hashed key by priority.

The first number is the vnode id, it's what we get on the ping response.

Next line just unpacks the pref list to get the vnode id and ignore the other part:

[{IndexNode, _Type}] = PrefList,

Finally we ask riak_core to call the ping command on the IndexNode we got back:

riak_core_vnode_master:sync_spawn_command(IndexNode, ping, tanodb_vnode_master).

Let's try it on the console:

(tanodb@127.0.0.1)5> [{IndexNode, _Type}] = PrefList.

[{{730750818665451459101842416358141509827966271488,
   'tanodb@127.0.0.1'}, primary}]

(tanodb@127.0.0.1)6> riak_core_vnode_master:sync_spawn_command(IndexNode, ping, tanodb_vnode_master).

{pong,730750818665451459101842416358141509827966271488}

You can see we get IndexNode back in the pong response, now let's try passing the second IndexNode:

(tanodb@127.0.0.1)7> [{IndexNode1, _Type1}, {IndexNode2, _Type2}] = PrefList2.

[{{730750818665451459101842416358141509827966271488,
   'tanodb@127.0.0.1'}, primary},
 {{753586781748746817198774991869333432010090217472,
   'tanodb@127.0.0.1'}, primary}]


(tanodb@127.0.0.1)9> riak_core_vnode_master:sync_spawn_command(IndexNode2, ping, tanodb_vnode_master).

{pong,753586781748746817198774991869333432010090217472}

We get the IndexNode2 back, that means that the request was sent to the second vnode instead of the first one.

But where does the command go?

Let's see the content of tanodb_vnode.erl (just the useful parts):

-module(tanodb_vnode).
-behaviour(riak_core_vnode).

-export([start_vnode/1,
         init/1,
         terminate/2,
         handle_command/3,
         is_empty/1,
         delete/1,
         handle_handoff_command/3,
         handoff_starting/2,
         handoff_cancelled/1,
         handoff_finished/2,
         handle_handoff_data/2,
         encode_handoff_item/2,
         handle_overload_command/3,
         handle_overload_info/2,
         handle_coverage/4,
         handle_exit/3]).

-record(state, {partition}).

%% API
start_vnode(I) ->
    riak_core_vnode_master:get_vnode_pid(I, ?MODULE).

init([Partition]) ->
    {ok, #state { partition=Partition }}.

%% Sample command: respond to a ping
handle_command(ping, _Sender, State) ->
    {reply, {pong, State#state.partition}, State};
handle_command(Message, _Sender, State) ->
    lager:warning("unhandled_command ~p", [Message]),
    {noreply, State}.

Let's go by parts, first we declare our module:

-module(tanodb_vnode).

We specify that we want to implement the riak_core_vnode behavior:

-behaviour(riak_core_vnode).

Behaviors in Erlang are like interfaces, a set of functions that a module must implement to satisfy the behaviour specification, you can read more in the Erlang documentation.

In this case riak_core defines a behavior with a set of functions we must implement to be a valid riak_core vnode, you can get an idea of the kind of functionality we need by looking at the exported functions:

-export([start_vnode/1,
         init/1,
         terminate/2,
         handle_command/3,
         is_empty/1,
         delete/1,
         handle_handoff_command/3,
         handoff_starting/2,
         handoff_cancelled/1,
         handoff_finished/2,
         handle_handoff_data/2,
         encode_handoff_item/2,
         handle_overload_command/3,
         handle_overload_info/2,
         handle_coverage/4,
         handle_exit/3]).

For the moment most of them have a "dummy" implementation where they just do the minimal amount of work to satisfy the behavior and not more, it's our job to change the default implementation to fit our needs.

We will have a record called state to keep info between callbacks, this is typical Erlang way of managing state so I won't cover it here:

-record(state, {partition}).

We implement the api to start the vnode:

%% API
start_vnode(I) ->
    riak_core_vnode_master:get_vnode_pid(I, ?MODULE).

Note that on init we store the Partition value on state so we can use it later, this is what I referred above as vnode id, it's the big number you saw before:

init([Partition]) ->
    {ok, #state { partition=Partition }}.

Now for the interesting part, here we have our ping command implementation, we match for ping in the Message position (the first argument):

handle_command(ping, _Sender, State) ->

Return a response with the second item in the tuple being the actual response that the caller will get where we reply with the atom pong and the partition number of this vnode, the last item in the tuple is the new state we want to have for this vnode, since we didn't change anything we pass the current value:

{reply, {pong, State#state.partition}, State};

We implement a catch all that will just log the unknown command and give no reply back:

handle_command(Message, _Sender, State) ->
    lager:warning("unhandled_command ~p", [Message]),
    {noreply, State}.

This is the roundtrip of a ping call, our task to add more commands will be:

Add a function on tanodb.erl that hides the internal work done to distribute the work
Add a new match on handle_command to match the command we added on tanodb.erl and provide a reply

ameo: Redis compatible GET,SET,DEL and PUBLISH/SUBSCRIBE on riak_core with WebSocket API

Mariano Guerra — Fri, 19 Jan 2018 10:08:04 GMT

Last week I was invited by Erlang Solutions to give a talk at the London's Erlang Meetup and to help at a riak_core themed hackathon at their offices the next day.

The talk slides: Building distributed applications: riak_core vs partisan in case you are interested.

After giving the introduction to riak_core the teams started to work on their projects and since they were pretty busy and didn't need much help I decided to "participate" too by implementing an idea I had in mind for a while.

The result is called ameo and can be described as:

a Redis compatible, distributed in-memory key-value store and pubsub server
implemented using riak_core that exposes the topics via WebSockets.

Why?

Many languages make it hard to provide websocket connections and live connections with clients, they work on a request/response basis and/or make it really hard/expensive to handle multiple persistent connections.

I've seen solutions that involve starting a redis server and putting usually nodejs in the front to expose redis topics via websockets, this involves two moving parts and for many projects, managing nodejs which they may not have experience with.

The solution is to just start one instance of ameo or a cluster of ameos and expose the WebSocket API to clients and the Redis API to the servers.

Servers can use their preferred Redis client library and as long as they only use GET, PUT, DEL, PUBLISH, SUBSCRIBE and UNSUBSCRIBE it will look like they are talking to a Redis server.

On my way back I had some extra time at the airport so I implemented a basic web UI to play with the websocket client and to provide a reference implementation others can use.

You can see the result in this screencast:

Implementation details:

As said earlier I use riak_core for clustering, for the Redis part I took some modules from an Erlang implementation of Redis called edis and I created a library called edis_proto that allows any project to expose a Redis compatible API to their servers with a couple lines of code.

The WebSocket part is implemented using the Cowboy Web Server.

rebar3 partisan template

Mariano Guerra — Sat, 09 Dec 2017 23:16:11 GMT

I created a rebar3 template to start a partisan project, the template is here:

rebar3 partisan template

What is partisan? from the docs:

Partisan is the technology that provides Lasp's scalable cluster
membership. Partisan bypasses the use of Distributed Erlang for manual
connection management via TCP, and has several pluggable backends for
different deployment scenarios.

To use:

mkdir -p ~/.config/rebar3/templates
git clone https://github.com/marianoguerra/rebar3_template_partisan.git ~/.config/rebar3/templates/rebar3_template_partisan
rebar3 new rebar3_template_partisan name=party
cd party
make release
make console

Clustering:

make devrel

# on 3 different shells
make dev1-console
make dev2-console
make dev3-console

# join all nodes:
make devrel-join

# check node members
make devrel-status

# join node1 to node2 manually:
./_build/dev1/rel/party/bin/party-admin cluster join party2@127.0.0.1

# check node1 members
./_build/dev1/rel/party/bin/party-admin cluster members

# check node1 connections
./_build/dev1/rel/party/bin/party-admin cluster connections

A video to show how to use:

Let's build a key value store in erlang

Mariano Guerra — Sun, 26 Nov 2017 02:04:25 GMT

The code for this post is at https://github.com/marianoguerra/akvs

We first need to have erlang installed, I will show you how to setup any version you want to use, and a way to have the version I will use for this without affecting any other installation you may have.

Setting up kerl

For this we will use kerl, from it's github README:

Easy building and installing of Erlang/OTP instances.

Kerl aims to be shell agnostic and its only dependencies, excluding what's
required to actually build Erlang/OTP, are curl and git.

So, first we need to fetch kerl:

# create bin folder in our home directory if it's not already there
mkdir -p ~/bin

# cd to it
cd ~/bin

# download kerl script
curl -O https://raw.githubusercontent.com/kerl/kerl/master/kerl

# set execution permitions for our user
chmod u+x kerl

You will need to add ~/bin to your PATH variable so your shell can find the kerl script, you can do it like this in your shell:

# set the PATH environment variable to the value it had before plus a colon
# (path separator) and a new path which points to the bin folder we just
# created
PATH=$PATH:$HOME/bin

If you want to make this work every time you start a shell you need to put it it the rc file of your shell of choice, for bash it's ~/.bashrc, for zsh it's .zshrc, check your shell's docs for other shells, you will have to add a line like this:

export PATH=$PATH:$HOME/bin

After this, start a new shell or source your rc file so that it picks up your new PATH variable, you can check that it's set correctly by running:

echo $PATH

Building an Erlang release with kerl

We have kerl installed and available in our shell, now we need to build an Erlang release of our choice, for this we will need a compiler and other tools and libraries needed to compile it:

This are instructions on ubuntu 17.10, check the names for those packages on your distribution.

# required: basic tools and libraries needed
# (compiler, curses for the shell, ssl for crypto)
sudo apt-get -y install build-essential m4 libncurses5-dev libssl-dev

# optonal: if you want odbc support (database connectivity)
sudo apt-get install unixodbc-dev

# optonal: if you want pdf docs you need apache fop and xslt tools and java (fop is a java project)
sudo apt-get install -y fop xsltproc default-jdk

# optional: if you want to build jinterface you need a JDK
sudo apt-get install -y default-jdk

# optional: if you want wx (desktop GUI modules)
sudo apt-get install -y libwxgtk3.0-dev

Now that we have everything we need we can finally build our erlang release.

First we fetch an updated list of releases:

kerl update releases

The output in my case:

The available releases are:

R10B-0 R10B-10 R10B-1a R10B-2 R10B-3 R10B-4 R10B-5 R10B-6 R10B-7 R10B-8
R10B-9 R11B-0 R11B-1 R11B-2 R11B-3 R11B-4 R11B-5 R12B-0 R12B-1 R12B-2 R12B-3
R12B-4 R12B-5 R13A R13B01 R13B02-1 R13B02 R13B03 R13B04 R13B R14A R14B01
R14B02 R14B03 R14B04 R14B_erts-5.8.1.1 R14B R15B01 R15B02
R15B02_with_MSVCR100_installer_fix R15B03-1 R15B03 R15B
R16A_RELEASE_CANDIDATE R16B01 R16B02 R16B03-1 R16B03 R16B 17.0-rc1 17.0-rc2
17.0 17.1 17.3 17.4 17.5 18.0 18.1 18.2.1 18.2 18.3 19.0 19.1 19.2 19.3
20.0 20.1

Let's build the 20.1 version:

# this will take a while
kerl build 20.1 20.1

And install it:

kerl install 20.1 ~/bin/erl-20.1

Now everytime we want to use this version of erlang we need to run:

. $HOME/bin/erl-20.1/activate

Setting up rebar3

Now we have erlang, we need a build tool, we are going to use rebar3:

# download rebar3 to our bin directory
wget https://s3.amazonaws.com/rebar3/rebar3 -O $HOME/bin/rebar3

# set execution permissions for our user
chmod u+x rebar3

Just in case you have problems running the rebar3 commands with a different version, here's the version I'm using:

rebar3 version

Output:

rebar 3.4.7 on Erlang/OTP 20 Erts 9.1

Setting up our project

We are ready to start our project, go to a folder where you keep your code and if you haven't done it yet, add $HOME/bin to your path and activate erlang 20.1 as shown above, then run:

rebar3 new release name=akvs

The output should be something like this:

===> Writing akvs/apps/akvs/src/akvs_app.erl
===> Writing akvs/apps/akvs/src/akvs_sup.erl
===> Writing akvs/apps/akvs/src/akvs.app.src
===> Writing akvs/rebar.config
===> Writing akvs/config/sys.config
===> Writing akvs/config/vm.args
===> Writing akvs/.gitignore
===> Writing akvs/LICENSE
===> Writing akvs/README.md

Let's see what each file does:

First of all, we created a release, which is a kind of project that can have more than one application (a common way to structure a project is into applications and libraries)

Under the apps folder are all the applications we mantain for this release, in our case we only have one application, named akvc.

Under the akvs application folder we have a src folder where all the source code for that application will live, we can add other folders there, for tests, header files, private files etc.

apps/akvs/src/akvs_app.erl

The $APPNAME_app module is called when starting and stopping the app to do the setup and tear down of the application.

Check Erlang's manual for application or the user's guide entry for application for more information.

apps/akvs/src/akvs_sup.erl

The $APPNAME_sup module defines the root supervisor for the application, it implements the supervisor behavior and will be "hooked" into the supervisor hierarchy of this release when initialized.

Check Erlang's manual for supervisor or the user's guide entry for supervisor for more information.

apps/akvs/src/akvs.app.src

The $APPNANE.app.src is a file that contains metadata about this app.

Check ERlang's manual for application resource file for more information.

rebar.config

Contains information about the project, dependencies, how to build it, test it, and how to build a release.

Check rebar3 docs for details.

config/sys.config

Configuration parameters for the application.

Check sys.config's manual page for more information.

config/vm.args

Configuration parameters for the Erlang VM.

.gitignore

Git specific, files to ignore.

LICENSE

The license for this project, you should change it if the Apache License 2.0 isn't the one you want.

README.md

Project's readme.

Starting it for the first time

First we need to build a release:

cd akvs

# build a release, the result will be at _build/default/rel/akvs
rebar3 release

# start the release and attach to the console
./_build/default/rel/akvs/bin/akvs console

The output in my case is (redacted for clarity):

Exec: bin/erl-20.1/erts-9.1/bin/erlexec
         -boot src/erl/akvs/_build/default/rel/akvs/releases/0.1.0/akvs
         -mode embedded -boot_var ERTS_LIB_DIR bin/erl-20.1/lib
         -config src/erl/akvs/_build/default/rel/akvs/releases/0.1.0/sys.config
         -args_file src/erl/akvs/_build/default/rel/akvs/releases/0.1.0/vm.args
         -pa -- console

Root: src/erl/akvs/_build/default/rel/akvs
src/erl/akvs/_build/default/rel/akvs

Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:4:4] [ds:4:4:10]
                          [async-threads:30] [kernel-poll:true]


=PROGRESS REPORT==== 25-Nov-2017::22:28:34 ===
                  supervisor: {local,sasl_safe_sup}
                         started: [{pid,<0.225.0>},
                                           {id,alarm_handler},
                                           {mfargs,{alarm_handler,start_link,[]}},
                                           {restart_type,permanent},
                                           {shutdown,2000},
                                           {child_type,worker}]

=PROGRESS REPORT==== 25-Nov-2017::22:28:34 ===
                  supervisor: {local,sasl_sup}
                         started: [{pid,<0.224.0>},
                                           {id,sasl_safe_sup},
                                           {mfargs,
                                                   {supervisor,start_link,
                                                           [{local,sasl_safe_sup},sasl,safe]}},
                                           {restart_type,permanent},
                                           {shutdown,infinity},
                                           {child_type,supervisor}]

=PROGRESS REPORT==== 25-Nov-2017::22:28:34 ===
                  supervisor: {local,sasl_sup}
                         started: [{pid,<0.226.0>},
                                           {id,release_handler},
                                           {mfargs,{release_handler,start_link,[]}},
                                           {restart_type,permanent},
                                           {shutdown,2000},
                                           {child_type,worker}]

=PROGRESS REPORT==== 25-Nov-2017::22:28:34 ===
                 application: sasl
                  started_at: akvs@ganesha
Eshell V9.1  (abort with ^G)
(akvs@ganesha)1>

There's not much we can do with our project at this stage, so we will just stop it and exit by running the q(). function in the shell:

(akvs@ganesha)1> q().
ok

Coding (and testing) the Key Value store modules

The way I usually code in erlang is to first build a stateless module that has an init function that returns some state, all other functions expect that state as first parameter, then those functions do something and return the state and the result.

This modules are really easy to use in the shell and test.

This will be our first module, we will call it akvs_kv and it will have the following API:

%% types:

-type error() :: {error, {atom(), iolist(), map()}}.
-type key()   :: binary().
-type value() :: any().

% we don't want other modules to know/care about the internal structure of
% the state type
-opaque state() :: map().

%% functions:

%% @doc create a new instance of a key value store
-spec new(map()) -> {ok, state()} | error().

%% @doc dispose resources associated with a previously created kv store
-spec dispose(state()) -> ok | error().

%% @doc set a value for a key in a kv store
-spec set(state(), key(), value()) -> {ok, state()} | error().

%% @doc get a value for a key or an error if not found
-spec get(state(), key()) -> {ok, value()} | error().

%% @doc get a value for a key or a default value if not found
-spec get(state(), key(), value()) -> {ok, value()} | error().

%% @doc remove a value for a key, if not found do nothing
-spec del(state(), key()) -> {ok, state()} | error().

Notice that to specify the API I used a specification of the types and functions, this is called spec, read more about it at the Types and Function Specifications sectio in the erlang reference manual.

Also for documentation comments I'm using the edoc format, read more about it at the edoc user's guide section.

You can see the full code of this module here: akvs_kv

But how do we know if it works?

At this point there are two ways: testing it in the shell, or writing tests for it, let's do the right thing and write some tests.

We are going to use Common Test for our tests.

First we need to create the test folder for our tests:

mkdir apps/akvs/test

Inside it we will create a module called akvs_kv_SUITE that will contain the tests for the akvs_kv module.

You can see the full code of this module here: akvs_kv_SUITE

To run the tests:

rebar3 ct

We can also use the type specs we defined to check our code using dialyzer:

rebar3 dialyzer

Everything seems to be right, let's move on to the next step.

But before that, in case you want to generate API docs for our code taking advantage of the edoc annotations, you can do so by running:

rebar3 edoc

And opening apps/akvs/doc/index.html with a browser.

Wrapping the state

Stateless modules are a good start and are really easy to test and use, but we don't want to pass the burden of threading the state to the users of our code, also we want to centralize the state management so that more than one process can call our module and see the state changes of other callers.

In this case we are using ETS to make it simpler but if our kv was backed by a map, or if we had some kind of cache, then state management would become really important to get right, otherwise the results seen by each caller would diverge.

To manage the state of our module we are going to wrap it in a process, a gen_server in this case.

The module will be called akvs_kv_s (_s for server, don't know if there's a convention for it).

The module is a basic gen_server that exposes a couple functions to call the kv API from the akvs_kv module, you can read the code here: akvs_kv_s.

We write tests for this module too, you can read the test's code here: akvs_kv_s_SUITE.

Run the tests:

rebar3 ct

An API for our key value stores

Now we can spawn a key value store in a gen_server and apply operations to it, but like with the stateless module, someone has to keep a reference to the process and provide a nicer way to find and operate on our key value stores, if it was only one it's easy to just start it as a registered process with a name and send messages to it by it's name, but in our case, we want to provide namespaces where each namespace holds a key value store of its own.

The abstract API or this module should be like this:

-type ns() :: binary().
-type key() :: akvs_kv:key().
-type value() :: akvs_kv:value().
-type error() :: akvs_kv:value().

%% @doc set Key to Value in namespace Ns
-spec set(ns(), key(), value()) -> ok | error().

%% @doc get Key from namespace Ns
-spec get(ns(), key()) -> {ok, value()} | error().

%% @doc get Key from namespace Ns or DefaultValue if Key not found
-spec get(ns(), key(), value()) -> {ok, value()} | error().

%% @doc delete  Key in namespace Ns
-spec del(ns(), key()) -> ok | error().

Right now we are going to solve the problem of who keeps the namespace to process mapping really simple so we can continue, we are going to setup a public ETS table at application startup and lookup the processes by namespace there, if not found we are going to start the process and register it under that namespace.

This solution is not recommendable at all but it will allow us to continue and since the API doesn't know a thing about the way we register/lookup namespaces we can explore different alternatives later.

You can view the source code for akvs module here: akvs and the tests here akvs_SUITE.

An HTTP API for our key value stores

We are at the point where we can expose our APIs to the world, we are going to do it by exposing a really basic HTTP API for it.

The API will look like this:

# set key in namespace to the binary value sent in body
# return status: 201
POST /kv/<namespace>/<key>
<body>

# get key in namespace
# return status:
#  200: if found
#  404: if not found
GET /kv/<namespace>/<key>

# delete key from namespace
# return status: 200
DELETE /kv/<namespace>/<key>

To create an HTTP API we need an HTTP server, in this case we will use Cowboy 2.

First we need to add it as a dependency in our rebar.config file in the deps section and in the release dependencies section.

Then we need to setup the routes in our application initialization code.

We are going to have only one route and handler, we are going to use a basic HTTP to keep it simple, you can read the handler's code here: akvs_h_kv.

Now we can test it by building a release, starting it and playing with the API using curl:

rebar3 release
_build/default/rel/akvs/bin/akvs console

In another shell:

curl http://localhost:8080/kv/foo/bar
Not Found

curl -X POST http://localhost:8080/kv/foo/bar -d "hello world"
Created

curl http://localhost:8080/kv/foo/bar
hello world

curl -X DELETE http://localhost:8080/kv/foo/bar
OK

curl http://localhost:8080/kv/foo/bar
Not Found

curl -X PUT http://localhost:8080/kv/foo/bar -d "hello world"
Method Not Allowed

Seems to work fine.

Now we can build a production release and try it:

rebar3 as prod release
cd _build/prod/rel
tar -czf akvs.tar.gz akvs
cd -
mv _build/prod/rel/akvs.tar.gz /tmp
cd /tmp
tar -xzf akvs.tar.gz
cd akvs
./bin/akvs start

The application is started, you can check it's running by pinging it:

./bin/akvs ping

In case you need, you can attach to it (you should exit with Ctrl+D, using q() won't only detach your console but also stop the system!):

./bin/akvs attach

You can try it again:

curl http://localhost:8080/kv/foo/bar
curl -X POST http://localhost:8080/kv/foo/bar -d "hello world"
curl http://localhost:8080/kv/foo/bar
curl -X DELETE http://localhost:8080/kv/foo/bar
curl http://localhost:8080/kv/foo/bar
curl -X PUT http://localhost:8080/kv/foo/bar -d "hello world"

When you are finished, you can stop it:

./bin/akvs stop

Now you can upload akvs.tar.gz to any bare server and start akvs there, as long as the operating system is similar (better if the same) as the one where you built the release, this is because when building the release we bundle the erlang runtime for simplicity, this assumes specific versions of libraries like libssl which may not be available on the target system if it's too different.

Another way is to build the release without bundling the erlang runtime and having it available on the target system, just make sure that the erlang runtime in the target system has the same version you used to build it, otherwise you may experience errors due to modules/functions not being available or bytecode incompatibility if the target runtime is older than the one used for the release.

Multi-Paxos with riak_ensemble Part 2

Mariano Guerra — Sat, 16 Sep 2017 17:29:18 GMT

In the previous post I showed how to use riak_ensemble from the interactive shell, now I will show how to use rebar3 to use riak_ensemble from a real project.

This post assumes that you have erlang and rebar3 installed, I'm using erlang 19.3 and rebar3 3.4.3.

The source code for this post is at https://github.com/marianoguerra/cadena check the commits for the steps.

Create Project

rebar3 new app name=cadena
cd cadena

The project structure should look like this:

.
├── LICENSE
├── README.md
├── rebar.config
└── src
        ├── cadena_app.erl
        ├── cadena.app.src
        └── cadena_sup.erl

1 directory, 6 files

Configuring Dev Release

We do the following steps, check the links for comments on what's going on for each step:

Add Dependencies
Configure relx section
1. Add overlay variables file vars.config
2. Add sys.config
3. Add vm.args

Build a release to test that everything is setup correctly:

$ rebar3 release

Run the release interactively with a console:

$ _build/default/rel/cadena/bin/cadena console

Output (edited and paths redacted for clarity):

Exec: erlexec
        -boot _build/default/rel/cadena/releases/0.1.0/cadena
        -boot_var ERTS_LIB_DIR erts-8.3/../lib
        -mode embedded
        -config    _build/default/rel/cadena/generated.conf/app.1.config
        -args_file _build/default/rel/cadena/generated.conf/vm.1.args
        -vm_args   _build/default/rel/cadena/generated.conf/vm.1.args
        -- console

Root: _build/default/rel/cadena
Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:4:4] [async-threads:64]
                      [kernel-poll:true]

18:31:12.150 [info] Application lager started on node 'cadena@127.0.0.1'
18:31:12.151 [info] Application cadena started on node 'cadena@127.0.0.1'
Eshell V8.3  (abort with ^G)
(cadena@127.0.0.1)1>

Quit:

(cadena@127.0.0.1)1> q().
ok

Non interactive start:

$ _build/default/rel/cadena/bin/cadena start

No output is generated if it's started, we can check if it's running by pinging the application:

$ _build/default/rel/cadena/bin/cadena ping

We should get:

pong

If we want we can attach a console to the running system:

$ _build/default/rel/cadena/bin/cadena attach

Output:

Attaching to /tmp/erl_pipes/cadena@127.0.0.1/erlang.pipe.1 (^D to exit)

(cadena@127.0.0.1)1>

If we press Ctrl+d we can dettach the console without stopping the system:

(cadena@127.0.0.1)1> [Quit]

We can stop the system whenever we want issuing the stop command:

$ _build/default/rel/cadena/bin/cadena stop

Output:

ok

Let's try it.

Non interactive start:

$ _build/default/rel/cadena/bin/cadena start

No output is generated if it's started, we can check if it's running by pinging the application:

$ _build/default/rel/cadena/bin/cadena ping

We should get:

pong

If we want we can attach a console to the running system:

$ _build/default/rel/cadena/bin/cadena attach

Output:

Attaching to /tmp/erl_pipes/cadena@127.0.0.1/erlang.pipe.1 (^D to exit)

(cadena@127.0.0.1)1>

Now let's quit with q():

(cadena@127.0.0.1)1> q().

Output:

ok

Now let's see if it's alive:

$ _build/default/rel/cadena/bin/cadena ping

Node 'cadena@127.0.0.1' not responding to pings.

Be careful with how you quit attached consoles in production systems :)

Configure Prod and Dev Cluster Releases

Building Prod Release

We start by adding a new section to rebar.config called profiles, and define 4 profiles that override the default release config with specific values, let's start by trying the prod profile, which we will use to create production releases of the project:

rebar3 as prod release

Output:

===> Verifying dependencies...
...
===> Compiling cadena
===> Running cuttlefish schema generator
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          _build/prod/lib
          erl-19.3/lib
===> Resolved cadena-0.1.0
===> Including Erts from erl-19.3
===> release successfully created!

Notice now that we have a new folder in the _build directory:

$ ls -1 _build

Output:

default
prod

The results of the commands run "as prod" are stored in the prod folder.

You will notice if you explore the prod/rel/cadena folder that there's a folder called erts-8.3 (the version may differ if you are using a different erlang version), that folder is there because of the include_erts option we overrided in the prod profile.

This means you can zip the _build/prod/rel/cadena folder, upload it to a server that doesn't have erlang installed in it and still run your release there.

This is a good way to be sure that the version running in production is the same you use in development or at build time in your build server.

Just be careful with deploying to an operating system too different to the one you used to create the release becase you may have problems with bindings like libc or openssl.

Running it is done as usual, only the path changes:

_build/prod/rel/cadena/bin/cadena console

_build/prod/rel/cadena/bin/cadena start
_build/prod/rel/cadena/bin/cadena ping
_build/prod/rel/cadena/bin/cadena attach
_build/prod/rel/cadena/bin/cadena stop

Building Dev Cluster Releases

To build a cluster we need at least 3 nodes, that's why the last 3 profiles are node1, node2 and node3, they need to have different node names, for that we use the overlay var files to override the name of each, that is achieved on config/vars_node1.config for node1, config/vars_node2.config for node2 and config/vars_node3.config for node3.

Now let's build them:

rebar3 as node1 release
rebar3 as node2 release
rebar3 as node3 release

The output for each should be similar to the one for the prod release.

Now on three different shells start each node:

./_build/node1/rel/cadena/bin/cadena console

Check the name of the node in the shell:

(node1@127.0.0.1)1>

Do the same for node2 and node3 on different shells:

./_build/node2/rel/cadena/bin/cadena console
./_build/node3/rel/cadena/bin/cadena console

You should get respectively:

(node2@127.0.0.1)1>

And:

(node3@127.0.0.1)1>

In case you don't remember, you can quit with q().

Joining the Cluster Together

Until here we built 3 releases of the same code with slight modifications to allow running a cluster on one computer, but 3 nodes running doesn't mean we have a cluster, for that we need to use what we learned in the Multi-Paxos with riak_ensemble Part 1 but now on code and not interactively.

For that we will create a cadena_console module that we will use to make calls from the outside and trigger actions on each node, the code is similar to the one presented in Multi-Paxos with riak_ensemble Part 1.

join([NodeStr]) ->
    % node name comes as a list string, we need it as an atom
    Node = list_to_atom(NodeStr),
    % check that the node exists and is alive
    case net_adm:ping(Node) of
        % if not, return an error
        pang ->
            {error, not_reachable};
        % if it replies, let's join him passing our node reference
        pong ->
            riak_ensemble_manager:join(Node, node())
    end.

create([]) ->
    % enable riak_ensemble_manager
    riak_ensemble_manager:enable(),
    % wait until it stabilizes
    wait_stable().

cluster_status() ->
    case riak_ensemble_manager:enabled() of
        false ->
            {error, not_enabled};
        true ->
            Nodes = lists:sort(riak_ensemble_manager:cluster()),
            io:format("Nodes in cluster: ~p~n",[Nodes]),
            LeaderNode = node(riak_ensemble_manager:get_leader_pid(root)),
            io:format("Leader: ~p~n",[LeaderNode])
    end.

We also need to add the riak_ensemble supervisor to our supervisor tree in cadena_sup:

init([]) ->
    % get the configuration from sys.config
    DataRoot = application:get_env(riak_ensemble, data_root, "./data"),
    % create a unique path for each node to avoid clashes if running more
    % than one node in the same computer
    NodeDataDir = filename:join(DataRoot, atom_to_list(node())),

    Ensemble = {riak_ensemble_sup,
                {riak_ensemble_sup, start_link,
                 [NodeDataDir]},
                permanent, 20000, supervisor, [riak_ensemble_sup]},

    {ok, { {one_for_all, 0, 1}, [Ensemble]} }.

Before building the dev cluster we need to add the crypto app to cadena.app.src since it's needed by riak_ensemble to create the cluster.

Now let's build the dev cluster, I created a Makefile to make it simpler:

make devrel

On three different shells run one command on each:

make node1-console
make node2-console
make node3-console

Let's make an rpc call to enable the riak_ensemble cluster on node1:

./_build/node1/rel/cadena/bin/cadena rpc cadena_console create

On node1 you should see something like:

[info] {root,'node1@127.0.0.1'}: Leading

Let's join node2 to node1:

./_build/node2/rel/cadena/bin/cadena rpc cadena_console join node1@127.0.0.1

On node1 you should see:

[info] join(Vsn): {1,152} :: 'node2@127.0.0.1' :: ['node1@127.0.0.1']

On node2:

[info] JOIN: success

Finally let's join node3:

./_build/node3/rel/cadena/bin/cadena rpc cadena_console join node1@127.0.0.1

Output on node1:

[info] join(Vsn): {1,453} :: 'node3@127.0.0.1' :: ['node1@127.0.0.1','node2@127.0.0.1']

On node3:

[info] JOIN: success

Let's check that the 3 nodes have the same view of the cluster, let's ask node1 what's the ensemble status:

./_build/node1/rel/cadena/bin/cadena rpc cadena_console ensemble_status

Nodes in cluster: ['node1@127.0.0.1','node2@127.0.0.1','node3@127.0.0.1']
Leader: 'node1@127.0.0.1'

node2:

$ ./_build/node2/rel/cadena/bin/cadena rpc cadena_console ensemble_status

Nodes in cluster: ['node1@127.0.0.1','node2@127.0.0.1','node3@127.0.0.1']
Leader: 'node1@127.0.0.1'

node3:

$ ./_build/node3/rel/cadena/bin/cadena rpc cadena_console ensemble_status

Nodes in cluster: ['node1@127.0.0.1','node2@127.0.0.1','node3@127.0.0.1']
Leader: 'node1@127.0.0.1'

Everything looks right, stop the 3 nodes (q().) and start them again, you will see that after starting up node1 logs:

[info] {root,'node1@127.0.0.1'}: Leading

And if you call ensemble_status on any node you get the same outputs as before, this means they remember the cluster topology even after restarts.