Skip to main content

Mariano Guerra's Log

This is my blog, more about me at marianoguerra.github.io

🦋 @marianoguerra.org 🐘 @marianoguerra@hachyderm.io 🐦 @warianoguerra

Riak Core on Partisan on Elixir Tutorial: Migrating Data with Handoff

Previous post: Riak Core on Partisan on Elixir Tutorial: We can make a Key Value Store out of that.

We are going to implement handoff by completing some of the remaining callbacks in our vnode.

Why? if a node dies and another takes his work or if we add a new node and the vnodes must be rebalanced we need to handle handoff.

The reasons to start a handoff are:

  • A ring update event for a ring that all other nodes have already seen.

  • A secondary vnode is idle for a period of time and the primary, original owner of the partition is up again.

When this happen riak_core will inform the vnode that handoff is starting, calling handoff_starting, if it returns false it’s cancelled, if it returns true it calls is_empty, that must return false to inform that the vnode has something to handoff (it’s not empty) or true to inform that the vnode is empty, in our case we ask for the first element of the ets table and if it’s the special value $end_of_table we know it’s empty, if it returns true the handoff is considered finished, if false then a call is done to handle_handoff_command.

The fold_req_v2 macro is defined in the riak_core_vnode.hrl header file as riak_core_fold_req_v2 which we include using the Record module.

This function must iterate through all the keys it stores and for each of them call foldfun with the key as first argument, the value as second argument and the latest acc_in value as third.

The result of the function call is the new acc_in you must pass to the next call to foldfun, the last accumulator value (acc_final) must be returned by handle_handoff_command.

For each call to foldfun(k, v, acc_in) riak_core will send it to the new vnode, to do that it must encode the data before sending, it does this by calling encode_handoff_item(k, v), where you must encode the data before sending it.

When the value is received by the new vnode it must decode it and do something with it, this is done by the function handle_handoff_data, where we decode the received data and do the appropriate thing with it.

When we sent all the key/values handoff_finished will be called and then delete so we cleanup the data on the old vnode.

You can decide to handle other commands sent to the vnode while the handoff is running, you can choose to do one of the following:

  • Handle it in the current vnode

  • Forward it to the vnode we are handing off

  • Drop it

What to do depends on the design of you app, all of them have tradeoffs.

A diagram of the flow:

+-----------+      +----------+        +----------+
|           | true |          | false  |          |
| Starting  +------> is_empty +--------> fold_req |
|           |      |          |        |          |
+-----+-----+      +----+-----+        +----+-----+
      |                 |                   |
      | false           | true              | ok
      |                 |                   |
+-----v-----+           |              +----v-----+     +--------+
|           |           |              |          |     |        |
| Cancelled |           +--------------> finished +-----> delete |
|           |                          |          |     |        |
+-----------+                          +----------+     +--------+

Replace the content of lib/civile_vnode.ex with:

defmodule Civile.VNode do
  require Logger
  @behaviour :riak_core_vnode

  require Record
  Record.defrecord :fold_req_v2, :riak_core_fold_req_v2, Record.extract(:riak_core_fold_req_v2, from_lib: "riak_core/include/riak_core_vnode.hrl")

  def start_vnode(partition) do
    :riak_core_vnode_master.get_vnode_pid(partition, __MODULE__)
  end

  def init([partition]) do
    table_name = :erlang.list_to_atom('civile_' ++ :erlang.integer_to_list(partition))

    table_id =
      :ets.new(table_name, [:set, {:write_concurrency, false}, {:read_concurrency, false}])

    state = %{
      partition: partition,
      table_name: table_name,
      table_id: table_id
    }

    {:ok, state}
  end

  def handle_command({:ping, v}, _sender, state = %{partition: partition}) do
    {:reply, {:pong, v + 1, node(), partition}, state}
  end

  def handle_command({:put, {k, v}}, _sender, state = %{table_id: table_id, partition: partition}) do
    :ets.insert(table_id, {k, v})
    res = {:ok, node(), partition, nil}
    {:reply, res, state}
  end

  def handle_command({:get, k}, _sender, state = %{table_id: table_id, partition: partition}) do
    res =
      case :ets.lookup(table_id, k) do
        [] ->
          {:ok, node(), partition, nil}

        [{_, value}] ->
          {:ok, node(), partition, value}
      end

    {:reply, res, state}
  end

  def handoff_starting(_dest, state = %{partition: partition}) do
    Logger.debug "handoff_starting #{partition}"
    {true, state}
  end

  def handoff_cancelled(state = %{partition: partition}) do
    Logger.debug "handoff_cancelled #{partition}"
    {:ok, state}
  end

  def handoff_finished(_dest, state = %{partition: partition}) do
    Logger.debug "handoff_finished #{partition}"
    {:ok, state}
  end

  def handle_handoff_command(fold_req_v2() = fold_req, _sender, state = %{table_id: table_id, partition: partition}) do
    Logger.debug "handoff #{partition}"
    foldfun = fold_req_v2(fold_req, :foldfun)
    acc0 = fold_req_v2(fold_req, :acc0)
    acc_final = :ets.foldl(fn {k, v}, acc_in ->
        Logger.debug "handoff #{partition}: #{k} #{v}"
        foldfun.(k, v, acc_in)
    end, acc0, table_id)
    {:reply, acc_final, state}
  end

  def handle_handoff_command(_request, _sender, state = %{partition: partition}) do
    Logger.debug "Handoff generic request, ignoring #{partition}"
    {:noreply, state}
  end

  def is_empty(state = %{table_id: table_id, partition: partition}) do
    is_empty = (:ets.first(table_id) == :"$end_of_table")
    Logger.debug "is_empty #{partition}: #{is_empty}"
    {is_empty, state}
  end

  def terminate(reason, %{partition: partition}) do
    Logger.debug "terminate #{partition}: #{reason}"
    :ok
  end

  def delete(state = %{table_id: table_id, partition: partition}) do
    Logger.debug "delete #{partition}"
    true = :ets.delete(table_id)
    {:ok, state}
  end

  def handle_handoff_data(bin_data, state = %{table_id: table_id, partition: partition}) do
    {k, v} = :erlang.binary_to_term(bin_data)
    :ets.insert(table_id, {k, v})
        Logger.debug "handle_handoff_data #{partition}: #{k} #{v}"
    {:reply, :ok, state}
  end

  def encode_handoff_item(k, v) do
     Logger.debug "encode_handoff_item #{k} #{v}"
     :erlang.term_to_binary({k, v})
  end

  def handle_coverage(_req, _key_spaces, _sender, state) do
    {:stop, :not_implemented, state}
  end

  def handle_exit(_pid, _reason, state) do
    {:noreply, state}
  end

  def handle_overload_command(_, _, _) do
    :ok
  end

  def handle_overload_info(_, _idx) do
    :ok
  end
end

Stop all nodes if you have them running.

Let's clean all the data on each node and the ring state so that we can join them again each time we want to try handoff, to make it simpler to run the commands and remember them, put this in a file called Makefile at the root of the project, make sure the indented lines are indented with tabs and not spaces:

start_single:
   iex --name dev@127.0.0.1 -S mix run

start_node1:
   MIX_ENV=dev1 iex --name dev1@127.0.0.1 -S mix run

start_node2:
   MIX_ENV=dev2 iex --name dev2@127.0.0.1 -S mix run

start_node3:
   MIX_ENV=dev3 iex --name dev3@127.0.0.1 -S mix run

clean:
   rm -rf data_1 data_2 data_3 data log ring_data_dir*

setup:
   mix deps.get

The project at this stage is available in the tag handoff-1

Now let's clean the date to start from scratch:

make clean

On a new terminal:

make start_node1

Inside iex run:

for i <- :lists.seq(1, 100) do
    Civile.Service.put("k#{i}", i)
end

This will insert k1: 1, k2: 2,..., k100: 100 , since we are running a single node they will all go to vnodes in node1.

On a new terminal:

make start_node2

Inside iex run:

for i <- :lists.seq(101, 200) do
    Civile.Service.put("k#{i}", i)
end

This will insert k101: 101, k102: 102,..., k200: 200 , since we haven't joined node2 to node1, they will all go to vnodes in node2.

On a new terminal:

make start_node3

Inside iex run:

for i <- :lists.seq(201, 300) do
    Civile.Service.put("k#{i}", i)
end

This will insert k201: 201, k202: 202,..., k300: 300 , since we haven't joined node3 to node1 or node2, they will all go to vnodes in node3.

Let's join node2 and node3 to node1 to form a cluster, on node2 and node3 run:

:riak_core.join('dev1@127.0.0.1')

You will see handoff transfering vnodes and their data between nodes and stopping the vnodes in the old nodes when it finishes transfering their data and removing it.

Periodically you can check the status with:

{:ok, ring} = :riak_core_ring_manager.get_my_ring
:riak_core_ring.pretty_print(ring, [:legend])

Riak Core on Partisan on Elixir Tutorial: We can make a Key Value Store out of that

Previous post: Riak Core on Partisan on Elixir Tutorial: Getting Started.

Now that we have the foundation of our service laid out, let's make it do something useful.

Given that riak_core is an implementation of the Dynamo Architecture which was created to build scalable key value stores, we should follow the same path and build one.

Let's start from the end, imagine our code is ready, what would be the interaction with it?

Let's imagine it's like this, first we ask the service to get the value for a key that doesn't exist yet:

iex(dev@127.0.0.1)1> Civile.Service.get(:k1)
{:ok, :"dev@127.0.0.1",
 913438523331814323877303020447676887284957839360, nil}

We get a tuple as response with 4 items:

  1. status = :ok

    • The request was handled correctly

  2. node = :"dev@127.0.0.1"

    • The node that handled the request

  3. partition = ...

  • The id of the partition that handled this request

  1. result = :nil

  • The operations' result (nil since the key has no value yet)

Let's set :k1 to 42:

iex(dev@127.0.0.1)2> Civile.Service.put(:k1, 42)
{:ok, :"dev@127.0.0.1",
 913438523331814323877303020447676887284957839360, nil}

We get a tuple as response with 4 items:

  1. status = :ok

    • The request was handled correctly

  2. node = :"dev@127.0.0.1"

    • The node that handled the request

  3. partition = ...

  • The id of the partition that handled this request (the same as before)

  1. result = :nil

  • Nothing of interest to result, use a 3 item tuple to keep the same shape as get

Now let's get :k1 again:

iex(dev@127.0.0.1)3> Civile.Service.get(:k1)
{:ok, :"dev@127.0.0.1",
 913438523331814323877303020447676887284957839360, 42}

Now we get 42 as result

Let's try with another key (:k2):

iex(dev@127.0.0.1)4> Civile.Service.get(:k2)
{:ok, :"dev@127.0.0.1",
 365375409332725729550921208179070754913983135744, nil}

Same response as before, but notice that the partition changed, we will see why later.

Let's set :k2 to :hello:

iex(dev@127.0.0.1)5> Civile.Service.put(:k2, :hello)
{:ok, :"dev@127.0.0.1",
 365375409332725729550921208179070754913983135744, nil}

Now get it:

iex(dev@127.0.0.1)6> Civile.Service.get(:k2)
{:ok, :"dev@127.0.0.1",
 365375409332725729550921208179070754913983135744, :hello}

Now let's build this, first let's create the API on lib/civile_service.ex:

defmodule Civile.Service do
  def ping(v \\ 1) do
    send_cmd("ping#{v}", {:ping, v})
  end

  def put(k, v) do
    send_cmd(k, {:put, {k, v}})
  end

  def get(k) do
    send_cmd(k, {:get, k})
  end

  defp send_cmd(k, cmd) do
    idx = :riak_core_util.chash_key({"civile", k})
    pref_list = :riak_core_apl.get_primary_apl(idx, 1, Civile.Service)

    [{index_node, _type}] = pref_list

    :riak_core_vnode_master.sync_command(index_node, cmd, Civile.VNode_master)
  end
end

The module was refactored to reuse the hashing logic in a common function called send_cmd/2. ping/1, put/2 and get/1 use send_cmd/2 by passing the value that will be used to hash as first argument and the command to send to the vnode as second argument.

On the vnode side we are going to use ets as the storage for our key value store, we need to initialize it in the Civile.VNode.init/1 function:

def init([partition]) do
  table_name = :erlang.list_to_atom('civile_' ++ :erlang.integer_to_list(partition))

  table_id =
    :ets.new(table_name, [:set, {:write_concurrency, false}, {:read_concurrency, false}])

  state = %{
    partition: partition,
    table_name: table_name,
    table_id: table_id
  }

  {:ok, state}
end

We create a table name unique for our app and partition, then create a new ets table that has read and write concurrency set to false since only this vnode will be reading and writing on it.

Now let's handle the put command:

def handle_command({:put, {k, v}}, _sender, state = %{table_id: table_id, partition: partition}) do
  :ets.insert(table_id, {k, v})
  res = {:ok, node(), partition, nil}
  {:reply, res, state}
end

Most of the code is extracting the values we need from the state to run the useful line:

:ets.insert(table_id, {k, v})

And then building the response to return.

The get command:

def handle_command({:get, k}, _sender, state = %{table_id: table_id, partition: partition}) do
  res =
    case :ets.lookup(table_id, k) do
      [] ->
        {:ok, node(), partition, nil}

      [{_, value}] ->
        {:ok, node(), partition, value}
    end

  {:reply, res, state}
end

It calls :ets.lookup/2 and handles the case where there's no value and when one is available.

The project at this stage is available in the tag kv-1

See if the code is correct by compiling the project:

mix compile

And then start one or 3 nodes and try the commands from the top of this chapter, let's do it with 3, first we clean the current state to start fresh:

rm -rf data_* data ring_data_dir*

Now let's run the 3 nodes by setting the right profile on MIX_ENV before running our commands:

MIX_ENV=dev1 iex --name dev1@127.0.0.1 -S mix run
MIX_ENV=dev2 iex --name dev2@127.0.0.1 -S mix run
MIX_ENV=dev3 iex --name dev3@127.0.0.1 -S mix run

On dev2 and dev3 console run the following to join the nodes:

:riak_core.join('dev1@127.0.0.1')

Now let's check the ring status until it settles since we are not migrating data on vnode migration yet:

{:ok, ring} = :riak_core_ring_manager.get_my_ring
:riak_core_ring.pretty_print(ring, [:legend])

From node2:

iex(dev2@127.0.0.1)3> Civile.Service.get(:k1)
{:ok, :"dev1@127.0.0.1",
 913438523331814323877303020447676887284957839360, nil}

From node3:

iex(dev3@127.0.0.1)12> Civile.Service.put(:k1, 42)
{:ok, :"dev1@127.0.0.1",
 913438523331814323877303020447676887284957839360, nil}

From node2:

iex(dev2@127.0.0.1)4> Civile.Service.get(:k1)
{:ok, :"dev1@127.0.0.1",
 913438523331814323877303020447676887284957839360, 42}

We can see that all requests where handled by node1 and the same partition since we used the same key on each request, which was used to decide which vnode/partion would handle it.

Riak Core on Partisan on Elixir Tutorial: Getting Started

Previous posts: Riak Core on Partisan on Elixir Tutorial: Introduction and Riak Core on Partisan on Elixir Tutorial: Setup.

Creating the project

mix new civiledb --app civile
cd civiledb

The project at this stage is available in the tag start-1

Add uniendo dependency

uniendo is a library that contains all required dependencies and overrides in a single place to make it easy to use from other projects, add it to the deps sections in mix.exs like this:

defp deps do
  [
    {:uniendo, git: "git://github.com/marianoguerra/uniendo.git", branch: "master"}
  ]
end

The project at this stage is available in the tag start-2

Add application configuration

We need to add riak_core to the list of applications to start and start our own supervisor tree, modify the application section in mix.exs like this:

def application do
  [
    extra_applications: [:riak_core, :logger],
    mod: {Civile, []}
  ]
end

The project at this stage is available in the tag start-3

First build

Let's try building the project as is, mainly to test that the dependencies are compiled correctly.

First we need to get the dependencies:

mix deps.get

If it's the first time it may ask you:

Could not find Hex, which is needed to build dependency :folsom
Shall I install Hex? (if running non-interactively, use "mix local.hex --force") [Yn]

Just hit enter to continue.

When it finishes, try to build the project:

mix compile

The project at this stage is available in the tag start-4

If it's the first time you build a project with dependencies written in erlang it may ask you:

Shall I install rebar3? (if running non-interactively, use "mix local.rebar --force") [Yn]

Just hit enter to continue.

If you get an error compiling you may want to try updating the rebar3 version mix uses with:

mix local.rebar rebar3 ~/bin/rebar3

If setup fails to build, try commenting the line on deps/setup/rebar.config:

{post_hooks, [{compile, "make escriptize"}]}.

Like this:

%{post_hooks, [{compile, "make escriptize"}]}.

Making it start

We need to start our application and riak_core, for that we need some initial setup:

mkdir priv config
cp deps/riak_core/priv/riak_core.schema priv/

The project at this stage is available in the tag start-5

Add the following content to the file in config/config.exs:

use Mix.Config

config :riak_core,
  ring_state_dir: 'ring_data_dir',
  handoff_port: 8099,
  handoff_ip: '127.0.0.1',
  schema_dirs: ['priv']

config :sasl,
  errlog_type: :error

The project at this stage is available in the tag start-6

Edit lib/civile.ex and change its content to:

defmodule Civile do
  use Application
  require Logger

  def start(_type, _args) do
    case Civile.Supervisor.start_link do
      {:ok, pid} ->
        {:ok, pid}
      {:error, reason} ->
        Logger.error("Unable to start Civile supervisor because: #{inspect reason}")
    end
  end
end

Create a new file called lib/civile_supervisor.ex with the following content:

defmodule Civile.Supervisor do
  use Supervisor

  def start_link do
    # riak_core appends _sup to the application name.
    Supervisor.start_link(__MODULE__, [], [name: :civile_sup])
  end

  def init(_args) do
    children = []
    supervise(children, strategy: :one_for_one, max_restarts: 5, max_seconds: 10)
  end

end

The project at this stage is available in the tag start-7

Now recompile the project:

mix compile

And start it:

iex --name dev@127.0.0.1 -S mix run

You should see something like this:

Erlang/OTP 22 [erts-10.5] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

00:48:07.530 [info] Starting reporters with []
00:48:07.577 [info] Using node name: 'dev@127.0.0.1'
00:48:07.581 [info] Resolving "127.0.0.1"...
00:48:07.583 [info] Resolved "dev@127.0.0.1" to {127,0,0,1}
00:48:07.583 [info] Resolved "127.0.0.1" to {127,0,0,1}
00:48:07.679 [info] Partisan listening on {127,0,0,1}:49489 listen_addrs: [#{ip => {127,0,0,1},port => 49489}]
00:48:07.686 [info] Not using container orchestration; disabling.
00:48:07.694 [info] node 'dev@127.0.0.1' choosing random seed: {52969312,-576460751045187665,-576460752303423453}
00:48:07.728 [info] node 'dev@127.0.0.1' choosing random seed: {52969312,-576460751045187665,-576460752303423453}
00:48:07.774 [info] Configuring partisan dispatch: false
dets: file "data/cluster_meta/manifest.dets" not properly closed, repairing ...
00:48:07.938 [info] New capability: {riak_core,vnode_routing} = proxy
00:48:07.942 [info] New capability: {riak_core,staged_joins} = true
00:48:07.947 [info] New capability: {riak_core,resizable_ring} = true
00:48:07.951 [info] New capability: {riak_core,fold_req_version} = v2
00:48:07.956 [info] New capability: {riak_core,security} = true
00:48:07.960 [info] New capability: {riak_core,bucket_types} = true
00:48:07.964 [info] New capability: {riak_core,net_ticktime} = true
Interactive Elixir (1.9.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(dev@127.0.0.1)1>

You may get a lager crash log at startup, ignore it.

Hit Ctrl+C twice to quit.

Implementing our VNode

Write the following on a new file called lib/civile_vnode.ex:

defmodule Civile.VNode do
  @behaviour :riak_core_vnode

  def start_vnode(partition) do
    :riak_core_vnode_master.get_vnode_pid(partition, __MODULE__)
  end

  def init([partition]) do
    {:ok, %{partition: partition}}
  end

  def handle_command({:ping, v}, _sender, state = %{partition: partition}) do
     {:reply, {:pong, v + 1, node(), partition}, state}
  end

  def handoff_starting(_dest, state) do
    {true, state}
  end

  def handoff_cancelled(state) do
    {:ok, state}
  end

  def handoff_finished(_dest, state) do
    {:ok, state}
  end

  def handle_handoff_command(_fold_req, _sender, state) do
    {:noreply, state}
  end

  def is_empty(state) do
    {true, state}
  end

  def terminate(_reason, _state) do
    :ok
  end

  def delete(state) do
    {:ok, state}
  end

  def handle_handoff_data(_bin_data, state) do
    {:reply, :ok, state}
  end

  def encode_handoff_item(_k, _v) do
  end

  def handle_coverage(_req, _key_spaces, _sender, state) do
    {:stop, :not_implemented, state}
  end

  def handle_exit(_pid, _reason, state) do
    {:noreply, state}
  end

  def handle_overload_command(_, _, _) do
    :ok
  end

  def handle_overload_info(_, _idx) do
    :ok
  end
end

Write the following on a new file called lib/civile_service.ex:

defmodule Civile.Service do

  def ping(v\\1) do
    idx = :riak_core_util.chash_key({"civile", "ping#{v}"})
    pref_list = :riak_core_apl.get_primary_apl(idx, 1, Civile.Service)

    [{index_node, _type}] = pref_list

    :riak_core_vnode_master.sync_command(index_node, {:ping, v}, Civile.VNode_master)
  end

end

In lib/civile_supervisor.ex, add the vnode master as a child to our supervisor

Change:

def init(_args) do
  children = []
  supervise(children, strategy: :one_for_one, max_restarts: 5, max_seconds: 10)
end

To:

def init(_args) do
  children = [
    worker(:riak_core_vnode_master, [Civile.VNode], id: Civile.VNode_master_worker)
  ]
  supervise(children, strategy: :one_for_one, max_restarts: 5, max_seconds: 10)
end

And register our vnode implementation on riak_core.

In lib/civile.ex change:

def start(_type, _args) do
  case Civile.Supervisor.start_link do
    {:ok, pid} ->
      {:ok, pid}
    {:error, reason} ->
      Logger.error("Unable to start Civile supervisor because: #{inspect reason}")
  end
end

To:

def start(_type, _args) do
  case Civile.Supervisor.start_link do
    {:ok, pid} ->
      :ok = :riak_core.register(vnode_module: Civile.VNode)
      :ok = :riak_core_node_watcher.service_up(Civile.Service, self())
      {:ok, pid}
    {:error, reason} ->
      Logger.error("Unable to start Civile supervisor because: #{inspect reason}")
  end
end

The project at this stage is available in the tag start-8

Now compile and run again:

mix compile
iex --name dev@127.0.0.1 -S mix run

Inside the shell try our new service:

iex(dev@127.0.0.1)1> Civile.Service.ping
{:pong, 2, :"dev@127.0.0.1",
 251195593916248939066258330623111144003363405824}

iex(dev@127.0.0.1)2> Civile.Service.ping 32
{:pong, 33, :"dev@127.0.0.1",
 342539446249430371453988632667878832731859189760}

The response is a tuple that contains the atom :pong, the number we passed (or 1 by default) incremented by one, the node and the partition that handled the reply. Right now we only have one node so that's not useful, but in the next steps it will make more sense.

Playing with clustering (in the same machine)

To build a cluster on the same machine we will need to make 3 builds with slightly different configurations to avoid each running node from reading another node's files or trying to use another node's ports, for this we will create 3 different configs and make 3 different builds.

First add a line at the end of config/config.exs:

import_config "#{Mix.env}.exs"

This will import the configuration for the given mix environment

Now create the following files:

config/dev1.exs

use Mix.Config

config :riak_core,
  node: 'dev1@127.0.0.1',
  web_port: 8198,
  handoff_port: 8199,
  ring_state_dir: 'ring_data_dir_1',
  platform_data_dir: 'data_1'

config/dev2.exs

use Mix.Config

config :riak_core,
  node: 'dev2@127.0.0.1',
  web_port: 8298,
  handoff_port: 8299,
  ring_state_dir: 'ring_data_dir_2',
  platform_data_dir: 'data_2'

config/dev3.exs

use Mix.Config

config :riak_core,
  node: 'dev3@127.0.0.1',
  web_port: 8398,
  handoff_port: 8399,
  ring_state_dir: 'ring_data_dir_3',
  platform_data_dir: 'data_3'

Since the default environment is dev we need to have a file for that one too, but without any config since the one in config.exs is ok:

config/dev.exs

use Mix.Config

The project at this stage is available in the tag start-9

Now let's run the 3 nodes by setting the right profile on MIX_ENV before running our commands:

MIX_ENV=dev1 iex --name dev1@127.0.0.1 -S mix run
MIX_ENV=dev2 iex --name dev2@127.0.0.1 -S mix run
MIX_ENV=dev3 iex --name dev3@127.0.0.1 -S mix run

On dev2 and dev3 console run the following to join the nodes:

:riak_core.join('dev1@127.0.0.1')

Now let's check the ring status:

{:ok, ring} = :riak_core_ring_manager.get_my_ring
:riak_core_ring.pretty_print(ring, [:legend])

It will take a while until the ring rebalances, run the two lines above periodically until they settle.

Now you can run ping from any node and it will work:

Civile.Service.ping 32

Given that the number we pass is used to decide the target vnode, this means that a call with the same number will end up on the same node, but from all nodes it will work transparently.

Running it on the 3 nodes after the cluster rebalanced at around 33% each I got:

iex(dev1@127.0.0.1)2> Civile.Service.ping 32
{:pong, 33, :"dev3@127.0.0.1",
 342539446249430371453988632667878832731859189760}

iex(dev2@127.0.0.1)5> Civile.Service.ping 32
{:pong, 33, :"dev3@127.0.0.1",
 342539446249430371453988632667878832731859189760}

iex(dev3@127.0.0.1)12> Civile.Service.ping 32
{:pong, 33, :"dev3@127.0.0.1",
 342539446249430371453988632667878832731859189760}

This means that the same value was handled by the same node and vnode no matter where the function was called.

Riak Core on Partisan on Elixir Tutorial: Setup

Previous post: Riak Core on Partisan on Elixir Tutorial: Introduction.

The Container Way

On the resources folder of civiledb there's a Dockerfile that creates a Docker image with ubuntu 18.04 and all the required dependencies setup, just clone the repo, install docker if you haven't yet and do this once:

cd resources
sudo docker build --build-arg UID=$(id -u) --build-arg GID=$(id -g) --build-arg UNAME=$(whoami) -t ubuntu-elixir:1804 ubuntu-1804

Then every time you want to do something with the project, from the root of the project run:

sudo docker run -it -v $PWD:/src -w /home/$(whoami) ubuntu-elixir:1804 bash

It will open a shell inside the ubuntu image with the project mounted in /src.

Install Erlang and rebar3

To start we need to have Erlang and rebar3 installed.

In this tutorial I assume Erlang 22 and rebar3 3.12.0 but it should work with Erlang 21 and newer versions of rebar3.

If you don't have Erlang installed or you don't have problem to install the latest one system wide you can try installing it with your package manager:

  • For Homebrew on OS X: brew install erlang

  • For MacPorts on OS X: port install erlang

  • For Ubuntu and Debian: apt-get install erlang

  • For Fedora: yum install erlang

  • For FreeBSD: pkg install erlang

Please check that the package version is 22.x, if not, check for instructions on how to install the Erlang Solutions package for Ubuntu, CentOS, Mac OS X, Debian or Fedora here: https://www.erlang-solutions.com/resources/download.html

Setting up rebar3

# download rebar3 to our bin directory
wget https://s3.amazonaws.com/rebar3/rebar3 -O $HOME/bin/rebar3

# set execution permissions for your user
chmod u+x $HOME/bin/rebar3

You will need to add $HOME/bin to your $PATH if it's not there already, run export PATH=$PATH:$HOME/bin to add it, you will need to run it on every new terminal session or add it to your shell's rc file (.bashrc, .zshrc or similar)

Just in case you have problems running the rebar3 commands with a different version, here's the version I'm using:

rebar3 version

Output:

rebar 3.12.0 on Erlang/OTP 22 Erts 10.5

Install Elixir with asdf

Go to https://asdf-vm.com, click Get Started and find the best way to start for you, I used this one:

git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.7.4

Then I installed the elixir plugin:

asdf plugin-add elixir

Then installed Elixir 1.9.1-otp-22:

asdf install elixir 1.9.1-otp-22

On every terminal I want to have this version of elixir available I do:

asdf local elixir 1.9.1-otp-22

To make sure you have everything setup try running:

iex

You should see something like this:

Erlang/OTP 22 [erts-10.5] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Interactive Elixir (1.9.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

Press Ctrl+C twice to exit.

Next post: Riak Core on Partisan on Elixir Tutorial: Getting Started.

Riak Core on Partisan on Elixir Tutorial: Introduction

This tutorial covers the practical aspects of building a distributed in memory key value store in Elixir using Riak Core, in this case a fork of it that uses Partisan as it's underlying distribution runtime instead of the default Distributed Erlang runtime.

This same tutorial can be used without change with standard Riak Core, I may add a chapter to set it up without partisan in the future, but it won't make much difference in terms of development.

What is Riak Core?

(From the Partisan website)

Riak Core is a distributed programming framework written in Erlang and based on the Amazon Dynamo system that influenced the design of the distributed database Riak, Apache Cassandra, and the distributed actor framework Akka.

In Riak Core, a distributed hash table is used to partition a hash space across a cluster of nodes. These virtual nodes—the division of the hash space into N partitions—are claimed by a node in the cluster, and the resulting ownership is stored in a data structure known as the ring that is periodically gossiped to all nodes in the cluster.

Requests for a given key are routed to a node in the cluster based on the current partitioning of virtual nodes to cluster nodes in the ring structure using consistent hashing, which minimizes the impact of reshuffling when nodes join and leave the cluster. Background processes are used for cluster maintenance; ownership handoff, (transferring virtual node ownership) metadata anti-entropy (an internal KVS for configuration metadata) and ring gossip (information about the cluster’s virtual node to node mapping).

/galleries/uniendo/riak-ring.png

What is Partisan?

(From the Partisan website)

Partisan is the design of an alternative runtime system for improved scalability and reduced latency in actor applications.

Partisan provides:

  • Better scalability by leveraging different network topologies for communication

  • Reduced latency through efficient parallel message scheduling for actor-to-actor communication

Partisan is provided as a user library in Erlang and achieves up to an order of magnitude increase in the number of nodes the system can scale to through runtime overlay selection, up to a 34.96x increase in throughput, and up to a 13.4x reduction in latency over Distributed Erlang.

Why Partisan?

I think it's an interesting project, and providing an easy way to set up a useful example on top of it may help more people learn about it, take advantage of it and maybe contribute to it.

Check the Scaling Riak Core section on the Partisan page for more information about the benefits of using Partisan.

I should mention that this fork is not the official one, so consider this if you want to put something based on this stack in production.

How to Follow this Tutorial

The tutorial provides the commands and code you need to add and change, so you can follow it by copying and pasting it in the right place.

It also provides a repository with one tag for each step, the tag corresponding to each step will be linked near the code changes.

Following the tutorial it's recommended that you don't change the project's name unless you want to make lot of renames and maybe get some errors if you forgot one. Of course for a real project based on this code you can rename it, usually you have to find and replace all mentions of 'civile', 'Civile' and 'civiledb'.

Here's the link to the project: https://gitlab.com/marianoguerra/civiledb/

Here's the link to the project tags: https://gitlab.com/marianoguerra/civiledb/-/tags

To follow the tutorial from the example repository you will need to have git installed, check the Git Website for instructions on how to install it.

You can clone the repository with the following command:

git clone https://gitlab.com/marianoguerra/civiledb.git

To change to another tag (replace start-1 with the tag you want to change to):

git checkout start-1

To go back to the last change:

git checkout master

Resources

This tutorial is based on many resources around the web, here's an incomplete list of them, some of them provide more details, theory and explanation, some are in Erlang, some don't run any more, but you can get something extra out of each of those.

Next post: Riak Core on Partisan on Elixir Tutorial: Setup.

Creemos en la Web: Estructura de un proyecto real

Hasta ahora hemos usado herramientas online para probar nuestros proyectos, en este capitulo vamos a ver como crear un proyecto completo en nuestra computadora.

Si preferis verlo en video:

Para empezar necesitamos crear una carpeta para nuestro proyecto, yo le voy a llamar miapp pero el nombre puede ser cualquiera que prefieras.

Para crear una carpeta hace click derecho en la carpeta donde querés crear tu proyecto y elegí del menú "Nueva Carpeta" o similar, renombrala a gusto y entra a la carpeta haciendole doble click.

Ahora necesitamos crear nuestro index.html, el archivo de entrada de nuestra aplicación, podes crearlo desde tu editor de texto preferido, si no tenes uno podes probar instalar notepad++ si estas en Windows y querés un editor liviano pero potente, si estas en linux fijate si tenes instalado gedit, kedit o kate, podes buscarlos escribiendo "editor" en el lanzador de aplicaciones.

Si querés algo mas potente podes instalar Visual Studio Code, va a tomar un poco mas de tiempo acostumbrarse a el y consume mas recursos pero es un editor completo usado por una gran parte de los programadores.

Volviendo a nuestro index.html, podes crearlo haciendo click derecho en la carpeta de proyecto y eligiendo "Nuevo Archivo" o similar y renombrandolo a index.html o podes crear un archivo nuevo desde tu editor y elegir "Guardar Como" y nombrarlo index.html en la carpeta de tu proyecto.

El contenido de index.html va a ser:

<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, minimum-scale=1.0, initial-scale=1, maximum-scale=1.0, user-scalable=no">
    <meta http-equiv="x-ua-compatible" content="ie=edge">
    <title>Mi App</title>
    <script src="./lib/vue.js"></script>
    <script type="module" src="./js/app.js"></script>
    <link rel="stylesheet" href="css/bootstrap.css" media="all"/>
    <link rel="shortcut icon" href="img/favicon.png">
  </head>
  <body>
    <div id="app">
       <p>Contador: <span>{{count}}</span></p>
       <button @click="count = count + 1">Incrementar</button>
    </div>
  </body>
</html>

Ahora dentro de la carpeta del proyecto crea los siguientes directorios:

js

donde vamos a escribir nuestro código javascript

lib

donde vamos a poner archivos javascript de otros que vamos a usar

css

donde vamos a poner nuestro código CSS

img

donde vamos a poner imagenes y otros archivos que necesitemos

Abri tu navegador y visita https://cdnjs.com/ en la barra de búsqueda de la pagina escribí "vue" te va a aparecer algo parecido a esto:

/galleries/cew/proyecto/vue-cdnjs.png

Hace click en el botón "Copy" que aparece a la derecha cuando acercas el mouse, abrí una pestaña nueva y pega la dirección (tecla Control y tecla v a la vez o click derecho y "Pegar"), apretá enter y te va a aparecer algo no muy legible, no importa, en el menú de tu navegador anda a "Archivo" y "Guardar pagina como" (o similar), elegí el directorio lib dentro de tu proyecto y nombra el archivo vue.js.

Este procedimiento sirve para cualquier otra biblioteca de funcionalidad que quieras usar en tu proyecto, guardas el archivo en el directorio lib y agregas el tag script en index.html, podes también poner la dirección directamente en el tag script como hemos hecho antes, solo que de esta forma podes programar incluso cuando no tenes acceso a internet.

Hagamos lo mismo con bootstrap anda a https://getbootstrap.com/ hace click en "Download", baja hasta la sección "BootstrapCDN", copia la dirección en el primer tag, en mi caso es "https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css", pegala en la barra de direcciones y guardala dentro del directorio css de tu proyecto como bootstrap.css.

Crea un archivo llamado app.js en el directorio js, pone el siguiente contenido dentro:

function main() {
   let app = new Vue({
      el: '#app',
      data: {count: 0}
   });
}

// llamar a la funcion main cuando la pagina termine de cargar
window.addEventListener('load', main);

Ahora para poder abrir nuestra aplicación en el navegador necesitamos un "servidor" web que le provea los archivos al navegador cuando este los pida, vamos a usar un servidor simple con una herramienta que vamos a necesitar de todas formas en los siguientes pasos.

Abrí tu navegador y visita https://nodejs.org/es/ hace click en el botón de descarga y seguí las instrucciones del instalador.

una vez instalado crea un archivo llamado miserver.js en el directorio del proyecto con el siguiente contenido:

var http = require('http');
var fs = require('fs');
var path = require('path');

http.createServer(function (request, response) {
    console.log('request ', request.url);

    var filePath = '.' + request.url;
    if (filePath == './') {
        filePath = './index.html';
    }

    var extname = String(path.extname(filePath)).toLowerCase();
    var mimeTypes = {
        '.html': 'text/html',
        '.js': 'text/javascript',
        '.css': 'text/css',
        '.json': 'application/json',
        '.png': 'image/png',
        '.jpg': 'image/jpg',
        '.gif': 'image/gif',
        '.wav': 'audio/wav',
        '.mp4': 'video/mp4',
        '.woff': 'application/font-woff',
        '.ttf': 'application/font-ttf',
        '.eot': 'application/vnd.ms-fontobject',
        '.otf': 'application/font-otf',
        '.svg': 'application/image/svg+xml',
        '.wasm': 'application/wasm'
    };

    var contentType = mimeTypes[extname] || 'application/octet-stream';

    fs.readFile(filePath, function(error, content) {
        if (error) {
            if(error.code == 'ENOENT') {
                fs.readFile('./404.html', function(error, content) {
                    response.writeHead(404, { 'Content-Type': contentType });
                    response.end(content, 'utf-8');
                });
            }
            else {
                response.writeHead(500);
                response.end('Sorry, check with the site admin for error: '+error.code+' ..\n');
            }
        }
        else {
            response.writeHead(200, { 'Content-Type': contentType });
            response.end(content, 'utf-8');
        }
    });

}).listen(8125);
console.log('Server running at http://127.0.0.1:8125/');

Mas adelante vamos a ver que significa ese código en detalle, por ahora a grandes rasgos implementamos nuestro propio servidor web usando javascript, que no es poca cosa :)

Guarda el archivo y abrí una terminal y escribí algo como:

cd miapp
node miserver.js

Vas a tener que cambiar "miapp" en cd miapp por el camino a la carpeta de tu proyecto, para saber donde estas podes correr el comando dir que te va a mostrar el contenido de la carpeta actual en la que estas.

Una vez que corrimos el comando node miserver.js tendría que mostrar lo siguiente:

Server running at http://127.0.0.1:8125/

Abrí tu navegador y visita esa dirección, debería abrirse nuestra aplicación.

Para seguir desarrollando simplemente cambia los archivos y recarga la pagina.

Eso es todo, así se desarrollan proyectos reales, no te preocupes si algunas cosas parecen complicadas, las vas a ir aprendiendo con el tiempo.

Cuando tengas algo listo para publicar simplemente copia la carpeta al sitio que aloje tu contenido, en capitulos siguientes vamos a ver algunas alternativas para alojar contenido.

A Simple, Understandable Production Ready Frontend Project Setup

I have low tolerance for complexity, I also like to understand the tools I use.

That's why over the years I've developed project setups myself after seeing towers of complex "boilerplate" fell apart once I tried to do some customization, or worse, for no apparent reason other tan bitrot or setting up the project on another machine.

Before the setup I will show you here I used to use requirejs as module system and the lowest common denominator of js features for development, the release step involved just minifying the module hierarchy.

With browsers keeping themselves up to date automatically and many "advanced" js features being available on them I started new projects by using some advanced features during development that are supported by the browsers I use to develop (Firefox and Chrome) and then only for the release I transpile those features to ES5.

I see a point in the near future where if we can target Chrome-based Edge and evergreen Firefox, Safari and Chrome, I can skip the tranpilation step and ship ES6 directly.

Here's the setup I use for my projects as an attempt to show a simpler alternative to current practices, or maybe to show how all the parts fit together by making them ourselves.

Initial Setup

Let's start by creating the folder for our project:

mkdir myapp
cd myapp

Let's create the basic structure:

mkdir js css lib img

Folder usage:

js

Our Javascript code

css

CSS files

lib

Libraries we use

img

Images, you can call it assets and put fonts and other things there too

Let's create our index.html with the following content:

<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, minimum-scale=1.0, initial-scale=1, maximum-scale=1.0, user-scalable=no">
    <meta http-equiv="x-ua-compatible" content="ie=edge">
    <title>My App</title>
    <script src="./lib/deps.js"></script>
    <script type="module" src="./js/app.js"></script>
    <link rel="stylesheet" href="css/bootstrap.css" media="all"/>
    <link rel="shortcut icon" href="img/favicon.png">
  </head>
  <body>
    <div id="app"></div>
  </body>
</html>

The file already gives you some ideas of the next steps, you may have noticed that the script tag has type="module" in it, that's because (new Date()).getFullYear() >= 2019 and we are going to use Javascript Modules supported by all evergreen browsers and easily translatable to old javascript syntax with a single command.

Let's create a file at js/app.js with the following content:

function main() {
  document.getElementById('app').innerHTML = 'Hello World';
}

// call main when the page completed loading
window.addEventListener('load', main);

Now we can try that it works by starting a server that will serve the files, I use python's builtin HTTP server when I'm starting, you can use others:

If you have python 2.x installed:

python -m SimpleHTTPServer

If you have python 3.x installed:

python3 -m http.server

Now open your browser at http://localhost:8000/

Let's see how to use modules, let's create a module at js/util.js and use it from js/app.js, write the following in js/util.js.

function byId(id) {
  return document.getElementById(id);
}

function setNodeText(node, text) {
  node.innerText = text;
}

export {byId, setNodeText};

And change js/app.js to use our new util module:

import {byId, setNodeText} from './util.js';

function main() {
  let node = byId('app');
  setNodeText(node, 'Hello World!');
}

window.addEventListener('load', main);

You can read more about import and export syntax on the MDN page about import and the MDN page about export.

Now let's add bootstrap to it:

wget https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css -O css/bootstrap.css

Yes, I just downloaded it to the css folder, I don't change my deps so often, when I do I want to do it manually and check that everything works, I also want my workflow to work when I don't have an internet connection and be fully versioned at every point so I can checkout any commit and dependencies will be at the right version.

Building a Release

To build a release we need to bundle all our modules together and minify it, to bundle them we are going to use rollup from the command line, first we need to install it if we don't have it:

npm install -g rollup

Then to use it:

# remove dist if it existed
rm -rf dist
# create it again
mkdir dist
# bundle app.js and all it's dependencies into dist/bundle.js
rollup js/app.js --file dist/bundle.js --format iife -n app

You can open dist/bundle.js and confirm that all our code is there as a single vanilla js file.

Now we want to minify it, for that we will use uglify-es, first we need to install it:

npm install -g uglify-es

Then we can use it:

uglifyjs dist/bundle.js -m -o dist/bundle.min.js

You can open dist/bundle.min.js and confirm that it's our code but minified.

What if we want to use new js features?

One option is to just use the features available on our minimum common denominator browser (which if we target Edge and evergreen Firefox and Chrome it's a lot!) and our build step stays as is (es modules -> rollup -> uglify-es), the alternative if we are targeting older browsers (I hope it's just IE 11) is to transpile our new javascript into an older version like ES5, we can achieve that with babel, first we install it:

npm install babel-cli babel-preset-es2015

We put a file at the root of our project called .babelrc to tell babel what options we want to use:

{"presets": ["babel-preset-es2015"]}

And then use it:

babel dist/bundle.js --minified -o dist/bundle.babel.js

Let's modify our code a little to see the transpiling in action, change the main function in js/app.js to look like this:

function main() {
  let node = byId('app'),
    items = [1, 2, 3, 4];

  setNodeText(node, items.map(v => '#' + v));
}

We are using arrow functions in items.map, now let's do the build:

rm -rf dist
mkdir dist
rollup js/app.js --file dist/bundle.js --format iife -n app
babel dist/bundle.js --minified -o dist/bundle.babel.js
# notice that we are now minifying the output of babel
uglifyjs dist/bundle.babel.js -m -o dist/bundle.min.js

If you check dist/bundle.babel.js you will notice that the let turned into var and the arrow function turned into a anonymous function.

You will also notice that the output from babel is already sort of minified, we can let it as is or pass it through uglify, that in my experience makes the output even smaller. In my case the outputs here are 312 bytes for babel and 231 for uglify.

There's a final step in the build process, if you remember our script tag said type="module" and our build is no longer using modules, so we need to get rid of it, let's do that:

sed 's/type="module" //g' index.html > dist/index.html

Done, we just remove type="module" in index.html and put the result in dist/index.html

So, now we have a production ready build process in 5 commands, let's work with dependencies.

Let's try using react, it will be similar with vue (even simpler), but I want to show how to use react without fancy things:

wget https://unpkg.com/react@16.8.6/umd/react.production.min.js -O lib/react.js
wget https://unpkg.com/react-dom@16.8.6/umd/react-dom.production.min.js -O lib/react-dom.js

Yes, I just downloaded the minified versions from cdnjs.

Now let's bundle all our dependencies:

cat lib/react.js lib/react-dom.js > lib/deps.js

If you check index.html is already loading lib/deps.js before our app.js so it will just work.

Our build needs an extra step, that is: copying the css, libs and images into the build directory:

mkdir -p dist/lib dist/css
cp lib/deps.js dist/lib/
cp css/bootstrap.css dist/css/
cp -r img dist/

Let's use react without a build step (aka no JSX), change js/app.js to look like this:

import {byId} from './util.js';
import {render, div, span, button} from './dom.js';

// global state
const STATE = {counter: 0};

function counterClicked(state) {
  state.counter += 1;
  doRender();
}

function renderRoot(state) {
  return div(
    {},
    span({}, 'Counter: ', state.counter),
    button({onClick: _e => counterClicked(state)}, 'Increment')
  );
}

let rootNode;
function doRender() {
  render(renderRoot(STATE), rootNode);
}

function main() {
  rootNode = byId('app');
  doRender();
}

window.addEventListener('load', main);

Here I'm doing a simplification with doRender calls, event handling and state management to avoid introducing any other library and make the example more complex, what you use for state management is up to you.

The important part is the renderRoot, it's all vanilla js calls, I don't have to learn a new syntax or mix two different syntaxes, I don't need a compiler running, I can use everything I know about javascript for my render code.

Let's see the "magic" behind the dom.js module:

/*globals React, ReactDOM*/
const c = React.createFactory.bind(React),
  div = c('div'),
  span = c('span'),
  button = c('button'),
  render = ReactDOM.render;

export {render, div, span, button};

Yep, that's all, of course in my real dom.js module I have all the HTML and SVG tags instead of just 3.

Now you may want to automate this instead of copying and pasting the commands, I have a Makefile that does the job for me, you can put it in shell scripts or whatever works for you, here's a Makefile for this project:

setup:
        npm install rollup uglify-es babel-cli babel-preset-es2015
        mkdir -p js lib css img
        wget https://unpkg.com/react@16.8.6/umd/react.production.min.js -O lib/react.js
        wget https://unpkg.com/react-dom@16.8.6/umd/react-dom.production.min.js -O lib/react-dom.js
        cat lib/react.js lib/react-dom.js > lib/deps.js
        wget https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css -O css/bootstrap.css

build:
        rm -rf dist
        mkdir -p dist/lib dist/css dist/js
        sed 's/type="module" //g' index.html > dist/index.html
        rollup js/app.js --file dist/bundle.js --format iife -n app
        babel dist/bundle.js --minified -o dist/bundle.babel.js
        uglifyjs dist/bundle.babel.js -m -o dist/bundle.min.js
        cp dist/bundle.min.js dist/js/app.js
        cp lib/deps.js dist/lib/
        cp css/bootstrap.css dist/css/
        cp -r img dist/

serve:
        python3 -m http.server

The "workflow" is, run make setup only once, change and reload as needed, when you want to ship, run make build, you can do cd dist; make serve and opening http://localhost:8000/ to check that it works.

To deploy remove dist/bundle*.js and copy the dist folder to its destination.

You may think that I'm lying and telling you something I don't use, well, this is the same workflow I use to build 4 apps at instadeq.

The main differences are:

  • I have a target in the makefile called setup-dev that fetches the non minified versions of the libraries to diagnose errors easily and be able to step into library code.

  • I have some extra dependencies

That's all, well... I also use eslint and prettier, but that's in my editor config and not in the project.

But.. live reloading?

I used to have it, basically by storing the current state in the window object, reloading the script tag and restoring the state if it was in the window object at startup (it was less than 30 lines of code), now I develop in a notebook style so I create an example of the thing I want to implement and just reload, the reload is fast and the example is in the initial state I want, no need to reimplement live reloading yet.

So there it is, a simple, modern, understandable and production ready setup for your frontends.

search erlang & efene code by pattern matching code with "holes"

Why and what?

One day, for weird reasons I was looking for a data structure that would allow me to pattern match it quick without being confused by existing data, I thought "nobody would have a tuple inside a one item tuple!", but I wasn't sure.

This wasn't the first time I wanted to search code by structure, not by text.

What I mean by "by structure"?

You may want to search for stuff like:

  • Calls to function f1 on module m1 with 3 arguments

    • Same but where some argument has a specific value

  • Tuples with a specific number of items where the first item is a given atom

I guess you get the idea, you want to match some expression's structure, but not the text, because of formatting, new lines, spacing and also because some of the values are not important, you would like to match any expression in some places.

Given that I have experience generating Erlang AST (abstract syntax tree) and walking AST trees (for parse transforms and some macro magic in efene)

I decided to give it a try, and the result worked, a year later I decided to write a blog post about it :)

erlplorer search "{{_@0, _@1, _@2}}" **/src/*.erl

asn1/src/asn1ct.erl:1366 {{decode,{Module,Type,Value},Error}}
dialyzer/src/dialyzer_plt.erl:675 {{M,_F,_A}}
kernel/src/pg2.erl:300 {{local_member,Name,Pid}}
kernel/src/pg2.erl:302 {{pid,Pid,Name}}
kernel/src/pg2.erl:343 {{local_member,Name,'$1'}}
kernel/src/pg2.erl:354 {{pid,Pid,'$1'}}
mnesia/src/mnesia_locker.erl:263 {{Tid,Oid,Op}}
mnesia/src/mnesia_locker.erl:266 {{Tid,Oid,read}}
mnesia/src/mnesia_locker.erl:269 {{Tid,Oid,write}}
mnesia/src/mnesia_locker.erl:309 {{Tid,Oid,{queued,Op}}}
mnesia/src/mnesia_locker.erl:508 {{Tid,Oid,{queued,Op}}}
mnesia/src/mnesia_locker.erl:524 {{'$1','_','_'}}
mnesia/src/mnesia_locker.erl:536 {{Tid,'_','_'}}
parsetools/src/yecc.erl:1075 {{From,Sym,Next}}
parsetools/src/yecc.erl:1164 {{From,Sym,To}}
reltool/src/reltool_server.erl:764 {{'$1','$2','$3'}}

Welp, yes, there are.

But what was that?

erlplorer is a command line tool built in efene that allows to search for Erlang and efene code by providing expressions with holes.

The first argument "{{_@0, _@1, _@2}}" is an Erlang expression, in this case a one item tuple holding a 3 item tuple, the weird looking variables are a specially named variables that erlplorer interprets as "match any AST node here, I don't care", they start with _@, any other variable will match that variable name in the code.

We can see that by searching for places that match a 3 item tuple ignoring the 3 places:

erlplorer search "{_, _, _}" **/src/*.erl

asn1/src/asn1ct_check.erl:917 {_,_,_}
... to many results to show

We can use this "meta variables" to do more pattern matching, let's search for 3 item tuples that have the same thing in the 3 places:

erlplorer search "{_@, _@, _@}" **/src/*.erl

asn1/src/asn1ct_check.erl:917 {_,_,_}
...
asn1/src/asn1ct_constructed_ber_bin_v2.erl:701 {[],[],[]}
...
asn1/src/asn1ct_constructed_per.erl:401 {false,false,false}
...
common_test/src/ct_framework.erl:1130 {_,_,_}
...
common_test/src/ct_logs.erl:1492 {"","",""}
...

Or search for identity functions:

erlplorer search "fun(_@) -> _@ end" **/src/*.erl

asn1/src/asn1ct.erl:2099 fun(D) -> D end
asn1/src/asn1ct_gen_ber_bin_v2.erl:650 fun(V) -> V end
common_test/src/test_server.erl:2067 fun(T) -> T end
compiler/src/beam_a.erl:147 fun(L) -> L end
compiler/src/beam_jump.erl:289 fun(Old) -> Old end
...

Places that add 0 to something:

erlplorer search "_@ + 0" **/src/*.erl

asn1/src/asn1rtt_per_common.erl:187 N + 0
asn1/src/asn1rtt_per_common.erl:197 N + 0
stdlib/src/ms_transform.erl:90 16 + 0
stdlib/src/ms_transform.erl:92 17 + 0
stdlib/src/ms_transform.erl:97 22 + 0
stdlib/src/ms_transform.erl:102 18 + 0
stdlib/src/ms_transform.erl:106 23 + 0
stdlib/src/ms_transform.erl:111 24 + 0
stdlib/src/ms_transform.erl:167 20 + 0
stdlib/src/ms_transform.erl:170 19 + 0
stdlib/src/ms_transform.erl:174 21 + 0

Places that add the same thing:

erlplorer search "_@ + _@" **/src/*.erl

dialyzer/test/small_SUITE_data/src/maps_redef2.erl:18 A + A
stdlib/src/dets_utils.erl:1138 1 + 1
stdlib/src/dets_utils.erl:1226 1 + 1
stdlib/src/rand.erl:1464 Y + Y
stdlib/src/zip.erl:1315 Sz + Sz

You get the idea...

How to use it

You need Erlang and rebar3 installed and in your $PATH

git clone https://github.com/marianoguerra/erlplorer
cd erlplorer
rebar3 escriptize
# ~/bin or any other folder in your $PATH
cp _build/default/bin/erlplorer ~/bin

How is it implemented?

Wrap the expression passed in a function [1]

Compile the module to Erlang AST [2]

Extract the AST body of the dummy function [3]

"abstract the AST", that is, I take an AST as Erlang data and I generate an AST that when compiled will generate that Erlang AST, I need that because I will put that AST in a pattern match position to pattern match AST nodes as I walk Erlang ASTs [4]. yeah, meta and too many AST references

An example is worth many of my words:

% the 42 is a fake "line" where the code was supposedly parsed
1> erl_parse:abstract({foo, 100, "hi"}, 42).
{tuple,42,[{atom,42,foo},{integer,42,100},{string,42,"hi"}]}

We have to take care of two things:

  • vars must match an AST node for a var with that name, not act as vars that will be bound on first match and pattern matched on successive matches

  • vars that start with _@ will be compiled to actual vars that behave as vars in a pattern match, that's how we can use them to pattern match

Then compile the abstracted AST into a module with a function we can pass to ast_walk/3 [5]

Load the compiled module [6]

Parse the files passed as last argument [7]

And walk the parsed AST with our compiled matcher [8]

For each match, since we have the AST, try to pretty print it [9]

Give it a try and let me know what you think.

PS: the code is a hack I did to use it when I needed it, don't judge efene by the code you see on that project :P

Elixir protocols, how do they work? the erlang perspective

We will start by creating a new project to learn more about Elixir protocols:

mix new learn --module Learn
cd learn

note: I'm using Erlang 21.3 and Elixir 1.8.1 and I never coded in Elixir before :)

I searched for Elixir protocols and found the official documentation with an example for a Size protocol I added it to the lib/learn.ex file and added some calls on the hello function to try it, it ended up looking like this:

defmodule Learn do
  @moduledoc """
  Documentation for Learn.
  """

  @doc """
  Hello world.

  ## Examples

  iex> Learn.hello()

  """
  def hello do
    Learn.Size.size("asd")
    Learn.Size.size(%{})
    Learn.Size.size({1, 2, 3})
  end

  defprotocol Size do
    @doc "Calculates the size (and not the length!) of a data structure"
    def size(data)
  end

  defimpl Size, for: BitString do
    def size(string), do: byte_size(string)
  end

  defimpl Size, for: Map do
    def size(map), do: map_size(map)
  end

  defimpl Size, for: Tuple do
    def size(tuple), do: tuple_size(tuple)
  end

end

Compiled the project:

mix compile

Opened an Elixir shell:

iex

Wrote a little script to decompile all beam files to Erlang (warning: Elixir flavored Erlang ahead!):

for f <- :filelib.wildcard('./_build/dev/lib/*/*/*.beam') do
  result = :beam_lib.chunks(f,[:abstract_code])
  {:ok,{_,[{:abstract_code,{_,ac}}]}} = result
  code = :erl_prettypr.format(:erl_syntax.form_list(ac))
  out_path = :string.replace(f, '.beam', '.erl')
  :file.write_file(out_path, code)
end

The results:

$  tree
.
├── _build
│   └── dev
│       └── lib
│           └── learn
│               ├── consolidated
│               │   ├── Elixir.Collectable.beam
│               │   ├── Elixir.Collectable.erl
│               │   ├── Elixir.Enumerable.beam
│               │   ├── Elixir.Enumerable.erl
│               │   ├── Elixir.IEx.Info.beam
│               │   ├── Elixir.IEx.Info.erl
│               │   ├── Elixir.Inspect.beam
│               │   ├── Elixir.Inspect.erl
│               │   ├── Elixir.Learn.Size.beam
│               │   ├── Elixir.Learn.Size.erl
│               │   ├── Elixir.List.Chars.beam
│               │   ├── Elixir.List.Chars.erl
│               │   ├── Elixir.String.Chars.beam
│               │   └── Elixir.String.Chars.erl
│               └── ebin
│                   ├── Elixir.Learn.beam
│                   ├── Elixir.Learn.erl
│                   ├── Elixir.Learn.Size.beam
│                   ├── Elixir.Learn.Size.BitString.beam
│                   ├── Elixir.Learn.Size.BitString.erl
│                   ├── Elixir.Learn.Size.erl
│                   ├── Elixir.Learn.Size.Map.beam
│                   ├── Elixir.Learn.Size.Map.erl
│                   ├── Elixir.Learn.Size.Tuple.beam
│                   ├── Elixir.Learn.Size.Tuple.erl
│                   └── learn.app

From the result it seems that it "consolidates" the protocols into the consolidated folder and then puts the modules at ebin (with protocol implementations named like the protocol plus the type they handle).

It's also clear that all Elixir modules are prefixed with Elixir., also that if I declare a protocol inside a module the protocol "belongs" to the module, in this case the "full qualified name" of the protocol is Elixir.Learn.Size.

Let's start exploring what code is generated by inspecting the main module we wrote (I will cleanup unneeded code from the examples):

-module('Elixir.Learn').
-export([hello/0]).

hello() ->
    'Elixir.Learn.Size':size(<<"asd">>),
    'Elixir.Learn.Size':size(#{}),
    'Elixir.Learn.Size':size({1, 2, 3}).

We can see that calling a function from a protocol implies calling the desired function on the consolidated module for the protocol itself.

Let's now see what the Elixir.Learn.Size module does:

-module('Elixir.Learn.Size').
-export(['__protocol__'/1, impl_for/1, 'impl_for!'/1, size/1]).

'impl_for!'(__@1) ->
    case impl_for(__@1) of
      __@2 when __@2 =:= nil orelse __@2 =:= false ->
      erlang:error('Elixir.Protocol.UndefinedError':exception([{protocol,
                                    'Elixir.Learn.Size'},
                                   {value,
                                    __@1}]));
      __@3 -> __@3
    end.

size(__@1) -> ('impl_for!'(__@1)):size(__@1).

struct_impl_for(_) -> nil.

impl_for(#{'__struct__' := __@1})
    when erlang:is_atom(__@1) ->
    struct_impl_for(__@1);
impl_for(__@1) when erlang:is_tuple(__@1) ->
    'Elixir.Learn.Size.Tuple';
impl_for(__@1) when erlang:is_map(__@1) ->
    'Elixir.Learn.Size.Map';
impl_for(__@1) when erlang:is_bitstring(__@1) ->
    'Elixir.Learn.Size.BitString';
impl_for(_) -> nil.

'__protocol__'(module) -> 'Elixir.Learn.Size';
'__protocol__'(functions) -> [{size, 1}];
'__protocol__'('consolidated?') -> true;
'__protocol__'(impls) ->
    {consolidated,
     ['Elixir.Map', 'Elixir.BitString', 'Elixir.Tuple']}.

The exported function for the protocol (size/1) does a simple thing, it asks the impl_for!/1 function for the module that knows how to handle Learn.Size.size/1 for the given argument and then calls that module's size/1 function:

size(__@1) -> ('impl_for!'(__@1)):size(__@1).

impl_for!/1 just calls impl_for/1 with the argument and handles the case where the value doesn't have a known implementation, in that case it raises an exception (Elixir.Protocol.UndefinedError), otherwise it just returns the module name.

impl_for/1 starts by checking if the argument is an Elixir struct, which underneath is just a map with a "well known" key __struct__ that contains the type of the struct as an atom:

impl_for(#{'__struct__' := __@1})
    when erlang:is_atom(__@1) ->

if it's a struct it calls struct_impl_for/1 with the struct type as argument:

struct_impl_for(__@1);

In our example, there's no struct that implements this protocol so the implementation of struct_impl_for/1 is simple:

struct_impl_for(_) -> nil.

After that it starts trying to find the implementation for non protocol types (mostly Erlang types), it tries to match using guards to check for the types, if none match, it returns nil like struct_impl_for/1:

impl_for(__@1) when erlang:is_tuple(__@1) ->
    'Elixir.Learn.Size.Tuple';

impl_for(__@1) when erlang:is_map(__@1) ->
    'Elixir.Learn.Size.Map';

impl_for(__@1) when erlang:is_bitstring(__@1) ->
    'Elixir.Learn.Size.BitString';

impl_for(_) -> nil.

Now that we got the module that handles the protocol function for each type, let's see their implementations:

Elixir.Learn.Size.BitString:

size(_string@1) -> erlang:byte_size(_string@1).

Elixir.Learn.Size.Map:

size(_map@1) -> erlang:map_size(_map@1).

Elixir.Learn.Size.Tuple:

size(_tuple@1) -> erlang:tuple_size(_tuple@1)

Now that we got the basic call and dispatch sequence let's try adding two structs and implement this protocol to see how it works for them:

I added two structs to the lib/learn.ex module:

defstruct name: "John", age: 27

defmodule User do
  defstruct name: "John", age: 27
end

Added calls to Size.size/1 in the hello/0 function:

def hello do
  Learn.Size.size("asd")
  Learn.Size.size(%{})
  Learn.Size.size({1, 2, 3})
  Learn.Size.size(%User{age: 27, name: "John"})
  Learn.Size.size(%Learn{age: 27, name: "John"})
end

And implemented the protocol Size for both structs:

defimpl Size, for: Learn do
  def size(learn), do: learn.age + 1
end

defimpl Size, for: User do
  def size(user), do: user.age + 2
end

Compiled with mix compile and inside iex pasted the script again, let's see what changed.

The hello world function looks like this:

hello() ->
        'Elixir.Learn.Size':size(<<"asd">>),
        'Elixir.Learn.Size':size(#{}),
        'Elixir.Learn.Size':size({1, 2, 3}),
        'Elixir.Learn.Size':size(#{age => 27,
                                   name => <<"John">>,
                                   '__struct__' => 'Elixir.Learn.User'}),
        'Elixir.Learn.Size':size(#{age => 27,
                                   name => <<"John">>,
                                   '__struct__' => 'Elixir.Learn'}).

Which confirms that Elixir structs are maps with a special __struct__ key.

Checking the generated files, there's a new file for our User struct (Elixir.Learn.User.erl), the other struct is defined inside Elixir.Learn.erl.

The module code relevant for the struct doesn't have anything specific to the protocols it implements:

-module('Elixir.Learn.User').
-export([_struct__'/0, '__struct__'/1]).

'__struct__'() ->
        #{'__struct__' => 'Elixir.Learn.User', age => 27,
          name => <<"John">>}.

'__struct__'(__@1) ->
        'Elixir.Enum':reduce(__@1,
                         #{'__struct__' => 'Elixir.Learn.User', age => 27,
                           name => <<"John">>},
                         fun ({__@2, __@3}, __@4) ->
                                 maps:update(__@2, __@3, __@4)
                         end).

Almost the same code is inside Elixir.Learn.erl for the other struct.

This shows that each struct has two "constructors", one without arguments that returns a struct with the default values for all fields and one that merges the arguments on the default values.

Let's see what changed on the consolidated protocol module:

struct_impl_for('Elixir.Learn.User') ->
        'Elixir.Learn.Size.Learn.User';
struct_impl_for('Elixir.Learn') ->
        'Elixir.Learn.Size.Learn';
struct_impl_for(_) -> nil.

Each struct type returns the module where the protocol is implemented, let's see both implementations:

Elixir.Learn.Size.Learn.User.erl:

size(_user@1) ->
        case _user@1 of
          #{age := __@1} -> __@1;
          __@1 when erlang:is_map(__@1) ->
          erlang:error({badkey, age, __@1});
          __@1 -> __@1:age()
        end
          + 2.

Elixir.Learn.Size.Learn.erl:

size(_learn@1) ->
        case _learn@1 of
          #{age := __@1} -> __@1;
          __@1 when erlang:is_map(__@1) ->
          erlang:error({badkey, age, __@1});
          __@1 -> __@1:age()
        end
          + 1.

Summary:

Elixir protocols are compiled to its own module whose content is the consolidated dispatch logic for it.

This logic is created by getting all the defimpl statements for it and adding a function clause to the struct_impl_for/1 function if the target type is an Elixir struct and a clause to the impl_for/1 function if the target type is any other type.

The function (impl_for!/1) returns the module that has the protocol implementation for the provided type.

Each protocol function asks for the module via impl_for!/1 and calls it with the given arguments.

This is just guessing, but the module indirection must be there to allow hot code reloading protocol implementations for each type independently without requiring also reloading the protocol consolidation. The struct_impl_for function is there to destructure the map only once.

I don't see traces of dynamic dispatch in case a module is loaded with a protocol implementation that was not known at consolidation time, I need to research this further.

An extra guess, this logic on the struct field to get the age field:

case _learn@1 of
  #{age := __@1} -> __@1;
  __@1 when erlang:is_map(__@1) ->
  erlang:error({badkey, age, __@1});
  __@1 -> __@1:age()
end

May be because Elixir allows to call a struct "method" without parenthesis and that's why it looks for the field first and the function with the same name second? I'm not entirely sure since my Elixir knowledge is basically non existent :)

If you have any questions or corrections I'm @warianoguerra my other accounts here: https://keybase.io/marianoguerra

Download frontend generated data to a file with clojurescript

I had to write this because I couldn't find it with a quick search, so here it his for future people like me (or me).

This is how you allow users to download a file from data generated in the browser with the save file dialog in cljs:

(defn to-json [v] (.stringify js/JSON v))

(defn download-object-as-json [value export-name]
        (let [data-blob (js/Blob. #js [(to-json value)] #js {:type "application/json"})
                  link (.createElement js/document "a")]
          (set! (.-href link) (.createObjectURL js/URL data-blob))
          (.setAttribute link "download" export-name)
          (.appendChild (.-body js/document) link)
          (.click link)
          (.removeChild (.-body js/document) link)))

You call it like this:

(download-object-as-json (clj->js {:hello "world"}) "myfile.json")

As an extra, it shows many idioms to interoperate with js and js objects.