Skip to main content

Hi, I'm Mariano Guerra, below is my blog, if you want to learn more about me and what I do check a summary here: marianoguerra.github.io or find me on twitter @warianoguerra or Mastodon @marianoguerra@hachyderm.io

Nikola Restructured Text Roles Plugin Example Project

This guide assumes you have python 3 and nikola installed.

See the Nikola Getting Started Guide for instructions to install it.

Environment with Nix

If you have Nix installed you can get the environment by running nix-shell on the root of this project with this shell.nix file:

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/d9448c95c5d557d0b2e8bfe13e8865e4b1e3943f.tar.gz") {} }:
with pkgs;

mkShell {
  LOCALE_ARCHIVE_2_27 = "${glibcLocales}/lib/locale/locale-archive";
  buildInputs = [
    glibcLocales
    python39
    python39Packages.Nikola
  ];
  shellHook = ''
    export LC_ALL=en_US.UTF-8
  '';
}

Setup a Nikola Site

In case you don't have a nikola site around or you want to play in a temporary project we will create one here:

nikola init my-site
cd my-site

I answered with the default to all of them by hitting enter on all questions, feel free to give different answers.

Create a New Plugin

In the base folder of the nikola site create a folder for your plugin, I will name mine my_rst_roles:

mkdir -p plugins/my_rst_roles

Create a configuration file with the plugin metadata and customize it at plugins/my_rst_roles/my_rst_roles.plugin:

[Core]
Name = my_rst_roles
Module = my_rst_roles

[Nikola]
PluginCategory = CompilerExtension
Compiler = rest
MinVersion = 7.4.0

[Documentation]
Author = Mariano Guerra
Version = 0.1.0
Website = https://marianoguerra.org
Description = A set of custom reStructuredText roles

Create a python module inside the folder that will contain the plugin logic at plugins/my_rst_roles/my_rst_roles.py:

from docutils import nodes
from docutils.parsers.rst import roles

from nikola.plugin_categories import RestExtension
from nikola.plugins.compile.rest import add_node

class Span(nodes.Inline, nodes.TextElement):
    pass

def visit_span(self, node):
    attrs = {}
    self.body.append(self.starttag(node, "span", "", **attrs))

def depart_span(self, _node):
    self.body.append("</span>")

add_node(Span, visit_span, depart_span)

class Plugin(RestExtension):

    name = "my_rst_roles"

    def set_site(self, site):
        self.site = site

        generic_docroles = {
            "my-foo": (Span, "my-base my-foo"),
            "my-bar": (Span, "my-base my-bar"),
        }

        for rolename, (nodeclass, classes) in generic_docroles.items():
            generic = roles.GenericRole(rolename, nodeclass)
            role = roles.CustomRole(rolename, generic, {"classes": [classes]})
            roles.register_local_role(rolename, role)

        return super(Plugin, self).set_site(site)

Create a New Post or Page to test it

Create a post:

nikola new_post -t "Nikola Restructured Text Roles Plugin Example Project"

Put some content in it:

echo 'Hi :my-foo:`hello`, :my-bar:`world`!' >> posts/nikola-restructured-text-roles-plugin-example-project.rst

Build the site

nikola build

Check the Generated HTML

You can serve it with:

nikola serve

And open http://localhost:8000 and inspect it with the browser's developer tools, here's a blog post friendly way with grep:

grep my-foo output/posts/nikola-restructured-text-roles-plugin-example-project/index.html

The output should be something like this:

Hi <span class="my-base my-foo">hello</span>, <span class="my-base my-bar">world</span>!</p>

More Advanced Transformations

I needed to process the children of the Span node by myself and stop docutils from walking the children between visit_span and depart_span, to do that here's a simplified version:

class Span(nodes.Inline, nodes.TextElement):
    def walk(self, visitor):
        visitor.dispatch_visit(self)
        # don't stop
        return False

    def walkabout(self, visitor):
        visitor.dispatch_visit(self)
        visitor.dispatch_departure(self)
        # don't stop
        return False

def visit_span(self, node):
    cls = " ".join(node.get('classes', []))
    child_str = " ".join([format_span_child(child) for child in node.children])
    self.body.append("<span class=\"" + cls + "\">" + child_str)

def depart_span(self, _node):
    self.body.append("</span>")

def format_span_child(node):
    return node.astext()

Here are links to the implementations of some of the relevant functions:

A Real World Use Case

I did this to embed inline UI components in the documentation for instadeq, you can see it in action in This Walkthrough Guide if you scroll a little.

I still have to improve the style and add some missing roles but it's much better than having to describe the position and look of ui components instead of just showing them.

As usual, thanks to Roberto Alsina for Nikola and for telling me how to get started with this plugin.

EFLFE: Elixir Flavoured Lisp Flavoured Erlang

History

In 2019 I was invited to give a talk at ElixirConfLA, at that point I didn't know Elixir so I decided to "make a joke" and instead of learning Elixir I would create a transpiler from Erlang to Elixir.

The Proof of Concept as a Joke was a lot of work but at least I learned a lot about Elixir and Pretty Printers.

One year later I was invited to CodeBEAM Brasil and I decided to push the PoC to completion to achieve the goal of transpiling Erlang/OTP and the transpiler itself.

This year I was invited again and I felt the pressure to continue with the tradition.

Last year my talk was with Robert Virding Co-creator of Erlang and creator of LFE (Lisp Flavoured Erlang), I had the idea to transpile LFE too.

At that point LFE 1.0 compiled to an internal representation (Core Erlang) that was one level below the one I was using in Elixir Flavoured Erlang.

I knew that the next major version of LFE was going to switch to the representation I was using, so it was just a matter of waiting for the release.

Timing helped and LFE 2.0 was released in June.

I went code diving and found how to get the data at the stage I needed and then fixing some corner cases around naming (LFE uses lispy Kebab case).

Result

The result is EFLFE: Elixir Flavoured Lisp Flavoured Erlang an Lisp Flavoured Erlang to Elixir transpiler.

Business in the Front

Aliens in the Back

Run it with a configuration file and a list of LFE files:

./efe pp-lfe file.conf my-code.lfe

And it will output Elixir files for each.

Example

Input:

(defmodule ping_pong
  (export
    (start_link 0)
    (ping 0))
  (export
    (init 1)
    (handle_call 3)
    (handle_cast 2)
    (handle_info 2)
    (terminate 2)
    (code_change 3))
  (behaviour gen_server))        ; Just indicates intent

(defun start_link ()
  (gen_server:start_link
    #(local ping_pong) 'ping_pong '() '()))

;; Client API

(defun ping ()
  (gen_server:call 'ping_pong 'ping))

;; Gen_server callbacks

(defrecord state
  (pings 0))

(defun init (args)
  `#(ok ,(make-state pings 0)))

(defun handle_call (req from state)
  (let* ((new-count (+ (state-pings state) 1))
         (new-state (set-state-pings state new-count)))
    `#(reply #(pong ,new-count) ,new-state)))

(defun handle_cast (msg state)
  `#(noreply ,state))

(defun handle_info (info state)
  `#(noreply ,state))

(defun terminate (reason state)
  'ok)

(defun code_change (old-vers state extra)
  `#(ok ,state))

Result:

defmodule :ping_pong do
  use Bitwise
  @behaviour :gen_server
  def start_link() do
    :gen_server.start_link({:local, :ping_pong}, :ping_pong, [], [])
  end

  def ping() do
    :gen_server.call(:ping_pong, :ping)
  end

  require Record
  Record.defrecord(:r_state, :state, pings: 0)

  def init(args_0) do
    {:ok, r_state(pings: 0)}
  end

  def handle_call(req_0, from_0, state_0) do
    new_count_0 = r_state(state_0, :pings) + 1

    (
      new_state_0 = r_state(state_0, pings: new_count_0)
      {:reply, {:pong, new_count_0}, new_state_0}
    )
  end

  def handle_cast(msg_0, state_0) do
    {:noreply, state_0}
  end

  def handle_info(info_0, state_0) do
    {:noreply, state_0}
  end

  def terminate(reason_0, state_0) do
    :ok
  end

  def code_change(old_vers_0, state_0, extra_0) do
    {:ok, state_0}
  end

  def unquote(:"LFE-EXPAND-EXPORTED-MACRO")(_, _, _) do
    :no
  end
end

Where The Code Gets Ugly

LFE and Elixir both share the fact that they support macros, macros are expanded at compile time and are a language feature, that means that when I get the code to transpile it, it's already expanded.

If the module you are transpiling uses macros you will transpile the macro expanded version of the code, which may be okay or not depending on the kind of code that the macro generates.

Remaining Work

The remaining work is to understand the details of variable scoping in LFE and see if it's compatible with Elixir so that I can translate it as is like I'm doing now.

If they differ I have to see if I can do some local analysis to transform it so that the resulting code behaves semantically like the original.

If you try it and have some questions let me know at @warianoguerra or in the repo's issue tracker.

How to transpile a complex Erlang project with Elixir Flavoured Erlang: erldns

A user (yes, there's another one!) of Elixir Flavoured Erlang asked why a project wasn't transpiling correctly, I went to look and here are the notes:

Assuming you have the efe escript in your PATH, cd to a folder of your choice and do:

git clone https://github.com/dnsimple/erldns.git
git clone https://github.com/dnsimple/dns_erlang.git

Create a file called erldns.conf with the following content:

#{
   includes => ["../include/", ".", "../priv/", "../../"],
   macros => #{},
   encoding => utf8,
   output_dir => "./out/"
}.

And then run:

efe pp erldns.conf erldns/src/*.erl

Now the context:

We clone dns_erlang because erldns includes headers files from it, like here include_lib("dns_erlang/include/dns_records.hrl")

Since we want to find files using include_lib that refer to dns_erlang, we also include ../../ (which will find any project cloned at the same level as erldns)

erldns also includes headers from the include and priv folders, paths in include are relative to the source file, that's why both start with ../

Currently efe will silently remove parts of the code that can't find or parse properly, that's why you may notice that code that references external records or constants in header files not found in the includes list will silently be missing, in the future I may warn about that.