Skip to main content

Hi, I'm Mariano Guerra, below is my blog, if you want to learn more about me and what I do check a summary here: marianoguerra.github.io or find me on twitter @warianoguerra or mastodon @marianoguerra@vis.social

A tour through the beam ADT representation zoo

The Languages

Dynamically Typed

Statically Typed

Column meaning

Record Type

named set of fields (key, value pairs), also refered as types, structs, records etc

Union Type

like a record type but with more than one "shape", also refered as discriminated unions, variants etc

Type Dispatch

a function that will have different implementations according to the type of one (or more) of its arguments, also refered as protocols or multi methods

TL;DR Table

Language

Inspiration

Record Type

Union Type

Type Dispatch

Dynamic

Clojerl

Clojure

Yes

No

Yes

Efene

Python/JS

Yes*

No*

No

Elixir

Ruby

Yes

No*

Yes

Erlang

Prolog

Yes*

No*

No

LFE

Common Lisp

Yes*

No*

No

Static

Alpaca

ML

Yes

Yes

No*

Fez

F#

Yes

Yes

Yes

Gleam

ML/Rust

Yes

Yes

Not Yet?

Hamler

Purescript

Yes

Yes

Yes

PureErl

Purescript

Yes

Yes

Yes

Groups

Languages that compile record types to erlang records

Languages that compile record types to erlang maps

  • Alpaca: __struct__ field for records, no extra field for anonymous records

  • Clojerl: __type__ field

  • Elixir: __struct__ field for structs

  • Purerl: no extra field for anonymous records

  • Hamler : no extra field for anonymous records

Languages with union types

  • Alpaca: tagged tuple if it has values, atom if not

  • Gleam: tagged tuple if it has values, atom if not

  • Fez: tagged tuple if it has values, atom if not

  • Purerl: tagged tuple (not sure about variants with no values, should be like Hamler)

  • Hanler: tagged tuple even with variant with no values

Languages that do type dispatch

Notes

Alpaca

Records: "anonymous records".

Records are compiled as maps with the KV '__struct__' => 'record'. Because Alpaca doesn't provide any reflection facilities, more type information isn't propagated to the generated Core Erlang.

In the tag (in the case of variants/discriminated unions), it's just the atom representation of the tag name itself.

E.g. Some_tag 1 gets compiled to {'Some_tag', 1}

Alpaca's records get an extra key-value pair of '__struct__' to a rough description of its type/structure if the "structure" is flagged as a record.

Tagged Unions: (variants in OCaml) get compiled as an atom if there's no associated value, and as a tuple if there is.

Type Dispatch: Author says: "Not currently. This sort of thing might get handled by type classes but I haven't gone too far down that line of thinking yet"

Clojerl

Records: (deftype) compiled to Erlang maps with a special __type__ field.

Tagged Unions: No

Type Dispatch: defprotocol and deftype, extend-type, extend-protocol work as in Clojure.

Protocols are not reified as in Clojure, there are no runtime protocol objects.

Elixir

Records: structs are compiled to Erlang maps with a special __struct__ field.

Tagged Unions: No (usually ad-hoc tagged tuples are used for this)

Type Dispatch: Protocols are collected and consolidated at compile time

Gleam

Records: Compiled to Erlang records (hrl files are generated)

Tagged Unions: Compiled to tagged tuples, they are just gleam custom types with multiple "constructors", if a variant has no values it gets compiled to an atom

Type Dispatch: Not Yet? Will Gleam have type classes?

Fez

Records: Compiled to Erlang records

Tagged Unions: Compiled to tagged tuples

Type Dispatch: Class method calls

LFE, Efene and Erlang

LFE and Efene are just "dialects" of Erlang, that's why they are covered together here.

Records: Erlang records, which are compiled to a tuple where the first value is an atom with the name of the record: LFE Records, Efene Records, Erlang Records

Tagged Unions: Since they are dynamically typed they can use tagged tuples for this, there's no need to declare them, examples are functions that return {ok, V} or {error, Reason}.

Type Dispatch: No

PureErl

Records: Compiled to Erlang maps (without an extra field), really similar to alpaca "anonymous records"

Tagged Unions: Compiled to tagged tuples

Type Dispatch: Type Classes

There are also newtype

Hamler

Records: Compiled to Erlang maps (without an extra field)

Tagged Unions: Compiled to tagged tuples

Type Dispatch: Type Classes

Why I don't like concept cars addendum

I don't like to glorify Steve Jobs, but sometimes he expresses ideas the right way and the fact that he, unlike me, showed clearly that those ideas can deliver good results may show that those ideas are valid.

Here's a part of an interview from him (emphasis mine):

... they have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product and they really have no feeling in their hearts usually about wanting to really help the customers there’s a just a tremendous amount of crafts ship in in-between a great idea and a great product and as you evolved that great idea it changes and grows it never comes out like it starts because you learn a lot more as you get into the subtleties of it and you also find there’s tremendous trade-offs that you have to make I mean you know there are there are just certain things you you can’t make electrons do there are certain things you can’t make plastic do or glass do or factories do or robots do and as you get into all these things designing a product is keeping 5,000 things in your brain these concepts and fitting them all together in in kind of continuing to push to fit them together in new and different ways to get what you want and every day you discover something new that is a new problem or a new opportunity to fit these things together a little differently and it’s that process that is the magic.

From here:

Two types of software prototypes and why I don't like concept cars

When I was a child I used to love concept cars, I loved how they were much better than regular cars.

I would listen carefully to when they would be available in the market, most of the time that information wasn't mentioned, sometimes the date was so far in the future that it wasn't interesting, by that time we would already have flying cars!

With time I started noticing that none of the concept cars from my childhood were on the streets, not even close. Worst was when a really bad version of them was available for sale, the worst deception ever.

Now when I see a company showing off with a concept car I think the opposite, that company is running out of real ideas or has lost the ability to execute novel designs, and tries to justify it by showing shiny things that it knows will never see the light of day.

Why I'm talking about concept cars? well, because there are different types of concept cars and different types of software prototypes, but they are almost the same.

The prototypes that require you to accept things that are never going to be feasible but are sold as if they are possible are the worst. Either because they violate basic laws of physics, materials, safety, regulations, performance, user experience or just because they focus on a single concept by disregarding others that are required when the thing migrates from prototype to production.

If you are upfront and tell that the prototype is an exploration of "what if we push this dimension to the extreme", I'm ok with it, it's a learning experience, you may learn a lot about that dimension and how it relate to others, what are its limitations and so on.

But the prototype should be clearly marked as such.

The other useful prototypes are as learning exercises, I like to build throw away prototypes as my first approach to something, as a way to learn more about something and have a better idea for next time. You should also mark them clearly as such.

The third useful prototype is the initial stage of something you want to grow into a product but want to show the potential along the way, you may throw some versions away in the early stages because you learned something that required a big reformulation and it's easier to start from scratch than to refactor it.

But from the beginning the prototype should be grounded in reality, what's possible and how the main concept relates to other features that may be a year or more in the future but they are going to be required for the prototype to turn into a product.

You can't have a slow/complex prototype in the early stages unless you have a clear idea on how it's going to get faster/simpler, small optimizations here and there aren't going to cut it once the performance/complexity penalty of all the extra features starts creeping up. You have to think up front how those other features will fit once you reach that point.

Of course, if you are building something new, at some point you will be in a new territory and some things may require some reformulation, that's good, but at that point you need to have buffers of performance and other metrics and a simple core architecture and code to be able to solve those problems, even if not in the optimal way. That's why you built this new thing, to push the boundary a little further.

If you get the chance to build a new prototype after the current one turned into a finished product, you can then incorporate those new insights at the beginning to be able to push a little further than before. Rinse and repeat.

What you should not do is to build a prototype that starts to fall apart even when only solving the single problem you care about and call the remaining just an implementation detail or "left as an exercise for the engineers".

It may sound like a contradiction, but to build good prototypes you have to be good at building complete products, otherwise you can't guide your initial design on constraints that you never experienced. A prototype should eventually be the foundation of something much bigger and complex, you can't build that on unstable foundations. If not you, somebody on the team must provide the experience that comes from completing, polishing and maintaining something that survives the contact with reality.

As Mike Tyson said: "Everybody has a plan until they get punched in the mouth."

Or put another way:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.

—John Gall

There's only one dimension you can ignore if you have the time/money to do so, and that's price: see the Experience curve effect. Just don't be too early ;)

inb4 appeal to visionaries, check Vannevar Bush's memex, it was a full design, it was easy to see how you could build it, even when some technologies were in the future, everything was feasible. Ivan Sutherland Sketchpad Demo was a complete runnable thing, same with Douglas Engelbart's Mother of all Demos, Xerox PARC's Alto and Bret Victor's Dynamicland.

Don't fool others, but most important, don't fool yourself.

On utopians and the fact that software must exist and solve real problems

Yet a man who uses an imaginary map, thinking that it is a true one, is likely to be worse off than someone with no map at all; for he will fail to inquire whenever he can, to observe every detail on his way, and to search continuously with all his senses and all his intelligence for indications of where he should go

—E. F. Schumacher, Small is Beautiful

There's an idea I first saw at Django Conf that I really liked, the opening keynote is called "Django Sucks" and it's about everything that is wrong with Django.

Since then I've been promoting the idea on every conference I've been and many times I fantasize about giving the "X Sucks" talk myself.

I'm part of more than one community in the utopian spectrum, 2 weeks ago I gave a talk at Bob Konf titled Programming by any other name where I showed how the future is already here, we just have to find it and help its creators gain adoption, here's the other side of that talk:

As a temporary member of the software utopians, an activity we share is to imagine alternative realities in which every wrong decision was instead replaced with the "right" one, with the benefit of hindsight, of course.

The word "right" in quotes because it may have yet to be tested in the real world and may require the suspension of disbelief on many aspects of reality. We may call this activity "counterfactual porn".

I say temporary member of the software utopians because I'm also a member of the software builders, people that build software that real users are willing to use (and pay for) to solve real problems.

One of the utopians' hobbies is to look down on systems that exist and solve real problems, pointing at its flaws.

These flaws are in contrast to systems that only exist as an idea, paper, or at most as a small prototype that proves a single point and solves a single problem carefully crafted to make it shine.

But there's a gap that utopians never seem to cross.

The gap from early prototype that at most proves a point or shows a new idea, to the point where that single thing is part of a whole that people can use to solve a large variety of complex problems.

Some utopians dared to cross the gap, General Magic, Pharo, Racket, Genera to name a few.

These Pioneers turned settlers faced the fact that "If you build it, they will come" is usually not true.

Those products were (and some are) things that you can get and use, but for some reason they failed to catch on.

Here utopians usually appeal to the "No true Scotsman fallacy", if not to the simpler "people are stupid/don't know what's good for them" and go back to the confortable possition of throwing stones at things others build and maintain.

When someone takes the failed idea (or prototype) and adapts it into something that people actually use, taking into account the limitations of reality, society, economics and people's behaviors, the utopians proceed to complain about how the new thing is a bastardized version of the original idea and how if they were to do it they would stay true to the original.

The exercise of actually doing either is almost never attempted, or is attempted and quickly abandoned.

In some cases it's completed and for "strange" reason it fails to gain adoption. GOTO No true Scotsman/People are stupid.

The implementation, its users or some aspect of reality are always blamed, the idea must be kept untouched.

I usually go around preaching ideas by thinkers I admire, I think it's really useful to consider them to improve the systems that builders create.

Pure Ideas are important, people stopping at prototypes too, but as much as I believe we need to learn more from history and study great thinkers, researchers and their ideas, I also think we should at least respect and listen to people building finished products that have to face the harsh reality of productive usage of working software.

While we are at it we could also listen to the users of such systems, which we love to talk about, but almost never talk to, let alone let them tell us something that may shape our ideas.

Ideas are only impactful when they get turned into things that real people can use to solve real problems.

During this process, pure ideas have to be adapted at each step, the end result is usually not as pure as one would wish, but that's the price to pay to get from idea to reality.

This involves at least 3 kinds of people, Utopians/Pioneers, Builders/Settlers and Users/Citizens, you need the 3 to collaborate, communicate and respect each other's roles and constraints.

If you are of one kind and think you could do a better job at being the other, then before telling them, try showing them, all the way. You may learn that it's not that easy.

If you don't want to show them, then collaborate and listen, you may learn something new that may improve the chances your idea gets adopted.

The process from idea to adoption also involves 3 different timescales, short, middle and long term (similar to operational, tactical and strategic levels of planning).

Execution has to work at each level individually but to achieve the long term vision, the short and middle term need to be aligned with the long term, even if in between it has to take some detours/shortcuts.

When the 3 roles collaborate and are willing to adapt the idea to consider each other's constraints and plan for the 3 timescales as a whole, they may have a better change to achieve the utopian objective, even when during the process it doesn't look like it.

The alternative is to stay forever at the idea level complaining at people trying to bring it to reality.

PS: I may not be only talking about software

RFC: Elixir Module and Struct Interoperability for Erlang

This is a RFC in the shape of a project that you can actually use, I'm interested in your feedback, find me as marianoguerra in the erlang and elixir slacks and as @warianoguerra on twitter.

The project on github: https://github.com/marianoguerra/exat and on hex.pm: https://hex.pm/packages/exat

See the exat_example project for a simple usage example.

Here's the description of the project:

Write erlang friendly module names and get them translated into the right Elixir module names automatically.

The project is a parse transform but also an escript to easily debug if the transformation is being done correctly.

Erlang Friendly Elixir Module Names

A call like:

ex@A_B_C:my_fun(1)

Will be translated automatically to:

'Elixir.A.B.C':my_fun(1)

At build time using a parse transform.

The trick is that the @ symbol is allowed in atoms if it's not the first character (thank node names for that).

We use the ex@ prefix to identify the modules that we must translate since no one[1] uses that prefix for modules in erlang.

Aliases for Long Elixir Module Names

Since Elixir module names tend to nest and be long, you can define aliases to use in your code and save some typing, for example the following alias declaration:

-ex@alias(#{ex@Baz => ex@Foo_Bar_Baz,
            bare => ex@Foo_Long}).

Will translate ex@Bar:foo() to ex@Foo_Bar_Baz:foo() which in turn will become 'Elixir.Foo.Bar.Baz:foo()

It will also translate the module name bare:foo() into ex@Foo_Long:foo() which in turn will become Elixir.Foo.Long:foo()

Creating Structs

The code:

ex:s@Learn_User(MapVar)

Becomes:

'Elixir.Learn.User':'__struct__'(MapVar)

The code:

ex:s@Learn_User(#{name => "bob", age => 42})

Becomes:

'Elixir.Learn.User':'__struct__'(#{name => "bob", age => 42})

Which in Elixir would be:

%Learn.User{name: 'bob', age: 42}

Aliases in Structs

The following alias declaration:

-ex@alias(#{ex@struct_alias => ex@Learn_User}).

Will expand this:

ex:s@struct_alias(#{name => "bob", age => 42})

Into this:

'Elixir.Learn.User':'__struct__'(#{name => "bob", age => 42})

Pattern Matching Structs

Function calls are not allowed in pattern match positions, for example on function/case/etc clauses or the left side of a =, for that there's a different syntax:

get_name({ex@struct_alias, #{name := Name}}) ->
    Name;
get_name({ex@struct_alias, #{}}) ->
    {error, no_name}.

Becomes:

get_name(#{'__struct__' := 'Elixir.Learn.User', name := Name}) ->
    Name;
get_name(#{'__struct__' := 'Elixir.Learn.User'}) ->
    {error, no_name}.

And:

{ex@struct_alias, #{name := _}} = ex:s@Learn_User(#{name => "bob", age => 42})

Becomes:

#{'__struct__' := 'Elixir.Learn.User', name := _} =
        'Elixir.Learn.User':'__struct__'(#{name => "bob", age => 42})

This is because that pattern will match maps that also have other keys.

Note on Static Compilation of Literal Structs

On Elixir if you pass the fields to the struct it will be compiled to a map in place since the compiler knows all the fields and their defaults at compile time, for now exat uses the slower version that merges the defaults against the provided fields using 'Elixir.Enum':reduce in the future it will try to get the defaults at compile time if the struct being compiled already has a beam file (that is, it was compiled before the current file).

Use

Add it to your rebar.config as a dep and as a parse transform:

{erl_opts, [..., {parse_transform, exat}, ...]}.
...
{deps, [exat, ...]}

Build

To build the escript:

$ rebar3 escriptize

Run

You can run it as an escript:

$ _build/default/bin/exat pp [erl|ast] path/to/module.erl

For example in the exat repo:

$ _build/default/bin/exat pp erl resources/example1.erl
$ _build/default/bin/exat pp ast resources/example1.erl

Syntax Bikesheding

The syntax I chose balances the need to not produce compiler/linter errors or warnings with the objective of avoiding accidentally translating something that shouldn't be translated.

Please let me know what you think!

[1] Famous last words

Riak Core on Partisan on Elixir Tutorial: Resources

A List of resources related to riak_core and partisan.

Riak Core

Project

  • riak_kv: Riak KV itself

  • riak_pg: Distributed process groups with riak_core

  • dalmatinerdb: A fast, distributed metric store

  • riak_test_core: riak_test fork which refactors riak_test to not bo targeted directly towards riak_kv and makes it more library like

  • nkdist: a library to manage Erlang processes evenly distributed in a riak_core cluster

  • riak_id: A clone of Twitter's Snowflake, built on riak_core

  • DottedDB: A prototype of a Dynamo-style distributed key-value database, implementing Server Wide Clocks as the main causality mechanism across the system

  • riak_pipe: riak_pipe allows you to pipe the output of a function on one vnode to the input of a function on another

Docs

Partisan

Partisan.cloud: Partisan Website

Projects

  • Lasp: Lasp is a suite of libraries aimed at providing a comprehensive programming system for planetary scale Elixir and Erlang applications

  • Vonnegut: an append-only log that follows the file format and API of Kafka 1.0

  • Erleans: Erlang Orleans

Presentations