How BEAMs are made

Mariano Guerra

BEAM BA Meetup

October 2016

Ego Slide

Start the project

gh:marianoguerra/otl to follow each step as a commit

commit 1:

rebar3 new fn_escript name=otl

cd otl

rebar3 escriptize

./_build/default/bin/otl

Implement erl2ast command

Because reading absform or using erl_syntax is too much work :)

commit 2:

$ ./_build/default/bin/otl erl2ast examples/mymod.erl

{ok,[{attribute,1,file,{"examples/mymod.erl",1}},
     {attribute,1,module,mymod},
     {attribute,2,export,[{main,0}]},
     {function,4,main,0,[{clause,4,[],[],[{atom,4,true}]}]},
     {eof,5}]}

Implement erl2ast2erl

commit 3:

$ ./_build/default/bin/otl erl2ast2erl examples/mymod.erl

-file("examples/mymod.erl", 1).

-module(mymod).

-export([main/0]).

main() -> true.

Implement erl2beam

commit 4:

$ ./_build/default/bin/otl erl2beam examples/mymod.erl .

{ok,[{module_name,mymod},{path,"./mymod.beam"}]}

Implement erl2beam

$ erl

Erlang/OTP 18
[kernel-poll:false]

Eshell V7.0  (abort with ^G)
1> mymod:main().
true
2> q().
ok

Tokenize true token, skip whites

commit 5:

$ cat examples/trues.otl

true true true


$ ./_build/default/bin/otl lex examples/trues.otl

{ok,[{boolean,1,true},{boolean,1,true},{boolean,1,true}],2}

Parse true token to AST

commit 6:

$ ./_build/default/bin/otl ast examples/trues.otl

{ok,[{atom,1,true},{atom,1,true},{atom,1,true}]}

Build Module with generated AST

commit 7:

$ ./_build/default/bin/otl mod examples/trues.otl mymod myfun

{ok,[{attribute,1,file,{"examples/trues.otl",1}},
     {attribute,1,module,mymod},
     {attribute,2,export,[{myfun,0}]},
     {function,3,myfun,0,
               [{clause,4,[],[],
                        [{atom,1,true},{atom,1,true},{atom,1,true}]}]}]}

Compile to BEAM

commit 8:

$ ./_build/default/bin/otl beam examples/trues.otl . mymod myfun

{ok,[{module_name,mymod},{path,"./mymod.beam"}]}

Compile to Erlang Source

commit 9:

$ ./_build/default/bin/otl erl examples/trues.otl mymod myfun

-file("examples/trues.otl", 1).
-module(mymod).
-export([myfun/0]).

myfun() -> true, true, true.

But I don't like your language!

rebar3 efene compile --format=erl

Add eval

commit 10:

$ ./_build/default/bin/otl eval examples/trues.otl

true

Implement a repl

commit 11:

$ ./otl-shell
Erlang/OTP 18

otl shell (Ctrl+g and then q to quit, Ctrl+g for Job Control Mode)

>>> true
true
>>> true true
true
>>>
>>> false
error: {1,otl_lexer,{illegal,"f"}}
>>> ^G
User switch command
 --> q

Parse Numbers Containing 1

commit 12:

$ ./otl-shell
Erlang/OTP 18

otl shell (Ctrl+g and then q to quit, Ctrl+g for Job Control Mode)

>>> 1
1
>>> 1 true
true
>>> 1 true 1111
1111
>>> ^G
User switch command
 --> q

Parse Numbers Containing 1

$ ./_build/default/bin/otl erl examples/one_true_language.otl one_true language

-file("examples/one_true_language.otl", 1).

-module(one_true).

-export([language/0]).

language() -> 1, true.

And we are done

We just created the one true language (TM)

Parse other numbers

commit 13:

$ ./otl-shell

>>> false true 1
1
>>> 92
92
>>> 42
42

Support Floats

commit 14:

>>> 42
42
>>> 12.4
12.4
>>> true
true
>>> false
false

Multiline Programs

commit 15:

true

false

12



13.4

Support + and -

commit 16:

1

1 + 2
1 - 3
4 + 3 - 2
4 + 3 - 2 + 1
3 + true

Support * / // %

commit 17:

1
2 * 3
2 * 3 / 4
2 * 3 / 4 % 5
1 + 2 * 3 - 4 / 5 + 6 % 7

Support unnary + and -

commit 18:

1
-1
+1
-1 * +1
1 * -1 * +1

Variables and Assignment

commit 19:

1
A = 1
1 = 1
1 = A
A = A
A = 2 - 1
B = A * 2 - 1

Comparisson Operators

commit 20:

1
1 < 2
1 < 2 * 3
2 * 3 < 1
A = 2 * 3 < 4 > 12
A is 5

Boolean Operators

commit 21:

true
true and false
true or false
true and false or true
true and 1 is 0
true and 1 > 0 or 2 > 4

Atom Data Type

commit 22:

otl shell (Ctrl+g and then q to quit, Ctrl+g for Job Control Mode)

>>> foo
foo
>>> bar
bar
>>> baz
baz
>>>

Parenthesis in Expressions

commit 23:

>>> 2 * (3 + 4)
14
>>> 2 * 3 + 4
10

Boolean not

commit 24:

>>> true
true
>>> false
false
>>> not true
false
>>> not false
true
>>> not other
error: badarg

Empty List

commit 25:

>>> []
[]

Note

[] -> {nil, Line} nil -> {atom, Line, nil}

One Item List

commit 26:

>>> []
[]
>>> [1]
[1]
>>> [true]
[true]
>>> [atom]
[atom]
>>> [[]]
[[]]

Cons Lists

The only lists you need/have ;)

commit 27:

>>> []
[]
>>> [foo]
[foo]
>>> [foo :: []]
[foo]
>>> [foo :: [bar]]
[foo, bar]
>>> [foo :: [bar :: []]]
[foo, bar]
>>> [foo :: [bar :: [baz :: []]]]
[foo, bar, baz]

Lists

commit 28:

>>> []
[]
>>> [1]
[1]
>>> [1, 2]
[1,2]
>>> [1, 2, 3]
[1,2,3]
>>> [1, 2, 3, [5, [6, 7]]]
[1,2,3,[5,[6,7]]]
>>> [1 :: []]
[1]
>>> [1 :: [2 :: []]]
[1,2]
>>> [1 :: 2]
[1|2]

Unify Lists and Cons Lists

And get an extra feature for free

commit 29:

>>> []
[]
>>> [1]
[1]
>>> [1 :: 2]
[1|2]
>>> [1 :: []]
[1]
>>> [1, 2, 3 :: []]
[1,2,3]
>>> [1, 2 :: [3 :: []]]
[1,2,3]
>>> [1, 2 :: [3 :: 4]]
[1,2,3|4]

Trailing Comma in Lists

commit 30:

>>> [1]
[1]
>>> [1,]
[1]
>>> [1,2]
[1,2]
>>> [1,2,]
[1,2]

List Operators ++ and --

commit 31:

>>> [] ++ []
[]
>>> [1] ++ [2, 3]
[1 ,2 ,3]
>>> [1] ++ [2] ++ [3]
[1, 2, 3]
>>> [1, 2, 3] -- [2]
[1, 3]

Strings

commit 32:

>>> ""
[]
>>> "this is a string"
"this is a string"
>>> "strings are lists of characters"
"strings are lists of characters"

Binary Strings

commit 33:

>>> ''
<<>>
>>> 'a binary string'
<<"a binary string">>

Atom with Special Characters

commit 34:

>>> an_atom
an_atom
>>> `an atom with spaces`
'an atom with spaces'
>>> `or weird characters!`
'or weird characters!'

Empty Tuple

commit 35:

>>> ()
{}

One Item Tuple

Need to distinguish from expression in parenthesis, trailing comma is required.

commit 36:

>>> ()
{}
>>> (1,)
{1}
>>> (foo,)
{foo}
>>> ((),)
{{}}

Multi Item Tuple

commit 37:

>>> ()
{}
>>> (1,)
{1}
>>> (1)
1
>>> (1, 2)
{1,2}
>>> (1, 2,)
{1,2}
>>> (1, 2, 3)
{1,2,3}
>>> (1, 2, 3,)
{1,2,3}

Function Calls

commit 38:

>>> now()
{1475,97317,459463}
>>> element(1, (foo,))
foo
>>> is_integer(1)
true
>>> `now`()
{1475,97351,894202}

Remote Calls

commit 39:

>>> erlang.now()
{1475,97590,786477}
>>> erlang.element(1, (foo,))
foo
>>> erlang.is_integer(1)
true
>>> `erlang`.`now`()
{1475,97619,94281}

Empty Map

commit 40:

>>> {}
#{}

One Item Map

commit 41:

>>> {}
#{}
>>> {a: 1}
#{a => 1}

Multi Item Map

commit 42:

>>> {}
#{}
>>> {a: 1}
#{a => 1}
>>> {a: 1, b: 2}
#{a => 1,b => 2}
>>> {a: 1,}
#{a => 1}
>>> {a: 1, b: 2,}
#{a => 1,b => 2}
>>> A = a
a
>>> {A: 1, b: A}
#{a => 1,b => a}

Update Map

commit 43:

>>> {} # {}
#{}
>>> {a: 1} # {b: 1}
#{a => 1,b => 1}
>>> {a: 1} # {a: 2}
#{a => 2}
>>> {a: 1, b: 2} # {b: 3, c: 4}
#{a => 1,b => 3,c => 4}
>>> A = {a: 1}
#{a => 1}
>>> A # {b: 2}
#{a => 1,b => 2}

Match Map

commit 44:

>>> A = {a: 1}
#{a => 1}
>>> {a = B} = A
#{a => 1}
>>> B
1
>>> A # {a = 2}
#{a => 2}
>>> A # {b = 3}
error: {badkey,b}

Single Clause Guard

it better be true :)

commit 45:

>>> when 42 > 4 { yes }
yes
>>> when true or false { A = 42 }
42
>>> when false { woops }
error: if_clause

Multi Clause Guard

Need to be entered all in the same line for now, wrapped for legibility commit 46:

>>> when false { shouldnt_happen }
    else when true { correct }
correct
>>> when true { correct }
correct
>>> when false { shouldnt_happen }
    else when 1 > 2 { not_yet }
    else when true { correct }
correct

Case Expression

Need to be entered all in the same line for now, wrapped for legibility commit 47:

>>> match 42:
      1 { one }
      2 { two }
      42 { the_answer }
    end
the_answer

>>> match 42:
      1 { one }
      42 { the_answer }
    end
the_answer

Case Expression

>>> match 42: 42 { the_answer } end
the_answer

>>> match 42: A { the_answer } end
the_answer

>>> match 42: A { A + 1 } end
43

>>> match 42: 1 { A + 1 } end
error: {case_clause,42}