Ir al contenido principal

Hi, I'm Mariano Guerra, below is my blog, if you want to learn more about me and what I do check a summary here: marianoguerra.github.io or find me on twitter @warianoguerra or Mastodon @marianoguerra@hachyderm.io

Speedrunning WebAssembly's History: A Fictional First-Person Perspective

Context

This is a true story. The events depicted in this film took place in Minnesota in 1987. At the request of the survivors, the names have been changed. Out of respect for the dead, the rest has been told exactly as it occurred.

Fargo (1996 film)

While writing WebAssembly from the Ground Up we reached the point where we needed to write an introduction. We decided to write one each and see which one worked better or if we could merge both into one.

From the conversations I knew Patrick was going to write a "Standard" introduction so I decided to try something else.

At the end Patrick's version ended up in the book so I'm posting mine here in case anyone finds it interesting.

Note that what I'm posting here was a draft, complete enough for patrick to check and provide feedback, it would have gone through a couple of rounds of revisions and edits if it ended up in the book :)

Introduction

It's 2013, web browsers are getting faster and enabling more use cases to be available on the web, you work at Mozilla and start thinking how to enable and accelerate the adoption of the web for more use cases.

You think "What kind of software is still not available on the web?", some ideas come to mind:

  • Games

  • Image / Video editors, encoders and decoders

  • Compression

  • Cryptography

  • CAD

  • Scientific visualization and simulation

  • Simulators, emulators

  • IDEs

You notice a pattern, they are all "compute intensive" and most of them are written in low level/system programming languages.

There are already many languages that compile to JavaScript, why not try to compile existing C/C++ codebases to JavaScript and see if it's enough?

After all when a C/C++ compiler does its job the result is mostly about calling functions, manipulating local variables and loading and storing numbers directly in memory.

You have an idea, if you start with an existing compiler like clang and swap the last stage to emit JavaScript instead of assembly, with the help of some shims you should be able to run some of those programs in the browser.

Since JavaScript doesn't have direct access to memory you decide to simulate memory by using an Array of numbers, pointers are compiled to numbers that are used to index into the memory array.

After some hacking you have a working prototype, you decide to call it emscripten.

After compiling and running some C/C++ programs you notice that they are really slow, the translation to JavaScript "throws away" a lot of information that is present in the original program, mainly the variable types.

The JavaScript runtime has to "rediscover" all that information by profiling and JITing, that rediscovery takes a while and is not always perfect, it would be nice to just pass the information you already have.

Another problem is that using an Array as memory is slow, JavaScript arrays are really flexible, but that comes at the cost of performance.

After some conversations and collaborations you get Typed Arrays implemented, this speeds up the memory operations a lot.

You notice that after the JIT has done its work peak performance is pretty good, but it takes a while and since each browser has its own engine the performance differs among them.

Using your connections at Mozilla you start talking with the Spidermonkey team to find a way to hint the runtime about the types of variables emitted by the compiler.

Low level languages have multiple numeric types, JavaScript has only one, but looking at the spec you notice that binary bitwise operators apply the ToInt32 conversion to the operands.

You can use that to hint the runtime that a variable is a 32 bit integer, using an operation that "does nothing" like binary or between a value and zero like a | 0 does exactly that.

Numbers in JavaScript are internally represented as doubles. If the compiler emits a variable of type double then you are almost done, but a variable can be of any type, like a String, a boolean, an Array etc. How do you hint to the runtime that this variable contains a Number and specifically a double?

In JavaScript there's a trick to coerce any value into a number, just put the plus sign in front of a variable like +a and it will convert its operand to the Number type.

Great, there's one type left, now you need to hint the runtime that a variable is a 32 bit floating point value, but you ran out of tricks.

You convince the Spidermonkey team to introduce a new function Math.fround that returns the nearest 32-bit single precision float representation of a number.

With the new hints in place you modify emscripten to emit the hints and notice a significant speed improvement.

But still it takes a while for the JIT to "warm up", the hints are there, why doesn't the JIT just optimize them the first time they are processed?

If there only was a way to hint the JIT that a piece of code is not written by humans but emitted by a compiler and it adheres to your hints it would be much faster and it could do the optimizations "Ahead of Time" instead of "Just in Time".

There's already a backward compatible way to hint JavaScript that a piece of code adheres to a restricted subset of JavaScript, "use strict", you could emit a similar backward compatible directive that a runtime may opt into and do the ahead of time optimizations, since the JavaScript you are emitting is replacing the assembly a compiler would emit at that stage you decide to call it asm.js, then the directive is defined as "use asm".

With this backward compatible changes you convince the Spidermonkey team to prototype the changes in a branch and the results are impressive, asm.js code runs only twice as slow as native code! After some extra rounds of optimizations you get it to 50% slower than native.

This is more than a toy, this may actually enable new types of applications on the web.

To show it to the world you decide to do a flashy demo, collaborating with Epic Games by compiling the Unreal 3 Engine to asm.js and running a 3D demo in the browser called Epic Citadel.

With the demo out you start working to specify asm.js and try to convince other browsers to adopt it.

But even if the results are impressive there are still problems, the amount of JavaScript generated is huge, a demo may generate over 40 MBs of asm.js, the bottleneck now moves to a new place, downloading and parsing that amount of code takes a while, on mobile it can take over 20 seconds just to parse!

Since asm.js is meant to be a compiler target and you are in the process of convincing other browser vendors to add a new capability to their engines you all agree that there's an opportunity to "do it correctly" and define a binary format that is more compact and can be decoded much faster.

By defining a binary format you also avoid the problem of JavaScript having to be two different things at the same time, a language for humans to write and a compile target. With a new binary format that can eventually diverge from JavaScript's semantics you can achieve the initial objective of enabling more applications to run on the web.

Since this is a compiler target for the web, and compilers usually emit assembly, the new project is named WebAssembly.

After some rounds of discussions all browser vendors get on board and a new collaboration is announced to standardize WebAssembly.

Playing with Code: Programming-Adjacent Games

Some weeks ago I was working on some explorable explanation for binary operators and I started thinking if there was any interesting work on making programming interactive, interesting, even fun.

I asked on twitter, mastodon and the future of coding slack the following question:

Do you know any games where the core game mechanic is about programming? Things like Zachtronics games, factorio or Robotopia

Below is an edited summary of the anwers I got.

Thanks to Ivan Reese, Cameron Yick, Joe Nash, Jeffrey Tao, George Campbell, Daniel Sosebee, Kartik Agaram, Richard Carlsson, Asbjorn, Janne Auki and Dragan Okanovic for the contributions.

Quadrilateral Cowboy

In Quadrilateral Cowboy, the player takes the role of a computer hacker in the 1980s, armed with a "top-of-the-line hacking deck outfitted with a 56.6k modem and a staggering 256k RAM".

The game is played from the first-person perspective. The player acts as the hacker overseeing one or more adept agents that have missions to infiltrate buildings and steal documents.

Human Resource Machine

Human Resource Machine is a visual programming-based puzzle video game developed by Tomorrow Corporation.

Human Resource Machine uses the concept of a corporate office worker assigned to perform tasks that involve moving objects between an inbox, an outbox, and to and from storage areas as a metaphor for assembly language concepts. The player works through some forty puzzles in constructing a program to complete a specific task.

Dreams

Dreams is a game creation system video game developed by Media Molecule.

Players can create and play user-generated content in the forms of games, audiovisual experiences and game assets, which can be shared or remixed to be used in other players' creations.

LightBot

Solve Puzzles using Programming!

LightBot is a puzzle game based on coding; it secretly teaches you programming logic as you play!

LightBot was designed with first-time coders. It's been played by over 20 million kids and has been used by tens of thousands of teachers worldwide.

Zoombinis (series)

Zoombinis was a series of educational puzzle computer games that were originally developed by TERC and published by Broderbund.

The series consists of three games: Logical Journey of the Zoombinis (1996), Zoombinis: Mountain Rescue (2001), and Zoombinis: Island Odyssey (2002). Logical Journey was remade as Zoombinis for modern operating systems in 2015. The series focuses on the Zoombinis, small blue creatures each with different appearances and personalities, which the player must guide through strange puzzle-filled lands.

TwilioQuest

TwilioQuest is an educational video game designed to teach a new generation of developers how to change the world with code.

TwilioQuest prepares you for real-world programming by helping you configure a local development environment and introducing tools used by professional programmers around the world. From learning how to use your terminal, to coding in Python, JavaScript, and PHP, TwilioQuest will help you develop practical engineering skills.

From the author:

I worked on a now defunct programming education game (TwilioQuest) and myself and another dev used to stream gameplay and interviews with programming game devs, there’s still some of the vods kicking around at twitch.tv/twilioquest, including a chat with Zach of Zachtronics

Nintendo game builder garage

In Game Builder Garage, the player uses a visual programming language centralized on the concept of creatures called Nodon. The Nodon represent various facets of input, game output, logic, and on-screen objects, such as a Stick Nodon that reports input from the Joy-Con analog stick or a Person Nodon that represents an on-screen character. The player builds a program by adding Nodon and making connections between the various nodes on Nodon, such as connecting the Stick Nodon to the Person Nodon as to tie the analog stick to movement of the character on-screen.[1] Nodon are available to interface nearly all features of the Switch and Joy-Con, including the infrared sensors and motion controls.

The game features a lesson mode to guide the player through using the Nodon language and to help them understand some of the principles of game development through a series of seven built-in games that the player can create.

Rabbids coding

Across 32 levels, players are tasked with cleaning up a spaceship that has been overrun by Rabbids, which is achieved by providing simple instructions to a Rabbid wearing a mind-control device.

Players drag instructions for their Rabbid from a menu and place them in order, before pressing the play button to test them out.

The goal in each level is to provide the simplest instructions possible to complete the task. Eventually players will unlock a sandbox environment, allowing them to explore and play with the instructions as they see fit.

Signal state

Set in a post-apocalyptic future, The Signal State challenges you with complex puzzles inspired by modular synthesizers. Repair machines, rebuild an abandoned farm, and be part of a revolution that will change the fate of agriculture once and for all.

Battlesnake

A competitive game where your code is the controller.

All you need is a web server that responds to the Battlesnake API.

Develop your own algorithm to find food, stay alive, and eliminate others. Battlesnakes are controlled by a web server you deploy, running the code you write.

Shenzhen I/O

Shenzhen I/O is a puzzle video game set in the near future in which players assume the role of an electronics engineer who has emigrated to Shenzhen, China to work for fictional technology company Shenzhen Longteng Electronics. The player is tasked with creating products for clients, which involves constructing circuits and then writing code to run them. The programming language used in the game is similar to assembly language and the circuit elements resemble simplified versions of real-world electronics.

The game allows players to create their own challenges by writing Lua scripts.

Exapunks

Exapunks takes place in an alternate timeline in the year 1997. The fictional world of Exapunks is heavily computerized, and a disease called "the phage" is ravaging the population, turning the bodies of those affected into computerized components. The player takes on the role of Moss, a hacker who breaks into computer systems in order to afford a $700/day drug to slow the progress of his phage affliction. His hacking missions are given to him by a mysterious artificial intelligence known as EMBER-2.

Each mission takes place inside a network of interconnected and specialized computer systems. Using programmable software agents called EXAs, the player must accomplish each given task by writing computer code to cleverly manipulate the data stored on the network's systems. The EXAs' instruction set features a few simple opcodes for movement, data processing, network messaging, and interfacing with files and registers. Due to their limited memory capacity, these tasks often require several agents working together in a highly coordinated fashion. EXA units also have the ability to replicate themselves inside the network. Typical missions include retrieving data from secured storage systems, hacking into company databases, and causing an automated teller machine to dispense free cash. Some puzzles also require the player to hack Moss's body to maintain his health. Some puzzles challenge the player to hacker battles, where they must pit their EXAs against an opponent's agents, for example altering a television station's program to broadcast Moss' content instead.

Lastcall BBS

A collection of ideas that weren’t big enough for full games in their own right but still absolute bangers, has a game that I think a lot of people here will get a kick out of, called “X’BPGH: The Forbidden Path”, which is kind of a cellular automata programming game where the rules of the automata are obscured by the eldritch horror dressings of the whole thing

Boot up your Z5 Powerlance and dial into Last Call BBS, the last game from Zachtronics! The Barkeep’s loaded up his retro computer with a full set of puzzle games for you to download and play. No need to worry about copy protection, they’re all fully cracked and ready to enjoy!

SpaceTraders

SpaceTraders is an API-based game where you acquire and manage a fleet of ships to explore, trade, and fight your way across the galaxy. Use any programming language with our API to control the most powerful fleet in universe.

Baba is You

Baba Is You is a puzzle game where the rules you have to follow are present as physical objects in the game world. By manipulating the rules, you can change how the game works, repurpose things you find in the levels and cause surprising interactions!

TIS-100

TIS-100 is an open-ended programming game by Zachtronics, the creators of SpaceChem and Infinifactory, in which you rewrite corrupted code segments to repair the TIS-100 and unlock its secrets. It’s the assembly language programming game you never asked for!

SineRider

graphing equations is the core mechanic

SineRider is a game about love and graphing, built by a global team of teenagers at Hack Club

Synthesis

Synthesis is I think something that doesn't look like programming, but seems to me like programming in a deep way. Seems similar to SineRider in that respect (which I love, but man it gets difficult quickly. Somebody should graph the learning curve of SineRider within SineRider :)

shapez.io

Shapez is a relaxed game in which you have to build factories for the automated production of geometric shapes. As the level increases, the shapes become more and more complex, and you have to spread out on the infinite map.

Turing Complete

Turing Complete is a game about computer science. If you enjoy the thrill of figuring things out and those moments where a deeper perspective is revealed about something you thought you understood, this game is for you.

ComputerCraft

ComputerCraft is a mod created for Minecraft by dan200 that adds Computers, Monitors, Modems, Turtles and more! ComputerCraft's Computers and Turtles are programmed with the easy-to-learn Lua programming language. You can use Redstone, RedPower or even MineFactory Reloaded alongside with your devices for the best experience.

Screeps

It's an open-source game for programmers, wherein the core mechanic is programming your units' AI. You control your colony by writing JavaScript.

Brawl.AI

The idea is: Surely there are some really smart people who can write the bot to beat all other bots. That could be you!

Here, on this site, you can write bots that play a turn-based-squad-based game inspired by XCOM. Especially the tactical layer. There is no strategy layer on this site as that is very game dependent.

Duskers

In Duskers you pilot drones into derelict spaceships to find the means to survive and piece together how the universe became a giant graveyard. In film terms it's The Road meets the first Alien movie. In game terms: It's a roguelike with elements of dungeon crawling and real time strategy, but in a survival horror setting that focuses on subterfuge, and adapting to survive.

Features:

  • Gritty retro digital atmosphere
  • Use a Command Line Interface to control drones & ship systems
  • Explore procedurally generated derelict ships and universe
  • Upgrade and modify drones with the salvage you find
  • Discover ship logs and piece together what happened
  • Find creative ways out of bad situations using tools and your environment

Old-school Games

Check Category:Programming games for a complete list.

Incredible machine (series - Rube Goldberg machines)

The Incredible Machine (TIM) is a series of video games in which players create a series of Rube Goldberg devices.

The general goal of the games is to create a series of Rube Goldberg devices: arrange a given collection of objects in a needlessly complex fashion so as to perform some simple task, such as "put the ball into a box" or "start a mixer and turn on a fan". Available objects range from simple ropes and pulleys to electrical generators, bowling balls, and even cats and mice to humans, most of which have specific interactions with or reactions to other objects: for example, mice will run towards nearby cheese, and light sources placed next to a magnifying glass will ignite wicks. Levels have a set of fixed objects that cannot be moved by the player, and the player must solve the puzzle by carefully arranging a provided set of objects around the fixed items. There is also a "freeform" option that allows the user to "play" with all the objects with no set goal or to also build their own puzzles with goals for other players to attempt to solve.

Rocky's Boots

Rocky's Boots is an educational logic puzzle game by Warren Robinett and Leslie Grimm, published by The Learning Company in 1982.

It was one of the first educational software products for personal computers to successfully use an interactive graphical simulation as a learning environment.

The object of the beginning part of Rocky's Boots is to use a mechanical boot to kick a series of objects (purple or green squares, diamonds, circles, or crosses) off a conveyor belt; each object will score some number of points, possibly negative. To ensure that the boot only kicks the positive objects, the player must connect a series of logic gates to the boot.

Robot Odyssey

Robot Odyssey is a programming game developed by Mike Wallace and Dr. Leslie Grimm and published by The Learning Company in December 1984.

It is a sequel to Rocky's Boots, and it was released for the Apple II, TRS-80 Color Computer, and MS-DOS.

The player is readying for bed when, suddenly, they fall through the floor into an underground city of robots, Robotropolis. The player begins in the sewers of the city with three programmable robots, and must make their way to the top of the city to try to find their way home again.

Core War

Core War is a 1984 programming game created by D. G. Jones and A. K. Dewdney in which two or more battle programs (called "warriors") compete for control of a virtual computer. These battle programs are written in an abstract assembly language called Redcode. The standards for the language and the virtual machine were initially set by the International Core Wars Society (ICWS), but later standards were determined by community consensus.

Tierra (computer simulation)

Tierra is a computer simulation developed by ecologist Thomas S. Ray in the early 1990s in which computer programs compete for time (central processing unit (CPU) time) and space (access to main memory). In this context, the computer programs in Tierra are considered to be evolvable and can mutate, self-replicate and recombine.

Halite AI Programming Competition

Halite is an open-source computer programming contest developed by the hedge fund/tech firm Two Sigma in partnership with a team at Cornell Tech. Programmers can see the game environment and learn everything they need to know about the game. Participants are asked to build bots in whichever language they choose to compete on a two-dimensional virtual battle field.

Ruby Warrior

Game written in Ruby for learning Ruby and artificial intelligence.

You play as a warrior climbing a tall tower to reach the precious Ruby at the top level. On each floor you need to write a Ruby script to instruct the warrior to battle enemies, rescue captives, and reach the stairs. You have some idea of what each floor contains, but you never know for certain what will happen. You must give the Warrior enough artificial intelligence up-front to find his own way.

Programming-Adjacent or Game-Adjacent

MockMechanics (arguably a game)

StarEdit or other level-editors that ship with a game

Pixel Starships

Pixel Starships has a rule-based AI engine that I think is pretty cool. You create a bunch of these rules for each crew member and give them an ordering. The highest-precedence rule whose condition is currently fulfilled is the one the crew member will take.

Games Big Enough to Require Explanation

Board Games

Robogem

A programming game, designed to teach kids (6+) to program. Players try to collect gems by programming a robot on the board.

The robot takes three commands: move forward, turn left and turn right. Players string these commands to guide the robots. Advanced games include functions that can contain many moves which can then be repeated when wanted.

First player to collect three gems and return them to their home base wins the game.

Comments

Cameron Yick:

On a meta note, this question made me think about what elements of game design are “enough” to qualify as programming eg

  • Opportunity for emergent behavior? (Conway)
  • Ability to manage state / control flow?
  • System simulations (city / tycoon) - you achieve goals by modifying the environment rather than the agents
  • Has elements that can be optimized or automated
  • No single path to “win”, but some are quantitatively better than others

Sorting like an ant

Edited from the transcript for readability:

Ants actually have a really cool grouping algorithm. One of the things that they do is they keep their larva sorted by age and frequently things will happen where the larva get disordered because they had to move the colony or something, but they do it through just seemingly random behavior, they just randomly walk around pick up larva and put them down and over time they end up perfectly sorted and they investigated this and what they found out was that if they found a larva in an area giving off a particular scent and that area has none of that scent around it they'll tend to pick it up and if they happen to be carrying a larva with a particular scent and they get into an area that's high in that scent from other larva they'll tend to put it down and basically those simple rules over time will sort amazingly efficiently.

From Will Wright's Dynamics for Designers

How ants know the queen is dead?

Edited from the transcript for readability:

Ants have this local communication where they drop pheromones and they can change nearby ant states that can smell a pheromone and say "Oh food is nearby" or "it's time to attack" or "time to do nesting" instinct.

So they have this very elaborate horizontal propagation of information through pheromones, but they do this other thing called trophallaxis, what happens is an ant will come up to another one and basically vomit up food and then the other ant slips it up, so ants in fact have two stomachs, they have this private stomach and they have the social stomach that they share the food with every other hand now what this ends up being it's their global variables it's like a global bulletin board system for the entire colony. For instance the Queen generates this very particular chemical that no other ant produces and she does trophallaxis just like all the other ants and what happens is over the course of maybe two hours that little trace of chemical will disseminate through the entire colony and if that chemical ever evaporates because the Queen died or was lost the entire ant colony instantly knows that they've lost their queen and then they actually start a whole new behavior promoting a new queen. So ants actually have a balance of communication here between the global and local the way they propagate information.

From Will Wright's Dynamics for Designers

Will Wright on Prototypes

Edited from the transcript for readability:

Just to illustrate the differences between the two modeling techniques, in traditional math if we had let's say a square mile of land and we planted some crops on it standard math would probably go in here, measure the size of each of these things that ellipsoid would be kind of problematic and there'd be a kind of a long equation that you'd run all the numbers through to calculate what area of this plot of land was covered by fields.

Using modeling techniques we probably do something much stupider, we start throwing darts at it randomly and after we threw a certain number of darts we would then measure how many of the darts landed in a field and how many landed outside of a field and that ratio would tell us what percentage of the area was occupied by crops.

As we throw more darts we get a more accurate answer and if you can see this in some sense on the surface is a kind of a stupid approach but using the power of a computer this can actually give you very good results.

This is called by the way the monte carlo technique for kind of obvious reasons. This is a stochastic method which in simulation stochastic means there's some amount of randomness involved.

The same run of the model won't always give the same exact result, so basically as designers we have these dynamic spaces. There's this vast landscape of dynamic spaces that we can put into our games and we cut off little chunks of that space with rulesets, then we build our games around.

So as a designer one of the things I like to do is explore this space and figure out which dynamics are out there that would make for interesting game play.

Part of this talk I'm going to be using prototypes to show you examples of how we map this space.

The way I view prototypes basically is the same as paratroopers, we just drop these little paratroopers where we think might be interesting spots on the landscape, when they land we can start playing with the prototype and seeing how interesting was that space actually and then start iterating the prototype uphill in regions and so the prototypes in some sense are kind of hill climbing things in local regions of this dynamic space.

From Will Wright's Dynamics for Designers

Quick and Easy Web Slides with RemarkJS

Say you (I) want to create a quick presentation, a plus if you can share a link to the slides, another plus if you can use your text editor and markdown instead of something more complex.

Say you (I) probably know about remarkjs but doing it each time starting from the previous presentation is getting annoying.

Say you (I) want a place to go to copy paste the things to get started.

Well, here it is, a quick and easy way to get started with a web presentation using remarkjs

First create a folder:

mkdir my-presentation
cd my-presentation/

Then download the latest version of remarkjs:

wget https://cdnjs.cloudflare.com/ajax/libs/remark/0.14.0/remark.min.js -O remark.js

Then copy this html and paste it into a file named index.html in the same folder:

<!DOCTYPE html>
<html>
  <head>
    <title>Presentation Title</title>
    <meta charset="utf-8">
    <style>
      *{ font-family: 'sans'; }
      .remark-slide-container, .remark-slide-content{
          color:white;
          background-color:#0d4774;
      }
      .remark-slide-content{
          font-size: 2em;
          background-position: center;
          background-repeat: no-repeat;
          background-size: contain;
      }
      .remark-slide-scaler{
        box-shadow:none;
      }
      a,a:visited{color:white;}
      code{font-size: 0.8em !important;}
      h1, h2, h3 {
        font-weight: normal;
      }
      .remark-slide-content.reverse{color:white;}
      .reverse-text h1{color: #fefefe; background-color: #111; padding: 0.2em;}
      .remark-code, .remark-code *{ font-family: 'monospace'; white-space:pre}
      .slide-video{padding-top:1vh}
      .slide-video video{height:61vh}
      .slide-table{padding:1em 0}
      .no-padding{padding:0}
    </style>
  </head>
  <body>
    <textarea id="source">

class: center, middle

# Hi

---

class: center, middle

![An Image](image.png)

---

background-image: url(a-slide-background.png)

---

class: center, slide-video

<video controls src="a-video.mp4">

---

class: center, middle

# Thanks

    </textarea>
    <script src="./remark.js">
    </script>
    <script>
      remark.create({
        ratio: '16:9',
        slideNumberFormat: '' // '%current% / %total%'
      });
    </script>
  </body>
</html>

Then start a webserver in the folder:

python3 -m http.server

Open your browser and go to http://localhost:8000/

Then:

  • Change the title

  • Maybe change the background color and style

  • Remove/Edit the slides

  • Play with remark settings at the bottom of the index.html file

I'm writing a WebAssembly book: Early Access for WebAssembly from the Ground Up

If my memory serves me well, last year I went to Munich to have some beers with Patrick Dubroy and I mentioned that I was planning to write a small book about WebAssembly by writing toy compilers for small weird languages.

He mentioned that he was thinking about something similar and if I wanted to write it together, I said yes and the rest is a blog post announcing the public release of the Early Access of WebAssembly from the Ground Up :)

The ideas evolved over time but we settled on a "digital first" book that would allow us to experiment with interactive elements inspired by the Explorable Explanations movement and similar ideas like Bret Victor's Learnable Programming.

The first 6 chapters are already available and after the launch we are going to go back to "writing mode" to complete the rest.

If you are interested in learning the fundamentals of WebAssembly by building a compiler for a simple programming language step by step using JavaScript and OhmJS this book may be for you.

"Screw it up all the way until the finish"

In a video about molten glass the artist says something I like a lot so I'm transcribing it here to reference it (slightly edited, emphasis mine):

A great piece is basically balanced right on the edge of failure and success.

It's just balanced right there.

But you don't really know how or where that line is.

So you're very excited about that idea, it's spectacular to you.

And you go and do it even though you don't seem like it.

You're going into it with a little bit of fear and trepidation to get too close to that line because you don't want to fail and lose it.

But once you do fail it... all that's gone.

Now it's game on.

It's all about just learning, right?

So if it's a piece that you know is going to take four and a half hours and at 3 hours, it's kind of screwed up.

And you just say, okay, let's stop and start over.

Well, you really don't know what happens in hour 3 to 5. You have no idea.

So when you get to three again. Now, you have no idea what's coming.

So my idea is usually if I screw up, screw it up all the way that I can to find out exactly what's hiding, what vocabulary of intuition has not been developed, what part of that language.

So now I've screwed it up, screwed it up, screwed it up all the way until the finish.

We know where things might happen.

So now, when I go back into it, I've got the intuition more developed.

I mean, failure ends up being a good space for discovery, right?

But it's like, if I'm going to fail,

let's keep failing,

let's keep screwing up.

Let's see what's there. Let's go find out.

You know, but if you just stop and put it away and start over, you're kind of missing out on a lot.

A Simple, Understandable Production Ready Preact Project Setup

I wrote a similar post in the past, it's time for an updated version. This is a setup similar to the one I'm using with GlooData.

Setup

First we need to create our project, change folder names accordingly:

mkdir myproj
cd myproj
mkdir js lib css img

Create a package.json file at the root of the folder, we will need it later to install one or more build tools:

npm init

fetch deps (you can put this in a makefile, a justfile, a shell script or whatever)

wget https://cdnjs.cloudflare.com/ajax/libs/preact/10.11.2/preact.module.min.js -O lib/preact.js

Now let's do our first and last edit on our index.html:

<!doctype html>
<html>
 <head>
   <meta charset="utf-8">
   <meta name="viewport" content="width=device-width, minimum-scale=1.0, initial-scale=1, maximum-scale=1.0, user-scalable=no">
   <title>My App</title>
   <script type=module src="./js/app.js?r=dev"></script>
   <link rel="shortcut icon" href="img/favicon.png">
 </head>
 <body>
   <div id="app"></div>
 </body>
</html>

Here's a simple entry point that shows how to use preact without transpilers, modify to your tastes, js/app.js:

import {h, Fragment, render} from '../lib/preact.js';

const c =
        (tag) =>
        (attrs, ...childs) =>
            h(tag, attrs, ...childs),
    // some tags here, you get the idea
    button = c('button'),
    div = c('div'),
    span = c('span');

function rCounter(count) {
    return div(
        {class: 'counter'},
        button({id: 'dec'}, '-'),
        span(null, count),
        button({id: 'inc'}, '+')
    );
}

function main() {
    const rootNode = document.getElementById('app'),
        dom = rCounter(0);

    render(dom, rootNode);
}

main();

Also, I'm not covering state management in this post, I may do it in a future one if there's interest in it.

Serving

For development I use basic-http-server, I just start it at the root of the project, open the browser at the address it logs and just edit, save, reload, no transpilation, no waiting, no watchers.

If you don't want to install it you can use any other, you probably have python at hand, I find it a little slower to load, specially when you have many js modules:

python3 -m http.server

Building

You could just publish it as it is, since any modern browser will load it.

If not there are three steps you can do, bundle it into a single file, minify it and provide it as an old style js file instead of an ES module. Let's do that.

First the bundling, there are two options I use, one if you have deno at hand, the other if you have npm at hand.

First the common part, create a dist folder to put the build artifacts:

rm -r dist
mkdir -p dist/js
cp index.html dist/
cp -r img dist/

Bundling with Deno

deno bundle js/app.js > dist/app.bundle.js

Bundling with Rollup

Do this once at the root of your project:

npm install --save-dev rollup

Then:

rollup js/app.js --file dist/app.bundle.js --format iife -n app

Minifying with Uglify-js

Do this once at the root of your project:

npm install --save-dev uglify-js

Then:

uglifyjs dist/app.bundle.js -m -o dist/js/app.js

And that's it.

Now some extra/optional things.

Reproducible Dev Environments with Nix

How do you get this environment up and running consistently and reproducible?

I use nix, you can if you want too.

First you need to install it.

Then create a shell.nix file at the root of your project with the following:

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/nixos-22.05.tar.gz") {} }:

pkgs.mkShell {
    LOCALE_ARCHIVE_2_27 = "${pkgs.glibcLocales}/lib/locale/locale-archive";
    buildInputs = [
        pkgs.glibcLocales
        pkgs.wget
        pkgs.nodejs
    ];
    shellHook = ''
        export LC_ALL=en_US.UTF-8
        export PATH=$PWD/node_modules/.bin:$PATH
    '';
}

Then whenever you want to develop in this project run the following command at the root of your project:

nix-shell

Now you have a shell with all the tools you need.

Task automation with just

I use just to automate tasks, here are some snippets, you should check the docs for more details:

set shell := ["bash", "-uc"]
cdn := "https://cdn.jsdelivr.net/npm/"

ver_immutable := "4.1.0"
ver_preact := "10.11.2"

ROLLUP := "node_modules/.bin/rollup"
UGLIFY := "node_modules/.bin/uglifyjs"

default:
    @just --list

server:
    deno run --allow-all server.js

static-serve:
    basic-http-server -a 127.0.0.1:8000

fetch-deps: clean-deps create-deps
    just fetch immutable.js immutable@{{ver_immutable}}/dist/immutable.es.js
    just fetch preact.js preact@{{ver_preact}}/dist/preact.module.js

clean-deps:
    rm -rf deps

create-deps:
    mkdir -p deps

fetch-full NAME URL:
    wget {{URL}} -O deps/{{NAME}}

fetch NAME URL:
    wget {{cdn}}/{{URL}} -O deps/{{NAME}}

clear-dist:
    rm -rf dist

mkdir-dist:
    mkdir -p dist/js dist/img

dist-bundle:
    {{ROLLUP}} js/app.js --file dist/app.bundle.js --format iife -n app
    {{UGLIFY}} dist/app.bundle.js -m -o dist/js/app.js
    rm dist/app.bundle.js
    cp img/favicon.png dist/img/
    sed "s/type=module //g;s/r=dev/r=$(git describe --long --tags --dirty --always --match 'v[0-9]*\.[0-9]*')/g" index.html > dist/index.html

dist: fetch-deps clear-dist mkdir-dist dist-bundle

Older browser and cache busting

The script tag with type=module will not work for really old browsers, you may also want to make sure the browsers load the latest bundle after a deploy, for that you can run a line similar to this to replace the script tag in the index.html in your dist folder with one that achieves the two objectives:

sed "s/type=module //g;s/r=dev/r=$(git describe --long --tags --dirty --always --match 'v[0-9]*\.[0-9]*')/g" index.html > dist/index.html

The Proxy trick to not repeat yourself ™️

Above you saw something like this:

import {h, Fragment, render} from '../lib/preact.js';

const c =
        (tag) =>
        (attrs, ...childs) =>
            h(tag, attrs, ...childs),
    // some tags here, you get the idea
    button = c('button'),
    div = c('div'),
    span = c('span');

There's a trick to avoid writing the tag name twice, it avoids mistakes and minifies better, here it is:

import {h, Fragment, render} from '../lib/preact.js';

const genTag = new Proxy({}, {
      get(_, prop) { return (attrs, ...childs) => h(prop, attrs, ...childs); },
    },),
    // some tags here, you get the idea
    {button, div, span} = genTag;

Reproducible Riak Core Lite Tutorial with Nix

Introduction

Over the years I've created and recreated guides, posts and tutorials on how to setup an environment to create riak_core based applications.

Most of the repetitive work and troubleshooting is around the moving target that is a current Erlang/Elixir/Rebar3/Mix setup.

With this attempt I hope people will be able to setup and follow the guides with a reproducible environment that always reflects the one I had when I wrote the guide.

For that I will use Nix, to follow it you just need git and nix.

Follow instructions here to Install Nix if you haven't done that already.

Riak Core Lite with Erlang

Clone the Riak Core Lite Rebar3 Template

mkdir -p ~/.config/rebar3/templates
git clone https://github.com/riak-core-lite/rebar3_template_riak_core_lite.git ~/.config/rebar3/templates/rebar3_template_riak_core_lite

Enter a nix-shell with the needed tools:

nix-shell ~/.config/rebar3/templates/rebar3_template_riak_core_lite/shell.nix

Now your shell should have Erlang and rebar3 available, try it:

erlc -h
rebar3 help

Now let's create a new Riak Core Lite application, go to the folder where you want to store your new project and then:

Create the project:

rebar3 new rebar3_riak_core_lite name=ricor

Build it:

cd ricor
rebar3 release

Try it:

./_build/default/rel/ricor/bin/ricor console
(ricor@127.0.0.1)1> ricor:ping().

The output should look something like, the number will probably be different:

{pong,'ricor@127.0.0.1', 1324485858831130769622089379649131486563188867072}

With this environment you should be able to follow previous tutorials and guides like the Riak Core Tutorial and maybe the Little Riak Core Book.

Riak Core Lite with Elixir

I recommend you to follow Rkv: Step by Step Riak Core Lite Key Value Store Project

Clone the project:

git clone https://github.com/riak-core-lite/rkv.git

Enter a nix-shell with the needed tools:

cd rkv
nix-shell shell.nix

Now your shell should have Erlang, Elixir, rebar3 and mix available, try it:

erlc -h
rebar3 help
elixirc -h
mix --help

Fetch deps, compile test and run:

mix deps.get
mix compile
mix test
iex --name dev@127.0.0.1 -S mix run

Play with the API:

Rkv.get(:k1)

# {:error, :not_found}

Rkv.delete(:k1)

# :ok

Rkv.put(:k2, :v2)

# :ok

Rkv.get(:k2)

# {:ok, :v2}

Rkv.delete(:k2)

# :ok

Rkv.get(:k2)

# {:error, :not_found}

Rkv.put(:k2, :v2)

Rkv.get(:k2)

# {:ok, :v2}

You can follow the guide by switching to each tag in order: https://github.com/riak-core-lite/rkv/tags

I will try to keep the shell.nix files on both languages up to date from time to time to keep with major Erlang/Elixir versions, you can try to update the nix hash yourself and see if it still builds by following the instructions here: Nix Possible URL values

Have fun!