Originally written by Tyr Chen and Riley Shu and posted on Erlang Solutions Blog

I first came to know Erlang/OTP through one of Joe Armstrong’s, where he broke down the world into processes that can talk to each other like humans. When I started at ArcBlock, we were tasked with building a blockchain platform, and we decided to use Erlang/OTP extensively for our backend services, as well as our blockchain framework - Forge. The reasons for that are described in this article, and because of the functions of OTP, we have been able to build a highly practical, production-ready blockchain framework that not only delivers critical services to run a blockchain network but greatly simplifies what is required for next-generation applications and services.

A Blockchain Framework Primer

Forge is a tool that significantly simplifies the process of building a framework to support multi-chain networks or the concept of Build Your Own Chain (BYOC). Before Forge, it was challenging to build a chain. If people wanted to start their own blockchain, they would first need to set up different components of a blockchain system, including a consensus algorithm, p2p network, and the many other parts. After they went to the effort of making the components work together, they would need to decide how to adjust the different parameters of the blockchain, like total token supplies and distribution, specific transaction settings and admin access control. If they were lucky enough to get the blockchain running and found even the slightest thing wrong, they would need to stop all the running nodes go through the process of setting it again.

With Forge, we used the already available features and benefits of Erlang/OTP to deliver a framework that does all the hard work for the developer. For example, if you start a blockchain with Forge, the only requirement is to set the behaviours by enabling or disabling them in the configuration or at runtime. What’s more, if you want to update something when the chain has started, individual parts of the system can be hot-upgraded without rebooting the entire node — a critical feature for any product-grade application or service.

During the design and planning phase for our framework, we also evaluated other popular languages in the blockchain community, such as Golang. Golang has its benefits, including some pretty advanced libraries; however, to build the robust platform we wanted to deliver to our customers, there are three things that really pushed us towards Elixir.

Processes Simplify Problems

First is the need to breakdown complicated problems into processes. Blockchain itself is a mixture of solutions to many problems, and OTP allows us to deconstruct and tackle them one by one. The flexibility of grouping different processes into applications also helps us to maintain our codebase.

For example, when a user needs to build a blockchain node, it’s a very similar process to building an operating system. We need to orchestrate a list of “applications” to work together for exchanging events (for example, transactions for a blockchain system), executing these events and then storing the updated states. To help everyone understand how this works, it is very easy to break down the structure into several core applications:

• consensus application: processes that manage consensus related tasks
• storage application: processes that manage file system related tasks
• Forge application: processes that execute smart contract and support RPC interface
• event application: processes that manage event subscription
• indexer application: processes that continuously pull data from states database and index them into a relational database

Within these applications, Forge has some additional processes that collaborate to help process and handle the transactional activity for the blockchain. For example:

• when a user sends a transaction, there is a gRPC server that will process it and push it to the queue of mempool
• if the transaction is valid, it will be inserted into mempool, then flooded to the entire network; otherwise, it gets dropped
• once a new block is synced to us, the transactions will be picked up one by one and executed by the smart contract engine

Orchestrating these activities can be difficult. However, with the help of OTP, we are able to easily manage the complexity of the processes, through a continuous divide and conquer approach - things are organized into applications, each application is organized into a supervision tree, and each tree consists of many small processes. When there’s a need for concurrency, we are able to output a pool of processes; when robust sequential processing is required, we use a single process - by nature, its inbox serves as a message queue, which guarantees the tasks are processed in the right order.

Crashing Made Easy

Second is the ‘Let it crash’ mentality. A blockchain system consists of many running entities that are connected through a network. It’s essential to have an appropriate error handling system to maintain everything when the network is unstable or other unexpected disruptions happen.

For example, if one process needs to read on-chain information to handle an RPC request and crashes due to network instability, where a few retries would have fixed the problem, the supervisor in OTP helps bring the process back. This a perfect illustration of “write once, run forever,” as described by Joe Armstrong.

Ready-Made for Blockchain

The third important reason we choose Erlang/OTP is that it comes with many great built-in features for a blockchain system, like hot-upgrade, concurrency and high-availability.

One of the blockchain framework’s responsibilities is to run both the framework code and the customer’s code in a mixed manner, which requires secure isolation to work appropriately.

For example, user-defined smart contracts might use the same variable names as framework-defined contracts. Other implementations might replace parts of the system (e.g. customers may replace a consensus engine with their own implementation), and new features could be added into the existing blockchain node at runtime, without compromising the availability and stability. Thus, Forge was built on the shoulders of giants, a battle-tested production system offering the features we needed.

Open Flexibility

Using Erlang/OTP allows a blockchain framework to have a very important advantage over other languages — flexibility. As a blockchain framework, Forge by design is open to extension: you can extend the framework by adding more applications to implement more complicated features, like using a different consensus engine.

Blockchain networks created within our Erlang/OTP based framework allow users to hot-upgrade their smart contracts when needed without taking down the whole system, which gives users great flexibility in run time. For example, if you need to take down a node in a blockchain system to upgrade parts of the code, all nodes need to be upgraded at the same time, so that they use the same set of code logic and output the same result. In this case, OTP allows the partial upgrade to be included in a transaction, and all nodes can execute this transaction and upgrade the code at the same time.

Summary

Given the choice to build a blockchain framework in the future, our team would use Erlang/OTP 100% of the time. While others are still struggling with building and maintaining complicated systems, Erlang/OTP is time tested and proven in high-stress environments. Today’s Erlang/OTP has solved most of the challenges for us, allowing our team to focus more on building high-level features as well as making them user-friendly.