Anatomy of an Ethereum distributed exchange

Is it a good idea to store all data in a blockchain? What do you think?

Frankly speaking, it is a very bad idea. Mainly because a lot of data is transient, especially exchanges’ order books, and this doesn’t really need to freeze resources on thousands of blockchain nodes. What needs to be kept is a record of transactions resulting from a match between the asks and the bids. So, the best solution seems to be mixing transient and out of the blockchain order books and permanently in the blockchain for recording transactions. If in addition, the amounts take the form of cryptocurrencies, the settlement will also occur on the blockchain. Some distributed exchange architectures like 0x are designed along this pattern.

So, what is transient and out of the blockchain and what is in the blockchain?

Transient data: the order Book

The order book consists of two collections:
• The bids. This is the supply of the market.
• The asks. This is the demand side of the market.

People place ask orders to publish to the market what they offer. For example, Frank wants to sell a certain number of ethers (ETH) for a certain number of bitcoins (BTC). Let’s pretend, 100 ethers for 1 bitcoin. On the other hand, two persons post their offer to exchange bitcoins (BTC) for ethers (ETH).
• John offers 1 bitcoin to 100 ethers
• Jimmy offers 0.5 bitcoin for 100 ethers
Frank (the guy who want to sell ethers) order will be placed in the ask data collection and both John and Jimmy (guys who want to buy ethers with bitcoins) orders will be placed in the bid data collection. The ETH-BTC market will then be represented by two columns, one for the asks and one for the bids.

At first glance, there is a match with john’s offer for 100 ETH for 1 BTC. This will lead to a transaction and both orders are fulfilled and removed from the asks and bids columns. Jimmy’s offer will stay in the bids column.

Off course, exchanges matching mechanisms can be more sophisticated and be based on different algorithms. But, as you most likely noticed, the asks and the bids are transient data. People can modify their orders in the asks column when they can’t sell their coins within a desired period. They will do it until somebody finally matches the ask order. On the other hand, if the demand is very strong, the seller may fragment their order in smalls chunks and increase their price. This whole process is behind market dynamics. Sometimes orders are simply canceled and people wait until the market is more favorable.

Because the asks and the bids order are basically transient, they are preferably stored outside of the blockchain.

But when a match occurs, the transaction matching the ask order and the bid order can be permanently stored in the blockchain.

Permanent data: the transaction

Several centralized exchanges will never store their transactions in the blockchain. A new wave of hybrid distributed exchanges like 0x are appearing in the marketplace. In contrast to centralized exchanges requiring from their clients to deposit funds into the exchange’s account, 0x collection of smart contracts communicate directly with the token smart contracts. In addition, the transaction is also recorded into the blockchain. The internals of this choreography of smart contracts is described in 0x github repository. In fact, it is the structure of ERC20 implementations which makes this possible.

Each ERC20 token is managed with an implementation of the ERC20 interface. This interface became an overnight standard for newly created tokens on Ethereum. The ERC20 proposes a common interface for applications needing to perform transactions on their contained account. Can we get a clue of the ERC20 token implementation? hold on, we are getting there.

Behind the ERC20 interface lies a collection of accounts. A client application can perform transfer of funds from one account to another in that collection. So, let’s imagine that John as an account in an ERC20 Token named “Gizmo” (GZM), Jimmy also has an account in the very same token. We can represent a token like a small bank (sometimes containing several million as we noticed in the recent months). This token bank contains a collection of accounts. John and Jimmy have each an account in this token bank. John can transfer gizmos to jimmy with an application using the transfer function, part of the ERC20 standard. And if the gizmo token bank is “exchange” friendly, is support the transferFrom function. The latter allows an exchange to request the permission to transform from one account to another. In other words, a third party, the exchange, requests the permission to perform a transaction on two gizmo accounts.

To recap, the ERC20 offers a function transfer to applications, let’s say a wallet for instance. This transfer function simply transfers amounts from one account to another. It can also present a transferFrom function to, let’s say an exchange. The latter obtained a procurement to do so with the approve function giving the right to a third party to transfer funds from one account (john) to another (jimmy).


What is interesting is that this smart contract organization allows transfer of funds between token is exchange funds. And if we consider that each token has its own bank, then this application interface offers the potential to perform operations between token banks. This time, if we imagine that Lucy wants to exchange gizmo coins for yooo coins, then through the 0x smart contract kit makes it possible.


Synthesis

At the moment, the ERC20 interface is only popular on the Ethereum blockchain, more than 6592 tokens support this interface. Some are minimalist and support only the transfer function, others are exchanged friendly and support also the transferFrom and approve functions. The 0x smart contract kit hybrid solution combines the fast response of a classical centralized server based order book and the settlement on slower blockchain which contains already the token banks. We can think of the 0x kit as a token interbank exchange protocol.

The ERC20 interface and the 0x kit can potentially be ported on other blockchain supporting solidity as their smart contract language. For instance, it could be ported to the Rootstock (RSK) network. This network supports the solidity language which at the heart of the ERC20 interface. Since rootstock has bitcoin as its main currency (Ethereum has Ether), creating a new token bank would require some bitcoin value to do so. At the moment of writing these lines, 11 nodes were active with the new Ginger release. There is also QTum, another blockchain supporting the Ethereum virtual machine (EVM). It can potentially support a solidity ERC20 implementation.

In fact, having the same interface on different platforms could be a certain advantage, isn’t it?

How to create a new Ethereum token with Magic Code Generator

Most of the actual tokens created on the Ethereum platform are based on the ERC20 interface standard, there are, at the moment of writing these words, 5736 tokens supporting the ERC20 interface. This number is growing considerably every month. However, not all tokens act as a store of value.  By defining a common set of rules for Ethereum-based tokens to adhere to, ERC-20 allows developers of wallets, exchanges and other smart contracts, to know in advance how any new token based on the standard will behave.

The ERC20 interface is defined and implemented as an Ethereum smart contract. However, it can potentially be defined in any other language like for instance in C# for the Neo platform

The ERC20 interface is defined as following

interface ERC20Interface {
    function totalSupply() constant returns (uint totalSupply);
    function balanceOf(address _owner) constant returns (uint balance);
    function transfer(address _to, uint _value) returns (bool success);
    function transferFrom(address _from, address _to, uint _value) returns (bool success);
    function approve(address _spender, uint _value) returns (bool success);
    function allowance(address _owner, address _spender) constant returns (uint remaining);
    event Transfer(address indexed _from, address indexed _to, uint _value);
    event Approval(address indexed _owner, address indexed _spender, uint _value);
}

A contract implementing this interface will inherit its specification as illustrated in the following example.

contract FixedSupplyToken is ERC20Interface

Each function defined in the interface needs to be implemented in the contract inheriting the interface. For example, the balance of function will be defined as follow.

function balanceOf(address _owner) constant returns (uint256 balance) {
   return balances[_owner];

}

A complete implementation of this interface is documented in the Ethereum Wiki.

Even if this interface allowed great progress by providing a single interface wallet and exchanges can use to interface with tokens, it, nonetheless, carry some problems as documented in the ERC233 proposition.

ERC20 implementation can interpret the standard interface in different ways.

A contract interface is what is presented to smart contracts clients. The smart contract implementing the procedures associated with the interface inherits from the interface. Instead of an interface which allows only functions as members of that interface, a base contract can be used instead. In contrast to an interface, a base class allows data to be inherited. For example, the ERC20 standard base contract could include some data as illustrated in the example below.

contract ERC20Interface {
    string name;
    uint8 decimals;
    string symbol;
    address owner;
    function totalSupply() constant returns (uint totalSupply);
    function balanceOf(address _owner) constant returns (uint balance);
    function transfer(address _to, uint _value) returns (bool success);
    function transferFrom(address _from, address _to, uint _value) returns (bool success);
    function approve(address _spender, uint _value) returns (bool success);
    function allowance(address _owner, address _spender) constant returns (uint remaining);
    event Transfer(address indexed _from, address indexed _to, uint _value);
    event Approval(address indexed _owner, address indexed _spender, uint _value);
}

Since, by default, the properties are public, a solidity compiler will generate getters and setters which basically are functions to this interface. For example, With Web3, the property name will be accessed from JavaScript with:

Contract.name();

Which returns the name stored in this account.

On the other hand, the implementation of this interface will include the needed map to manage the collection of accounts for that created token. Something like:

Mapping (address => uint256) public balances;

Mapping (address => mapping (address => uint 256)) public allowances;

For more details, examine the ERC20 reference implementation in the Ethereum Wiki.

Using Magic Code Generator to create the ERC20 code.

On Magic Code Generator, because it supports a model driven approach, the first step is to specify a model as illustrated below for ERC20:

model = [
    {
        // service specifications
        service: {
            url: 'http://localhost:8545',
            name: 'ERC20Tokens',  
            type: 'ethereum', 
            contractAddress: '0xa588b75c081159c798feb5b8b64dffcce06f1805'
        }
    },
    {
        // singleton specifications
        singleton: {
            name: 'ERC20Interface',
            model: {
                symbol: { type: String, label: "Symbol", default:'MGX' },
                decimals: { type: Number, label: "Decimals", default: 18 },
                supply: { type: Number, label: "Supply", default: 1000000000},
                owner: { type: Address, label: "Owner" }
            },
            interfaces: {
                solidity: [
                    { Function: 'totalSupply() constant returns (uint totalSupply)' },
                    { Function: 'balanceOf(address _owner) constant returns (uint balance)' },
                    { Function: 'transfer(address _to, uint _value) returns (bool success)' },
                    { Function: 'transferFrom(address _from, address _to, uint _value) returns (bool success)' },
                    { Function: 'approve(address _spender, uint _value) returns (bool success)' },
                    { Function: 'allowance(address _owner, address _spender) constant returns (uint remaining)' },
                    { Event: 'Transfer(address indexed _from, address indexed _to, uint _value)' },
                    { Event: 'Approval(address indexed _owner, address indexed _spender, uint _value)' }
                ]
            }
        }
    }
];

At the moment, from this model, the Magic Code Generator automatically produces 2 files. So, less code to write. However, we are working toward the more ambitious goal to produce also the web service and the client application allowing you to set the default values in the smart contract. The data specified in the model will serve as templates to create an angular form communicating with a generated web service which in turn will produce the base contract with the default values set. This base contract can then be inherited into an implementation contract.

Thus, the code architecture proposed by the Magic code Generator is based on the notion of a base contract (which includes data) or an interface contract (which includes only functions). The latter is inherited by an implementation contract. The base contract or interface contract are automatically produced from the model and may also be used to generate a form to fulfill the default contract property values. The contract inheriting from the base contract implements each published function procedures.

contract ERC20Implementation is ERC20Interface {…}

So, to recap, Magic Code Generator produces the Solidity base contract file from which the implementation contract can inherit and implement the procedure associated with each function defined in the interface. Magic Code also generates the JavaScript class encapsulating the interaction with the contract.

The philosophy behind this structure is to separate the concerns in two different areas:

  • The base contract or interface is what is exposed to the outside world, how other contracts and javascript clients will interact with the implemented contract.
  • The implementation contract imports the base contract specification and implements the functions.

Because Magic Code Generator will generate only the base contract, it is not destroyed when the interface is modified. It will simply create a coding discrepancy and produce a compile error.

Actually, Magic Code Generator is packaged as a Visual Studio Code extension. The project is a work in progress as we add new functionalities. To generate the files from a model is quite easy, just a right mouse click (function click on the mac) over the model file name (with a .mdl extension) and select “Magic Code Generator”. It will automatically generate the files from the model as illustrated below.

The Magic Code Generator Visual Studio Code Extension can be installed from the Visual Studio Code marketplace.

Blockchain parts catalog

 

Have you ever considered that a blockchain is not a monolith but more a whole made of parts?

So, what are these parts?

The main components of a blockchain are:

  • A state storage.
  • A processing unit
  • A transaction linked list.
  • A consensus process.
  • An inter node communication protocol.
  • An interface to applications.

Storage state

In 2009, when bitcoin was introduced to the world, it stored a very simple data model to implement a ledger consisting of an account balance. A data model replicating the familiar bank account. This account allows deposits and withdrawals to be performed on it. The bitcoin network can be envisioned as a digital bank storing bitcoin accounts. After, the phenomenal success this technology encountered on the marketplace, some developers started to wonder how this technology can be improved.

In 2014 Vitalik, a young developer introduced Ethereum, a programmable blockchain. By expanding the instruction set of the main interpreter and by developing a programming language to let developers make applications, Vitalik created a more versatile blockchain, inventing a new term: the smart contracts. Basically, a smart contract is a piece of code to be replicated and interpreted on each node of the network. With this new technology, developers where given a more versatile data model no longer limited to a simple account balance. A data structure usual to programming languages like, structures, arrays, mapped arrays, etc. This data structure can be massaged by the smart contract code through a programming interface.

Since then, the notion of programmable blockchain has made its way with new variation on the same theme. One of them is making its way to the heart of developers, Hyperledger Fabric incubated and hosted by the Linux foundation. The data model supported by this blockchain is the data model supported by the languages used to implement the chaincode, the equivalent of Ethereum’s smart contract. Because the code runs into a Docker container, interpreted or compiled languages can be used as chaincode. Hence, more versatility has been added to blockchain applications developer’s arsenal. Actually, only the interpreted language GO is supported, but some developers are working on a Java and Javascript implementation. Thus, the data model supported by this new incarnation of blockchain technology is the one supported by the languages used to implement the chaincodes.

A processing unit

Since the beginning, with the bitcoin implementation, a virtual machine is mainly used as a processing engine. At first, in the case of bitcoin, the instruction set is limited to few instructions. This restriction also reduces the footprint to malicious account manipulations. With the appearance of Ethereum, also appeared an expansion of the instruction set and a more sophisticated virtual machine interpreter. A main advantage of using a virtual machine is to act as a sandbox to limit what operations are possible on an Ethereum node (i.e. server). For example, the instruction set prevent smart contracts (i.e. programs) to write or read files on the host. The only allowed operations are the one to be performed in the memory reserved to a smart contract and not elsewhere. Other blockchain processing unit are based on Java with the very same characteristics. Both Ethereum and Java virtual machine interpreters run pieces of code in a sandbox limiting what is permitted to be performed by a program.

With Hyperledger Fabric a new paradigm allows a different approach. The sandbox, in that case, is a Docker container. This technology introduced in 2013 provides the equivalent of a virtual machine without the overhead. We are not talking here of software interpreters also named virtual machines but more of low level implemented virtual machines based on hardware processor instructions set. Docker container rely on operating system sandboxes like LXC on lInux or Hyper-V on Windows. In a nutshell, this technology allows compiled or interpreted programs to run in a safe sandbox without interfering with other piece of software running on a host. Thus, each chaincode is provided with a Docker container to execute its either compiled or interpreted code.

Thus, the two main technologies used to execute blockchain code is:

  • Interpreted code running in a software implemented virtual machine. For example, a Java, Javascript or Ethereum virtual machine.
  • Compiled code running in a container. For example, the GO language.

A transaction linked list

This is the linked list which also gave the name blockchain to this type of technology. All transactions are mostly packaged as blocks and these blocks are linked together. Each block as a backlink to its predecessor. Such a link would be easily manipulated if it were not the case of the hash value of the transaction’s content. To prevent malicious manipulation of the transaction list, a hash value of each block is maintained. Each block also contains the hash value of the precedent block. Changing any block in this list would also change the hash value and hence indicate that the list was pirated.

A consensus Process

Hackers can include fraudulent transactions, and if money is involved, some may want to take advantage of the system. If you put together a group of network participants who doesn’t know and even trust each other, then you must find a mechanism to determine what is good and what is bad. This is precisely what the consensus mechanism is intended for. An automated process decides what is to be included into the blockchain and what the order of the transactions should be. In the case of bitcoin and ethereum networks, a group of network participants named miners fulfill this role. The specialized nodes have to resolve a puzzle, the first who comes to a satisfying result will be allowed to include a new block to the blockchain and be rewarded for the work. This process is called proof of work. Other means also exist like, proof of stake, proof of elapse time, etc. All these processes are in place to determine who will decide of the right order and what is included and what is discarded.

An inter node communication protocol

Network participant need a certain protocol to communicate together. They also need a protocol to discover each other. Some network requires to know the address of the neighbor nodes to be specified into a configuration file others uses a discovery mechanism to discover who to talk to. In a blockchain network, nodes are linked together with a peer to peer links. This means that all broadcasted transactions are transmitted from node to node to the very last one.

An interface to applications

A good way to perceive blockchains is to envision them as the equivalent of databases. Hence they need a front-end application service. Blockchains provide an API to communicate with. Most of the blockchains applications are three tiers. The blockchain is the storage tier, the application server, the second tier and the client, usually a browser is the third tier.