External Accounts vs. Smart Contracts

When a smart contract is deployed on the Ethereum network, it is created at a new address. This address acts as an identifier for a smart contract: whenever a user (or another contract) wants to interact with it, they send a transaction to that particular address.

Externally owned accounts (often abbreviated EOAs), on the other hand, are accounts owned by real-life users – or by any agent outside the network. They are also represented by addresses, which use the exact same format as the ones that identify smart contracts. As such, references to a user or a smart contract are equal in Ethereum: they are just addresses. This allows sending ETH to a recipient, without needing to differentiate whether it is an address backed by a smart contract or managed by an end user.

Code and State

A smart contract has two main properties: its code and its state. A contract’s state is composed of its ETH balance (since all Ethereum addresses have an associated balance) and its storage (where the value of its variables is persisted).

Code in a smart contract is typically short, since its execution “time” has a tight upper bound, defined by the Ethereum network. The code is run every time a transaction is received by the contract, and has access to the contract’s local storage, and the transaction’s context.

All Ethereum code is not run natively, but executed by Ethereum nodes on the Ethereum Virtual Machine, or EVM. The EVM executes a low-level stack-based assembly that operates with 32-byte words, typically referred to as EVM assembly. This assembly has opcodes for traditional arithmetic and logic operations, basic control flow, and some Ethereum-specific operations such as accessing storage and memory, or querying and managing ETH balance. There are also primitives for computing hashes or working with elliptic curve signatures.⁴ It is worth mentioning that the EVM has no support for floating-point arithmetic, and all operations are done on 256-bit integers used as fixed point decimals, to minimize the risk of numerical errors.

The execution model of the EVM is designed to favor simplicity over performance. All transactions are executed in a serial fashion (i.e. one after the other) and always in a single execution thread. This makes reasoning on smart contracts much easier: while a contract is executing a piece of code in response to a transaction, you can be sure that it will not receive a simultaneous transaction that could affect the current thread.

State in a smart contract is comprised of its storage and balance. The latter is the most straightforward of the two: all address types in Ethereum, regardless of being externally owned accounts or smart contracts, have an associated balance in ETH. Ethereum provides primitives for querying such balances (both from within a smart contract and from outside the network), and for easily transferring it.

As for the storage space in a smart contract, it is extremely large: it is an addressable space of 2^256 slots of 32 bytes each. However, writing to storage in the EVM is very costly, so it should always be used with care.

Since storage usage is expensive, the EVM also provides another 256-bit-addressable transient space called the memory, which is equivalent to a memory heap in other environments, and is guaranteed to be cleared in-between transactions.

Gas Usage

Executing code in the Ethereum network costs gas. Every operation run by a smart contract consumes a predefined amount of gas, where more complex operations consume more gas than simpler ones. All in all, gas is just a measure of execution cost, designed to prevent excessively complex computations on Ethereum. Since every transaction needs to be executed by every full node on the network to verify it, it is critical to keep them as simple as possible. This is also why operations that create new data on the blockchain, such as writing to storage or creating a new contract, are among the most costly ones in terms of gas.

How is gas obtained? The process is handled automatically on every transaction. Whenever a user sends a new transaction, they specify a gas price, which is the conversion rate between ETH and gas. After the transaction is run, the total amount of gas used is calculated, which gets converted to ETH using this gas price, and then deducted from the sender’s balance. Note that there are no requirements on gas price, and it can be nearly arbitrarily high or low. However, transactions with a very high gas price will be extremely costly to send; on the other hand, transactions with a very low gas price will be unattractive to miners, and will most likely never be included in the blockchain.

Additionally to the gas price, the transaction sender must specify a maximum gas allowance to be used during execution. If the transaction reaches a point where it has used all the gas allowed, it stops running and reports an out-of-gas error. This allows a user to control up to how much they are willing to spend on a transaction. Conversely, this also allows the network to check that a user has enough ETH for paying forexecution before actually running the code by checking that the sender’s balance is at least the maximum gas allowance times the specified gas price.

Note that Ethereum nodes can be used to query an estimation of the gas required to run a transaction, assuming the context where it runs does not change. This allows dynamic calculation of how much gas should be attached to a transaction instead of needing to hard-code it for every call issues by your system.

If the estimation is run when the contract’s balance is below 1 ETH, then the gas estimation will be low, and the user may send the transaction to the network using that value. However, before the transaction is actually picked up by a miner, another transaction may front-run the original one and increase the contract’s balance to be over 1 ETH. This would cause the original transaction to actually require a much higher amount of gas, and end up failing with an out-of-gas error. You should be aware of these situations when coding interactions with the network by always adding a reasonable buffer to the gas allowance on top of the estimated amounts and retrying transactions with updated estimations if needed.

Transactions

To recap, in order to interact with a smart contract, an external account must sign and broadcast a transaction directed to the contract’s address. The network then executes the smart contract’s code, with all data contained in the transaction (and the contract’s state) as context.

Transactions can also be sent to another externally owned account. In these cases, data is typically left empty, as the purpose is only to transfer ETH between accounts. However, they also consume gas, albeit a small amount compared to those sent to smart contracts.

The only field in a transaction that we have not yet reviewed is the nonce. This is an incremental integer value that ensures that all transactions sent from an account are processed in order: a nonce cannot skip a value and is always equal to the number of executed plus pending transactions sent from the address. It is also part of the network’s replay protection. Typically, you will not need to deal with nonces explicitly.

The lifecycle of a transaction is a bit complex, since transactions need to be picked up by a miner and confirmed in order to be considered final (Figure 3-1).

The first step in a transaction’s lifecycle is to be sent to an Ethereum node. This could be a private node owned by the user or a public node with no accounts associated. On the former, signing is typically handled by the node, which holds the user’s private keys; on the latter, transactions are signed by client software and then sent to the node. In either case, the node checks that the transaction is valid by trying to execute it locally, and if it is, it broadcasts it to the network.

Broadcasted transactions are said to be pending, since they have not yet been included in a block, but are waiting on what is called the mempool. The time until the transaction is processed by a miner and added to the blockchain will typically depend on network congestion and the gas price of the transaction itself – as we mentioned before, higher gas prices lead to more attractive transactions which are mined faster.

Eventually, the transaction is mined and included in a block. However, due to how the consensus algorithm in Ethereum works, it is still possible that a chain reorganization occurs, and the block that included this transaction is replaced by a different one. This is only likely to occur in very recently mined blocks. With every new block mined on top of it, the chance of a block being plucked out of the chain gets slimmer. A dozen confirmations (i.e., new blocks mined) is good enough for most scenarios, but you may want to wait for even more depending on your use case.

It is possible for a single account to have multiple pending transactions, since it is not required by the protocol to wait for a transaction to be mined or confirmed before sending the next one. The nonce ensures that all pending transactions will be processed by miners in the correct order.

Transactions in Ethereum may not always be successful. A transaction can fail due to a variety of reasons, such as running out of gas during execution, or because of a failed precondition check in the smart contract code. Smart contracts can enforce checks on the parameters with which they are called, which may cause a transaction to fail if it does not pass all preconditions. Transactions are atomic, meaning that they are all or nothing in terms of changes to state. In other words, a failing transaction will not persist any changes to the blockchain, except for the deduction of the gas execution fee from the sender’s balance. As such, when a transaction you sent fails, you can be confident that the state of your contracts on-chain was not altered in any way.

During execution, a transaction may log arbitrary information. These logs cannot be accessed from another smart contract and are only visible from outside the Ethereum network itself, such as from a front-end interface. Logged data may be structured and even indexed, allowing clients to search for specific events. We will work with logs more in-depth when we tackle Solidity’s events.

Calls

While transactions are the only way to perform a change in the Ethereum blockchain, they are not the only way to interact with smart contracts. Any off-chain client can perform a query on a smart contract by making a static call to it.

Calls are different from transactions in that they do not need to be signed and are not broadcasted to the Ethereum network, and thus cannot make any changes to the blockchain state and do not cost any gas. Calls are always resolved by the node that receives them and are only used for querying data from a smart contract.

A call executes smart contract code just like a transaction does, the only difference is that any changes performed during a call are not persisted, and the return value of a call is sent back to the sender (unlike transactions, where the sender has no way to get a return value back). If transactions can be thought of as setters that change the state of a smart contract, calls would be the getters.

Calls can even be issued on older blocks. Since all data in the blockchain is persisted, the state of the chain on every block,⁷ so a call to a smart contract can be made in the context of an older block. This feature is not used very often, but can be used to reconstruct the history of a contract, although logs are the preferred method to do this.

Remix

Before going into Solidity itself, we will introduce Remix,¹⁰ a tool for quickly prototyping Solidity code (Figure 3-2). Remix is a full in-browser IDE for Solidity development. It bundles a Solidity code editor, a compiler, and an EVM runtime. The EVM runtime allows you to locally deploy and test your smart contracts in a mock environment. Remix can also be connected to any Ethereum node, allowing you to manage your smart contract on any network, whether it is a local development network or the main Ethereum network (also referred to as mainnet).

Remix has several features for developing, analyzing, deploying, interacting with, and debugging smart contracts. We will focus on the most fundamental ones, but feel free to play around with the tool.

To start, add a new file by clicking the plus sign on the top left of the IDE, and create a new MyContract.sol file. On the right side of the screen, in the Compile tab, make sure you are using the compiler version 0.5.0, and choose to auto-compile your contracts. We will use this to test our first Solidity contract.

Your First Solidity Contract

Let’s go through this contract. First thing to notice is the pragma directive to set a required Solidity compiler version that corresponds to the code. A compiler that does not match the required version will refuse to compile the file. In particular, ^0.5.0 indicates any version that starts with 0.5.¹¹

Next is a contract block, which defines a smart contract to be deployed. The contract can define several state variables, such as myNumber in the example, which will be saved to storage in the EVM.

A contract can define multiple functions that will be executed in its context and will have access to its storage. Functions must always define the type of their arguments and their return type, if any. Also, functions can have different visibilities, depending on whether they can only be called internally within the contract or from outside, and can be restricted to not modify the contract’s storage, like getValue in the example. A constructor can be optionally defined and is run when the contract is deployed.

Another interesting keyword in this sample contract is require. It allows you to check for a condition and throw an error (an EVM revert) if it doesn’t hold. It is commonly used to check for preconditions in functions.

Remember that your contract is exposed to everyone on the blockchain. This means that any attacker can send a transaction to any public function with any parameters they wish. This makes a very compelling case for adding as many require statements as you need to always validate the inputs to your functions.

Before delving deeper into Solidity code, let’s try out our first contract in Remix. Copy the code for MyContract into the newly created MyContract.sol file tab in Remix, and wait for it to auto-compile. Then open the Run tab on the right side of the IDE (Figure 3-3). This will allow you to configure the environment where you want to deploy your contract: choose JavaScript VM to run the code in a simulated blockchain in your browser, and pick any of the provided Accounts.

To deploy your contract, choose MyContract from the contracts drop-down, enter an initial value to be used for the constructor we had defined, and accept the transaction. This will deploy the contract to your in-browser environment, which will execute almost instantly. Remember that when working in a real blockchain, the deployment will actually take several seconds.

You will notice that a new log entry showed up at in the middle-bottom panel (Figure 3-4). This has detailed information on the transaction executed. Take some time to go through it and understand all the info listed, referring back to the “Transactions” section in this chapter.

Also, on the bottom of the right sidebar of the IDE, you will now see that an instance of MyContract is listed under the Deployed Contracts section, including the address at which it was deployed. Expanding it will give you access to the public functions of the contract: increase and getValue. Try calling both of them, using different values for increase, to play around with the contract and check out the transactions generated.

Remember the distinction we made earlier between transactions and calls to contracts: the former broadcasts a transaction to the entire network that may change the state of a contract or the balance of an address, while the latter simply queries a single node to retrieve a value. Since getValue is flagged as a method that does not modify the contract (via the view keyword), Remix automatically issues a call to the contract when you execute it instead of a transaction. On the other hand, since increase does alter the contract’s state, it spawns a new transaction every time you call it.

We will now go more in depth into Solidity. Feel free to copy the code samples into Remix, deploy them, and interact with them. Remember that if you change the code of a contract, you will need to deploy a new instance of it in order to interact with the new version, since already deployed contracts cannot be changed. Also, refer to the Solidity documentation if you want to explore a particular topic in detail.

What’s in a Function?

Value Data Types

Reference Types

Reference types in Solidity include arrays, strings, mappings, and structs. Unlike value types, which are always handled by copy when assigning them or passing them as a parameter, reference types usually pass a handle to an object that can be aliased or modified from another function. We will review how this works with the most common reference type: the array.

Arrays, Bytes, and Strings

Solidity supports both fixed size and dynamic arrays. Array types are parametric, which means they are defined as an array of a base type. This allows you to define dynamic integer arrays like uint256[], or fixed size address arrays like address[4]. You can even work with arrays of arrays, but keep in mind that in Solidity the notation is reversed as compared to other languages: bool[][4] is an fixed size array of four dynamic arrays. Furthermore, you cannot return arrays of arrays in an external function call.

Arrays have a length method to query their size and provide an indexing operator to access or modify a position in the array. Dynamic arrays also have push and pop methods to add or remove elements. Arrays are typically iterated using a for-loop in Solidity (Listing 3-11).

contract ArrayTest {

  uint256[] array;

  function sum() public view returns (uint256) {

    uint256 total = 0;

    for (uint256 i = 0; i < array.length; i++) {

      total += array[i];

    }

    return total;

  }

  function add(uint256 value) public {

    array.push(value);

  }

}

Listing 3-11

Sample code for appending elements to an array and iterating them. Note that this example is subject to arithmetic overflow, since it is not using SafeMath for computing the sum over the array

Warning

Using a for-loop over an unbounded array is risky, since it may consume an arbitrarily high amount of gas, potentially more that fits in a single block, rendering the function impossible to call. Always avoid looping over an array that can grow out of control, or at least provide methods for iterating it by batches of controllable size.

Remember that arrays are reference types instead of value types. Reference types contain, as their name indicates, a reference to an object instead of the actual value. This means that, depending on the context, assigning an array variable to another will not create a copy but hand over a reference to the same array.

Whether the array is copied or a reference is passed depends on the data location. Data location may be a confusing concept, since it does not have a direct equivalent in other languages, and is an abstraction leakage from the EVM. Instead of trying to hide it and lead to potentially surprising results, Solidity opts for surfacing this distinction and forcing the programmer to be conscious about this important implementation detail.

As we mentioned before, every contract has access to an internal storage that can be used to persist data. And since this space is very expensive to use, the EVM provides access to a memory heap for transient operations. These are precisely the two main data locations that Solidity defines: storage and memory. The third location is calldata, which refers to the space where data is supplied in a transaction. For all practical purposes, calldata works just like memory, only that it is immutable.

Data locations need to be specified for every local variable or function parameter of a reference type. The only case where data location is unneeded is when declaring contract state variables, since these are always kept in storage. Note that when specifying locations for function arguments, you need to adhere to the following rules:

• External functions can only accept calldata reference types.

• Public functions can only accept memory reference types.

• Internal or private functions can only accept memory or storage reference types.

Assignment semantics then depend on the location of a reference type (Listing 3-12). An assignment from a memory reference to another will just pass a reference to the same object, and the same happens when assigning from a storage reference to another. However, when assigning from a memory reference to a storage one, Solidity will copy the entire memory array into storage.

contract DataLocations {

  uint256[] public storageArray;

  function test(uint256[] memory memoryArray) public {

    // We alias memoryArray to localMemory

    uint256[] memory localMemory = memoryArray;

    localMemory[0] = 42;

    require(localMemory[0] == memoryArray[0]);

    // We copy memoryArray into storageArray

    storageArray = memoryArray;

    require(storageArray[0] == 42);

    // We alias storageArray to localStorage

    uint256[] storage localStorage = storageArray;

    localStorage[0] = 21;

    require(localStorage[0] == storageArray[0]);

    // And changes to storageArray don't affect

    // the original memoryArray

    require(storageArray[0] != memoryArray[0]);

  }

}

Listing 3-12

Demo of how the memory and storage location modifiers work in Solidity

A special case of an array is bytes , which behaves exactly as a byte[] (i.e., a dynamic array of byte). However, this type is optimized and tightly packed in memory or storage, so it should always be preferred over byte[].

Another special case are strings. A string is an immutable UTF-8-encoded byte array, which doesn’t allow indexed access. String literals are defined using double quotes. Keep in mind that Solidity ships with almost no string manipulation functions, so strings are mostly stored as immutable identifiers or descriptions.

string myString = "foo";

Unlike value types, when an array state variable is defined as public, the implicit getter generated by Solidity accepts an index parameter, to identify which item in the array is to be retrieved. This only holds for regular dynamic arrays: bytes and strings are returned in a single call.

As an example, given the following contract with a public dynamic array, string, and bytes, the following getters (Figure 3-5) are available:

contract PublicArrays {

    uint256[] public numbers = [20,30,40];

    string public text = "foo";

    bytes public data = hex"20";

}

Mappings

Mappings, also referred to as hashes or dictionaries in other languages, are an associative reference type in Solidity.

Like arrays, they are parametric, in that they contain elements from other types. Mappings go from keys to values, and they accept any value type (plus bytes or strings) as keys, and can handle any type whatsoever (including other mappings) as values. Unlike arrays, however, the only valid location for mappings is storage, not memory or calldata.

Note

Under the hood, mappings are hash tables that rely on the fact that the storage space of a contract is large enough to ensure there will be no collisions for two different keys, so they guarantee that access to a value is always in constant time.

Autogenerated public getters for mappings (Listing 3-13) are similar to those of arrays, only that instead of accepting an index, they accept a key (Figure 3-6). In the case of nested mappings, a getter for a nested mapping will require a parameter for each key in each nesting level and only return the innermost value.

contract PublicMappings {

  mapping(uint256 => string)

    public num2str;

  mapping(uint256 => mapping(uint256 => string))

    public num2num2str;

  constructor() public {

    num2str[10] = "foo";

    num2num2str[10][20] = "bar";

  }

}

Listing 3-13

Sample contract with autogenerated getters for two mappings: a simple one and a nested one

An important caveat about mappings in Solidity is that, unlike other languages, there is no way to iterate the keys or values present in the mapping. This is related to how mappings are implemented. If you do need to keep track of the keys inserted into a mapping, you will need to keep a separate array to store them.

Emitting Events

The event is declared using the event keyword and fired using emit. Note that, in the event declaration, some of its arguments can be flagged as indexed. These allow watching or querying events that have a certain value for that parameter. Whether to flag an argument as indexed or not will depend strictly on your use case.

Events are useful not just for monitoring a contract for changes but also as a replacement for return values in a transaction. Since it is not possible for a client to retrieve a return value from a method called in a transaction, it is common to emit an event with the value that needs to be obtained. The client then retrieves the events attached to the transaction receipt and extracts the value from there.

Imports, Inheritance, and Libraries

A file can be imported not only to refer to another contract but also to extend from it. Solidity has support for multiple inheritance. This makes inheritance the default mechanism to extend functionality, or to pull in features from another contract, using base contracts as if they were mixins (Listing 3-17).

Application Binary Interface

The Application Binary Interface (ABI) of a contract is the set of public methods exposed by a contract. Think of it as its public API that can be called from an external account or another contract.

The key concept behind the ABI is that it is language independent. It is a specification on how function calls, arguments, and return values should be encoded. This allows a contract written in Solidity to seamlessly interact with a contract written in another high-level language, such as Vyper.

The ABI has a set of data types fairly close to those of Solidity, including addresses, integers (signed and unsigned), booleans, strings, arrays, and so on. The main exceptions are contract types, which are handled as plain addresses, and structs, which are encoded as tuples with all their fields.

EIPs and ERCs

Being a decentralized protocol, all improvements to Ethereum often start as a proposal (or EIP, Ethereum Improvement Proposal) to be discussed by the community. These proposals encompass from changes to the core protocol itself to application-level standards defined for compatibility.

The latter are referred to as Ethereum Request for Comments (or ERC, following the RFC nomenclature used by the Internet Engineering Task Force). These are of particular importance, since they define the common ABIs and semantics of contracts to be used. They act as building blocks for larger applications and foster reusability by setting a common interface agreed upon by the community.

Two of the most popular smart contract standards, fungible and non-fungible tokens, are defined as ERCs – ERC20 and ERC721, respectively.

ERC20 Tokens

Tokens, defined in the ERC20 standard,¹⁴ are probably the most common building block of Ethereum applications. In its core, an ERC20 contract keeps track of a balance for every token holder address and provides methods for querying and managing such balances (Listing 3-20).

A token can act as a decentralized currency for any project. As such, any team can easily roll out their own cryptocurrency on top of the Ethereum network, without needing to set up their own blockchain.

Additionally, the standard includes three optional getters: name, symbol, and decimals. These are often used by wallets or other client software to display information about a token given its address.

The standard does not specify how tokens are initially distributed or how their total supply evolves over time. Certain tokens have a fixed supply set when created and assigned to a single address which manually distributes them. Others can be minted over time and distributed based on certain rules.

ERC721 Non-fungible Tokens

The ERC721 standard (Listing 3-21) defines the specification for digital collectibles, also called non-fungible tokens (often abbreviated NFTs). NFTs are different from the traditional ERC20 token in that each token is identifiable and different from the other. As such, a user no longer has a number of tokens, but has a particular set of unique identifiable tokens, each with its own metadata associated to it. As an analogy, if ERC20 tokens can be used to represent a currency, ERC721 tokens can be used to represent collectible cards.

Throughout the standard, tokens are identified by an opaque uint256 value. While some implementations use incremental numbers for IDs, this is not required at all.

To avoid returning an arbitrarily large array with all the tokens created, or all the tokens that belong to a user, this Enumerable extension provides means to know the total number of tokens (or the number of tokens that belong to a user) and to iterate through them via an index.

The safe transfer methods check that the recipient of the token can actually manage them, by calling into a specified onERC721Received method in the recipient (Figure 3-7). If the recipient does not implement this method, the transfer is aborted. This prevents tokens from being accidentally lost by sending to contracts that cannot manage them, thus locking them forever, which is a common problem in ERC20. As such, it is recommended to always prefer this method over the plain transferFrom.

An overload of this method includes an extra data parameter. This data is forwarded on the onERC721Received call and can be used by the recipient to decide whether to accept the token to be transferred.

While the format of tokenURI is not defined and is left for implementers to choose,¹⁶ it provides a standard way to obtain information on a particular token instance, such as an image or a blurb of text that describes it. Token URI often points to an off-chain¹⁷ resource that contains a manifest for the token.