In this article we are benchmarking several auditing tools. The smart contract security audit is a critical phase in the development of smart contracts. The DAO hack was just one trip in the odyssey to secure Ethereum smart contracts and compatible blockchains like RSK and Cardano. It is important to highlight that back in 2016 A Call for a Temporary Moratorium on The DAO was published but it fell on deaf ears. Lesson learned: never deploy a smart contract without one or more security audits. It is tempting to use automated security tools to save time and money but they cannot replace human brains right now.
Linters and analysis tools are used to quickly point out errors in the syntax and in the smart contract logic. More complex tools can be used to better analyze in depth the behavior of smart contracts.
In this article we will examine how some of these tools fare against well known vulnerabilities extracted from the Smart Contract Weakness Registry.
Tools
We selected a set of tools that have different ways to analyse and report errors and are well known in the community. We have linters like Solhint and Solium that will perform an analysis at the source code level, and other more advanced like Oyente that acts at the opcode level to detect erroneous behavior.
Manticore
A symbolic execution tool that provides an API for writing your own tests from within Python. It comes with a set of heuristics for detecting some common mistakes, and we checked those.
Mythril
Mythril is a tool defined specifically for security and does seem to fare well against common bugs, yet still wasn’t perfect. It can also generate a control flow graph and can install a certain version of solc.
Oyente
An Analysis Tool for Smart Contracts. It performs the evaluation a the opcode level so problems at the Solidity syntax will not be detected.
Our tests were done with the latest release available as a docker container which includes Oyente v0.2.7.3.
SmartCheck
SmartCheck is an online tool that automatically checks Smart Contracts for vulnerabilities and bad practices. We testested against v0.0.9 released on October 25.
Solhint
This tool examines syntax and style errors, and a few security problems. We ran our tests against Solhint v1.4.0.
Solium
This is a linter designed to highlight style issues with the code. However, it also accepts some plugins that help detect security issues too. We tested against version 1.1.8. This tool has a plugin system, which provides the ability to extend it.
SonarSolidity
This is a tool provided as a plugin to the SonarQube console to analyse the code. It classifies what it finds in “code smells”, vulnerabilities and bugs. It can also import Solium reports, which can be further enhanced with Solium’s own plugins. We tested version 3.2.0.1227 by itself to see what we found.
Tests
We selected a representative subset from the SWC registry to run our test against. We specially chose some from the ranking in the Decentralized Application Security Project of vulnerabilities with a largest impact.
-
- SWC-100: Function Default Visibility
In solidity functions without visibility are marked as public and it will expose them, .
- SWC-100: Function Default Visibility
-
- SWC-101: Integer Overflow and Underflow
The result of arithmetic operations might exceed the maximum type value, or fall below the minimum if not properly addresses these will trigger unintended behavior.
- SWC-101: Integer Overflow and Underflow
-
- SWC-106: Unprotected SELFDESTRUCT Instruction
The SELFDESTRUCT operation causes the smart contract calling it to be deleted from the state and it cannot be called afterward. The attack to the Parity Multi-Sig wallet was caused by a contract that wasn’t properly initialized and left it unprotected.
- SWC-106: Unprotected SELFDESTRUCT Instruction
-
- SWC-107: Reentrancy
When a contracts makes a calls to external contract it giving up control and it is possible for another function on the same contract to be called.
- SWC-107: Reentrancy
-
- SWC-108: State Variable Default Visibility
Explicitly defining storage variable visibility prevent the misuse of them from inheriting contracts.
- SWC-108: State Variable Default Visibility
-
- SWC-109: Uninitialized Storage Pointer
Uninitialized local storage variables can point to existing storage variables in the contract incorrect use will cause inadvertently overwrites.
- SWC-109: Uninitialized Storage Pointer
-
- SWC-112: Delegatecall to Untrusted Callee
To call an untrusted contracts with delegatecall enables the target to modify the storage of the caller.
- SWC-112: Delegatecall to Untrusted Callee
-
- SWC-113: DoS with Failed Call
Making calls to external contracts might fail. If several calls are executed inside a loop a single failure will cause the whole transaction to revert.
- SWC-113: DoS with Failed Call
-
- SWC-114: Transaction Order Dependence
Miners are allowed to reorder transactions. A contract that depends on certain transaction order might be susceptible to attacks, for example Front Running.
- SWC-114: Transaction Order Dependence
-
- SWC-116: Timestamp Dependence
The timestamp in the block is controlled by the miner and it can manipulate it at his convenience. It should not be used for critical calculations.
- SWC-116: Timestamp Dependence
-
- SWC-119: Shadowing State Variables
Solidity allow using the same names for variables in different contexts. Using the same name is confusing and might cause subtle bugs.
- SWC-119: Shadowing State Variables
- SWC-120: Weak Sources of Randomness from Chain Attributes
The possible sources of randomness available to smart contracts is limited and an incorrect calculation can cause the results not to be random. The SmartBillions lottery was attacked through the incorrect assumption that all block hashes were accessible to smart contracts, only the last 256 hashes are available.
Results
The following table resumes the execution of the differents tools against our tests.
| SWC | Description | Manticore | Mythril | Oyente | Smartec | Solhint | Solium | SonarSolidity |
|---|---|---|---|---|---|---|---|---|
| 100 | Function Default Visibility | ✘ | ✔ | n/a (1) | ✔ | ✔ | ✔ (2.a) | ✘ |
| 101 | Integer Overflow and Underflow | ✔ | ✔ | ✔ | ✘ | ✘ | ✔ (2.b) | ✘ |
| 106 | Unprotected SELFDESTRUCT Instruction | ✔ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ |
| 107 | Reentrancy | ✘ | ✔ | ✔ | ✘ | ✘ | ✘ | ✘ |
| 108 | State Variable Default Visibility | ✔ | ✘ | n/a (1) | ✔ | ✔ | ✔ (2.c) | ✘ |
| 109 | Uninitialized Storage Pointer | ✔ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ |
| 112 | Delegatecall to Untrusted Callee | ✔ | ✔ | ✘ | ✘ | ✘ | ✘ (2.d) | ✘ |
| 113 | DoS with Failed Call | ✘ | ✔ | ✘ | ✔ | ✘ | ✘ | ✘ |
| 114 | Transaction Order Dependence | ✘ | ✔ | ✘ | ✘ | ✘ | ✘ | ✘ |
| 116 | Timestamp Dependence | ✔ | ✘ | ✘ | ✘ | ✔ | ✔ | ✘ |
| 119 | Shadowing State Variables | ✘ | ✘ | n/a (1) | ✘ | ✘ | ✔ (2.e) | ✘ |
| 120 | Weak Sources of Randomness from Chain Attributes | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ | ✔ |
Notes:
-
- Oyente analysis is at the EVM level so it will not be able to detect issues with the Solidity language.
- Solium uses a plugin architecture, to detect some of the attacks a separate plugin has to be enabled.
-
- security/enforce-explicit-visibility
-
- zeppelin/no-arithmetic-operations
-
- zeppelin/constant-candidates
-
- security/no-low-level-calls
- zeppelin/no-state-variable-shadowing
-
Conclusion
Most of the tools analyzed find the more common errors like function or variable visibility or arithmetic overflows/underflows. Some event will identify and recommend better practices.
Most of them will fail against more complex attacks like transaction reordering, or denial of service attacks. These attacks are complex to detect for a competent auditor and often requires interaction between several contracts.
In general the outcome was far from perfect. They find the more simple errors and failed against more complex attacks. Humans win.
