Building a tool to detect security vulnerabilities is all about finding bugs, while helping the developer team not to make mistakes and not imposing a high burden in terms of reviewing findings.
Last time we introduced this tool for finding security vulnerabilities in ink! smart contracts. In Scout Audit’s inception, then, we were set to build a tool that detects some of the more critical vulnerabilities in ink! smart contracts. After doing a cost-benefit analysis, we decided to go for a linter based off dylint, doing simple syntactic predicates, and targeting the top vulnerability classes ordered using Ethereum statistics.
In this second run, our aim is to improve precision and coverage, and also going from the proof-of-concept work we had done, to a more robust prototype that is ready to be used by devs. This post records the work undergone towards the second milestone with ink! Scout, CoinFabrik’s vulnerability detection tool for the ink! Smart contract programming language.
This work was funded by a grant from the Web3 Foundation Grants Program.
The Sum of it All
“You gotta be kidding. I don’t got any time to break in any newcomers. Why don’t you do this boy a favor … if I need a partner, I’ll get me someone who knows what the hell he’s doin’.” Harry Callahan (Dirty Harry, 1971 film).
In order to build a sound foundation we improved the code of the proof of concept we had developed first, adding tests, fixing bugs, and adding two components that would make Scout Audit usable and appealing: a command-line interface and a VS Code extension.
Scout Audit thus includes:
- The CLI
- The VS Code extension with line squiggles over the security issues discovered and hover-over messages
- A folder where each subfolder is named after a vulnerability class, and each of these subfolders includes at least one pair of smart contracts (one vulnerable and its remediated counterpart).
- A list of detectors one per vulnerability class here.
We want False Positives Out
“If it looks like a duck and quacks like a duck but it needs batteries, you probably have the wrong abstraction.” Bailey, Derick. “SOLID Development Principles – In Motivational Pictures” when describing the Liskov Substitution Principle.
Reducing the rate of false positives is a difficult task: one needs to have a detection mechanism that does not trigger an alarm when there is no vulnerability. In the case of ink! smart contracts the problem worsens as the amount of deployed smart contracts in use is not huge, and of those very few have public audits.
For the first milestone, our aim was to build a good amount of detectors, and having no publicly-available set of pairs of vulnerability class-vulnerability examples, we concocted the database ourselves. We curated a taxonomy of vulnerability classes, and for each of these, we developed a “small” smart contract including an example of the said vulnerability class.
Hence, the lints we prepared may have a precision and recall of 100% in the sample of smart contracts we had created, but have much lower precision and recall when compared against all possible smart contracts. Nevertheless, it would be pointless to build a tool for all the possible smart contracts. We need much less.
Statistics. We only need to reduce false positives for those smart contracts that are developed and deployed. And, in order to have that, our plan was to canvas deployed projects for which we could also get the source code, run the tool against those smart contracts, analyze alerts in order to signal out false positive alarms, isolate these cases and suggest changes to the underlying detectors so that the alarms are no longer triggered in these cases.
In order to improve on precision, we made a measured effort by running the tool against the following five smart contracts, one in the Phala Network, called Phat Bricks, and four in the Panorama Swap Platform: psp22, trading_pair_psp22, vesting_contract, multi_sig (all in the Panorama Swap INK SC repository). We found false positive alarms which lead to improvements in two of our detectors.
First, there was a problem with the unused return enum detector, which warns if a function that returns a Result type does not return a Result enum variant (Ok/Err). Our first version of this detector was based off the High-Level Intermediate Representation (the primary IR used in rustc), which would not propagate the error when using the question mark operator (?), and as a result the detector was detecting a vulnerability when in fact the error was properly caught. We switched to the Abstract Syntax Tree (AST) which is also available via rustc_ast and allows for correct propagation of errors.
A second, the set-contract-storage detector which validates that the owner is calling env::set_contract_storage was missing something. While it was checking if the caller was equal to the owner, it failed to check if the owner was equal to the callee.
In short, in this stage of our project we corroborated that the path to a precise tool is paved by using the tool with different interesting projects and using the toolset provided by rustc to improve detectors.
We want to detect bugs
“If it looks like a duck, and quacks like a duck, we have at least to consider the possibility that we have a small aquatic bird of the family Anatidae on our hands.” Douglas Adams. Dirk Gently’s Holistic Detective Agency (1987).
We carefully generated a list of vulnerability classes that can happen in ink! smart contracts. The attempt was to do something which is exhaustive and sufficiently-detailed so that examples are easy to create. This work stands on the shoulders of a list (summarized here) that our Security Audits Team has been preparing based on their experience and other works. The list is alive, and keeps growing as we add best practices, and modify some items to better reflect the vulnerability class.
With that list in mind, we developed during the first stage of this project, 7 examples of smart contracts, each including an instance of a new security vulnerability. In this second stage, we added 6 more examples and detectors. For each of the thirteen vulnerability classes we have at least one smart contract with an instance of the vulnerability, a second smart contract that “fixes” this vulnerability, and a detector which correctly detects the vulnerability in the first one and does not trigger an alarm in what we call the remediated example.
Outro
We nicknamed this stage “bigger, better and faster” because our intent here was to develop more vulnerabilities (bigger), a prototype with CLI and VS Code integration that on top has less false positives (better), while learning how to do this more efficiently (faster).
As we iterate, we learn how to do this bigger, better and faster. That is how we envision the work ahead. We want the community of ink! developers to use Scout Audit, and give us feedback. And we want to use this feedback to grow the tool more efficiently, reducing the pains of false positives and enhancing the user experience. We are releasing the Scout Audit to the public with our eyes and ears open. We welcome feedback and suggestions.
