Overview
To effectively flatten an anvil, one must identify its imperfections and apply the necessary heat and pressure to correct them. A flat anvil is essential for fixing other tools and armors. Similarly, in ink! smart contracts, we regard integration and end-to-end (e2e) testing environments as anvils for resolving smart contract imperfections. However, during our research on vulnerability detection techniques for ink! in collaboration with LAFHIS in March 2013, we observed that while ink! integration tests are quicker, they lack certain functionalities compared to e2e tests. Addressing these gaps could streamline the development process and potentially reduce testing and quality assurance times.
Over the last three months, from September to November 2023, we diligently worked to identify and implement missing functionalities in the ink! integration testing environment, our anvil. We accomplished this thanks to a series of grants from the Web3 Foundation.
The first part of our journey was focused on identifying and documenting missing functionalities in integration tests, as well as implementation differences when compared to e2e tests. Our effort was centered around the 24 functions exposed in the env_access.rs file of the ink! repository. These functions are exposed in this file for their usage in ink! integration (IT) and e2e tests. As we documented each function, we created a set of test cases evaluating their behaviour against their e2e equivalent. These test cases and an in depth analysis of each function can be found in our analysis repository.
From this analysis we determined that 11 functions showed a consistent implementation across the integration and e2e testing environments. The other 13 functions showed missing implementations for integration tests, or implementation differences when compared to e2e tests.
Flattening the Anvil
Our work on the 7 functions with missing but feasible implementations in the integration testing environment started with the implementation of instantiate_contract(), allowing calls to contract constructors. This was the cornerstone of our development, as it allowed us to then work on contract invocation, implementing functions invoke_contract() and invoke_contract_delegate(). Also connected with contract instantiation were functions related to the contract’s code hash: set_code_hash(), code_hash() and own_code_hash(); we implemented these functions to have an equivalent behaviour to what is observed in the e2e environment. Finally, we developed caller_is_origin(), which detects caller changes every time calls are made between contracts. These developments were posted as pull requests to the ink! repository in PR #1988 and PR #1991.
On the other side, we resolved issues with 3 functions which were implemented on integration tests, but showed differences when compared to their equivalents on e2e. Starting with default_accounts(), we updated account names and addresses in integration tests for them to coincide with e2e tests, making them depend as well on the library sp_keyring::sr25519::Keyring and ensuring a consistent implementation in both environments. For set_contract_storage(), we added a missing validation in integration tests that was present in e2e, which checked that the size of the storage set did not exceed 16380 bytes. Lastly, we corrected the balance() function, ensuring the initial balance is the same in the integration and e2e environments. The proposed corrections for these functions were posted respectively through pull requests PR #1955, PR #1961 and PR #1982.
The 2 functions with missing and unfeasible implementations in integration tests were gas_left() and call_runtime(). The implementation of gas_left() was deemed unfeasible because integration tests are performed on native code rather than WASM code, and because gas cost is based on the number of WASM instructions executed. As for call_runtime(), implementing this function would require emulating practically the entire node to get consistent results, which is impractical.
Finally, the only function with a problem on the end-to-end (e2e) side was weight_to_fee(), which returned different values in integration and e2e tests. However, the large size of the runtime significantly slowed down debugging. We reported this issue (#1985) along with an associated detailed report to facilitate discussion with the ink! development team. The issue was eventually resolved in pull request PR #219 to substrate-contracts-node.
All in all, of the universe of 24 functions under our analysis, we have successfully flattened 22, correcting their implementation differences between integration and e2e tests. We also justified the infeasibility of developing gas_left() and call_runtime().
Igniting the Forge
This project started as a spinoff from our research on vulnerability detection techniques that we performed for building Scout, our ink! bug detection tool. As we evaluated cargo-fuzz, we discovered that integration tests are a useful foundation for building testing harnesses for fuzzing detectors. However, we soon realized that our work was constrained by the limitations of the integration testing environment itself, as many functions were unimplemented for testing purposes. This marked our first encounter with ink! integration tests and initially motivated us to work on improving them.
With a few exceptions, we can now proudly state that we have contributed significantly to completing the ink! integration testing environment. Having successfully tackled the challenges of ink! integration and end-to-end tests, we hope our contributions will not only help lift constraints for research teams like ours but also enhance testing practices in general within this environment.
