Acala Engineers Deliver Open-Source ‘Chopsticks’ Client to Improve Testing in Polkadot Ecosystem
Background
Developers who have worked with Ethereum will understand how useful it is to be able to fork the Ethereum mainnet and simulate transactions with it. Similarly, we wanted a similar tool for the networks using the Polkadot SDK, Substrate. We have had various tools to simulate transactions, fork mainnet into testnet, but not yet a tool to achieve a developer experience similar to what Ethereum development tools provide. I am presenting Chopsticks — the testing client to help you create a parallel reality of any Substrate-based network.
Goal: The goal of Chopsticks is to allow developers to fork any Substrate-based network instantly and to be able to modify the on-chain state of the fork-net arbitrary, as well as stub host functions. For example, to replace council members, or stub the signature verification host call to make it always return true.
Chopsticks is only ~2 months old, but it is already a working tool that is able to fork any Substrate-based network that’s using the “AURA” or “BABE” consensus algorithm (i.e. relaychains and most of the parachains). Note: I have only tested on Acala & Karura, but the codebase has zero references to Acala, so it should work with other parachains using AURA. Please open an issue if that’s not the case.
In the Polkadot ecosystem, many parachain teams have Kusama-based parachains to help serve as a canary network for deploying code there first before deploying to the Polkadot-based chain. This has been very helpful for teams in many regards. Canary networks are good for game theory or economic attack vector testing. However, there are always differences between the canary network and the mainnet and therefore a piece of code working on one doesn’t always mean it will work on the other network (e.g. logic involves one specific asset). As a result, we are still missing a testing environment that is an exact replica of the mainnet. In addition to that, the canary network is still a mainnet with real assets. This means it is not suitable for experiments that may cause availability or integrity issues.
There are some tools that provide the ability to fork an existing network for testing purposes, but they all have various limitations. For example, downloading the whole on-chain state can be time consuming for networks with a large on-chain state. And yet, the historical data can’t be easily forked. This means developers cannot test certain scenarios that involve block number related logic. Several issues stemming from this issue also inspired Chopsticks. For example, many Governance or Treasury votes have had to be redone due to errors, as there were no tools for people to test accurately and easily.
Until Chopsticks, there was no testing environment that could simulate the exact same environment as the mainnet for which you’re testing. This has now been solved, and the tool is now available open-source for all Polkadot developers to begin using.
This release marks another open-source software built for the community by the Acala engineering team, some examples here:
https://github.com/AcalaNetwork/parachain-utilities
https://github.com/open-web3-stack/parachain-launch
https://github.com/open-web3-stack/guardian
https://github.com/open-web3-stack/open-runtime-module-library
What is Chopsticks & How Does it Work?
At its most basic level, Chopsticks is a testing client that allows developers to fork a mainnet environment for testing that can simulate the exact same environment as the mainnet, including block number.
How?
Chopsticks reimplements a subset of a Substrate RPC, and deliberately does not implement most of the verification mechanism, allowing it to accept invalid blocks. For example, it does not check inherents, so there is no required sproof relaychain to create para blocks.
There are two components:
- A nodejs server that implements a subset of Substrate RPC and most of the logic
- A rust Wasm executor that is utilizing smoldot to execute Wasm calls and send results back to the nodejs server
The nodejs server implements a WebSocket Substrate JSONRPC client, a simple blockchain data structure and state overlay that transparently forwards requests to a remote RPC node to fetch data. This means it only fetches required data to avoid pulling all the on-chain state, which can be large. Optionally, the fetched data can be cached & persisted into a sqlite database, to avoid redundant query and speed up future usage.
The nodejs server will spawn the rust Wasm executor to execute runtime calls. The executor uses smoldot to execute the wasm runtime call and callback into the nodejs server when it is trying to access state. The state is either coming from the remote RPC node, or the local blockchain representation. The nodejs server is able to intercept and modify the data if needed. A simple block production logic is implemented to be able to create new block on demand. The block production logic deliberately produces invalid block if validated by Substrate node to simplify the implementation and gain more freedom on the implementation.
Chopsticks Milestones
- Build Chopsticks and test with a single chain ✅
- Improve support of relaychain (50% done)
- Support parachain runtime upgrade and improve runtime migration test support
- Support launching multiple networks and establish the XCM channels to support testing XCM related code
- Try-runtime integration to validate integrality of new blocks
- Library mode to allow other code to integrate with Chopsticks directly. For example, perform end-to-end tests
- Browser mode to allow web apps to simulate transactions before submitting them
- Implement more Substrate RPC
How to get involved with Chopsticks?
- Check out the repo: https://github.com/AcalaNetwork/chopsticks
- Questions or Feedback? Message Bryan in our teams’ shared chat or get in touch with our team.
Treasury Funding for Chopsticks
Another upcoming milestone for Chopsticks is to apply to the Polkadot Treasury to help fund further development milestones of Chopsticks and to ensure we can have the scope not only focus on Acala/Karura but also the whole ecosystem, including Polkadot, Kusama, and other parachains.
About Acala
Acala is a decentralized finance network powering the aUSD ecosystem. The core product, Acala USD, is a decentralized, multi-collateral, crypto-backed stablecoin serving as the native stablecoin of the Polkadot ecosystem. Acala’s Ethereum-compatible blockchain has built-in DeFi protocols for application developers to leverage, including a trustless staking derivative (liquid DOT — LDOT), a decentralized exchange, and the EVM+, a hybrid EVM offering fully Ethereum-compatible development environment plus full compatibility with Substrate.
Linktree | Discord | Website | Twitter | GitHub | Wiki | Newsletter | YouTube
