Analysis & Remediation of the Precompile Attack on the Hedera Network
T035 PD799 T9 U04 JPLN3 M7 Y 189f4abcd835 512 2023 03 11 001610 wxdn
Mar 10, 2023
by Joe Blanchard
Chief Information & Chief Information Security Officer, Swirlds Labs
TEAM Alex 2023 03 11 001410 xwcy
by Alex Popowycz
Chief Information & Chief Information Security Officer, Hedera

Update March 15, 2023:

There have been some questions from the Hedera community regarding how proxy access to the network was disabled. In April 2020, Hedera’s Technical Steering & Product Committee approved, via updates to Hedera’s Node Policy, the placement of IP proxies in front of all Hedera network nodes as a precautionary measure to protect the network against attacks. As shared in a May 2020 Hedera blog post, Hedera network operations staff would initially control those proxies, with full ownership and control of the proxies to be transitioned to Council member node operators over time. Currently, 14 Council members have deployed their own proxies to their nodes. Hedera network operations staff (composed of Hedera and Swirlds Labs employees) still have delegated authority and operational responsibility to retain SSH access to all the proxies on the network.

When the attacker began stealing tokens from the DEXs on March 9th, the Hedera network operations team (Hedera’s CIO/CISO Alex Popowycz, Swirlds Labs’s Dr. Leemon Baird (who also co-chairs the Council’s Technical Steering & Product Committee), and Swirlds Labs’ DevOps staff), together made the decision to disable the proxies, which prevented the exploit from being used to illicitly gain privileges to tokens managed by smart contracts (which could allow them to be stolen) and prevented the possibility of it being deployed to attack other potentially vulnerable implementations across the network. This decisive action limited the loss to ~$600K (USD equivalent).

In addition, there have been questions raised regarding whether or not the unauthorized privileges that the hacker had gained for the attack had been revoked. The Hedera community, including developers and DEXs affected by the attack, worked together with the Hedera team to construct and review a whitehat smart contract that used the same exploit as the attacker. The purpose was to remove the privileges that the attacker had illicitly obtained to the DEX’s accounts. The DEXs then oversaw the execution of the smart contract to ensure that the attacker no longer had any unauthorized privileges that could have been used to continue the attack. The ability for a smart contract to be deployed to edit these privileges, even by the Hedera whitehat team, was permanently removed once the mainnet code was updated by the Hedera Council.

***

On Thursday, March 9, an attacker exploited the Smart Contract Service code of the Hedera mainnet to transfer Hedera Token Service (HTS) tokens held by certain DEXs’ accounts to the attacker’s own account. The following is a summary of the attack: how it happened, steps taken to pause the attack and then permanently prevent it from happening again.

TL;DR

  • On March 9, an attacker exploited the Smart Contract Service code of the Hedera mainnet to transfer HTS tokens from targeted accounts to the attacker’s own account, targeting accounts at multiple DEXs, including Pangolin, SaucerSwap, and HeliSwap.

  • No retail user Hedera accounts were ever at risk.

  • No Hedera wallets were ever at risk.

  • DEXs and bridges worked together to stop tokens flowing over the bridge within an hour of initial notification of the attack.

  • To the best of our knowledge, the following tokens (valued at just under $600K USD at the time of attack) were stolen across multiple accounts/DEXs before the attacker was stopped.
    • DAI Stablecoin: 1,001 DAI [hts]

    • Tether USD: 66,997 USDT [hts]

    • USD Coin: 287,998 USDC [hts]

    • Wrapped HBAR: 3,630,000 WHBAR

  • After initial analysis of the attack and notifying relevant DEXs, Hedera DevOps shut off proxy access to the Hedera mainnet at 20:18 UTC, eleven hours after being notified of the attack by the DEXs. This prevented users from accessing the mainnet (and the attacker from draining additional tokens), but the mainnet remained up.

  • The core maintainers of the Hedera open-source software developed and tested a fix within 13 hours of discovering the vulnerability.
    • The fix prevents a smart contract from using a delegate call to call an HTS precompiled contract.

  • Time to resolution: Subsequently, the node operators (the Hedera Governing Council members) signed transactions to update the network’s codebase and the mainnet upgrade was completed at 02:04 UTC on March 11th, 41 hours from initial discovery of the attack.

Analysis of the Attack

The attacker targeted accounts used as liquidity pools at multiple DEXs that use Uniswap v2-derived contract code ported over from Ethereum to use the Hedera Token Service, including Pangolin, SaucerSwap, and HeliSwap.

The attack was conducted in multiple stages:

  1. Using one contract [0.0.2015837], the attacker exploited a bug in the precompiled contract code on Hedera to illicitly grant themselves authorization to withdraw tokens from specific DEX liquidity pools.

  2. Using a second contract [0.0.2015850], the attacker interacted with a target DEX to extract tokens into the attacker’s account [0.0.2015717].

  3. On Tuesday, March 7, the attacker created a new account, [0.0.2015705], which was to form the base of operations for gaining illicit authorization on liquidity pools. They funded their account from Binance [0.0.1030878], with about 2000 HBAR (First and Second transactions).

  4. The attacker then created two contracts: 0.0.2015743 and 0.0.2015837. Essential to the attack is the fact that most smart contracts, such as any Uniswap V2 smart contract derivation (of which several DEXs on Hedera use), have a swap function that takes an arbitrary contract to invoke as part of the swap.

  5. The attacker calls the swap, passing its attack contract 0.0.2015837 as the contract to be called as a delegate call with enhanced privilege. The DEX calls this contract, which then performs a delegate call on the Hedera HTS precompile to grant approval/allowance for the tokens owned by the DEXs liquidity pool.

  6. Critically, a bug in the Hedera mainnet code allowed the attacker to request approval with the credentials of the liquidity pool instead of with their own contracts’ credentials. Consequently, the attacker was able to gain approval/allowance on the DEX’s smart contract.

  7. The attacker then created and funded their second account [0.0.2015717] from Binance with a small amount of HBAR. This account would be the base from which they would launch a second smart contract to actually remove the tokens from the liquidity pools. They created and deployed a contract [0.0.2015850] that they used for the attack.

  8. The attacker used the smart contract against HeliSwap. They started by extracting a very small amount and then attacked for 1000 USDC / DAI. The attacker then used HashPort to transfer the stolen tokens off of the Hedera network.

  9. Next, the attacker targeted Pangolin’s USDC/USDT pool. They stole funds over the course of several contract calls (starting small and scaling up) while transferring tokens out of Hedera using HashPort.

  10. They then went after the Pangolin USDC/WHBAR pool. Initially they were met with failure (SPENDER_DOES_NOT_HAVE_ALLOWANCE) when extracting the USDC, but were able to extract WHBAR. Two minutes later, they were able to extract both.

  11. When the attackers moved tokens obtained through these attacks over the HashPort bridge, the bridge operators detected the activity and took swift action to disable it. HashPort was shut down, and the attacker resorted to sending funds to exchanges or other accounts.

Resolution

The Ethereum state is stored strictly in individual smart contracts, whereas Hedera tokens are stored in a system-wide state map. When making a delegate call, the EVM will execute the called contract’s code on the calling contract’s state, and not the contract storage storing canonical balances. However, when calling a precompiled contract, execution leaves the EVM and enters the core layer one logic, where the balance state is shared.

Hedera, SwirldsLabs, Limechain, Heliswap, and Pangolin collaborated to troubleshoot the situation and design a safe and accommodating approach to resolve the issue for the community. As a result of the collaboration between these decentralized stakeholders, a code change was authored for Hedera, which prevents a smart contract from using a delegate call to call a precompiled contract.

Under some circumstances it is possible for an EVM delegatecall() instruction invoking a pre-compile contract to result in one contract impersonating another. In 0.34.5, contracts may no longer use delegateCall() to invoke a pre-compiled contract. Contracts should instead use the call() method. The software upgrade blocks the HTS precompile calls only when they are called from the smart contract as a delegateCall() and not as a call().

In addition, to ensure that some of the already deployed contracts don’t break after the upgrade, we have also taken precautionary steps to create exception rules for these. A search of all smart contracts on the Hedera network was done to find any that would be impacted negatively by this new restriction (such as the HTS ERC-20 redirect contract) and, in two very specific router contracts, preserved the old behavior.

A Community Effort

The Hedera community and broader ecosystem came together quickly to minimize damage to users, and collaborated on the best way to fix the issues in as timely a manner as possible. We cannot thank the community enough for your patience, support, and understanding as we worked together to mitigate and resolve the vulnerability.

Appendix: How the Attack Unfolded

Using the account information and links to hashscan.io below anyone can investigate the attack for themselves.

Description

Explorer Link

Attacker Account

0.0.2015717

HashPort Bridge Account

0.0.540219

HeliSwap Contract 1
(USDC[hts] / DAI[hts])

0.0.1321537

Pangolin Contract 1
(USDC [hts] / USDT [hts])

0.0.1742018

Pangolin Contract 2
(USDC [hts] / WHBAR)

0.0.1739269

Transactions

Description

Explorer Link

Attacker receives 0.000001 USDC[hts] and 0.00000001 DAI[hts] from HeliSwap Contract 1

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678295022-842244363

Attacker receives 1 USDC[hts] and 1 DAI[hts] from HeliSwap Contract 1

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678295148-068622167

Attacker receives 1,000 USDC[hts] and 1,000 DAI[hts] from HeliSwap Contract 1

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678295172-757995189

Attacker transfers 1,000 USDC[hts] to HashPort Bridge

https://hashscan.io/mainnet/transaction/1678295664.069914036?tid=0.0.2015717-1678295649-215868455

Attacker transfers 1,000 DAI[hts] to HashPort Bridge

https://hashscan.io/mainnet/transaction/1678295912.615877003?tid=0.0.2015717-1678295902-741951792

Attacker receives 8,999 USDC[hts] and 8,999 USDT[hts] from Pangolin Contract 1

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678296437-766051588

Attacker transfers 8,999 USDT[hts] to HashPort Bridge

https://hashscan.io/mainnet/transaction/1678296652.235995003?tid=0.0.2015717-1678296640-930416009

Attacker transfers 9,000 USDC[hts] to HashPort Bridge

https://hashscan.io/mainnet/transaction/1678350360.193044504?tid=0.0.2015717-1678350340-965277504

Attacker receives 18,999 USDC[hts] and 18,999 USDT[hts] from Pangolin Contract 1

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678351699-768949791

Attacker receives 38,999 USDC[hts] and 38,999 USDT[hts] from Pangolin Contract 1

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678351720-769566523

Attacker transfers 57,998 USDC[hts] to HashPort Bridge

https://hashscan.io/mainnet/transaction/1678351949.709515968?tid=0.0.2015717-1678351926-753743020

Attacker receives 0.00000001 WHBAR from Pangolin Contract 2

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678353898-994119518

Attacker receives 165,000 WHBAR from Pangolin Contract 2

https://hashscan.io/mainnet/transaction/1678354038.265016005?tid=0.0.2015717-1678354026-839531487

Attacker receives 20,000 USDC[hts] and 165,000 WHBAR from Pangolin Contract 2

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678354145-559706938

Attacker receives 100,000 USDC[hts] and 1,650,000 WHBAR from Pangolin Contract 2

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678354301-249281606

Attacker swaps 1,650,000 WHBAR for native HBAR (BURN)

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678355241-437626974

Attacker transfers 1,600,000 HBAR to exchange (without memo)

https://hashscan.io/mainnet/transaction/1678355493.196910382?tid=0.0.2015717-1678355481-495505905

Attacker transfers 50,000 HBAR to exchange

https://hashscan.io/mainnet/transaction/1678355872.464304795?tid=0.0.2015717-1678355859-165367604

Attacker receives 100,000 USDC[hts] and 1,650,000 WHBAR from Pangolin Contract 2

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678356542-610671579

Attacker swaps 1,650,000 WHBAR for native HBAR (BURN)

https://hashscan.io/mainnet/transactionsById/0.0.2015717-1678356562-608926695

Attacker transfers 500,000 HBAR to exchange

https://hashscan.io/mainnet/transaction/1678356652.295369824?tid=0.0.2015717-1678356638-323550842

Attacker transfers 500,000 HBAR to exchange

https://hashscan.io/mainnet/transaction/1678356917.833889003?tid=0.0.2015717-1678356905-115689684

Attacker transfers 650,000 HBAR to exchange

https://hashscan.io/mainnet/transaction/1678357900.255395829?tid=0.0.2015717-1678357887-685718795