Eth2 Slashing Prevention Tips
This post regards information on slashing in the Eth2 network — what it is, why it’s used, how to avoid being slashed, and what to do if slashing occurs. Most of the information presented within this documentation is implementation agnostic, meaning it will be applicable to all users, regardless of whether they choose to run Prysm, Lighthouse, Teku, Nimbus, or another implementation for their validator setup.
*Important Disclaimer*
This post, as well as the Prysm eth2 implementation, are covered by our official Terms of Service. Within the terms, it is specified that Prysmatic Labs holds no liability in the case of lost funds, regardless of cause, including client bugs or errors in documentation. This removal of liability extends to the slashing process, and users are the sole legally responsible party for their validator(s). While each eth2 team does their best to ensure the codebase is secure and production grade, as with any other development project, there is always the potential for error. Participating in Ethereum 2.0 is a risky activity and it is critical participants research and understand their responsibilities, legal and otherwise.
This post is merely meant as a set of tips for security best practices. It does not guarantee slashing prevention.
What is slashing
Slashing occurs when a validator has provably acted against the Ethereum network. Slashing doesn’t need to have malicious intent necessarily for example, it could happen from misconfiguration. The validator acted in a way the can confuse or disrupt the integrity of the system It removes, or ‘slashes’, a portion of the offending validators existing stake, causing a gradual loss of ETH over time until the validator is forcefully ejected and marked as SLASHED. This is irreversible.
Slashing is not to be confused with inactivity penalties, which are normal loss of funds incurred over time the longer your validator is offline and unable to perform its duties
The goal of slashing is to disincentive those who seek to harm the eth2 network, and inversely reward those who maintain, police, and operate the network as intended. The main purpose of slashing is to mitigate attacks that might be otherwise cheap to perform on the eth2 network, such as trivially creating conflicting forks of validators attesting on a different view of historical checkpoints.
A validator that correctly follows the protocol never emits a slashable vote in normal operations. Validators will not be slashed for simply being offline.
How slashing works
The act of slashing is destroying a portion of an active validators stake for acting against the Ethereum network and forcefully ejecting them. The goal of slashing is to disincentive those who seek to harm the Eth2 network, and to bolster the security of the chain by promoting good actors.
Though slashing is intended to be a method of punishment, even as a benevolent user, it is still possible to get slashed if you have a validator has been configured improperly; thus, it is critical to understand how to properly set up and manage a validator client and operating system, and to avoid attempting advanced instructions without understanding its entire functionality.
Slashing occurs in the following scenarios:
- A validator proposes two conflicting blocks at the same slot with a different root (essentially a hash of the inner data). If this action were not penalized, it would be trivial for validators to try to create unnecessary forks and chaos in the chain. Note: simply proposing the same, exact block twice is not a slashable offense.
- A validator attests to two conflicting blocks at the same slot. This is known as a double vote, and also signifies that a validator could be attempting to create conflicting forks of the chain. Note: simply voting for the same, exact block twice is not a slashable offense
- A validator casts a vote which “surrounds” or is “surrounded” by a previous vote, meaning the validator is trying to vote against history, which is a slashable offense.
What happens after a slashing
Despite multiple slashing prevention mechanisms in place, it is critical that only a single validator is used per validator key pair. This sounds easy enough, but when multiple validators are configured, it is very easy to restore a validator improperly and duplicate an already running validator. Always confirm the correct validator keys are being used/restored/setup anytime you are making changes.
A couple of additional resources can be found here and here.
In the event a validator is slashed, a number of things will happen:
- The slashed validator is forced to exit the beacon chain ~36 days in the future
- Three penalties types are earned by the slashed validator (source):
- A minimum penalty is incurred at the moment a proposed whistleblower includes the whistleblowing message in a block
- A penalty at the beginning of every epoch until the validator leaves the exit queue
- A special penalty is applied midway between the time when the whistleblowing message is included in a block, and the time when the slashed offender can withdraw. This special penalty is proportional to how many other validators have also been slashed over the period. The maximum applied can be as high as the whole offender’s effective balance.
This means if a validator is slashed, it will immediately see a penalty, and will continue to incur penalties for ~36 days until it can exit, AND also receive a one time penalty at day ~18 in the ~36 day period. The amount slashed is also influenced by the number of validators who are also found to be guilty of slashable offenses (in addition to the number online) during the same epoch (ie, is this a coordinated attack or a single bad actor). For a more detailed breakdown, please visit here.
Finally, it is worth noting slashed validators cannot re-enter the validator set. A new stake and keys would need to be generated to continue validating. Slashing is irreversible!
Being slashed essentially results in a slow bleed of ETH staked, with a big hit taken at the halfway point as well. An exit from the beacon chain is possible after ~36 days and any remaining ETH staked can also be withdrawn, but a sizable portion of ETH will be lost during this ~36 day time period.
Common user mistakes leading to slashing
Even though the slashing scenarios previously mentioned seem impossible to do as a normal user, it is likely an improper configuration can lead you to get slashed as an honest validator! Here are some common scenarios where this may arise:
The same validating keys running on two servers at the same time, with one of them perhaps serving as a failover in case the first one is down
Explanation: This is the EASIEST way to get yourself slashed. If your failover system has a false positive your first node is down, you can find yourself in a situation where you commit slashable offenses.
DO NOT RUN YOUR VALIDATING KEYS IN TWO PLACES AT THE SAME TIME.
You migrate your keys to another machine or another eth2 client without also migrating your slashing protection history
Explanation: your other node might have an incorrectly synced clock, leading you to commit a slashable offense that could have easily been avoided if you also imported your slashing protection history.
You delete or lose your slashing protection history in your validator client:
Explanation: not having a slashing protection history could lead some issues such as your clock being messed up to create a slashable block or vote.
Validating using a containerized environment that does not have a persistent volume
Explanation: If you are running with Docker or perhaps in a cloud environment such as Kubernetes, you need to setup persistent volumes for your validators, such that if a pod or container is restart, the slashing protection history will not be erased.
Protocol bug that may cause slashing errors (this is very unlikely, worst case scenario)
Explanation: catalysts for mass slashing events that have occured on the testnets were often due to bugs in implementations. However, validators which had a slashing protection database and proper configurations were not affected. These examples encompass bugs with time server failures, and improper handling of block IDs. When the time server failures occurred in the Medalla testnet, most slashed validators got slashed because they did not have a slashing history database persisted.
When choosing a validator, it is critical to understand how to set up, configure, upgrade, and troubleshoot any installed software. A great resource to better understand the risks of staking Eth2 can be found here at ethereumprice.org. Another great resource explaining how to detect slashing conditions can be found here.
Who Does the Slashing?
Slasher
The slasher refers to a separate piece of software with its main purpose of detecting slashable events. You can think of a slasher as the “police” of the network. Due to the extra data and processes required to detect malicious messages, usually it is run separate from the beacon node. In order to detect slashable messages, the slasher records the attesting and proposing history for every validator on the network, and cross references this history with what is broadcasted to find slashable messages such as double blocks or surrounding votes.
All the network needs is 1 honest slasher client to monitor the network because any slashings found are propagated to the entire network for it to be put into a block as soon as possible.
Whistleblower Rewards
In order to incentivize slashing detection, validators are given a “whistleblower reward” which is a reward on the beacon chain for submitting a block with any valid slashing. These rewards are given for each validator in the slashing, and are usually ~0.1 ETH each validator.
While incentivizing detection is valuable, simply running a slasher client will not have you earn whistleblower rewards if you find a slashing in Prysm. By default, any slashings found are propagated to the network to be included in the block ASAP so usually the reward goes to the proposer immediately after the slashing is detected, not to the validator running the slasher.
Running a slasher is not meant to be profitable. Slashing is meant to be rare and whistleblower rewards are low on purpose. Running a slasher is meant to be an altruistic action, and once again, only a single honest, properly functioning slasher needs to be active in the network to catch slashable offenses. Thankfully, this is a low bar to entry, and we envision quite a lot of users and entities will run slashers to ensure network security.
Slashing Prevention
We have some good news — slashing is a preventable offense! There are a number of best practices to ensure slashing does not occur, but it is critical they are understood to be used at their best capacity.
Local slashing protection database
A slashing prevention method that is implemented by a number of clients is a local signing history database. This functionality is enabled by default in Prysm’s validator client. This database ensures the validator does not sign a message that would be considered a slashable message according to its own history; more simply, the validator sees the database as its sole source of truth when deciding if a message should be signed. This approach ensures the single validator does not perform duplicative actions.
Local slashing protection does not protect against running multiple validator instances with the same validator keys, as the database is local to the validator it is paired with.
This database only tracks the signing events of the validators in that local instance. This also means if the user changes their validator to a different client (for example, moving from Lighthouse to Nimbus), or moves to a new hardware setup (installing on a fresh linux machine, moving to a hosted solution) it is critical to also migrate the signing history database. This will ensure the history of past actions is preserved on any new client moving forward.
Remote slashing protection
An alternative implementation for slashing prevention is the use of a slasher. The slasher records all attestations and blocks it receives, and the validator references it before agreeing to sign a message. This method is a stronger form of slashing protection than local signing history databases, but as with the databases, this method cannot protect from running multiple instances of the same validator.
An alternative implementation for slashing prevention is the use of slasher itself as a middleware between your beacon node and validator client. Before a validator client submits a block or an attestation, it asks the slasher if it is slashable first. If the check passes, the data will go through to the beacon node. This is the most advanced form of slashing protection as slasher is, ideally, aware of everything happening in the network and has recorded history of blocks and attestations for every validator. There is a caveat, however, that remote slashing protection is not yet optimized in Prysm as our slasher is still a resource hog. Given the high resource consumption of our slasher, it might cause your validator to miss attestations or blocks. Remote slashing protection is meant to run as an added layer on top of local slashing protection. For security reasons, local slashing protection cannot be disabled in Prysm.
Migrating Your Slashing Protection History
An important activity stakers may need to perform at a certain point in their validating journey is to migrate their validator keys to a different machine or to a different eth2 client. Sometimes, you might want to switch staking machines or perhaps you would like to migrate to a different eth2 implementation which suits your needs. Regardless, you should always take your slashing protection history with you.
The slashing protection standard: EIP-3076
There is an official standard to migrate slashing protection histories between eth2 clients known as EIP-3076. This standard recommends implementations to represent their validators’ slashing protection history in a JSON file. At a high level, this file contains:
- Information about the genesis state of the chain the validating history is for (to differentiate between testnets and mainnet)
- Information about all signed blocks and signed attestations for validator public keys the user is running
By exporting this file and importing it when you migrate to another computer or eth2 client, you get a ton of benefits and are able to remain protected against simple slashing conditions that may occur if you did not have this local history stored.
Importing existing validating keys into Prysm
At the time of writing, in Prysm’s v1.0.1 release, Prysm does not allow you to import the EIP-3076 slashing protection JSON file into your validator client. This is our biggest priority for our team and we are aiming to have this as soon as possible, possibly in our next release after genesis. The foundations for import exist, but we would rather work on comprehensive tests to ensure stakers are kept safe.
Exporting your slashing protection history from Prysm
At the time of writing, in Prysm’s v1.0.1 release, Prysm does not allow you to export your slashing protection history into a EIP-3076 slashing protection JSON file. This is our biggest priority for our team and we are aiming to have this as soon as possible, possibly in our next release after genesis. The foundations for import exist, but we would rather work on comprehensive tests to ensure stakers are kept safe.
Important files to assist your migration from one machine running Prysm to another
We recommend waiting until Prysm supports exporting its slashing protection history if you really wish to migrate machines. In the meantime, if you absolutely need to migrate, here are some important things to keep in mind:
- Turn off your beacon node and validator on machine 1, make sure it is not running as a system process. You can check this using the process monitor tools of your OS, or a command line tool such as top or htop and check for anything containing the name “prysm” “validator” or “beacon”
- Note the location of your wallet directory. If you used the default when you started Prysm, you can view its path at the top of the output of `validator accounts list`, which varies based on your operating system
- Take that entire directory and move it over to your next machine
- If you modified your validators’ — datadir, also migrate that directory to your next machine
- Wait at least a few epochs, sync your beacon node on your second machine, then start your validator client on your second machine
- Ensure you never run the same keys again on machine 1 or anywhere else
Acknowledgement
We want to personally thank Kyle Weiss for writing up good portions of this guide and aiding us on helping stakers in their journey. Thank you, Kyle! If this guide was helpful to you, send Kyle some kudos or a tip on Gitcoin here.
