Validator Migration
How-To guide to migrate Ethereum validator account keys from a source server host to a destination host
This validator migration guide is a rudimentary example of migrating validator accounts (also known as keys), from one server hosting (or running) an ETH2 validator client, to a new destination server. These practices are commonly useful as an exercise to migrate out from degrading servers, rebalancing a server hosting too many validator accounts, for disaster recovery (DR) exercises during DDoS events, or when a server is deemed compromised.
In this example, the validator ETH2 client example uses Prysm. Validator accounts are private keys stored within a server which is hosting the ETH2 client. You may find more details at the official reference document of Prysm at https://docs.prylabs.network/docs/advanced/migrating-keys
This operator guide is a public service to the Ethereum community. If you have any questions, please feel free to reach out to us at devops@kysenpool.io
Step 1: Export Slashing Protection history
⚠️ This is highly recommended ⚠️ -- to export the JSON blob containing all the historical attestations (and block proposals) for all the validator accounts to be migrated from the ETH client node that have been performed in the history of the ETH2 Proof-of-Stake's chain history.
You can optionally shutdown and remove the validator in question. Best to review the status of Beaconchain validator or monitor the beaconcha.in dashboard with the validator address(es).
Step 2: Backup Validator accounts
You'll want to backup your validator accounts to be then migrated to the new server.
Follow the prompts by entering a wallet password (if any) to unlock your validator account database and select the accounts to backup. Towards the end, you'll be prompted for a password to encrypt the backup file. We recommend you to add a password for the backup file. You'll now receive keystore-file JSON blobs containing the encrypted validator accounts.
Step 3: Shutdown the Validator node
You can go ahead and shutdown the service (and the node). Assuming you have a systemd setup to run Prysm client called prysmvalidator.service like this
If this is a full migration, you should stop the validator from running
You should then check for server processes to see whether Prysm is still operating. Best yet, delete the validator db directory, and shutdown the entire node itself.
DO NOT issue the exit validator command as you place your validators into the exit queue instead, and you'll not be able to earn any attestation or block production rewards even when you've imported it in the destination node.
Step 4: Delete Validator(s) from your ETH2 client
Run the delete command below, as with Prysm's use case below. Select the validator keys you'd like to remove, if you have more than one list.
This deletion is to remove the validator from your server's validator database only. This does not exit the validators of interest from the Beacon chain.
After deletion is completed, verify account deletion again:
The validators that are being migrated should be gone from the account list.
Common rule of thumb to be safe, is to start import process on the destination validator node after 2 missed attestations.
Step 5: Import Slashing protection data
Over at the destination server to migrate into, assuming you have set up the server with the ETH1 client and ETH2 Prysm client to your satisfaction including the slashing protection file and the validator database files copied over to the new server, you should first and foremost above all else import the slashing protection data.
Verify that the slashing protection is available
Step 6: Import Validator accounts
Now that the slashing protection history is imported to your ETH2 client, you can now import validator accounts of interest to the new ETH2 client node
Monitor the Prysm client to look for Validator activated keyword in the service journal
For additional verification, use the Beaconcha.in service, to see whether your validator account(s) are now attesting properly without incurring any slashing penalties.
👏 Congratulations, you have now successfully completed the migration of validator accounts to your new node.
Notes from KysenPool Engineering
We have tested and ran more than a dozen permutations of ETH1 and ETH2 client combinations. Managing validator accounts via ETH2 clients are a common practice but we prefer using external signers such as Web3Signer, an open-source remote signing service by Consensys (the Prysm team also mentions this here).
We take security seriously. Storing validator account keys on ETH2 client nodes are not desirable but may be slightly more performant. We prefer to script our key management in order to reach an immutable infrastructure framework, to easily tear down and stand up nodes, and migrating validator accounts safely and securely across data centers.
We have also looked at a slashing protection implementation using TEEs (Trusted Execution Environments). We disagree with associating validator account keys with TEEs to sign for attestations and block proposals. If the dependent hardware degrades then there is no way but to exit all validators created under a TEE. If there is a way to migrate out the same TEE created accounts into a new environment, this defeats the whole purpose of slashing protection. This would mean a loss in customer earning potentials reducing their staking APY by 10% or more due to the entrance and exit queue wait times. Hence we have our reservations about using TEEs as a slashing protection mechanism.
KysenPool operates a Hardware Security Module (HSM) to validate for the Cosmos Hub. There are significant differences between HSM and TEEs, and the HSM model may in the future fit for Ethereum as long as the encryption models are using the same BLS signature scheme in its elliptical curve cryptography.
This operator guide is a public service to the Ethereum community. If you have any questions, please feel free to reach out to us at devops@kysenpool.io
Last updated