🔄Advanced TMKMS Integration
The default consensus engine available within the SDK is Tendermint Core. See Tendermint notes on running in production and notes on setting up a validator
Validator block signing should be via tmkms
Setting up AWS Nitro Enclaves + Tendermint KMS for signing blocks
CAUTION The setup isn't yet ready for production use:
It is not yet audited
The tmkms prototype fork isn't meant to be maintained in the long term
Background
TMKMS, initially targeting Cosmos Validators, provides High-availability, Double-signing prevention and Hardware security module.
Currently, TMKMS provides both hardware signing and softsign. However, it is hard or impossible to plug your own Hardware Security Modules(HSM) into the major cloud providers when one wants to run it on the cloud for hardware signing. On the other hand, it is also insecure to use softsign as your generated signing key is actually in plain text on the machine.
What we want to achieve is just running TMKMS securely and provision validator conveniently on the cloud. To meet this end, we now can leverage AWS Nitro Enclaves to execute TMKMS and TMKMS then decrypts (during initialization) the signing via AWS KMS. Read more details here
Note that this is still a work in progress and this document only describes a basic setup, so it is not yet ready for production use. We recommend looking at other materials for additional setups, such as the Security best practices for AWS KMS whitepaper.
Step 1. Set up supported EC2 instance types
Virtualized Nitro-based instances with at least four vCPUs. t3, t3a, t4g, a1, c6g, c6gd, m6g, m6gd, r6g, and r6gd instances are not supported.
We recommend m5a.xlarge
and Amazon Linux 2 AMI
for easier installation for AWS Nitro Enclaves CLI.
Step 2. Installing the Nitro Enclaves CLI
One needs to install Docker + Nitro Enclaves CLI.
Follow this doc to proceed.
Step 3. Prepare TMKMS Enclave images on EC2
You can either follow this compiling-tmkms-for-aws-ne to build TMKMS Enclave images from scratch or simply use our published image.
After building the enclave image, you should obtain 3 enclave's measurements(PCRs): PCR0 (SHA384 hash of the image), PCR1 (SHA384 hash of the OS kernel and the bootstrap process), and PCR2 (SHA384 hash of the application). Take note of the PCR0 value. One can also use PCR3 and PCR8, for more details, please find this link
And also create and take a note of PCR4 manually which is unique across ec2.
Step 4. Preparing IAM instance role and AWS KMS policy
Step 4.1. Create an IAM role for EC2
Create an IAM role for the created EC2 previously without permissions policies attached. We will allow this role to decrypt with CMK inside nitro enclave in KMS key policy instead.
Attach this role to the previously created EC2. Check this guide.
Step 4.2. Create your CMK
Create your symmetric CMK
Define key administrative permissions and key usage permissions that user can admin, encrypt and decrypt the signing key in your local or a trusted machine via AWS CLI.
Edit key policy to allow only TMKMS inside nitro enclave to decrypt instead of entire EC2 and encrypt on EC2 You should have a generated policy shown in the console.
For the decryption action, you should add the following snippet in "Statement" as:
Change EC2_IAM_ROLE
,PCR0
and PCR4
to what we just created in previous steps.
If you plan to run the tmkms enclave in the debug mode, set the recipient attestation value to: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
(instead of the PCR0 value).
Step 5. Prepare encrypted validator signing key on your local machine
Step 5.1. Install tmkms-nitro-helper on EC2
Install tmkms-nitro-helper from source code.
Step 5.2. Generate a new encrypted validator signing key
bech32-prefix
is crocnclconspub
for mainnet and tcrocnclconspub
for testnet
It should generate a bech32 public key
to stdout and an encrypted private key in relative path secrets/secret.key
. We need bech32 public key for node join and secret.key to decrypt inside enclave.
Step 6. Configure tmkms.toml for enclave TMKMS on EC2
Move above generated secrets/secret.key
to ~/.tmkms
directory
Create tmkms.toml
under ~/.tmkms
directory as:
Example: tmkms.toml for testnet
Step 7. Create TMKMS enclave service
To launch the TMKMS enclave, one needs to execute several commands to make it work.
In order to have a resilient validator, one should run the TMKMS enclave as a service.
Step 7.1. Create a script to run TMKMS enclave
One should adjust the <KMS_REGION>
in the script if set differently in tmkms.toml
eg. us-east-1
Create the script run_tmkms_nitro_helper.sh
with executable permissions under ~/.tmkms
directory
Step 7.2. Create systemd service for TMKMS enclave
One should adjust the path in the systemd file if set different paths for the binary and script.
Create /lib/systemd/system/tmkms.service
and run the service
Step 8. Running chain-maind
One should follow the same steps in Croeseid Testnet: Running Nodes
Except for one last thing one needs to further configure ~/.chain-maind/config/config.toml
to enable enclave tmkm to sign.
In ~/.chain-maind/config/config.toml
, priv_validator_key_file
and priv_validator_state_file
should be commented and uncomment priv_validator_laddr
to value unix://...
which should match the address
in tmkms.toml
. e.g. unix:///home/ec2-user/sockets/validator.socket
Last updated