Etcd

Production - Etcd on External Virtual Machines

For production installations, StorageOS strongly recommends running etcd outside of Kubernetes on a minimum of 3 dedicated virtual machines. This topology offers strong guarantees of resilience and uptime. We recommend this architecture in all environments, including those where Kubernetes is being deployed as a managed service.

StorageOS doesn’t require a high performance etcd cluster, as the throughput of metadata to the cluster is low. However, we recommend a careful assessment of IOPS capacity best practices to ensure that etcd operates normally.

Depending on the level of redundancy you feel comfortable with you can install etcd on the Kubernetes Master nodes. Take extreme care to avoid collisions of the StorageOS etcd installation with the Kubernetes etcd when using the Kubernetes Master nodes. Precautions such as changing the default configuration for the client and peer ports, and ensuring the etcd data directory is modified. The ansible playbook below will default the etcd installation directory to /var/lib/storageos-etcd.

You can choose between two installation options.

• Manual Installation
• Ansible Installation

Installation - Manual

This section documents the steps required for manual installation of etcd using standard package management commands and systemd manifests.

Repeat the following steps on all the nodes that will run etcd as a systemd service.

1. Configure Etcd version and ports

   export ETCD_VERSION="3.4.9"
   export CLIENT_PORT="2379"
   export PEERS_PORT="2380"
   

   If targeting Kubernetes Master nodes, you must change CLIENT_PORT, PEERS_PORT

2. Download Etcd from CoreOS official site

   curl -L https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz
   mkdir -p /tmp/etcd-v${ETCD_VERSION}-linux-amd64
   tar -xzvf /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz -C /tmp/etcd-v${ETCD_VERSION}-linux-amd64 --strip-components=1
   rm /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz
   

3. Install Etcd binaries

   cd /tmp/etcd-v${ETCD_VERSION}-linux-amd64
   mv etcd /usr/local/sbin/etcd3
   mv etcdctl /usr/local/sbin/etcdctl
   chmod 0755 /usr/local/sbin/etcd3 /usr/local/sbin/etcdctl
   

4. Set up persistent Etcd data directory

   mkdir /var/lib/storageos-etcd
   

5. Create the systemd environment file

   On all nodes that will run etcd create a systemd environemnt file /etc/etcd.conf which has the IPs of all the nodes. The NODE_IP will need to change to correspond to the node IP where the environment file resides. NODE1_IP, NODE2_IP and NODE3_IP will remain the same across all three files.

   $ cat <<END > /etc/etcd.conf
   # NODE_IP is the IP of the node where this file resides.
   NODE_IP=10.64.10.228
   # Node 1 IP
   NODE1_IP=10.64.10.228
   # Node 2 IP
   NODE2_IP=10.64.14.233
   # Node 3 IP  
   NODE3_IP=10.64.12.111
   CLIENT_PORT=${CLIENT_PORT}
   PEERS_PORT=${PEERS_PORT}
   END
   
   # Verify that variables are expanded in the file
   $ cat /etc/etcd.conf
   
   

6. Create the systemd unit file for etcd3 service

   Create a systemd unit file /etc/systemd/system/etcd3.service with the following information:

   [Unit]
   Description=etcd3
   Documentation=https://github.com/coreos/etcd
   Conflicts=etcd2.service
   
   [Service]
   Type=notify
   Restart=always
   RestartSec=5s
   LimitNOFILE=40000
   TimeoutStartSec=0
   EnvironmentFile=/etc/etcd.conf
   
   ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \
       --heartbeat-interval 500 \
       --election-timeout 5000 \
       --max-snapshots 10 \
       --max-wals 10 \
       --data-dir /var/lib/storageos-etcd \
       --quota-backend-bytes 8589934592 \
       --snapshot-count 100000 \
       --auto-compaction-retention 20000 \
       --auto-compaction-mode revision \
       --initial-cluster-state new \
       --initial-cluster-token etcd-token \
       --listen-client-urls http://${NODE_IP}:${CLIENT_PORT},http://127.0.0.1:${CLIENT_PORT} \
       --advertise-client-urls http://${NODE_IP}:${CLIENT_PORT} \
       --listen-peer-urls http://${NODE_IP}:${PEERS_PORT} \
       --initial-advertise-peer-urls http://${NODE_IP}:${PEERS_PORT} \
       --initial-cluster etcd-${NODE1_IP}=http://${NODE1_IP}:${PEERS_PORT},etcd-${NODE2_IP}=http://${NODE2_IP}:${PEERS_PORT},etcd-${NODE3_IP}=http://${NODE3_IP}:${PEERS_PORT}
   
   
   [Install]
   WantedBy=multi-user.target
   

   $NODE_IP is the IP address of the machine you are installing etcd on.`

   Note that setting the advertise-client-urls incorrectly will cause any client connection to fail. StorageOS will fail to communicate to Etcd.

   If enabling TLS, it is recomended to generate your own CA certificate and key. You will need to distribute the keys and certificates for the client auth on all etcd nodes. Moreover, the ExecStart value should look as below:

       ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \
       --heartbeat-interval 500 \
       --election-timeout 5000 \
       --max-snapshots 10 \
       --max-wals 10 \
       --data-dir /var/lib/storageos-etcd \
       --quota-backend-bytes 8589934592 \
       --snapshot-count 100000 \
       --auto-compaction-retention 20000 \
       --auto-compaction-mode revision \
       --peer-auto-tls \
       --client-cert-auth --trusted-ca-file=/path/to/client-cert.pem \
       --cert-file=/path/to/ca.pem \
       --key-file=/path/to/client-key.pem \
       --initial-cluster-state new \
       --initial-cluster-token etcd-token \
       --listen-client-urls https://${NODE_IP}:${CLIENT_PORT} \
       --advertise-client-urls https://${NODE_IP}:${CLIENT_PORT} \
       --listen-peer-urls https://${NODE_IP}:${PEERS_PORT} \
       --initial-advertise-peer-urls https://${NODE_IP}:${PEERS_PORT} \
       --initial-cluster etcd-${NODE1_IP}=https://${NODE1_IP}:${PEERS_PORT},etcd-${NODE2_IP}=https://${NODE2_IP}:${PEERS_PORT},etcd-${NODE3_IP}=https://${NODE3_IP}:${PEERS_PORT}
   

7. Reload and start the etc3 systemd service

   $ systemctl daemon-reload
   $ systemctl enable etcd3.service
   $ systemctl start  etcd3.service
   

8. Installation Verification

   The etcdctl binary is installed at /usr/local/bin on the nodes.

   $ ssh $NODE # Any node running the new etcd
   $ ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:${CLIENT_PORT} member list # $NODE_IP - the IP of the node
   66946cff1224bb5, started, etcd-b94bqkb9rf,  http://172.28.0.1:2380, http://172.28.0.1:2379
   17e7256953f9319b, started, etcd-gjr25s4sdr, http://172.28.0.2:2380, http://172.28.0.2:2379
   8b698843a4658823, started, etcd-rqdf9thx5p, http://172.28.0.3:2380, http://172.28.0.3:2379
   

   Read the etcd operations page for our etcd recommendations.

Installation - Ansible

For a repeatable and automated installation, use of a configuration management tool such as ansible is recommended. StorageOS provides an ansible playbook to help you deploy etcd on standalone virtual machines.

1. Clone StorageOS deployment repository

   git clone https://github.com/storageos/deploy.git
   cd k8s/deploy-storageos/etcd-helpers/etcd-ansible-systemd
   

2. Edit the inventory file

   The inventory file targets the nodes that will run etcd. The file hosts is an example of such an inventory file.

   $ cat hosts
   [nodes]
   centos-1 ip="10.64.10.228" fqdn="ip-10-64-10-228.eu-west-2.compute.internal"
   centos-2 ip="10.64.14.233" fqdn="ip-10-64-14-233.eu-west-2.compute.internal"
   centos-3 ip="10.64.12.111" fqdn="ip-10-64-12-111.eu-west-2.compute.internal"
   
   # Edit the inventory file
   $ vi hosts # Or your own inventory file
   

   The ip or fqdn are used to expose the advertise-client-urls of Etcd. Failing to provide valid ip/fqdn will cause any client connection to fail. StorageOS will fail to communicate to Etcd.

3. Edit the etcd configuration

   If targeting Kubernetes Master nodes, you must change etcd_port_client, etcd_port_peers

   $ cat group_vars/all
   etcd_version: "3.4.9"
   etcd_port_client: "2379"
   etcd_port_peers: "2380"
   etcd_quota_bytes: 8589934592  # 8 GB
   etcd_auto_compaction_mode: "revision"
   etcd_auto_compaction_retention: "1000"
   members: "{{ groups['nodes'] }}"
   installation_dir: "/var/lib/storageos-etcd"
   advertise_format: 'fqdn' # fqdn || ip
   backup_file: "/tmp/backup.db"
   
   tls:
     enabled: false
     ca_common_name: "eu-west-2.compute.internal"
     etcd_common_name: "*.eu-west-2.compute.internal"
     cert_dir: "/etc/etcdtls"
     ca_cert_file: "etcd-ca.pem"
     etcd_server_cert_file: "server.pem"
     etcd_server_key_file: "server-key.pem"
     etcd_client_cert_file: "etcd-client.crt"
     etcd_client_key_file: "etcd-client.key"
   
   $ vi group_vars/all
   

   Choose between using IP addressing or FQDN in the advertise_format parameter. It allows you to decide how Etcd advertises its address to clients. This is particularly relevant when using TLS.

   If enabling TLS, it is recomended to generate your own CA certificate and key. You can do it by generating the CA from the machine running Ansible by: ansible-playbook create_ca.yaml.

4. Install

   ansible-playbook -i hosts install.yaml
   

5. Installation Verification

   The playbook installs the etcdctl binary on the nodes, at /usr/local/bin.

   $ ssh $NODE # Any node running the new etcd
   $ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 member list
   66946cff1224bb5, started, etcd-b94bqkb9rf,  http://172.28.0.1:2380, http://172.28.0.1:2379
   17e7256953f9319b, started, etcd-gjr25s4sdr, http://172.28.0.2:2380, http://172.28.0.2:2379
   8b698843a4658823, started, etcd-rqdf9thx5p, http://172.28.0.3:2380, http://172.28.0.3:2379
   

Benefits of Running External to Kubernetes

Etcd is a distributed key-value store database focused on strong consistency. That means that etcd nodes perform operations across the cluster to ensure quorum. In the case that quorum is lost, an etcd node stops and marks its contents as read-only. Another peer might have a newer version that has not been committed to the database. Quorum is fundamental for etcd operations.

In a Kubernetes environment, applications are scheduled across and in some scenarios such as “DiskPressure” they may need to be evicted from a node, and be scheduled onto a different node. With an application such as etcd, the scenario described can result in quorum being lost, making the cluster unable to recover automatically. Usually a 3 node etcd cluster can survive losing one node and recover. However, losing a second node at the same time or even having a network partition between them will result in quorum lost.

Bind Etcd IPs to Kubernetes Service

Kubernetes external services use a DNS name to reference external endpoints, making them easy to reference from inside the cluster. You can use the example from the helper github repository to deploy the external Service. Using an external service can make monitoring of etcd from Prometheus easier.