Etcd setup¶

In our solutions, we use etcd distributed configuration store. Refresh your knowledge about etcd.

Install etcd¶

Install etcd on all PostgreSQL nodes: node1, node2 and node3.

On Debian / Ubuntu On RHEL and derivatives

Install etcd:

$ sudo apt install etcd etcd-server etcd-client

Stop and disable etcd:

$ sudo systemctl stop etcd
$ sudo systemctl disable etcd

Install etcd.

$ sudo yum install
etcd python3-python-etcd\

Stop and disable etcd:

$ sudo systemctl stop etcd
$ systemctl disable etcd

Note

If you installed etcd from tarballs, you must first enable it before configuring it.

Configure etcd¶

To get started with etcd cluster, you need to bootstrap it. This means setting up the initial configuration and starting the etcd nodes so they can form a cluster. There are the following bootstrapping mechanisms:

Static in the case when the IP addresses of the cluster nodes are known
Discovery service - for cases when the IP addresses of the cluster are not known ahead of time.

Since we know the IP addresses of the nodes, we will use the static method. For using the discovery service, please refer to the etcd documentation .

We will configure and start all etcd nodes in parallel. This can be done either by modifying each node’s configuration or using the command line options. Use the method that you prefer more.

Method 1. Modify the configuration file¶

Create the etcd configuration file on every node. You can edit the sample configuration file /etc/etcd/etcd.conf.yaml or create your own one. Replace the node names and IP addresses with the actual names and IP addresses of your nodes.

node1node2node3

/etc/etcd/etcd.conf.yaml

name: 'node1'
initial-cluster-token: PostgreSQL_HA_Cluster_1
initial-cluster-state: new
initial-cluster: node1=http://10.104.0.1:2380,node2=http://10.104.0.2:2380,node3=http://10.104.0.3:2380
data-dir: /var/lib/etcd
initial-advertise-peer-urls: http://10.104.0.1:2380 
listen-peer-urls: http://10.104.0.1:2380
advertise-client-urls: http://10.104.0.1:2379
listen-client-urls: http://10.104.0.1:2379

/etc/etcd/etcd.conf.yaml

name: 'node2'
initial-cluster-token: PostgreSQL_HA_Cluster_1
initial-cluster-state: new
initial-cluster: node1=http://10.104.0.1:2380,node2=http://10.104.0.2:2380,     node3=http://10.104.0.3:2380
data-dir: /var/lib/etcd
initial-advertise-peer-urls: http://10.104.0.2:2380 
listen-peer-urls: http://10.104.0.2:2380
advertise-client-urls: http://10.104.0.2:2379
listen-client-urls: http://10.104.0.2:2379

/etc/etcd/etcd.conf.yaml

name: 'node3'
initial-cluster-token: PostgreSQL_HA_Cluster_1
initial-cluster-state: new
initial-cluster: node1=http://10.104.0.1:2380,node2=http://10.104.0.2:2380,     node3=http://10.104.0.3:2380
data-dir: /var/lib/etcd
initial-advertise-peer-urls: http://10.104.0.3:2380 
listen-peer-urls: http://10.104.0.3:2380
advertise-client-urls: http://10.104.0.3:2379
listen-client-urls: http://10.104.0.3:2379

Enable and start the etcd service on all nodes:
```
$ sudo systemctl enable --now etcd
$ sudo systemctl status etcd
```
During the node start, etcd searches for other cluster nodes defined in the configuration. If the other nodes are not yet running, the start may fail by a quorum timeout. This is expected behavior. Try starting all nodes again at the same time for the etcd cluster to be created.
Check the etcd cluster members. Use etcdctl for this purpose. Ensure that etcdctl interacts with etcd using API version 3 and knows which nodes, or endpoints, to communicate with. For this, we will define the required information as environment variables. Run the following commands on one of the nodes:
```
export ETCDCTL_API=3
HOST_1=10.104.0.1
HOST_2=10.104.0.2
HOST_3=10.104.0.3
ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
```

Now, list the cluster members and output the result as a table as follows:

$ sudo etcdctl --endpoints=$ENDPOINTS -w table member list

Sample output

+------------------+---------+-------+------------------------+----------------------------+------------+
|        ID        | STATUS  | NAME  |         PEER ADDRS     |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------+------------------------+----------------------------+------------+
| 4788684035f976d3 | started | node2 | http://10.104.0.2:2380 | http://192.168.56.102:2379 |      false |
| 67684e355c833ffa | started | node3 | http://10.104.0.3:2380 | http://192.168.56.103:2379 |      false |
| 9d2e318af9306c67 | started | node1 | http://10.104.0.1:2380 | http://192.168.56.101:2379 |      false |
+------------------+---------+-------+------------------------+----------------------------+------------+

To check what node is currently the leader, use the following command

$ sudo etcdctl --endpoints=$ENDPOINTS -w table endpoint status

Sample output

+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT   |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.104.0.1:2379 | 9d2e318af9306c67 |  3.5.16 |   20 kB |      true |      false |         2 |         10 |                 10 |        |
| 10.104.0.2:2379 | 4788684035f976d3 |  3.5.16 |   20 kB |     false |      false |         2 |         10 |                 10 |        |
| 10.104.0.3:2379 | 67684e355c833ffa |  3.5.16 |   20 kB |     false |      false |         2 |         10 |                 10 |        |
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Method 2. Start etcd nodes with command line options¶

On each etcd node, set the environment variables for the cluster members, the cluster token and state:

TOKEN=PostgreSQL_HA_Cluster_1
CLUSTER_STATE=new
NAME_1=node1
NAME_2=node2
NAME_3=node3
HOST_1=10.104.0.1
HOST_2=10.104.0.2
HOST_3=10.104.0.3
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Start each etcd node in parallel using the following command:

node1node2node3

THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
    --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} &

THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
    --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} &

THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
    --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} &

Check the etcd cluster members. Use etcdctl for this purpose. Ensure that etcdctl interacts with etcd using API version 3 and knows which nodes, or endpoints, to communicate with. For this, we will define the required information as environment variables. Run the following commands on one of the nodes:
```
export ETCDCTL_API=3
HOST_1=10.104.0.1
HOST_2=10.104.0.2
HOST_3=10.104.0.3
ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
```

Now, list the cluster members and output the result as a table as follows:

$ sudo etcdctl --endpoints=$ENDPOINTS -w table member list

Sample output

+------------------+---------+-------+------------------------+----------------------------+------------+
|        ID        | STATUS  | NAME  |         PEER ADDRS     |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------+------------------------+----------------------------+------------+
| 4788684035f976d3 | started | node2 | http://10.104.0.2:2380 | http://192.168.56.102:2379 |      false |
| 67684e355c833ffa | started | node3 | http://10.104.0.3:2380 | http://192.168.56.103:2379 |      false |
| 9d2e318af9306c67 | started | node1 | http://10.104.0.1:2380 | http://192.168.56.101:2379 |      false |
+------------------+---------+-------+------------------------+----------------------------+------------+

To check what node is currently the leader, use the following command

$ sudo etcdctl --endpoints=$ENDPOINTS -w table endpoint status

Sample output

+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT   |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.104.0.1:2379 | 9d2e318af9306c67 |  3.5.16 |   20 kB |      true |      false |         2 |         10 |                 10 |        |
| 10.104.0.2:2379 | 4788684035f976d3 |  3.5.16 |   20 kB |     false |      false |         2 |         10 |                 10 |        |
| 10.104.0.3:2379 | 67684e355c833ffa |  3.5.16 |   20 kB |     false |      false |         2 |         10 |                 10 |        |
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Get expert help¶

If you need assistance, visit the community forum for comprehensive and free database knowledge, or contact our Percona Database Experts for professional support and services.

Community Forum Get a Percona Expert Join Percona Squad