1.概念
etcd 是由GO语言编写的分布式的、可靠的键值存储系统,主要用于分布式系统中关键数据的存储和服务发现。
2.核心概念
节点(Node)
每个运行 etcd 的实例被称为一个节点。一个或多个节点可以组成一个集群。
集群(Cluster)
由多个节点组成的集合,这些节点共同工作以提供一致的数据存储服务。通过 Raft 共识算法确保集群中各节点间数据的一致性。
键值对(Key-Value Pair)
etcd 存储的基本单位是键值对,其中键和值都是字节数组。键用于唯一标识存储的数据项,而值则包含实际的数据内容。
3.etcd
集群准备
节点 | IP地址 | 操作系统版本 | etcd版本 |
---|---|---|---|
etcd-node1 | 192.168.100.5 | Ubuntu 24.04.2 LTS | v3.6.4 |
etcd-node2 | 192.168.100.6 | Ubuntu 24.04.2 LTS | v3.6.4 |
etcd-node3 | 192.168.100.7 | Ubuntu 24.04.2 LTS | v3.6.4 |
3.1 配置IP地址
配置etcd-node1节点IP
sudo cat /etc/netplan/ens32-cloud-init.yaml
network:version: 2ethernets:ens32:dhcp4: falseaddresses:- "192.168.100.5/24"routes:- to: defaultvia: 192.168.100.254nameservers:addresses:- 114.114.114.114
sudo netplan apply
配置etcd-node2节点IP
sudo cat /etc/netplan/ens32-cloud-init.yaml
network:version: 2ethernets:ens32:dhcp4: falseaddresses:- "192.168.100.6/24"routes:- to: defaultvia: 192.168.100.254nameservers:addresses:- 114.114.114.114
sudo netplan apply
配置etcd-node3节点IP
sudo cat /etc/netplan/ens32-cloud-init.yaml
network:version: 2ethernets:ens32:dhcp4: falseaddresses:- "192.168.100.7/24"routes:- to: defaultvia: 192.168.100.254nameservers:addresses:- 114.114.114.114
sudo netplan apply
3.2 配置主机名
配置etcd-node1节点主机名
sudo hostnamectl set-hostname etcd-node1
配置etcd-node2节点主机名
sudo hostnamectl set-hostname etcd-node2
配置etcd-node3节点主机名
sudo hostnamectl set-hostname etcd-node3
3.3 配置主机名与IP解析
3个节点均需要执行
sudo cat >> /etc/hosts <<EOF
192.168.100.5 etcd-node1
192.168.100.6 etcd-node2
192.168.100.7 etcd-node3
EOF
3.4 关闭防火墙
3个节点均需要执行
sudo ufw stop
sudo ufw status
3.5 时钟同步
3个节点均需要执行
sudo apt install chrony
sudo sed -i '/pool.*ubuntu\.pool\.ntp\.org/s/^/# /' /etc/chrony/chrony.conf
sudo sed -i 's/^pool ntp\.ubuntu\.com.*$/server ntp.aliyun.com iburst/' /etc/chrony/chrony.conf
sudo systemctl restart chrony
sudo chronyc sources
3.6 配置节点互信
3个节点均需要执行
sudo ssh-keygen
sudo cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
在etcd-node1节点上执行
sudo for i in 5 6 7
> do
> scp -r /root/.ssh 192.168.100.$i:/root/
> done
4.etcd
集群部署
4.1 下载etcd
3个节点均需要执行
sudo wget https://github.com/etcd-io/etcd/releases/download/v3.6.4/etcd-v3.6.4-linux-amd64.tar.gz
4.2解压etcd
3个节点均需要执行
sudo tar xzvf etcd-v3.6.4-linux-amd64.tar.gz -C /usr/local
sudo ln -s /usr/local/etcd-v3.6.4-linux-amd64/ /usr/local/etcd
4.3复制解压文件至系统标准可执行文件路径中
3个节点均需要执行
sudo cp /usr/local/etcd/etcd* /usr/local/bin/
4.4创建etcd
用户
3个节点均需要执行
sudo useradd --system --shell /bin/false --home-dir /var/lib/etcd etcd
4.5创建数据目录
3个节点均需要执行
sudo mkdir -p /var/lib/etcd
sudo mkdir /var/lib/etcd/default.etcd
sudo chown -R etcd:etcd /var/lib/etcd /usr/local/etcd
4.6 创建etcd
配置文件
配置etcd-node1节点配置文件
sudo cat > /usr/local/etcd/etcd.conf <<EOF
ETCD_NAME="etcd-node1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.5:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.5:2379,http://127.0.0.1:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.5:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.5:2379"
ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.100.5:2380,etcd-node2=https://192.168.100.6:2380,etcd-node3=https://192.168.100.7:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
配置etcd-node2节点配置文件
sudo cat > /usr/local/etcd/etcd.conf <<EOF
ETCD_NAME="etcd-node2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.6:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.6:2379,http://127.0.0.1:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.6:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.6:2379"
ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.100.5:2380,etcd-node2=https://192.168.100.6:2380,etcd-node3=https://192.168.100.7:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
配置etcd-node3节点配置文件
sudo cat > /usr/local/etcd/etcd.conf <<EOF
ETCD_NAME="etcd-node3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.7:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.7:2379,http://127.0.0.1:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.7:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.7:2379"
ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.100.5:2380,etcd-node2=https://192.168.100.6:2380,etcd-node3=https://192.168.100.7:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
相关参数说明
参数 | 说明 |
---|---|
ETCD_NAME | 当前etcd节点名称 |
ETCD_DATA_DIR | 数据存储目录 |
ETCD_LISTEN_CLIENT_URLS | 当前节点通过该地址监听客户端发送的信息 |
ETCD_LISTEN_PEER_URLS | 当前节点通过该地址监听集群其他节点发送的信息 |
ETCD_INITIAL_ADVERTISE_PEER_URLS | 集群的其他节点通过该地址与当前节点通信 |
ETCD_ADVERTISE_CLIENT_URLS | 客户端通过该地址与当前节点通信 |
ETCD_INITIAL_CLUSTER | 当前集群的所有节点信息,当前节点根据此信息与其他节点取得联系 |
ETCD_INITIAL_CLUSTER_TOKEN | 用于区分不同的集群,同一集群的所有节点配置相同的值 |
ETCD_INITIAL_CLUSTER_STATE | 本次是否为新建集群,取值为 new 或者 existing |
4.7下载cfssl
证书生成工具
在etcd-node1节点上下载cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssl_1.6.5_linux_amd64
在etcd-node1节点上下载cfssljson
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssljson_1.6.5_linux_amd64
4.8授权并移至系统标准可执行文件路径中
在etcd-node1节点上执行
chmod +x cfssl_1.6.5_linux_amd64chmod +x cfssljson_1.6.5_linux_amd64mv cfssl_1.6.5_linux_amd64 /usr/local/bin/cfssljsonmv cfssljson_1.6.5_linux_amd64 /usr/local/bin/cfssljson
4.9创建CA证书
在etcd-node1节点上配置CA证书策略
mkdir /usr/local/etcd/ssl
cat > /usr/local/etcd/ssl/ca-config.json <<EOF
{"signing": {"default": {"expiry": "87600h"},"profiles": {"etcd-server": {"usages": ["signing","key encipherment","client auth","server auth"],"expiry": "87600h"}}}
}
EOF
在etcd-node1节点上配置CA证书请求文件
cat > /usr/local/etcd/ssl/ca-csr.json <<EOF
{"CN": "My etcd CA","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "LANZHOU","O": "LZ","ST": "LANZHOU","OU": "CN"}],"ca": {"expiry": "87600h"}
}
EOF
在etcd-node1节点上生成CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
4.10创建etcd证书
在etcd-node1节点上配置etcd请求文件
cat > /usr/local/etcd/ssl/etcd-server.json <<EOF
{"CN": "etcd","hosts": ["127.0.0.1","192.168.100.5","192.168.100.6","192.168.100.7"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "LANZHOU","ST": "LANZHOU","OU": "CN"}]
}
EOF
在etcd-node1节点上生成 Etcd 证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-server etcd-server.json | cfssljson -bare etcd-server
说明:
参数 | 说明 |
---|---|
-ca-key | 指定CA证书机构的私钥 |
-config | 指定CA证书策略 |
-profile | 指定使用CA证书策略 |
etcd-server.pem | 证书/公钥 |
etcd-server-key.pem | 私钥 |
4.7创建 systemd 服务
3个节点均需要执行
sudo cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target[Service]
Type=notify
EnvironmentFile=-/usr/local/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \--cert-file=/usr/local/etcd/ssl/etcd-server.pem \--key-file=/usr/local/etcd/ssl/etcd-server-key.pem \--trusted-ca-file=/usr/local/etcd/ssl/ca.pem \--peer-cert-file=/usr/local/etcd/ssl/etcd-server.pem \--peer-key-file=/usr/local/etcd/ssl/etcd-server-key.pem \--peer-trusted-ca-file=/usr/local/etcd/ssl/ca.pem \--peer-client-cert-auth \--client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF
4.8 启动etcd
3个节点均需要执行
sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd
sudo systemctl status etcd
4.9查看集群成员
任意节点执行
sudo etcdctl member list
4.10查看集群节点健康状态
ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/usr/local/etcd/ssl/ca.pem --cert=/usr/local/etcd/ssl/etcd-server.pem --key=/usr/local/etcd/ssl/etcd-server-key.pem --endpoints=https://192.168.100.5:2379,https://192.168.100.6:2379,https://192.168.100.7:2379 endpoint health