概述 什么是Kubernetes? Kubernetes是Google开源的一个容器编排引擎,它支持自动化部署、大规模可伸缩、应用容器化管理。在生产环境中部署一个应用程序时,通常要部署该应用的多个实例以便对应用请求进行负载均衡。
在Kubernetes中,我们可以创建多个容器,每个容器里面运行一个应用实例,然后通过内置的负载均衡策略,实现对这一组应用实例的管理、发现、访问,而这些细节都不需要运维人员去进行复杂的手工配置和处理。
Kubernetes 官方提供的三种部署方式 minikube Minikube是一个工具,可以在本地快速运行一个单点的Kubernetes,仅用于尝试Kubernetes或日常开发的用户使用。部署地址:https://kubernetes.io/docs/setup/minikube/
kubeadm Kubeadm也是一个工具,提供kubeadm init和kubeadm join,用于快速部署Kubernetes集群。
Kubeadm 降低部署门槛,但屏蔽了很多细节,遇到问题很难排查。如果想更容易可控,推荐使 用二进制包部署 Kubernetes 集群,虽然手动部署麻烦点,期间可以学习很多工作原理,也利 于后期维护。
部署地址:https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
二进制包 推荐,从官方下载发行版的二进制包,手动部署每个组件,组成Kubernetes集群。下载地址:https://github.com/kubernetes/kubernetes/releases
这里采用二进制包的方式部署Kubernetes集群
安装要求
在开始之前,部署 Kubernetes 集群机器需要满足以下几个条件:
一台或多台机器,操作系统 CentOS7.x-86_x64
硬件配置:2GB 或更多 RAM,2 个 CPU 或更多 CPU,硬盘 30GB 或更多
集群中所有机器之间网络互通
可以访问外网,需要拉取镜像,如果服务器不能上网,需要提前下载镜像并导入节点
禁止 swap 分区
集群环境规划 软件环境
软件
版本
操作系统
CentOS Linux release 7.8.2003 (Core)
Docker
19-ce
Etcd
3.4.9
Kubernetes
1.20
集群规划
角色
IP
组件
mster
10.150.1.35
kube-apiserver,kube-controller-manager,kube-scheduler, etcd
node1
10.150.1.36
kubelet,kube-proxy,docker,etcd
node2
10.150.1.37
kubelet,kube-proxy,docker,etcd
系统初始化 所有节点均需执行
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 systemctl stop firewalld systemctl disable firewalld sed -i 's/enforcing/disabled/' /etc/selinux/config setenforce 0 swapoff -a sed -ri 's/.*swap.*/#&/' /etc/fstab hostnamectl set-hostname <hostname> cat >> /etc/hosts << EOF 10.150.1.35 k8s-master 10.150.1.36 k8s-node1 10.150.1.37 k8s-node2 EOF cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system yum install ntpdate -y ntpdate time.windows.com
部署etcd集群 Etcd 是一个分布式键值存储系统,Kubernetes 使用 Etcd 进行数据存储,所以先准备一个 Etcd 数据库,为解决Etcd 单点故障,应采用集群方式部署,这里使用 3 台组建集群,可容忍 1 台 机器故障,当然,你也可以使用 5 台组建集群,可容忍 2 台机器故障。
节点名称
IP
etcd-1
10.150.1.35
etcd-2
10.150.1.36
etcd-3
10.150.1.37
注:为了节省机器,这里与 K8s 节点机器复用。也可以独立于 k8s 集群之外部署,只要 apiserver 能连接到就行。
准备 cfssl 证书生成工具 cfssl 是一个开源的证书管理工具,使用 json 文件生成证书,相比 openssl 更方便使用。 找任意一台服务器操作,这里用 Master 节点。
1 2 3 4 5 6 7 wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
生成 etcd 证书 自签证书颁发机构(CA) 创建工作目录:
1 2 mkdir -p ~/TLS/{etcd,k8s} cd TLS/etcd
自签CA:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 cat > ca-config.json << EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "www": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json << EOF { "CN": "etcd CA", "key": {"algo": "rsa", "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" } ] } EOF
生成证书:
1 2 3 cfssl gencert -initca ca-csr.json | cfssljson -bare ca - ls *pem ca-key.pem ca.pem
使用自签CA签发Etcd HTTPS证书 创建证书申请文件:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 cat > server-csr.json << EOF { "CN": "etcd", "hosts": [ "10.150.1.35", "10.150.1.36", "10.150.1.37" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF
注:上述文件 hosts 字段中 IP 为所有 etcd 节点的集群内部通信 IP,一个都不能少!为了方便 后期扩容可以多写几个预留的 IP。
生成证书:
1 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
从 Github 下载二进制文件 下载地址: https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
部署 Etcd 集群 以下在节点 1 上操作,为简化操作,待会将节点 1 生成的所有文件拷贝到节点 2 和节点3
创建工作目录并解压二进制包 1 2 3 mkdir /opt/etcd/{bin,cfg,ssl} -p tar zxvf etcd-v3.4.9-linux-amd64.tar.gz mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
创建etcd配置文件 1 2 3 4 5 6 7 8 9 10 11 12 13 cat > /opt/etcd/cfg/etcd.conf << EOF # [Member] ETCD_NAME="etcd-1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://10.150.1.35:2380" ETCD_LISTEN_CLIENT_URLS="https://10.150.1.35:2379" # [Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.150.1.35:2380" ETCD_ADVERTISE_CLIENT_URLS="https://10.150.1.35:2379" ETCD_INITIAL_CLUSTER="etcd-1=https://10.150.1.35:2380,etcd-2=https://10.150.1.36:2380,etcd-3=https://10.150.1.37:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" EOF
ETCD_NAME:节点名称,集群中唯一
ETCD_DATA_DIR:数据目录
ETCD_LISTEN_PEER_URLS:集群通信监听地址
ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址
ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
ETCD_INITIAL_CLUSTER:集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN:集群 Token
ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new 是新集群,existing 表示加入已有 集群
systemd管理etcd 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 cat > /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/opt/etcd/cfg/etcd.conf ExecStart=/opt/etcd/bin/etcd \ --cert-file=/opt/etcd/ssl/server.pem \ --key-file=/opt/etcd/ssl/server-key.pem \ --peer-cert-file=/opt/etcd/ssl/server.pem \ --peer-key-file=/opt/etcd/ssl/server-key.pem \ --trusted-ca-file=/opt/etcd/ssl/ca.pem \ --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \ --logger=zap Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
拷贝刚才生成的证书 把刚才生成的证书拷贝到配置文件中的路径:
1 cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/
启动并设置开机启动 1 2 3 systemctl daemon-reload systemctl start etcd systemctl enable etcd
注:1节点的etcd需要等2节点和3节点启动后才可启动,不然会起不来
将上面节点 1 所有生成的文件拷贝到节点 2 和节点 3
1 2 3 4 scp -r /opt/etcd/ root@k8s-node1:/opt/ scp /usr/lib/systemd/system/etcd.service root@k8s-node1:/usr/lib/systemd/system/ scp -r /opt/etcd/ root@k8s-node2:/opt/ scp /usr/lib/systemd/system/etcd.service root@k8s-node2:/usr/lib/systemd/system/
然后在节点 2 和节点 3 分别修改 etcd.conf 配置文件中的节点名称和当前服务器 IP:
1 2 3 4 5 6 7 8 9 10 11 12 13 vi /opt/etcd/cfg/etcd.conf # [Member] ETCD_NAME="etcd-1" # 修改此处,节点 2 改为 etcd-2,节点 3 改为 etcd3 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://10.150.1.35:2380" # 修改此处为当前服务器 IP ETCD_LISTEN_CLIENT_URLS="https://10.150.1.35:2379" # 修改此处为当前服务器 IP # [Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.150.1.35:2380" # 修改此处为当前服务 器 IP ETCD_ADVERTISE_CLIENT_URLS="https://10.150.1.35:2379" # 修改此处为当前服务器 IP ETCD_INITIAL_CLUSTER="etcd-1=https://10.150.1.35:2380,etcd-2=https://10.150.1.36:2380,etcd-3=https://10.150.1.37:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
最后启动 etcd 并设置开机启动,同上。注:首先需要启动2节点和3节点后才可启动1节点
查看集群状态 1 2 3 4 5 ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://10.150.1.35:2379,https://10.150.1.36:2379,https://10.150.1.37:2379" endpoint health https://10.150.1.35:2379 is healthy: successfully committed proposal: took = 25.510959ms https://10.150.1.37:2379 is healthy: successfully committed proposal: took = 32.2567ms https://10.150.1.36:2379 is healthy: successfully committed proposal: took = 34.589319ms
如果输出上面信息,就说明集群部署成功。如果有问题第一步先看日志:/var/log/message 或 journalctl -u etcd
安装Docker 下载地址: https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz
以下在所有节点操作。这里采用二进制安装,用 yum 安装也一样
解压二进制包 1 2 tar zxvf docker-19.03.9.tgz mv docker/* /usr/bin
systemd管理docker 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 cat > /usr/lib/systemd/system/docker.service << EOF [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.serviceWants=network-online.target [Service] Type=notify ExecStart=/usr/bin/dockerd ExecReload=/bin/kill -s HUP \$MAINPID LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TimeoutStartSec=0 Delegate=yes KillMode=process Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target EOF
创建配置文件 1 2 3 4 5 6 mkdir /etc/docker cat > /etc/docker/daemon.json << EOF { "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"] } EOF
registry-mirrors 阿里云镜像加速器
启动并设置开机启动 1 2 3 systemctl daemon-reload systemctl start docker systemctl enable docker
部署Master Node 部署 kube-apiserver 生成kube-apiserver证书(CA) 1.自签证书颁发机构(CA)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 cat > ca-config.json << EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 cat > ca-csr.json << EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing", "O": "k8s", "OU": "System" } ] } EOF
生成证书:
1 2 3 cfssl gencert -initca ca-csr.json | cfssljson -bare ca - ls *pem ca-key.pem ca.pem
使用自签CA签发kube-apiserver HTTPS证书 创建证书申请文件:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 cat > server-csr.json << EOF { "CN": "kubernetes", "hosts": [ "10.0.0.1", "127.0.0.1", "10.150.1.35", "10.150.1.36", "10.150.1.37", "10.150.1.38", "10.150.1.39", "10.150.1.40", "10.150.1.41", "10.150.1.42", "10.150.1.43", "10.150.1.44", "10.150.1.45", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF
生成证书:
1 2 3 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server ls server*pem server-key.pem server.pem
从 Github 下载二进制文件 https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md
注:打开链接你会发现里面有很多包,下载一个 server 包就够了,包含了 Master 和 Worker Node 二进制文件。
解压二进制包 1 2 3 4 5 6 wget https://dl.k8s.io/v1.20.6/kubernetes-server-linux-amd64.tar.gz mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} tar zxvf kubernetes-server-linux-amd64.tar.gz cd kubernetes/server/bin cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin cp kubectl /usr/bin/
部署kube-apiserver 创建配置文件 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF KUBE_APISERVER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --etcd-servers=https://10.150.1.35:2379,https://10.150.1.36:2379,https://10.150.1.37:2379 \\ --bind-address=10.150.1.35 \\ --secure-port=6443 \\ --advertise-address=10.150.1.35 \\ --allow-privileged=true \\ --service-cluster-ip-range=10.0.0.0/24 \\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\ --authorization-mode=RBAC,Node \\ --enable-bootstrap-token-auth=true \\ --token-auth-file=/opt/kubernetes/cfg/token.csv \\ --service-node-port-range=30000-32767 \\ --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\ --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\ --tls-cert-file=/opt/kubernetes/ssl/server.pem \\ --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\ --client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --service-account-issuer=api \\ --service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem \\ --etcd-cafile=/opt/etcd/ssl/ca.pem \\ --etcd-certfile=/opt/etcd/ssl/server.pem \\ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \\ --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\ --proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\ --requestheader-allowed-names=kubernetes \\ --requestheader-extra-headers-prefix=X-Remote-Extra- \\ --requestheader-group-headers=X-Remote-Group \\ --requestheader-username-headers=X-Remote-User \\ --enable-aggregator-routing=true \\ --audit-log-maxage=30 \\ --audit-log-maxbackup=3 \\ --audit-log-maxsize=100 \\ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log" EOF
注:上面两个\ \ 第一个是转义符,第二个是换行符,使用转义符是为了使用 EOF 保留换行符
–logtostderr:启用日志
—v:日志等级
–log-dir:日志目录
–etcd-servers:etcd集群地址
–bind-address:监听地址
–secure-port:https安全端口
–advertise-address:集群通告地址
–allow-privileged:启用授权
–service-cluster-ip-range:Service虚拟IP地址段
–enable-admission-plugins:准入控制模块
–authorization-mode:认证授权,启用RBAC授权和节点自管理
–enable-bootstrap-token-auth:启用TLS bootstrap机制
–token-auth-file:bootstrap token文件
–service-node-port-range:Service nodeport类型默认分配端口范围
–kubelet-client-xxx:apiserver访问kubelet客户端证书
–tls-xxx-file:apiserver https证书
1.20版本必须加的参数:–service-account-issuer,–service-account-signing-key-file
–etcd-xxxfile:连接Etcd集群证书
–audit-log-xxx:审计日志
启动聚合层相关配置:–requestheader-client-ca-file,–proxy-client-cert-file,–proxy-client-key-file,–requestheader-allowed-names,–requestheader-extra-headers-prefix,–requestheader-group-headers,–requestheader-username-headers,–enable-aggregator-routing
拷贝刚才生成的证书 把刚才生成的证书拷贝到配置文件中的路径:
1 cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/
启用 TLS Bootstrapping 机制 TLS Bootstraping:Master apiserver 启用 TLS 认证后,Node 节点 kubelet 和 kube-proxy 要 与 kube-apiserver 进行通信,必须使用 CA 签发的有效证书才可以,当 Node 节点很多时,这 种客户端证书颁发需要大量工作,同样也会增加集群扩展复杂度。为了简化流程,Kubernetes 引入了 TLS bootstraping 机制来自动颁发客户端证书,kubelet 会以一个低权限用户自动向 apiserver 申请证书,kubelet 的证书由 apiserver 动态签署。所以强烈建议在 Node 上使用这 种方式,目前主要用于 kubelet,kube-proxy 还是由我们统一颁发一个证书。
TLS bootstraping 工作流程:
创建上述配置文件中 token 文件:
1 2 3 cat > /opt/kubernetes/cfg/token.csv << EOF 741b5f19602dedbbcd4b1fb5c5248225,kubelet-bootstrap,10001,system:node-bootstrapper EOF
格式:token,用户名,UID,用户组
注意: 741b5f19602dedbbcd4b1fb5c5248225,kubelet-bootstrap,10001,“system:node-bootstrapper” 引号会导致error: invalid authentication config: parse error on line 1, column 82: extraneous or missing “ in quoted-field
token 也可自行生成替换:
1 head -c 16 /dev/urandom | od -An -t x | tr -d ' '
systemd管理apiserver 1 2 3 4 5 6 7 8 9 10 11 cat > /usr/lib/systemd/system/kube-apiserver.service << EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF
注:“\$KUBE_APISERVER_OPTS” \号为cat命令$符号转义所需,实际生成的文件里面是没有\符号的。
启动并设置开机启动 1 2 3 systemctl daemon-reload systemctl start kube-apiserver systemctl enable kube-apiserver
部署 kube-controller-manager 创建配置文件 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --leader-elect=true \\ --kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\ --bind-address=127.0.0.1 \\ --allocate-node-cidrs=true \\ --cluster-cidr=10.244.0.0/16 \\ --service-cluster-ip-range=10.0.0.0/24 \\ --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --root-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --cluster-signing-duration=87600h0m0s" EOF
–kubeconfig:连接apiserver配置文件
–leader-elect:当该组件启动多个时,自动选举(HA)
–cluster-signing-cert-file/–cluster-signing-key-file:自动为kubelet颁发证书的CA,与apiserver保持一致
生成kubeconfig文件
生成kube-controller-manager证书:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 # 切换工作目录 cd ~/TLS/k8s # 创建证书请求文件 cat > kube-controller-manager-csr.json << EOF { "CN": "system:kube-controller-manager", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:masters", "OU": "System" } ] } EOF # 生成证书 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
生成kubeconfig文件(以下是shell命令,直接在终端执行):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig" KUBE_APISERVER="https://10.150.1.35:6443" kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG} kubectl config set-credentials kube-controller-manager \ --client-certificate=./kube-controller-manager.pem \ --client-key=./kube-controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG} kubectl config set-context default \ --cluster=kubernetes \ --user=kube-controller-manager \ --kubeconfig=${KUBE_CONFIG} kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
systemd管理controller-manager 1 2 3 4 5 6 7 8 9 10 11 cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF
启动并设置开机启动 1 2 3 systemctl daemon-reload systemctl start kube-controller-manager systemctl enable kube-controller-manager
部署kube-scheduler 创建配置文件
1 2 3 4 5 6 7 8 cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF KUBE_SCHEDULER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --leader-elect \\ --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\ --bind-address=127.0.0.1" EOF
–master:通过本地非安全本地端口 8080 连接 apiserver。
–leader-elect:当该组件启动多个时,自动选举(HA)
生成kubeconfig文件 生成kube-scheduler证书:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 # 切换工作目录 cd ~/TLS/k8s # 创建证书请求文件 cat > kube-scheduler-csr.json << EOF { "CN": "system:kube-scheduler", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:masters", "OU": "System" } ] } EOF # 生成证书 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
生成kubeconfig文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig" KUBE_APISERVER="https://10.150.1.35:6443" kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG} kubectl config set-credentials kube-scheduler \ --client-certificate=./kube-scheduler.pem \ --client-key=./kube-scheduler-key.pem \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG} kubectl config set-context default \ --cluster=kubernetes \ --user=kube-scheduler \ --kubeconfig=${KUBE_CONFIG} kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
systemd 管理 scheduler 1 2 3 4 5 6 7 8 9 10 11 cat > /usr/lib/systemd/system/kube-scheduler.service << EOF [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF
启动并设置开机启动 1 2 3 systemctl daemon-reload systemctl start kube-scheduler systemctl enable kube-scheduler
查看集群状态 创建集群证书
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 cat > admin-csr.json <<EOF { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:masters", "OU": "System" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
生成kubeconfig文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 mkdir /root/.kube KUBE_CONFIG="/root/.kube/config" KUBE_APISERVER="https://10.150.1.35:6443" kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG} kubectl config set-credentials cluster-admin \ --client-certificate=./admin.pem \ --client-key=./admin-key.pem \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG} kubectl config set-context default \ --cluster=kubernetes \ --user=cluster-admin \ --kubeconfig=${KUBE_CONFIG} kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
1 2 3 4 5 6 7 kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-2 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"}
部署Worker Node 下面还是在Master Node 上操作,即同时作为Worker Node
创建工作目录并拷贝二进制文件
在所有worker node创建工作目录:
1 mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
从master节点拷贝:
1 2 cd kubernetes/server/bin cp kubelet kube-proxy /opt/kubernetes/bin
部署kubelet 创建配置文件 1 2 3 4 5 6 7 8 9 10 11 12 cat > /opt/kubernetes/cfg/kubelet.conf << EOF KUBELET_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --hostname-override=k8s-master \\ --network-plugin=cni \\ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\ --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\ --config=/opt/kubernetes/cfg/kubelet-config.yml \\ --cert-dir=/opt/kubernetes/ssl \\ --pod-infra-container-image=lizhenliang/pause-amd64:3.0" EOF
-–hostname-override:显示名称,集群中唯一
–network-plugin:启用 CNI
–kubeconfig:空路径,会自动生成,后面用于连接 apiserver
–bootstrap-kubeconfig:首次启动向 apiserver 申请证书
–config:配置参数文件
–cert-dir:kubelet 证书生成目录
–pod-infra-container-image:管理 Pod 网络容器的镜像
配置参数文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 0.0 .0 .0 port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - 10.0 .0 .2 clusterDomain: cluster.local failSwapOn: false authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s evictionHard: imagefs.available: 15 % memory.available: 100Mi nodefs.available: 10 % nodefs.inodesFree: 5 % maxOpenFiles: 1000000 maxPods: 110 EOF
生成kubelet初次加入集群引导kubeconfig文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 KUBE_CONFIG="/opt/kubernetes/cfg/bootstrap.kubeconfig" KUBE_APISERVER="https://10.150.1.35:6443" # apiserver IP:PORT TOKEN="741b5f19602dedbbcd4b1fb5c5248225" # 与token.csv里保持一致 # 生成 kubelet bootstrap kubeconfig 配置文件 kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG} kubectl config set-credentials "kubelet-bootstrap" \ --token=${TOKEN} \ --kubeconfig=${KUBE_CONFIG} kubectl config set-context default \ --cluster=kubernetes \ --user="kubelet-bootstrap" \ --kubeconfig=${KUBE_CONFIG} kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
systemd 管理 kubelet 1 2 3 4 5 6 7 8 9 10 11 12 cat > /usr/lib/systemd/system/kubelet.service << EOF [Unit] Description=Kubernetes Kubelet After=docker.service [Service] EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
授权kubelet-bootstrap用户允许请求证书 1 kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
启动并设置开机启动 1 2 3 systemctl daemon-reload systemctl start kubelet systemctl enable kubelet
批准 kubelet 证书申请并加入集群 1 2 3 4 5 6 7 8 9 10 11 12 [root@k8s-master cfg]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-OPLNZD3CWwDvP7PZv0pyJhJl1jEL9M9733IiVSzfWUc 11s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending [root@localhost cfg]# kubectl certificate approve node-csr-OPLNZD3CWwDvP7PZv0pyJhJl1jEL9M9733IiVSzfWUc certificatesigningrequest.certificates.k8s.io/node-csr-OPLNZD3CWwDvP7PZv0pyJhJl1jEL9M9733IiVSzfWUc approved [root@k8s-master cfg]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-OPLNZD3CWwDvP7PZv0pyJhJl1jEL9M9733IiVSzfWUc 14m kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued [root@k8s-master cfg]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master NotReady <none> 75s v1.20.6
尚未部署网络插件,故node状态会显示NotReady,待网络插件部署完成,就会显示Ready
部署kube-proxy 创建配置文件 1 2 3 4 5 6 7 cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF KUBE_PROXY_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --config=/opt/kubernetes/cfg/kube-proxy-config.yml" EOF
配置参数文件 1 2 3 4 5 6 7 8 9 10 11 cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0 .0 .0 metricsBindAddress: 0.0 .0 .0 :10249 clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig hostnameOverride: k8s-master clusterCIDR: 10.0 .0 .0 /24 EOF
生成kube-proxy证书: 切换到工作目录
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 cat > kube-proxy-csr.json << EOF { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
生成kubeconfig文件: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 生成kubeconfig文件: KUBE_CONFIG="/opt/kubernetes/cfg/kube-proxy.kubeconfig" KUBE_APISERVER="https://10.150.1.35:6443" kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG} kubectl config set-credentials kube-proxy \ --client-certificate=./kube-proxy.pem \ --client-key=./kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG} kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=${KUBE_CONFIG} kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
systemd 管理 kube-proxy 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cat > /usr/lib/systemd/system/kube-proxy.service << EOF [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
启动并设置开机启动 1 2 3 systemctl daemon-reload systemctl start kube-proxy systemctl enable kube-proxy
部署网络组件 Calico是一个纯三层的数据中心网络方案,是目前Kubernetes主流的网络方案。
部署Calico:
1 2 3 wget https://docs.projectcalico.org/manifests/calico.yaml kubectl apply -f calico.yaml kubectl get pods -n kube-system
等Calico Pod都Running,节点也会准备就绪:
授权 apiserver 访问 kubelet
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 cat > apiserver-to-kubelet-rbac.yaml << EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-apiserver-to-kubelet rules: - apiGroups: - "" resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics - pods/log verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:kube-apiserver namespace: "" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubelet subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kubernetes EOF kubectl apply -f apiserver-to-kubelet-rbac.yaml
新增加 Worker Node 拷贝已部署好的 Node 相关文件到新节点 在 master 节点将 Worker Node 涉及文件拷贝到新节点 k8s-node1/k8s-node2
1 2 3 4 scp -r /opt/kubernetes root@k8s-node1:/opt/ scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@k8s-node1:/usr/lib/systemd/system scp -r /opt/cni/ root@k8s-node2:/opt/ scp /opt/kubernetes/ssl/ca.pem root@k8s-node2:/opt/kubernetes/ssl
删除 kubelet 证书和 kubeconf 1 2 rm /opt/kubernetes/cfg/kubelet.kubeconfig rm -f /opt/kubernetes/ssl/kubelet*
注:这几个文件是证书申请审批后自动生成的,每个 Node 不同,必须删除重新生成。
修改主机名 1 2 3 4 vi /opt/kubernetes/cfg/kubelet.conf --hostname-override=k8s-node1 vi /opt/kubernetes/cfg/kube-proxy-config.yml hostnameOverride: k8s-node1
启动并设置开机启动 1 2 3 4 5 systemctl daemon-reload systemctl start kubelet systemctl enable kubelet systemctl start kube-proxy systemctl enable kube-proxy
在 Master 上批准新 Node kubelet 证书申请 1 2 3 4 5 6 [root@k8s-master ~]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-0bcLNxV1KJ56XaNq2acgAK77rPhwOsfCsl6lqverGoM 52s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending node-csr-KzSmPqstZsx8uQUfcCWKiipMY6x8az8CH7xcCSQRo9s 52s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending node-csr-OPLNZD3CWwDvP7PZv0pyJhJl1jEL9M9733IiVSzfWUc 29m kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
查看 Node 状态 1 2 3 4 5 [root@k8s-master ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready <none> 17m v1.20.6 k8s-node1 Ready <none> 96s v1.20.6 k8s-node2 Ready <none> 84s v1.20.6
至此,Kubernetes搭建完成。
故障排查 kubectl get csr No resources found 检查 KUBE_APISERVER 和TOKEN和实际的是否一致,如不一致,重新执行后,重启kubectl restart kubelet。然后kubectl get csr
kube-apiserver起不来 检测配置文件网段是否与物理网段重复,检查配置是否生效
etcd节点起不来 需要首先启动2节点和3节点 才可启动1节点
error: invalid authentication config: parse error on line 1, column 82: extraneous or missing “ in quoted-field 创建token.csv文件的时候需去掉“”号
参考来源 1.搭建一套生产级K8s高可用集群
2.https://www.cnblogs.com/digdeep/p/12252073.html
3.http://blog.leanote.com/post/wang2020/k8s%E9%83%A8%E7%BD%B2