Kubernetes
大约 74 分钟
Kubernetes
概述
Kubernetes 是一个管理容器和服务的开源平台，用于在多主机上高效部署，升级，回滚，扩缩容及管理容器应用，以实现容器应用和服务的负载均衡及故障转移。
相比于传统单主机部署应用及虚拟化多主机部署应用，使用容器化部署应用既可以实现资源隔离，还可以更高效的使用资源。类似于使用 VMware vSphere 管理多个主机上的虚拟机，使用 Kubernetes 管理多个主机上的容器，以实现一个可弹性运行的分布式系统，以满足容器应用的高可用，高性能和高扩展。
架构
Kubernetes 集群使用客户端-服务器架构。
其中：
- Control Plane：控制平面，管理集群，包括：
- kube-apiserver：控制平面的前端接口，暴露 Kubernetes API。
- kube-controller-manager：控制器管理器，运行控制器进程，包括：
- Node controller
- Job controller
- EndpointSlice controller
- ServiceAccount controller
- cloud-controller-manager：用于云服务的控制器管理器，本地部署则没有该组件。
- kube-scheduler：调度器，根据资源及配置来调度，决定将容器部署到哪个工作节点。
- etcd：存储集群数据。
- Node：工作节点，包括：
- kubelet：运行在每个工作节点上的代理，拉取镜像，运行容器。
- kube-proxy：运行在每个工作节点上的网络代理，维护节点上的网络规则。
除了以上 Kubernetes 自带的组件外，为便于系统的扩展，还开放了以下接口：
- CRI（Container Runtime Interface）：容器运行时接口，提供容器和镜像服务，常用的实现有 Docker，containerd，CRI-O 等。
- CNI（Container Network Interface）：容器网络接口，为容器提供网络服务，常用的实现有 Flannel 和 Calico 等。
- CSI（Container Storage Interface）：容器存储接口，为容器提供存储服务。
部署
本节介绍使用 kubeadm 在 CentOS 上部署 Kubernetes 集群。
环境
主机规划：
| No. | Host | IP | CPU | Memory | Role |
|---|---|---|---|---|---|
| 1 | k8sm1 | 192.168.92.140 | 2 | 4 GB | Control Plane |
| 2 | k8sn1 | 192.168.92.141 | 2 | 4 GB | Worker Node 1 |
| 3 | k8sn2 | 192.168.92.142 | 2 | 4 GB | Worker Node 2 |
注意：
CPU 数量至少为 2，内存建议至少 4 GB。
软件版本：
| No. | Name | Release |
|---|---|---|
| 1 | CentOS | 7.9 |
| 2 | Docker | 20.10.23 |
| 3 | Containerd | 1.6.15 |
| 4 | kubernates | 1.23.17 |
| 5 | Flannel | 0.22.2 |
| 6 | Ingress-Nginx | 1.6.4 |
| 7 | Dashboard | 2.5.1 |
注意：
Docker 和 Containerd 的版本对应关系可在 Docker 查看。
Kubernetes 和 Docker 的版本对应关系可在 Github 查看。
Kubernetes 和 Flannel 的版本对应关系可在 Github 查看。
Kubernetes 和 Ingress-Nginx 的版本对应关系可在 Github 查看。
Kubernetes 和 Dashboard 的版本对应关系可在 Github 查看。
准备
安装 CentOS 操作系统，分别配置 3 台主机的主机名，网络地址：
Control Plane：
[root@localhost ~]# hostnamectl set-hostname k8sm1.stonecoding.net
[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPADDR=192.168.92.140
NETMASK=255.255.255.0
GATEWAY=192.168.92.2
NAME="ens33"
DEVICE="ens33"
ONBOOT="yes"
UUID="9b6cf18b-7e51-4d1f-818e-142bb03ee6eb"
[root@localhost ~]# systemctl restart network
Node 1：
[root@localhost ~]# hostnamectl set-hostname k8sn1.stonecoding.net
[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPADDR=192.168.92.141
NETMASK=255.255.255.0
GATEWAY=192.168.92.2
NAME="ens33"
DEVICE="ens33"
ONBOOT="yes"
UUID="4e6bae92-d721-4188-ab0d-71b1b5d8711e"
[root@localhost ~]# systemctl restart network
Node 2：
[root@localhost ~]# hostnamectl set-hostname k8sn2.stonecoding.net
[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPADDR=192.168.92.142
NETMASK=255.255.255.0
GATEWAY=192.168.92.2
NAME="ens33"
DEVICE="ens33"
ONBOOT="yes"
UUID="3ac41122-f782-4a69-bc93-9df7846db136"
[root@localhost ~]# systemctl restart network
然后创建脚本 k8s-pre.sh：
#!/bin/bash
# 配置 /etc/hosts
echo "=========edit /etc/hosts============="
cat >> /etc/hosts << EOF
192.168.92.140 k8sm1 k8sm1.stonecoding.net
192.168.92.141 k8sn1 k8sn1.stonecoding.net
192.168.92.142 k8sn2 k8sn2.stonecoding.net
EOF
# 配置 DNS
echo "=========edit /etc/resolv.conf ============="
cat >> /etc/resolv.conf << EOF
nameserver 192.168.92.2
EOF
# 安装基础软件
echo "=========install basic sofeware============="
yum -y install wget
# 关闭防火墙
echo "=========stop firewalld============="
systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld
# 关闭 NetworkManager
echo "=========stop NetworkManager ============="
systemctl stop NetworkManager
systemctl disable NetworkManager
systemctl status NetworkManager
# 关闭 SELinux
echo "=========disable selinux============="
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
setenforce 0
getenforce
# 关闭 SWAP
echo "=========close swap============="
sed -ri 's/.*swap.*/#&/' /etc/fstab
swapoff -a
free -m
# 时间同步
echo "=========sync time============="
yum -y install chrony
cat >> /etc/chrony.conf << EOF
server ntp.aliyun.com iburst
EOF
systemctl start chronyd
systemctl enable chronyd
chronyc sources
# 将桥接的 IPv4 流量传递到 iptables 的链
echo "=========Letting iptables see bridged traffic============="
cat > /etc/modules-load.d/k8s.conf << EOF
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
# 配置 YUM
echo "=========config yum============="
cat > /etc/yum.repos.d/nginx.repo << EOF
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/rhel/7/x86_64/
gpgcheck=0
enabled=1
EOF
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
# 如果是 RedHat，则需要执行下面替换命令
#sed -i "s/\$releasever/7/g" /etc/yum.repos.d/CentOS-Base.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum clean all
yum makecache
yum -y install net-tools telnet tree nmap sysstat lrzsz dos2unix bind-utils ipvsadm bash-completion nfs-utils ftp nginx git
# 安装 Docker
echo "=========install docker============="
wget http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
# 如果是 RedHat，则需要执行下面替换命令
#sed -i "s/\$releasever/7/g" /etc/yum.repos.d/docker-ce.repo
sed -i "s/https/http/g" /etc/yum.repos.d/docker-ce.repo
yum -y install docker-ce-20.10.23 docker-ce-cli-20.10.23 containerd.io-1.6.15
systemctl enable docker
systemctl start docker
docker version
cat > /etc/docker/daemon.json << EOF
{
"insecure-registries": ["harbor.stonecoding.net"],
"live-restore": true,
"registry-mirrors": ["https://dockerproxy.com","https://registry.cn-hangzhou.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl restart docker
# 安装 kubelet kubeadm kubectl
echo "=========install kubeadm============="
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum -y install kubelet-1.23.17 kubeadm-1.23.17 kubectl-1.23.17
systemctl enable kubelet
分别在 3 台主机上运行：
[root@k8sm1 ~]# sh k8s-pre.sh
[root@k8sn1 ~]# sh k8s-pre.sh
[root@k8sn2 ~]# sh k8s-pre.sh
控制平面
在主机 k8sm1 上初始化控制平面：
[root@k8sm1 ~]# kubeadm init \
--control-plane-endpoint=192.168.92.140 \
--apiserver-advertise-address=192.168.92.140 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--kubernetes-version v1.23.17 \
--image-repository=registry.aliyuncs.com/google_containers
[init] Using Kubernetes version: v1.23.17
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8sm1.stonecoding.net kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.92.140]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8sm1.stonecoding.net localhost] and IPs [192.168.92.140 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8sm1.stonecoding.net localhost] and IPs [192.168.92.140 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 5.509988 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8sm1.stonecoding.net as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8sm1.stonecoding.net as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: xo6ca1.0wa790qn33gx9mwb
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join 192.168.92.140:6443 --token xo6ca1.0wa790qn33gx9mwb \
--discovery-token-ca-cert-hash sha256:63b42c8fda1a3a954c66de806fdd7ba32e6986b433e78f565ede301cd1b9e932 \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.92.140:6443 --token xo6ca1.0wa790qn33gx9mwb \
--discovery-token-ca-cert-hash sha256:63b42c8fda1a3a954c66de806fdd7ba32e6986b433e78f565ede301cd1b9e932
其中：
--control-plane-endpoint：指定控制平面接入点，如果控制平面为多节点组成的高可用集群，则该接入点应该为集群的负载均衡地址。--apiserver-advertise-address：指定apiserver的地址。--service-cidr：指定服务网络地址段，默认为 10.96.0.0/12。--pod-network-cidr：指定 Pod 网络地址段，不能与主机网络段相同。--kubernetes-version：指定 Kubernetes 版本。--image-repository：指定镜像仓库，需要指定国内的地址。
根据网络情况，有可能需要较多时间下载镜像，最终下载的镜像如下：
[root@k8sm1 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-apiserver v1.23.17 62bc5d8258d6 5 months ago 130MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.23.17 1dab4fc7b6e0 5 months ago 120MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.23.17 bc6794cb54ac 5 months ago 51.9MB
registry.aliyuncs.com/google_containers/kube-proxy v1.23.17 f21c8d21558c 5 months ago 111MB
registry.aliyuncs.com/google_containers/etcd 3.5.6-0 fce326961ae2 8 months ago 299MB
registry.aliyuncs.com/google_containers/coredns v1.8.6 a4ca41631cc7 22 months ago 46.8MB
registry.aliyuncs.com/google_containers/pause 3.6 6270bb605e12 24 months ago 683kB
根据输出日志，执行以下命令配置 kubectl：
[root@k8sm1 ~]# mkdir -p $HOME/.kube
[root@k8sm1 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8sm1 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
完成后，就可以使用 kubectl 命令查看资源情况：
[root@k8sm1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8sm1.stonecoding.net NotReady control-plane,master 64m v1.23.17
[root@k8sm1 ~]# kubectl get componentstatus
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true","reason":""}
[root@k8sm1 ~]# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 64m
[root@k8sm1 ~]# kubectl get service -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 64m
[root@k8sm1 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6d8c4cb4d-qqk4w 0/1 Pending 0 65m
coredns-6d8c4cb4d-vz5fs 0/1 Pending 0 65m
etcd-k8sm1.stonecoding.net 1/1 Running 0 65m
kube-apiserver-k8sm1.stonecoding.net 1/1 Running 0 65m
kube-controller-manager-k8sm1.stonecoding.net 1/1 Running 0 65m
kube-proxy-8k87h 1/1 Running 0 65m
kube-scheduler-k8sm1.stonecoding.net 1/1 Running 0 65m
由于还没有安装网络插件，coredns 处于 Pending 状态，安装网络插件后会创建完成。
网络插件
根据输出日志，需要安装网络插件。常用的网络插件有 Flannel 和 Calico，这里使用 Flannel。
下载 Flannel 资源配置清单文件（适用于 Kubernetes 1.17 及以后版本）：
[root@k8sm1 ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
如果直接下载失败，可以尝试使用代理下载：
[root@k8sm1 ~]# wget https://ghproxy.com/https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
然后根据初始化控制平面时指定的 Pod 网络配置进行修改，最终的资源配置清单文件如下：
---
kind: Namespace
apiVersion: v1
metadata:
name: kube-flannel
labels:
k8s-app: flannel
pod-security.kubernetes.io/enforce: privileged
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: flannel
name: flannel
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
- apiGroups:
- networking.k8s.io
resources:
- clustercidrs
verbs:
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: flannel
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-flannel
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: flannel
name: flannel
namespace: kube-flannel
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-flannel
labels:
tier: node
k8s-app: flannel
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-flannel
labels:
tier: node
app: flannel
k8s-app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni-plugin
image: docker.io/flannel/flannel-cni-plugin:v1.2.0
command:
- cp
args:
- -f
- /flannel
- /opt/cni/bin/flannel
volumeMounts:
- name: cni-plugin
mountPath: /opt/cni/bin
- name: install-cni
image: docker.io/flannel/flannel:v0.22.2
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: docker.io/flannel/flannel:v0.22.2
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: EVENT_QUEUE_DEPTH
value: "5000"
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
- name: xtables-lock
mountPath: /run/xtables.lock
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni-plugin
hostPath:
path: /opt/cni/bin
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
- name: xtables-lock
hostPath:
path: /run/xtables.lock
type: FileOrCreate
部署：
[root@k8sm1 ~]# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[root@k8sm1 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
flannel/flannel v0.22.2 d73868a08083 47 hours ago 70.2MB
flannel/flannel-cni-plugin v1.2.0 a55d1bad692b 4 weeks ago 8.04MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.23.17 62bc5d8258d6 5 months ago 130MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.23.17 1dab4fc7b6e0 5 months ago 120MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.23.17 bc6794cb54ac 5 months ago 51.9MB
registry.aliyuncs.com/google_containers/kube-proxy v1.23.17 f21c8d21558c 5 months ago 111MB
registry.aliyuncs.com/google_containers/etcd 3.5.6-0 fce326961ae2 8 months ago 299MB
registry.aliyuncs.com/google_containers/coredns v1.8.6 a4ca41631cc7 22 months ago 46.8MB
registry.aliyuncs.com/google_containers/pause 3.6 6270bb605e12 24 months ago 683kB
查看：
[root@k8sm1 ~]# kubectl get all -n kube-flannel
NAME READY STATUS RESTARTS AGE
pod/kube-flannel-ds-bq4f8 1/1 Running 0 87s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kube-flannel-ds 1 1 1 1 1 <none> 74m
[root@k8sm1 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6d8c4cb4d-qqk4w 1/1 Running 0 15h
coredns-6d8c4cb4d-vz5fs 1/1 Running 0 15h
etcd-k8sm1.stonecoding.net 1/1 Running 1 (3h21m ago) 15h
kube-apiserver-k8sm1.stonecoding.net 1/1 Running 1 (3h21m ago) 15h
kube-controller-manager-k8sm1.stonecoding.net 1/1 Running 1 (3h21m ago) 15h
kube-proxy-8k87h 1/1 Running 1 (3h21m ago) 15h
kube-scheduler-k8sm1.stonecoding.net 1/1 Running 1 (3h21m ago) 15h
网络插件 Flannel 运行后，coredns 处于 Running 状态。
工作节点
根据初始化输出日志，在主机 k8sn1 和 k8sn2 执行加入到集群的命令：
[root@k8sn1 ~]# kubeadm join 192.168.92.140:6443 --token xo6ca1.0wa790qn33gx9mwb --discovery-token-ca-cert-hash sha256:63b42c8fda1a3a954c66de806fdd7ba32e6986b433e78f565ede301cd1b9e932
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@k8sn2 ~]# kubeadm join 192.168.92.140:6443 --token xo6ca1.0wa790qn33gx9mwb --discovery-token-ca-cert-hash sha256:63b42c8fda1a3a954c66de806fdd7ba32e6986b433e78f565ede301cd1b9e932
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
在主机 k8sm1 上查看：
[root@k8sm1 ~]# kubectl get pods -n kube-flannel
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-5tf8x 1/1 Running 0 4m26s
kube-flannel-ds-h25mg 1/1 Running 0 7m11s
kube-flannel-ds-ncrrd 1/1 Running 0 4m37s
[root@k8sm1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8sm1.stonecoding.net Ready control-plane,master 15h v1.23.17
k8sn1.stonecoding.net Ready <none> 5m33s v1.23.17
k8sn2.stonecoding.net Ready <none> 4m41s v1.23.17
[root@k8sm1 ~]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-6d8c4cb4d-qqk4w 1/1 Running 0 15h 10.224.0.2 k8sm1.stonecoding.net <none> <none>
coredns-6d8c4cb4d-vz5fs 1/1 Running 0 15h 10.224.0.3 k8sm1.stonecoding.net <none> <none>
etcd-k8sm1.stonecoding.net 1/1 Running 1 (3h31m ago) 15h 192.168.92.140 k8sm1.stonecoding.net <none> <none>
kube-apiserver-k8sm1.stonecoding.net 1/1 Running 1 (3h31m ago) 15h 192.168.92.140 k8sm1.stonecoding.net <none> <none>
kube-controller-manager-k8sm1.stonecoding.net 1/1 Running 1 (3h31m ago) 15h 192.168.92.140 k8sm1.stonecoding.net <none> <none>
kube-proxy-69pt2 1/1 Running 0 5m25s 192.168.92.141 k8sn1.stonecoding.net <none> <none>
kube-proxy-8k87h 1/1 Running 1 (3h31m ago) 15h 192.168.92.140 k8sm1.stonecoding.net <none> <none>
kube-proxy-rs82r 1/1 Running 0 4m33s 192.168.92.142 k8sn2.stonecoding.net <none> <none>
kube-scheduler-k8sm1.stonecoding.net 1/1 Running 1 (3h31m ago) 15h 192.168.92.140 k8sm1.stonecoding.net <none> <none>
Ingress
需要安装 Ingress 控制器，以便 Kubernetes 对外提供服务。常用的 Ingress 控制器有 Ingress-Nginx 和 Traefik，这里使用 Ingress-Nginx。
注意：
部署前，需要确认工作节点的 80 和 443 端口没有被占用。
下载 Ingress-Nginx 资源配置清单文件：
[root@k8sm1 ~]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.6.4/deploy/static/provider/cloud/deploy.yaml -O ingress-nginx.yaml
如果直接下载失败，可以尝试使用代理下载：
[root@k8sm1 ~]# wget https://ghproxy.com/https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.6.4/deploy/static/provider/cloud/deploy.yaml -O ingress-nginx.yaml
修改资源配置清单文件中的镜像：
[root@k8sm1 ~]# sed -i "s|registry.k8s.io/ingress-nginx/controller:v1.6.4@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f|registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.6.4|g" ingress-nginx.yaml
[root@k8sm1 ~]# sed -i "s|registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f|registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20220916-gd32f8c343|g" ingress-nginx.yaml
然后调整资源配置清单文件中 ingress-nginx-controller 这个 Deployment ：
- 将 Deployment 修改为 DaemonSet
- 增加
hostNetwork: true，将 Ingress 控制器暴露到宿主机的 80 和 443 端口 - 增加参数
--watch-ingress-without-class=true，避免定义IngressClass - 修改
dnsPolicy为ClusterFirstWithHostNet - 为
nodeSelector增加ingress: nginx，将ingress-nginx-controller部署到指定节点上
最终的资源配置清单文件如下：
apiVersion: v1
kind: Namespace
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
name: ingress-nginx
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- coordination.k8s.io
resourceNames:
- ingress-nginx-leader
resources:
- leases
verbs:
- get
- update
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-admission
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
- namespaces
verbs:
- list
- watch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-admission
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-admission
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-admission
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: v1
data:
allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-controller
namespace: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
externalTrafficPolicy: Local
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
ports:
- appProtocol: https
name: https-webhook
port: 443
targetPort: webhook
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: ClusterIP
---
apiVersion: apps/v1
#kind: Deployment
kind: DaemonSet
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
minReadySeconds: 0
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
spec:
hostNetwork: true
containers:
- args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-nginx-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
- --watch-ingress-without-class=true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.6.4
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: controller
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 8443
name: webhook
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 100m
memory: 90Mi
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
runAsUser: 101
volumeMounts:
- mountPath: /usr/local/certificates/
name: webhook-cert
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
nodeSelector:
ingress: nginx
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-admission-create
namespace: ingress-nginx
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-admission-create
spec:
containers:
- args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20220916-gd32f8c343
imagePullPolicy: IfNotPresent
name: create
securityContext:
allowPrivilegeEscalation: false
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccountName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-admission-patch
namespace: ingress-nginx
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-admission-patch
spec:
containers:
- args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20220916-gd32f8c343
imagePullPolicy: IfNotPresent
name: patch
securityContext:
allowPrivilegeEscalation: false
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccountName: ingress-nginx-admission
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: nginx
spec:
controller: k8s.io/ingress-nginx
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.6.4
name: ingress-nginx-admission
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
service:
name: ingress-nginx-controller-admission
namespace: ingress-nginx
path: /networking/v1/ingresses
failurePolicy: Fail
matchPolicy: Equivalent
name: validate.nginx.ingress.kubernetes.io
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
sideEffects: None
先为需要部署 ingress-nginx-controller 的工作节点打标签：
[root@k8sm1 ~]# kubectl label nodes k8sn1.stonecoding.net ingress=nginx
node/k8sn1.stonecoding.net labeled
[root@k8sm1 ~]# kubectl label nodes k8sn2.stonecoding.net ingress=nginx
node/k8sn2.stonecoding.net labeled
部署：
[root@k8sm1 ~]# kubectl apply -f ingress-nginx.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
daemonset.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
查看：
[root@k8sm1 ~]# kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-xltbp 0/1 Completed 0 5m54s
pod/ingress-nginx-admission-patch-gh9p5 0/1 Completed 0 5m54s
pod/ingress-nginx-controller-j49qh 1/1 Running 0 5m54s
pod/ingress-nginx-controller-whd5f 1/1 Running 0 5m54s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.99.217.210 <pending> 80:31412/TCP,443:30818/TCP 5m54s
service/ingress-nginx-controller-admission ClusterIP 10.109.220.22 <none> 443/TCP 5m54s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/ingress-nginx-controller 2 2 2 2 2 ingress=nginx,kubernetes.io/os=linux 5m54s
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 4s 5m54s
job.batch/ingress-nginx-admission-patch 1/1 5s 5m54s
此时在工作节点上就可以看到监听的 80 和 443 端口：
[root@k8sn1 ~]# netstat -natp |egrep ":80|:443"
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 74480/nginx: master
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 74480/nginx: master
[root@k8sn2 ~]# netstat -natp |egrep ":80|:443"
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 43741/nginx: master
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 43741/nginx: master
控制平台节点已经安装了 Nginx，在 Nginx 上配置业务流量的反向代理，凡是访问 *.stonecoding.net 域的请求，都代理到工作节点映射出来的 80 端口，然后由 Ingress 根据服务名称分发到对应的 Pod。
[root@k8sm1 ~]# vi /etc/nginx/conf.d/stonecoding.net.conf
upstream default_backend_ingress {
server 192.168.92.141:80 max_fails=3 fail_timeout=10s;
server 192.168.92.142:80 max_fails=3 fail_timeout=10s;
}
server {
server_name *.stonecoding.net;
client_max_body_size 100m;
location / {
proxy_pass http://default_backend_ingress;
proxy_set_header Host $http_host;
proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
}
}
[root@k8sm1 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@k8sm1 ~]# systemctl start nginx
[root@k8sm1 ~]# systemctl enable nginx
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
Dashboard
安装 Dashboard ，使用图形界面管理 Kubernetes。
获取资源配置清单文件：
[root@k8sm1 ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml -O dashboard.yaml
如果直接下载失败，可以尝试使用代理下载：
[root@k8sm1 ~]# wget https://ghproxy.com/https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml -O dashboard.yaml
修改资源配置清单文件，将 kubernetes-dashboard 这个 Service 的端口类型修改为 NodePort，最终的资源配置清单文件如下：
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
type: NodePort
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kubernetes-dashboard
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-csrf
namespace: kubernetes-dashboard
type: Opaque
data:
csrf: ""
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-key-holder
namespace: kubernetes-dashboard
type: Opaque
---
kind: ConfigMap
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-settings
namespace: kubernetes-dashboard
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
rules:
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster", "dashboard-metrics-scraper"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
# Allow Metrics Scraper to get metrics from the Metrics server
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: kubernetes-dashboard
image: kubernetesui/dashboard:v2.5.1
imagePullPolicy: Always
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
- --namespace=kubernetes-dashboard
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
nodeSelector:
"kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
ports:
- port: 8000
targetPort: 8000
selector:
k8s-app: dashboard-metrics-scraper
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: dashboard-metrics-scraper
image: kubernetesui/metrics-scraper:v1.0.7
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- mountPath: /tmp
name: tmp-volume
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
serviceAccountName: kubernetes-dashboard
nodeSelector:
"kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: tmp-volume
emptyDir: {}
部署：
[root@k8sm1 ~]# kubectl apply -f dashboard.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
创建 dashboard-serviceaccount 服务账户：
[root@k8sm1 ~]# kubectl create serviceaccount dashboard-serviceaccount -n kubernetes-dashboard
serviceaccount/dashboard-serviceaccount created
为 dashboard-serviceaccount 服务账户授予 cluster-admin 角色：
[root@k8sm1 ~]# kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-serviceaccount
clusterrolebinding.rbac.authorization.k8s.io/dashboard-cluster-admin created
查看：
[root@k8sm1 ~]# kubectl get all -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
pod/dashboard-metrics-scraper-799d786dbf-bm5gt 1/1 Running 0 5m32s
pod/kubernetes-dashboard-fb8648fd9-t4c7k 1/1 Running 0 5m32s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dashboard-metrics-scraper ClusterIP 10.110.86.109 <none> 8000/TCP 5m32s
service/kubernetes-dashboard NodePort 10.107.3.191 <none> 443:32490/TCP 5m32s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dashboard-metrics-scraper 1/1 1 1 5m32s
deployment.apps/kubernetes-dashboard 1/1 1 1 5m32s
NAME DESIRED CURRENT READY AGE
replicaset.apps/dashboard-metrics-scraper-799d786dbf 1 1 1 5m32s
replicaset.apps/kubernetes-dashboard-fb8648fd9 1 1 1 5m32s
获取 token：
[root@k8sm1 ~]# kubectl describe secrets -n kubernetes-dashboard $(kubectl -n kubernetes-dashboard get secret | awk '/dashboard-serviceaccount/{print $1}')
Name: dashboard-serviceaccount-token-n4222
Namespace: kubernetes-dashboard
Labels: <none>
Annotations: kubernetes.io/service-account.name: dashboard-serviceaccount
kubernetes.io/service-account.uid: 6ec8d67b-14b6-4369-a330-4b80eda90f25
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1099 bytes
namespace: 20 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6Ijc4NndjNWJSUjYxMDZEcmU3amtGak1OYXpwODFuX2RYQWFQeVVBOEpDRFEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtc2VydmljZWFjY291bnQtdG9rZW4tbjQyMjIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLXNlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNmVjOGQ2N2ItMTRiNi00MzY5LWEzMzAtNGI4MGVkYTkwZjI1Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmRhc2hib2FyZC1zZXJ2aWNlYWNjb3VudCJ9.rq3Z0YvWBaJEn3YJTuD83mXQBgrwJ2bK8XpoFFC592XuFZr4pqvjis54QezAsgvJH4MNkkCeXWYG4w9LjOeWl10v3PqJxqWfzYA8AkFLOnIARJ0f4MoeW0nwKx8hotPVaa5iDa84TaytWwTUCC6PjUtFMx_B-RNgJnsYEvkvv7ImvSFMhLcypHieouEnLCTG7977MR9U7M2zqzeao7r-tIph6AeMCoaSwpKk0QXWNb4tuWLhdNTmKKWUdQamF0k9UE6Vl6Nz_ZkZMGtueCkctj_3pAucLl6_AYgeFjltC1WC_QGrSWg_dL8D9OjobxfMp929LvQ6hqjmpUGm-tOeGw
使用 Firefox 浏览器访问 https://192.168.92.141:32490 或者 https://192.168.92.142:32490，输入 token，点击 “登录”：
切换命名空间为 kube-system，点击 “工作负载”，可以看到各类工作负载的运行情况：
资源
在 Kubernetes 中，一切皆资源，管理 Kubernetes 就是管理 Kubernetes 中的资源。
[root@k8sm1 ~]# kubectl api-resources
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
componentstatuses cs v1 false ComponentStatus
configmaps cm v1 true ConfigMap
endpoints ep v1 true Endpoints
events ev v1 true Event
limitranges limits v1 true LimitRange
namespaces ns v1 false Namespace
nodes no v1 false Node
persistentvolumeclaims pvc v1 true PersistentVolumeClaim
persistentvolumes pv v1 false PersistentVolume
pods po v1 true Pod
podtemplates v1 true PodTemplate
replicationcontrollers rc v1 true ReplicationController
resourcequotas quota v1 true ResourceQuota
secrets v1 true Secret
serviceaccounts sa v1 true ServiceAccount
services svc v1 true Service
mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration
customresourcedefinitions crd,crds apiextensions.k8s.io/v1 false CustomResourceDefinition
apiservices apiregistration.k8s.io/v1 false APIService
controllerrevisions apps/v1 true ControllerRevision
daemonsets ds apps/v1 true DaemonSet
deployments deploy apps/v1 true Deployment
replicasets rs apps/v1 true ReplicaSet
statefulsets sts apps/v1 true StatefulSet
tokenreviews authentication.k8s.io/v1 false TokenReview
localsubjectaccessreviews authorization.k8s.io/v1 true LocalSubjectAccessReview
selfsubjectaccessreviews authorization.k8s.io/v1 false SelfSubjectAccessReview
selfsubjectrulesreviews authorization.k8s.io/v1 false SelfSubjectRulesReview
subjectaccessreviews authorization.k8s.io/v1 false SubjectAccessReview
horizontalpodautoscalers hpa autoscaling/v2 true HorizontalPodAutoscaler
cronjobs cj batch/v1 true CronJob
jobs batch/v1 true Job
certificatesigningrequests csr certificates.k8s.io/v1 false CertificateSigningRequest
leases coordination.k8s.io/v1 true Lease
endpointslices discovery.k8s.io/v1 true EndpointSlice
events ev events.k8s.io/v1 true Event
flowschemas flowcontrol.apiserver.k8s.io/v1beta2 false FlowSchema
prioritylevelconfigurations flowcontrol.apiserver.k8s.io/v1beta2 false PriorityLevelConfiguration
ingressclasses networking.k8s.io/v1 false IngressClass
ingresses ing networking.k8s.io/v1 true Ingress
networkpolicies netpol networking.k8s.io/v1 true NetworkPolicy
runtimeclasses node.k8s.io/v1 false RuntimeClass
poddisruptionbudgets pdb policy/v1 true PodDisruptionBudget
podsecuritypolicies psp policy/v1beta1 false PodSecurityPolicy
clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding
clusterroles rbac.authorization.k8s.io/v1 false ClusterRole
rolebindings rbac.authorization.k8s.io/v1 true RoleBinding
roles rbac.authorization.k8s.io/v1 true Role
priorityclasses pc scheduling.k8s.io/v1 false PriorityClass
csidrivers storage.k8s.io/v1 false CSIDriver
csinodes storage.k8s.io/v1 false CSINode
csistoragecapacities storage.k8s.io/v1beta1 true CSIStorageCapacity
storageclasses sc storage.k8s.io/v1 false StorageClass
volumeattachments storage.k8s.io/v1 false VolumeAttachment
常用的资源：
| 类别 | 名称 |
|---|---|
| 集群 | Node、Namespace |
| 负载 | Pod、ReplicaSet、Deployment、StatefulSet、DaemonSet、Job、CronJob |
| 服务 | Service、Ingress |
| 配置 | ConfigMap、Secret、HorizontalPodAutoscaling、CustomResourceDefinition |
| 存储 | Volume、PersistentVolume、LimitRange |
| 身份 | ServiceAccount、Role、ClusterRole、RoleBinding、ClusterRoleBinding |
Namespace
Namespace 是 Kubernetes 的命名空间，用于分组命名空间级别的资源对象。可以为不同的项目创建对应的命名空间。
支持命名空间的资源有：
[root@k8sm1 ~]# kubectl api-resources --namespaced=true
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
configmaps cm v1 true ConfigMap
endpoints ep v1 true Endpoints
events ev v1 true Event
limitranges limits v1 true LimitRange
persistentvolumeclaims pvc v1 true PersistentVolumeClaim
pods po v1 true Pod
podtemplates v1 true PodTemplate
replicationcontrollers rc v1 true ReplicationController
resourcequotas quota v1 true ResourceQuota
secrets v1 true Secret
serviceaccounts sa v1 true ServiceAccount
services svc v1 true Service
controllerrevisions apps/v1 true ControllerRevision
daemonsets ds apps/v1 true DaemonSet
deployments deploy apps/v1 true Deployment
replicasets rs apps/v1 true ReplicaSet
statefulsets sts apps/v1 true StatefulSet
localsubjectaccessreviews authorization.k8s.io/v1 true LocalSubjectAccessReview
horizontalpodautoscalers hpa autoscaling/v2 true HorizontalPodAutoscaler
cronjobs cj batch/v1 true CronJob
jobs batch/v1 true Job
leases coordination.k8s.io/v1 true Lease
endpointslices discovery.k8s.io/v1 true EndpointSlice
events ev events.k8s.io/v1 true Event
ingresses ing networking.k8s.io/v1 true Ingress
networkpolicies netpol networking.k8s.io/v1 true NetworkPolicy
poddisruptionbudgets pdb policy/v1 true PodDisruptionBudget
rolebindings rbac.authorization.k8s.io/v1 true RoleBinding
roles rbac.authorization.k8s.io/v1 true Role
csistoragecapacities storage.k8s.io/v1beta1 true CSIStorageCapacity
查看所有命名空间：
[root@k8sm1 ~]# kubectl get namespaces
NAME STATUS AGE
default Active 24h
ingress-nginx Active 166m
kube-flannel Active 24h
kube-node-lease Active 24h
kube-public Active 24h
kube-system Active 24h
kubernetes-dashboard Active 119m
其中：
default：默认的命名空间，在创建资源时，如果不指定命名空间，则会创建在此命名空间中。kube-system：集群管理相关组件所在的命名空间。
查看命名空间的 Label：
[root@k8sm1 ~]# kubectl get namespaces --show-labels
NAME STATUS AGE LABELS
default Active 24h kubernetes.io/metadata.name=default
ingress-nginx Active 166m app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,kubernetes.io/metadata.name=ingress-nginx
kube-flannel Active 24h k8s-app=flannel,kubernetes.io/metadata.name=kube-flannel,pod-security.kubernetes.io/enforce=privileged
kube-node-lease Active 24h kubernetes.io/metadata.name=kube-node-lease
kube-public Active 24h kubernetes.io/metadata.name=kube-public
kube-system Active 24h kubernetes.io/metadata.name=kube-system
kubernetes-dashboard Active 119m kubernetes.io/metadata.name=kubernetes-dashboard
创建命名空间：
[root@k8sm1 ~]# kubectl create namespace app
namespace/app created
使用资源配置清单文件创建命名空间：
apiVersion: v1
kind: Namespace
metadata:
name: app
字段含义如下：
| 名称 | 类型 | 是否必要 | 说明 |
|---|---|---|---|
apiVersion | String | Required | API 的版本，使用 kubectl api-resources 命令查询 |
kind | String | Required | 资源类型，这里为 Namespace |
metadata | Object | Required | 元数据 |
metadata.name | String | Required | Namespace 的名字 |
Pod
Pod 是 Kubernetes 最小部署和调度单位，运行在物理节点上。
架构
一个 Pod 包含一个或者多个容器，共享文件系统和网络。
在 Docker 中，我们知道容器之间是隔离的，有自己的网络和文件系统，为实现 Pod 中多个容器共享文件系统和网络，Kubernetes 使用了一个称为 Pause 的容器：
[root@k8sm1 ~]# docker ps --filter "name=flannel"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3e28576512e7 d73868a08083 "/opt/bin/flanneld -…" About an hour ago Up About an hour k8s_kube-flannel_kube-flannel-ds-h25mg_kube-flannel_f94326b5-c1e4-408e-9dd5-7af526f5d059_2
d5c27e570141 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" About an hour ago Up About an hour k8s_POD_kube-flannel-ds-h25mg_kube-flannel_f94326b5-c1e4-408e-9dd5-7af526f5d059_2
Pause 容器的作用主要有 2 点：
- 在 Pod 中作为 Linux Namespace 共享的基础容器
- 在 PID Namespace 共享的前提下，作为每个 Pod 中的 PID 1，回收僵尸进程
下面来模拟多个容器使用 Pause 容器共享资源，先创建一个 Pause 容器：
[root@k8sm1 ~]# docker run -d --name pause --ipc=shareable -p 8080:80 registry.aliyuncs.com/google_containers/pause:3.6
0ef2e7ba8946da71166ae6b8374c7b837bb9360c49c682440c021d6588cb31ca
其中：
-p 8080:80：将容器的 80 端口映射到主机的 8080 端口--ipc=shareable：拥有私有的 IPC 命名空间, 且可以共享给其他容器
再创建一个 Nginx 容器：
[root@k8sm1 ~]# cat <<EOF >> nginx.conf
error_log stderr;
events { worker_connections 1024; }
http {
access_log /dev/stdout combined;
server {
listen 80 default_server;
server_name example.com www.example.com;
location / {
proxy_pass http://127.0.0.1:2368;
}
}
}
EOF
[root@k8sm1 ~]# docker run -d --name nginx -v /root/nginx.conf:/etc/nginx/nginx.conf --net=container:pause --ipc=container:pause --pid=container:pause nginx
cc600daa1bf90cbdb75ad0b174b45259b9354061379cefdda4ccd443b51107a1
其中：
--net：指定共享 Pause 的网络--ipc：指定共享 Pause 的 IPC--pid：指定共享 Pause 的 PID
再创建一个 Ghost 容器：
[root@k8sm1 ~]# docker run -d --name ghost --net=container:pause --ipc=container:pause --pid=container:pause ghost:2
448b9f646d6e45c54723959ab41cde6cc32a599f710f2984ef3d0a6780d057e5
完成后，在浏览器中访问 http://192.168.92.140:8080/，发现打开了 Ghost 网页。
从上面的步骤可知：
- Pause 容器将内部 80 端口映射到宿主机 8080 端口并共享网络。
- Nginx 容器使用
--net=container:pause加入到 Pause 网络中。 - Ghost 容器使用
--net=container:pause加入到 Pause 网络中。 - 这样三个容器共享网络后，互相之间就可以使用 localhost 直接通信。
--ipc=container:pause，--pid=container:pause使三个容器的 IPC 和 PID 处于同一个 Namespace中，Init 进程为 Pause，PID 为 1。
[root@k8sm1 ~]# docker exec -it nginx /bin/bash
root@0ef2e7ba8946:/# pidof pause
1
[root@k8sm1 ~]# docker exec -it ghost /bin/bash
root@0ef2e7ba8946:/var/lib/ghost# pidof pause
1
网络
每个 Pod 都会被分配一个唯一的 IP 地址。Pod 中的所有容器共享网络空间，包括 IP 地址和端口，可以使用 localhost 互相通信。
同一宿主机上的 Pod 通过 cni0 进行通信，访问流程如下：
- Pod 1 (10.244.1.2) 向 Pod 2 (10.244.1.3) 发送数据，通过
eth0接口离开 Pod 1。 - 数据通过
veth0到达cni0，寻找 Pod 2 的地址。 - 数据通过
veth1离开cni0，发往 Pod 2。 - 数据通过
eth1到达 Pod 2。
不同宿主机上的 Pod 通信，则需要使用 CNI 网络插件，这里使用的是 Flannel。
Flannel 的部署文件中包含一个 ConfigMap，其中包含 Flannel CNI 的配置文件以及 Network 和 Backend 配置。Flannel 通过 DaemonSet 调度到每个节点上。Flannel 的 Pod 有一个 Init 容器，负责将 ConfigMap 中的 cni-conf.json 复制到宿主机上的 /etc/cni/net.d/10-flannel.conflist：
[root@k8sm1 ~]# cat /etc/cni/net.d/10-flannel.conflist
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
Flannel 会根据 ConfigMap 中的 Network 来为每个 Node 分配单独的子网，这样就能保证不同 Node 上的 Pod IP 不会冲突。并将集群网络和 Subnet 的配置信息写入文件 /run/flannel/subnet.env：
[root@k8sm1 ~]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
一般配置 ConfigMap 中的 Backend 为 vxlan（使用类似 VLAN 的封装技术将 OSI 第 2 层以太网帧封装在第 4 层 UDP 数据报中，并使用 4789 作为默认 IANA 分配的目标 UDP 端口号），宿主机会增加 flannel.1 网口，作为 Pod 与外部网络的网关。
3 个节点的网络和路由情况如下：
[root@k8sm1 ~]# ip addr | egrep 'mtu|link|inet'
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:fa:5b:e5 brd ff:ff:ff:ff:ff:ff
inet 192.168.92.140/24 brd 192.168.92.255 scope global ens33
inet6 fe80::20c:29ff:fefa:5be5/64 scope link
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:8e:8d:41:6b brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
inet6 fe80::42:8eff:fe8d:416b/64 scope link
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether fa:9d:c6:a2:46:3a brd ff:ff:ff:ff:ff:ff
inet 10.244.0.0/32 scope global flannel.1
inet6 fe80::f89d:c6ff:fea2:463a/64 scope link
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether b2:7a:c6:02:36:44 brd ff:ff:ff:ff:ff:ff
inet 10.244.0.1/24 brd 10.244.0.255 scope global cni0
inet6 fe80::b07a:c6ff:fe02:3644/64 scope link
6: veth71670adb@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether c2:5f:ae:2c:0b:c9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::c05f:aeff:fe2c:bc9/64 scope link
7: vethb988663a@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 0e:48:71:3a:1b:bc brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::c48:71ff:fe3a:1bbc/64 scope link
15: vetha420398@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 62:9b:9f:84:c6:73 brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::609b:9fff:fe84:c673/64 scope link
[root@k8sm1 ~]# ip route
default via 192.168.92.2 dev ens33
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
169.254.0.0/16 dev ens33 scope link metric 1002
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.92.0/24 dev ens33 proto kernel scope link src 192.168.92.140
[root@k8sn1 ~]# ip addr | egrep 'mtu|link|inet'
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:63:91:cc brd ff:ff:ff:ff:ff:ff
inet 192.168.92.141/24 brd 192.168.92.255 scope global ens33
inet6 fe80::20c:29ff:fe63:91cc/64 scope link
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:d3:ab:d7:ba brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether a2:ad:25:ef:4d:9b brd ff:ff:ff:ff:ff:ff
inet 10.244.1.0/32 scope global flannel.1
inet6 fe80::a0ad:25ff:feef:4d9b/64 scope link
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether f6:71:9e:02:b3:8f brd ff:ff:ff:ff:ff:ff
inet 10.244.1.1/24 brd 10.244.1.255 scope global cni0
inet6 fe80::f471:9eff:fe02:b38f/64 scope link
6: veth3c45aab8@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 86:ba:d2:db:17:d8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::84ba:d2ff:fedb:17d8/64 scope link
7: veth9f1c97d6@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether fa:2e:28:2b:1c:11 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::f82e:28ff:fe2b:1c11/64 scope link
[root@k8sn1 ~]# ip -d link show flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether a2:ad:25:ef:4d:9b brd ff:ff:ff:ff:ff:ff promiscuity 0
vxlan id 1 local 192.168.92.141 dev ens33 srcport 0 0 dstport 8472 nolearning ageing 300 noudpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
[root@k8sn1 ~]# ip route
default via 192.168.92.2 dev ens33
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink
10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
169.254.0.0/16 dev ens33 scope link metric 1002
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.92.0/24 dev ens33 proto kernel scope link src 192.168.92.141
[root@k8sn2 ~]# ip addr | egrep 'mtu|link|inet'
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:23:91:65 brd ff:ff:ff:ff:ff:ff
inet 192.168.92.142/24 brd 192.168.92.255 scope global ens33
inet6 fe80::20c:29ff:fe23:9165/64 scope link
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:f2:73:8f:f1 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 5e:ae:41:8d:4b:9f brd ff:ff:ff:ff:ff:ff
inet 10.244.2.0/32 scope global flannel.1
inet6 fe80::5cae:41ff:fe8d:4b9f/64 scope link
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 92:7d:fb:49:42:0a brd ff:ff:ff:ff:ff:ff
inet 10.244.2.1/24 brd 10.244.2.255 scope global cni0
inet6 fe80::907d:fbff:fe49:420a/64 scope link
6: veth52664c07@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 6a:ca:b4:3f:da:50 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::68ca:b4ff:fe3f:da50/64 scope link
[root@k8sn2 ~]# ip route
default via 192.168.92.2 dev ens33
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 dev cni0 proto kernel scope link src 10.244.2.1
169.254.0.0/16 dev ens33 scope link metric 1002
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.92.0/24 dev ens33 proto kernel scope link src 192.168.92.142
可以看到每台宿主机上都有两条使用 flannel.1 网口的路由，作为访问其他宿主机上 Pod 的网关。
Pod 跨宿主机访问流程如下：
Pod 1 (10.244.1.2) 向 Pod 4 (10.244.2.3) 发送数据。因为 Pod 1 和 Pod 4 的 IP 不在一个子网，因此走默认路由表，直接发向接口
cni0的地址10.244.1.1。IP 包到达
cni0接口后，根据主机路由表10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink，下一跳是10.244.2.0，通过flannel.1发送。此时需要知道
10.244.2.0的 MAC 地址，因此检查节点 1 的 ARP 表：[root@k8sn1 ~]# arp Address HWtype HWaddress Flags Mask Iface 10.244.1.6 ether 1e:a2:1e:4d:7e:11 C cni0 10.244.0.0 ether fa:9d:c6:a2:46:3a CM flannel.1 10.244.1.5 ether 62:1d:b8:54:c9:7c C cni0 10.244.2.0 ether 5e:ae:41:8d:4b:9f CM flannel.1 192.168.92.1 ether 00:50:56:c0:00:08 C ens33 k8sm1 ether 00:0c:29:fa:5b:e5 C ens33 gateway ether 00:50:56:e9:3d:8f C ens33发现
10.244.2.0 ether 5e:ae:41:8d:4b:9f CM flannel.1，因此要发送的帧如下：
二层帧的转发需要查找主机的 FDB 表（使用
bridge fdb show命令查看）。这里匹配到5e:ae:41:8d:4b:9f dev flannel.1 dst 192.168.92.142 self permanent。内核将数据封装成vxlan的包从ens33发出去。发出去的报文如下：
节点 2 的
ens33收到包后，发现是vxlan，内核对包解封装，转发到flannel.1上：此时三层目标地址是
10.244.2.3，因此匹配主机的路由表10.244.2.0/24 dev cni0 proto kernel scope link src 10.244.2.1，转发给cni0。cni0和 Pod 是二层互通的，将包发给 Pod。Pod 收到包。三层的来源地址是
10.244.1.2，二层的来源地址是cni0的 MAC 地址。
配置
通过资源配置清单文件来创建 Pod，使用以下命令查看相关配置项：
[root@k8sm1 ~]# kubectl explain pods
[root@k8sm1 ~]# kubectl explain pods.metadata
[root@k8sm1 ~]# kubectl explain pods.spec
[root@k8sm1 ~]# kubectl explain pods.status
语法如下：
apiVersion: v1
kind: Pod
metadata:
name: string
namespace: string
labels:
- name: string
annotations:
- name: string
spec:
containers:
- name: string
image: string
imagePullPolicy: [Always | Never | IfNotPresent]
command: [string]
args: [string]
workingDir: string
volumeMounts:
- name: string
mountPath: string
readOnly: boolean
ports:
- name: string
containerPort: int
hostPort: int
protocol: [TCP | UDP]
env:
- name: string
value: string
resources:
limits:
cpu: string
memory: string
requests:
cpu: string
memory: string
livenessProbe:
exec:
command: [string]
httpGet:
path: string
port: number
host: string
scheme: string
httpHeaders:
- name: string
value: string
tcpSocket:
port: number
initialDelaySeconds: 0
timeoutSeconds: 0
periodSeconds: 0
successThreshold: 0
failureThreshold: 0
securityContext:
provoleged: false
restartPolicy: [Always | Never | OnFailure]
nodeSelector: object
imagePullSecrets:
- name: string
hostNetwork: false
dnsPolicy: ClusterFirst
hostAliases:
- ip: "127.0.0.1"
hostnames:
- "foo.local"
- "bar.local"
volumes:
- name: string
emptyDir: {}
hostPath:
path: string
secret:
secretName: string
items:
- key: string
path: string
comfigMap:
name: string
items:
- key: string
path: string
字段含义如下：
| 名称 | 类型 | 是否必要 | 说明 |
|---|---|---|---|
apiVersion | String | Required | API 的版本，使用 kubectl api-resources 命令查询 |
kind | String | Required | 资源类型，这里为 Pod |
metadata | Object | Required | 元数据 |
metadata.name | String | Required | Pod 的名字 |
metadata.namespace | String | Required | Pod 的命名空间，默认为 default 命名空间 |
metadata.labels[] | List | 自定义标签列表 | |
metadata.annotation[] | List | 自定义注解列表 | |
Spec | Object | Required | Pod 的详细定义 |
spec.containers[] | List | Required | Pod 的容器定义 |
spec.containers[].name | String | Required | 容器名称 |
spec.containers[].image | String | Required | 容器的镜像名称和版本 |
spec.containers[].imagePullPolicy | String | 镜像拉取策略，包括： Always：默认值，表示每次都尝试重新下载镜像 Never：表示仅使用本地镜像 IfNotPresent：本地没有镜像才下载镜像 | |
spec.containers[].command[] | List | 容器的启动命令列表，如果不指定，则使用镜像的启动命令 | |
spec.containers[].args[] | List | 容器的启动命令参数列表 | |
spec.containers[].workingDir | String | 容器的工作目录 | |
spec.containers[].volumeMounts[] | List | 挂载到容器内部的存储卷配置 | |
spec.containers[].volumeMounts[].name | String | 在 spec.volumes[] 部分定义的共享存储卷名称 | |
spec.containers[].volumeMounts[].mountPath | String | 存储卷在容器内挂载的绝对路径 | |
spec.containers[].volumeMounts[].readOnly | Boolean | 是否为只读模式，默认值为读写模式 | |
spec.containers[].ports[] | List | 容器是否需要暴露的端口号列表 | |
spec.containers[].ports[].name | String | 端口的名称 | |
spec.containers[].ports[].containerPort | Int | 容器需要监听的端口号 | |
spec.containers[].ports[].hostPort | Int | 容器所在宿主机需要监听的端口号，默认与 containerPort 相同。设置 hostPort 时，同一台宿主机将无法启动该容器的第二份副本 | |
spec.containers[].ports[].protocol | Srting | 端口协议，支持 TCP 和 UDP，默认值为 TCP | |
spec.containers[].env[] | List | 容器的环境变量列表 | |
spec.containers[].env[].name | String | 环境变量的名称 | |
spec.containers[].env[].valume | String | 环境变量的值 | |
spec.containers[].resources | Object | 资源限制和资源请求 | |
spec.containers[].resources.limits | Object | 资源限制 | |
spec.containers[].resources.limits.cpu | String | CPU 限制 | |
spec.containers[].resources.limits.memory | String | 内存限制 | |
spec.containers[].resources.requests | Object | 资源请求 | |
spec.containers[].resources.requests.cpu | String | CPU 请求 | |
spec.containers[].resources.requests.memory | String | 内存请求 | |
spec.containers[].livenessProbe | Object | 容器存活探针 | |
spec.containers[].livenessProbe.exec | Object | 使用 exec 方式对 Pod 容器进行探测 | |
spec.containers[].livenessProbe.exec.command[] | String | exec 方式需要执行的命令或脚本 | |
spec.containers[].livenessProbe.httpGet | Object | 使用 httpGet 方式对 Pod 容器进行探测 | |
spec.containers[].livenessProbe.tcpSocket | Object | 使用 tcpSocket 方式对 Pod 容器进行探测 | |
spec.containers[].livenessProbe.initialDelaySeconds | Number | 容器启动完成后到进行初次探测的时间间隔，单位为秒 | |
spec.containers[].livenessProbe.timeoutSeconds | Number | 探测的超时秒数，默认值为 1 秒，超时将重启该容器 | |
spec.containers[].livenessProbe.periodSeconds | Number | 定期探测时间间隔，单位为 10 秒 | |
spec.containers[].readinessProbe | Object | 容器就绪探针 | |
spec.containers[].readinessProbe.exec | Object | 使用 exec 方式对 Pod 容器进行探测 | |
spec.containers[].readinessProbe.exec.command[] | String | exec 方式需要执行的命令或脚本 | |
spec.containers[].readinessProbe.httpGet | Object | 使用 httpGet 方式对 Pod 容器进行探测 | |
spec.containers[].readinessProbe.tcpSocket | Object | 使用 tcpSocket 方式对 Pod 容器进行探测 | |
spec.containers[].readinessProbe.initialDelaySeconds | Number | 容器启动完成后到进行初次探测的时间间隔，单位为秒 | |
spec.containers[].readinessProbe.timeoutSeconds | Number | 探测的超时秒数，默认值为 1 秒，超时将该容器从服务中剔除 | |
spec.containers[].readinessProbe.periodSeconds | Number | 定期探测时间间隔，单位为 10 秒 | |
spec.containers[].startupProbe | Object | 容器启动探针 | |
spec.containers[].startupProbe.exec | Object | 使用 exec 方式对 Pod 容器进行探测 | |
spec.containers[].startupProbe.exec.command[] | String | exec 方式需要执行的命令或脚本 | |
spec.containers[].startupProbe.httpGet | Object | 使用 httpGet 方式对 Pod 容器进行探测 | |
spec.containers[].startupProbe.tcpSocket | Object | 使用 tcpSocket 方式对 Pod 容器进行探测 | |
spec.containers[].startupProbe.initialDelaySeconds | Number | 容器启动完成后到进行初次探测的时间间隔，单位为秒 | |
spec.containers[].startupProbe.timeoutSeconds | Number | 探测的超时秒数，默认值为 1 秒 | |
spec.containers[].startupProbe.periodSeconds | Number | 定期探测时间间隔，单位为 10 秒 | |
spec.volumes[] | List | 共享存储卷列表 | |
spec.volumes[].name | String | 共享存储卷的名称，用于 spec.containers[].volumeMounts[].name | |
spec.volumes[].emptyDir | Object | 类型为 emptyDir 的存储卷，表示与 Pod 同生命周期的一个临时目录，其值为一个空对象：emptyDir: {} | |
spec.volumes[].hostPath | Object | 类型为 hostPath 的存储卷，表示挂载 Pod 所在宿主机的目录 | |
spec.volumes[].hostPath.path | String | Pod 所在宿主机的目录，将用于容器中挂载的目录 | |
spec.volumes[].secret | Object | 类型为 secret 的存储卷，表示挂载集群预定义的 secret 对象到容器内部 | |
spec.volumes[].configMap | Object | 类型为 configMap 的存储卷，表示挂载集群预定义的 configMap 对象到容器内部 | |
spec.restartPolicy | String | Pod 中所有容器的重启策略，包括： Always：默认值，始终重启 OnFailure：以非零退出码终止时才会重启 Never：不重启 | |
spec.nodeSelector | Object | 表示将该 Pod 调度到包含对应 Label 的 Node 上 | |
spec.imagePullSecrets | Object | 拉取镜像时使用的 secret 名称 | |
spec.hostNetwork | Boolean | 是否使用宿主机的地址和端口 | |
spec.dnsPolicy | String | Pode 的 DNS 策略，包括以下四种：Default：从宿主机继承域名解析配置，只能解析注册到互联网上的外部域名，无法解析集群内部域名ClusterFirst：默认策略，应用对接 Kube-DNS/CoreDNS。这种场景下，容器既能够解析 Service 注册的集群内部域名，也能够解析发布到互联网上的外部域名。不过 ClusterFirst 还有一个特例，如果 Pod 设置了hostNetwork: true，则 ClusterFirst 就会被强制转换成 DefaultClusterFirstWithHostNet：如果 Pod 设置了hostNetwork: true，则应显式设置 DNS 策略为 ClusterFirstWithHostNetNone：允许 Pod 忽略 Kubernetes 环境中的 DNS 设置，会使用其 dnsConfig 字段提供的 DNS 设置 | |
spec.hostAliases | Object | 向 Pod 的 /etc/hosts 文件中添加条目 |
管理
使用如下资源配置清单文件定义 Pod：
apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: app
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
创建 Pod：
[root@k8sm1 ~]# kubectl apply -f simple-pod.yaml
pod/nginx created
查看 Pod 运行情况：
[root@k8sm1 ~]# kubectl get pods -n app
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 63s
查看 Pod 的详细信息：
[root@k8sm1 ~]# kubectl get pods -n app -o yaml
apiVersion: v1
items:
- apiVersion: v1
kind: Pod
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"nginx","namespace":"app"},"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx","ports":[{"containerPort":80}]}]}}
creationTimestamp: "2023-08-24T05:09:58Z"
name: nginx
namespace: app
resourceVersion: "134938"
uid: 6aea2cfe-3547-4a9e-b559-d39fccb98282
spec:
containers:
- image: nginx:1.14.2
imagePullPolicy: IfNotPresent
name: nginx
ports:
- containerPort: 80
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-glhq7
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: k8sn2.stonecoding.net
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: kube-api-access-glhq7
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2023-08-24T05:09:58Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2023-08-24T05:10:47Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2023-08-24T05:10:47Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2023-08-24T05:09:58Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://1d19c07aff70c409a3d3c9b303a67853f97ce77cf76848aff5a60d7f01b898c4
image: nginx:1.14.2
imageID: docker-pullable://nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
lastState: {}
name: nginx
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2023-08-24T05:10:47Z"
hostIP: 192.168.92.142
phase: Running
podIP: 10.244.2.6
podIPs:
- ip: 10.244.2.6
qosClass: BestEffort
startTime: "2023-08-24T05:09:58Z"
kind: List
metadata:
resourceVersion: ""
selfLink: ""
查看 Pod 运行信息：
[root@k8sm1 ~]# kubectl describe pods nginx -n app
Name: nginx
Namespace: app
Priority: 0
Node: k8sn2.stonecoding.net/192.168.92.142
Start Time: Thu, 24 Aug 2023 13:09:58 +0800
Labels: <none>
Annotations: <none>
Status: Running
IP: 10.244.2.6
IPs:
IP: 10.244.2.6
Containers:
nginx:
Container ID: docker://1d19c07aff70c409a3d3c9b303a67853f97ce77cf76848aff5a60d7f01b898c4
Image: nginx:1.14.2
Image ID: docker-pullable://nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 24 Aug 2023 13:10:47 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-glhq7 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-glhq7:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m19s default-scheduler Successfully assigned app/nginx to k8sn2.stonecoding.net
Normal Pulling 2m18s kubelet Pulling image "nginx:1.14.2"
Normal Pulled 91s kubelet Successfully pulled image "nginx:1.14.2" in 47.377873524s (47.377878779s including waiting)
Normal Created 90s kubelet Created container nginx
Normal Started 90s kubelet Started container nginx
删除 Pod：
[root@k8sm1 ~]# kubectl delete pod nginx -n app
pod "nginx" deleted
Deployment
在实际工作中，往往不会单独创建一个 Pod，而是使用控制器来创建和管理 Pod。最常用的控制器就是 Deployment，用来创建和管理无状态的 Pod。
在介绍 Deployment 控制器之前，先了解一下 ReplicaSet 控制器。ReplicaSet 的目的是维护一组 Pod，确保任何时间都有指定数量的 Pod 副本在运行，如果有容器异常退出，会自动创建新的 Pod 来替代。可以修改 ReplicaSet 资源配置清单文件中的 spec.replica 数量来创建或删除 Pod，让 Pod 数量符合要求。
Deployment 为 Pod 和 ReplicaSet 提供了一个声明式更新，用于创建或删除 Pod，让 Pod 数量符合要求。通过在 Deployment 对象中描述一个期望的状态，Deployment 就会按照一定的控制速率把实际状态改成期望状态。通过定义一个 Deployment 控制器创建一个新的 ReplicaSet 控制器，通过 ReplicaSet 控制器创建 Pod，删除 Deployment 控制器，也会删除 Deployment 控制器下对应的 ReplicaSet 控制器和 Pod 资源。
使用 Deployment 而不直接创建 ReplicaSet 是因为 Deployment 拥有 ReplicaSet 没有的特性，例如：
- 滚动升级
- 回滚应用
创建
使用如下资源配置清单文件 nginx-deployment.yaml 定义 Deployment：
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: app
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
其中：
metadata.name：指定 Deployment 名称，为后续创建 ReplicaSet 和 Pod 的命名基础metadata.labels：指定 Deployment 标签spec.replicas：指定 Pod 数量，默认为 1spec.selector：spec的必要字段，指定 Pod 的标签选择器，必须与spec.template.metadata.labels匹配spec.strategy：指定用新 Pod 替换旧 Pod 的策略spec.strategy.type：指定策略类型，可以是Recreate（删除当前所有 Pod，再新建）和RollingUpdate（滚动更新），默认为RollingUpdatespec.strategy.rollingUpdate.maxSurge：指定更新过程中可以创建的超出期望的 Pod 个数，默认为 25%，例如原 Pod 为 4 个，则更新过程中，Pod 数量最多为 5 个spec.strategy.rollingUpdate.maxUnavailable：指定更新过程中不可用的 Pod 的个数上限，默认为 25%，例如原 Pod 为 4 个，则更新过程中，Pod 不可用数量最多为 1 个spec.template：spec的必要字段，定义 Pod，字段含义与 Pod 的配置一致，但没有apiVersion和kind字段spec.template.metadata.labels：指定 Pod 标签
创建 Deployment ：
[root@k8sm1 ~]# kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx-deployment created
查看 Deployment：
[root@k8sm1 ~]# kubectl get deployments -n app
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 0/3 3 0 48s
[root@k8sm1 ~]# kubectl get deployments -n app
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 3 3 2m55s
其中：
NAME：Deployment 的名称READY：可用副本数，格式为就绪数量/期望数量UP-TO-DATE：成功更新的 Pod 数量AVAILABLE：可用副本数AGE：Deployment 运行时间
查看创建的 ReplicaSet：
[root@k8sm1 ~]# kubectl get replicasets -n app --show-labels
NAME DESIRED CURRENT READY AGE LABELS
nginx-deployment-9456bbbf9 3 3 3 21m app=nginx,pod-template-hash=9456bbbf9
其中：
NAME：ReplicaSets 的名称，其格式为[DEPLOYMENT-NAME]-[HASH]，作为其 Pod 名称的基础，其HASH与标签pod-template-hash的值相同DESIRED：期望副本数量，即spec.replicasCURRENT：当前运行的副本数READY：用户可用的副本数AGE：ReplicaSet 运行时间
查看创建的 Pod：
[root@k8sm1 ~]# kubectl get pods -n app --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-deployment-9456bbbf9-9pssm 1/1 Running 0 18m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-9456bbbf9-gwlf8 1/1 Running 0 18m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-9456bbbf9-lnxq8 1/1 Running 0 18m app=nginx,pod-template-hash=9456bbbf9
可以看到每个 Pod 都有标签 pod-template-hash。
查看 Deployment 详细信息：
[root@k8sm1 ~]# kubectl get deployments nginx-deployment -n app -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"nginx-deployment","namespace":"app"},"spec":{"replicas":3,"selector":{"matchLabels":{"app":"nginx"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app":"nginx"}},"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx","ports":[{"containerPort":80}]}]}}}}
creationTimestamp: "2023-08-25T05:49:21Z"
generation: 1
labels:
app: nginx
name: nginx-deployment
namespace: app
resourceVersion: "173682"
uid: 976e6f8c-f5e9-4691-9316-a8ff20b09533
spec:
progressDeadlineSeconds: 600
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
imagePullPolicy: IfNotPresent
name: nginx
ports:
- containerPort: 80
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 3
conditions:
- lastTransitionTime: "2023-08-25T05:49:23Z"
lastUpdateTime: "2023-08-25T05:49:23Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2023-08-25T05:49:21Z"
lastUpdateTime: "2023-08-25T05:49:23Z"
message: ReplicaSet "nginx-deployment-9456bbbf9" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 3
replicas: 3
updatedReplicas: 3
更新
只有 Deployment 的 Pod 模版（spec.template）发生改变才会触发 Rollout，例如更新了容器镜像的版本，其它诸如扩容 Deployment，不会触发 Rollout。
更新 Deployment，使用 nginx:1.16.1 镜像替换 nginx:1.14.2 ：
[root@k8sm1 ~]# kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 -n app
deployment.apps/nginx-deployment image updated
也可以使用以下命令直接编辑资源配置清单或者修改资源配置清单文件后再 Apply：
[root@k8sm1 ~]# kubectl edit deployment/nginx-deployment -n app
查看 Rollout 状态：
[root@k8sm1 ~]# kubectl rollout status deployment/nginx-deployment -n app
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out
查看 Deployment 变化，可以看到 UP-TO-DATE 从 1 变到 3：
[root@k8sm1 ~]# kubectl get deployments -n app
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 1 3 77m
[root@k8sm1 ~]# kubectl get deployments -n app
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 2 3 77m
[root@k8sm1 ~]# kubectl get deployments -n app
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 3 3 80m
查看 ReplicaSet 变化，可以看到创建了一个新的 ReplicaSet，先在新的 ReplicaSet 中创建一个（spec.strategy.rollingUpdate.maxSurge）新版本的 Pod，再在老的 ReplicaSet 中删除一个（spec.strategy.rollingUpdate.maxUnavailable）旧版本的 Pod，直到新的 ReplicaSet 中的 Pod 数量达到设定值，老的 ReplicaSet 中的Pod 数量变为 0：
[root@k8sm1 ~]# kubectl get replicasets -n app --show-labels
NAME DESIRED CURRENT READY AGE LABELS
nginx-deployment-9456bbbf9 3 3 3 76m app=nginx,pod-template-hash=9456bbbf9
[root@k8sm1 ~]# kubectl get replicasets -n app --show-labels
NAME DESIRED CURRENT READY AGE LABELS
nginx-deployment-9456bbbf9 3 3 3 77m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-ff6655784 1 1 0 15s app=nginx,pod-template-hash=ff6655784
[root@k8sm1 ~]# kubectl get replicasets -n app --show-labels
NAME DESIRED CURRENT READY AGE LABELS
nginx-deployment-9456bbbf9 2 2 2 77m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-ff6655784 2 2 1 32s app=nginx,pod-template-hash=ff6655784
[root@k8sm1 ~]# kubectl get replicasets -n app --show-labels
NAME DESIRED CURRENT READY AGE LABELS
nginx-deployment-9456bbbf9 0 0 0 77m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-ff6655784 3 3 3 48s app=nginx,pod-template-hash=ff6655784
查看 Pod 变化，可以看到新版本 Pod 的创建和老版本 Pod 的删除，只有在足够数量的新版本 Pod 创建成功后才会删除老版本 Pod，也只有在足够数量的老版本 Pod 删除成功后才会再去创建新版本 Pod，以确保至少有 3 个 Pod 可用，最多有 4 个 Pod 可用，也就是可用 Pod 数量在 replicas - maxUnavailable 和 replicas + maxSurge 之间：
[root@k8sm1 ~]# kubectl get pods -n app --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-deployment-9456bbbf9-9pssm 1/1 Running 0 76m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-9456bbbf9-gwlf8 1/1 Running 0 76m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-9456bbbf9-lnxq8 1/1 Running 0 76m app=nginx,pod-template-hash=9456bbbf9
[root@k8sm1 ~]# kubectl get pods -n app --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-deployment-9456bbbf9-9pssm 1/1 Running 0 77m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-9456bbbf9-gwlf8 1/1 Running 0 77m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-9456bbbf9-lnxq8 1/1 Running 0 77m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-ff6655784-vx5fr 0/1 ContainerCreating 0 17s app=nginx,pod-template-hash=ff6655784
[root@k8sm1 ~]# kubectl get pods -n app --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-deployment-9456bbbf9-9pssm 1/1 Running 0 77m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-9456bbbf9-gwlf8 1/1 Running 0 77m app=nginx,pod-template-hash=9456bbbf9
nginx-deployment-ff6655784-vx5fr 1/1 Running 0 35s app=nginx,pod-template-hash=ff6655784
nginx-deployment-ff6655784-wlvrk 0/1 ContainerCreating 0 10s app=nginx,pod-template-hash=ff6655784
[root@k8sm1 ~]# kubectl get pods -n app --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-deployment-ff6655784-qfz68 1/1 Running 0 4s app=nginx,pod-template-hash=ff6655784
nginx-deployment-ff6655784-vx5fr 1/1 Running 0 51s app=nginx,pod-template-hash=ff6655784
nginx-deployment-ff6655784-wlvrk 1/1 Running 0 26s app=nginx,pod-template-hash=ff6655784
查看 Deployment 运行信息：
[root@k8sm1 ~]# kubectl describe deployments nginx-deployment -n app
Name: nginx-deployment
Namespace: app
CreationTimestamp: Thu, 24 Aug 2023 19:48:31 +0800
Labels: app=nginx
Annotations: deployment.kubernetes.io/revision: 2
Selector: app=nginx
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.16.1
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-ff6655784 (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 22m deployment-controller Scaled up replica set nginx-deployment-ff6655784 to 1
Normal ScalingReplicaSet 21m deployment-controller Scaled down replica set nginx-deployment-9456bbbf9 to 2
Normal ScalingReplicaSet 21m deployment-controller Scaled up replica set nginx-deployment-ff6655784 to 2
Normal ScalingReplicaSet 21m deployment-controller Scaled down replica set nginx-deployment-9456bbbf9 to 1
Normal ScalingReplicaSet 21m deployment-controller Scaled up replica set nginx-deployment-ff6655784 to 3
Normal ScalingReplicaSet 21m deployment-controller Scaled down replica set nginx-deployment-9456bbbf9 to 0
回滚
如果发现当前版本有问题，可以先回滚到之前的版本。
先查看 Deployment 历史：
[root@k8sm1 ~]# kubectl rollout history deployment/nginx-deployment -n app
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 <none>
2 <none>
[root@k8sm1 ~]# kubectl rollout history deployment/nginx-deployment -n app --revision=1
deployment.apps/nginx-deployment with revision #1
Pod Template:
Labels: app=nginx
pod-template-hash=9456bbbf9
Containers:
nginx:
Image: nginx:1.14.2
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
[root@k8sm1 ~]# kubectl rollout history deployment/nginx-deployment -n app --revision=2
deployment.apps/nginx-deployment with revision #2
Pod Template:
Labels: app=nginx
pod-template-hash=ff6655784
Containers:
nginx:
Image: nginx:1.16.1
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
回滚到前一个版本：
[root@k8sm1 ~]# kubectl rollout undo deployment/nginx-deployment -n app
deployment.apps/nginx-deployment rolled back
查看：
[root@k8sm1 ~]# kubectl get all -n app
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-9456bbbf9-7wcrw 1/1 Running 0 16s
pod/nginx-deployment-9456bbbf9-rw9cx 1/1 Running 0 14s
pod/nginx-deployment-9456bbbf9-t6srk 1/1 Running 0 12s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 3/3 3 3 29m
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-9456bbbf9 3 3 3 29m
replicaset.apps/nginx-deployment-ff6655784 0 0 0 28m
[root@k8sm1 ~]# kubectl describe deployment nginx-deployment -n app
Name: nginx-deployment
Namespace: app
CreationTimestamp: Fri, 25 Aug 2023 10:35:03 +0800
Labels: app=nginx
Annotations: deployment.kubernetes.io/revision: 3
Selector: app=nginx
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.14.2
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-9456bbbf9 (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 30m deployment-controller Scaled up replica set nginx-deployment-ff6655784 to 1
Normal ScalingReplicaSet 28m deployment-controller Scaled down replica set nginx-deployment-9456bbbf9 to 2
Normal ScalingReplicaSet 28m deployment-controller Scaled up replica set nginx-deployment-ff6655784 to 2
Normal ScalingReplicaSet 25m deployment-controller Scaled down replica set nginx-deployment-9456bbbf9 to 1
Normal ScalingReplicaSet 25m deployment-controller Scaled up replica set nginx-deployment-ff6655784 to 3
Normal ScalingReplicaSet 25m deployment-controller Scaled down replica set nginx-deployment-9456bbbf9 to 0
Normal ScalingReplicaSet 102s deployment-controller Scaled up replica set nginx-deployment-9456bbbf9 to 1
Normal ScalingReplicaSet 100s deployment-controller Scaled down replica set nginx-deployment-ff6655784 to 2
Normal ScalingReplicaSet 100s deployment-controller Scaled up replica set nginx-deployment-9456bbbf9 to 2
Normal ScalingReplicaSet 98s (x2 over 31m) deployment-controller Scaled up replica set nginx-deployment-9456bbbf9 to 3
Normal ScalingReplicaSet 98s deployment-controller Scaled down replica set nginx-deployment-ff6655784 to 1
Normal ScalingReplicaSet 96s deployment-controller Scaled down replica set nginx-deployment-ff6655784 to 0
再次先查看 Deployment 历史：
[root@k8sm1 ~]# kubectl rollout history deployment/nginx-deployment -n app
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
2 <none>
3 <none>
回滚到指定版本：
[root@k8sm1 ~]# kubectl rollout undo deployment/nginx-deployment --to-revision=2 -n app
deployment.apps/nginx-deployment rolled back
查看：
[root@k8sm1 ~]# kubectl get all -n app
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-ff6655784-h8qtv 1/1 Running 0 32s
pod/nginx-deployment-ff6655784-j7bj7 1/1 Running 0 36s
pod/nginx-deployment-ff6655784-kwsx7 1/1 Running 0 34s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 3/3 3 3 35m
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-9456bbbf9 0 0 0 35m
replicaset.apps/nginx-deployment-ff6655784 3 3 3 34m
[root@k8sm1 ~]# kubectl describe deployment nginx-deployment -n app
Name: nginx-deployment
Namespace: app
CreationTimestamp: Fri, 25 Aug 2023 10:35:03 +0800
Labels: app=nginx
Annotations: deployment.kubernetes.io/revision: 4
Selector: app=nginx
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.16.1
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-ff6655784 (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 30m deployment-controller Scaled down replica set nginx-deployment-9456bbbf9 to 1
Normal ScalingReplicaSet 30m deployment-controller Scaled up replica set nginx-deployment-ff6655784 to 3
Normal ScalingReplicaSet 30m deployment-controller Scaled down replica set nginx-deployment-9456bbbf9 to 0
Normal ScalingReplicaSet 6m30s deployment-controller Scaled up replica set nginx-deployment-9456bbbf9 to 1
Normal ScalingReplicaSet 6m28s deployment-controller Scaled up replica set nginx-deployment-9456bbbf9 to 2
Normal ScalingReplicaSet 6m28s deployment-controller Scaled down replica set nginx-deployment-ff6655784 to 2
Normal ScalingReplicaSet 6m26s (x2 over 36m) deployment-controller Scaled up replica set nginx-deployment-9456bbbf9 to 3
Normal ScalingReplicaSet 6m26s deployment-controller Scaled down replica set nginx-deployment-ff6655784 to 1
Normal ScalingReplicaSet 6m24s deployment-controller Scaled down replica set nginx-deployment-ff6655784 to 0
Normal ScalingReplicaSet 55s (x2 over 34m) deployment-controller Scaled up replica set nginx-deployment-ff6655784 to 1
Normal ScalingReplicaSet 53s (x2 over 33m) deployment-controller Scaled up replica set nginx-deployment-ff6655784 to 2
Normal ScalingReplicaSet 53s (x2 over 33m) deployment-controller Scaled down replica set nginx-deployment-9456bbbf9 to 2
Normal ScalingReplicaSet 49s (x3 over 51s) deployment-controller (combined from similar events): Scaled down replica set nginx-deployment-9456bbbf9 to 0
扩缩容
扩容：
[root@k8sm1 ~]# kubectl scale deployment/nginx-deployment --replicas=4 -n app
deployment.apps/nginx-deployment scaled
[root@k8sm1 ~]# kubectl get all -n app
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-ff6655784-djxbf 1/1 Running 0 109s
pod/nginx-deployment-ff6655784-h8qtv 1/1 Running 0 21m
pod/nginx-deployment-ff6655784-j7bj7 1/1 Running 0 21m
pod/nginx-deployment-ff6655784-kwsx7 1/1 Running 0 21m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 4/4 4 4 56m
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-9456bbbf9 0 0 0 56m
replicaset.apps/nginx-deployment-ff6655784 4 4 4 55m
缩容：
[root@k8sm1 ~]# kubectl scale deployment/nginx-deployment --replicas=2 -n app
deployment.apps/nginx-deployment scaled
[root@k8sm1 ~]# kubectl get all -n app
\NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-ff6655784-j7bj7 1/1 Running 0 21m
pod/nginx-deployment-ff6655784-kwsx7 1/1 Running 0 21m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 2/2 2 2 56m
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-9456bbbf9 0 0 0 56m
replicaset.apps/nginx-deployment-ff6655784 2 2 2 55m
Service
Service 是访问 Kubernetes 集群内部多个 Pod 的入口，可以理解为多个 Pod 的 VIP。
从前面创建的 Deployment 可知，在 Kubernetes 集群中，无状态的 Pod 副本是可以随时删除、随时创建的，并且重新创建的 Pod 不再保留旧 Pod 的任何信息，包括 IP 地址。在这种情况下，前端 Pod 如何使用后端 Pod 来提供服务就成了问题，因此 Kubernetes 实现了 Service 这样一个抽象概念。对于有 Selector 的 Service，在被创建的时候会自动创建 EndPoint 资源，包含了对应 Lable 的所有 Pod 的 IP 和端口，并且在之后的 Pod 的删除、创建中，也会立即更新相关 Pod 的 IP 和端口信息。同时，Service 的 IP 地址是永远固定的，Service 和 EndPoint 是一一对应的关系。这样，如果前端 Pod 通过固定的 Service IP 来访问后端 Pod 提供的服务，那么就可以在 EndPoint 中找到一个可用 Pod 的 IP 和端口，再将数据包转发到指定的 Pod 上即可。
创建服务
使用如下资源配置清单文件 nginx-service.yaml 定义 Service：
apiVersion: v1
kind: Service
metadata:
name: nginx-service
namespace: app
labels:
app: nginx
spec:
type: ClusterIP
selector:
app: nginx
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
sessionAffinity: None
其中:
metadata.name：Service 名称metadata.labels：Service 标签spec.type：Service 类型，有以下几种：ClusterIP：默认类型，通过集群的内部 IP 暴露服务，服务只能够从集群内部访问，其他类型的基础NodePort：是ClusterIP类型的超集。使用 NAT 方式，通过每个节点上的 IP 和端口（默认 30000-32767）暴露服务，可以不指定由系统自动分配节点暴露的端口，也可以通过spec.ports[*].nodePort指定节点暴露的端口。前面创建 Dashboard 时就是采用的NodePort方式，直接将服务暴露到所有节点主机的端口上LoadBalancer：用于支持云服务商提供的外部负载均衡器，是NodePort类型的超集ExternalName：将服务映射到 DNS 名称，而不是 Selector
spec.selector：指定要访问的 Pod 的标签spec.ports：指定端口信息，包括：spec.ports[*].name：服务端口名称spec.ports[*].protocol：端口协议spec.ports[*].port：服务端口号spec.ports[*].targetPort：对应的 Pod 端口，可以是端口号，也可以是创建 Pod 时指定的端口名称
spec.sessionAffinity：会话亲和性，默认为None，可以配置为ClientIP将同一客户端会话连接到同一 Pod
创建 Service ：
[root@k8sm1 ~]# kubectl apply -f nginx-service.yaml
service/nginx-service created
查看 Service：
[root@k8sm1 ~]# kubectl get services -n app -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
nginx-service ClusterIP 10.97.154.99 <none> 80/TCP 52s app=nginx
[root@k8sm1 ~]# kubectl describe services nginx-service -n app
Name: nginx-service
Namespace: app
Labels: app=nginx
Annotations: <none>
Selector: app=nginx
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.97.154.99
IPs: 10.97.154.99
Port: http 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.1.20:80,10.244.2.19:80,10.244.2.20:80
Session Affinity: None
Events: <none>
可以看到该 Service 对接的 Endpoints 为 10.244.1.20:80,10.244.2.19:80,10.244.2.20:80。
查看 Endpoint：
[root@k8sm1 ~]# kubectl get endpoints nginx-service -n app -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
endpoints.kubernetes.io/last-change-trigger-time: "2023-08-28T02:33:38Z"
creationTimestamp: "2023-08-28T02:33:38Z"
labels:
app: nginx
name: nginx-service
namespace: app
resourceVersion: "198840"
uid: 04d89ad4-c0be-4107-8595-847b56e50072
subsets:
- addresses:
- ip: 10.244.1.20
nodeName: k8sn1.stonecoding.net
targetRef:
kind: Pod
name: nginx-deployment-9456bbbf9-b5b79
namespace: app
resourceVersion: "193892"
uid: 15681a8a-7a57-48bc-b7c3-f569a3be0ccd
- ip: 10.244.2.19
nodeName: k8sn2.stonecoding.net
targetRef:
kind: Pod
name: nginx-deployment-9456bbbf9-f4dxq
namespace: app
resourceVersion: "193991"
uid: 384b52ec-1c49-4c68-b76a-572b0560ec29
- ip: 10.244.2.20
nodeName: k8sn2.stonecoding.net
targetRef:
kind: Pod
name: nginx-deployment-9456bbbf9-j95b6
namespace: app
resourceVersion: "193988"
uid: e9be4521-9f4a-4e09-9d63-2b5dd6857a75
ports:
- name: http
port: 80
protocol: TCP
将 Deployment 扩容为 4 副本：
[root@k8sm1 ~]# kubectl scale deployment/nginx-deployment --replicas=4 -n app
deployment.apps/nginx-deployment scaled
查看 Endpoint，可以看到将新增的 Pod 加入到 Endpoint 中：
[root@k8sm1 ~]# kubectl get endpoints nginx-service -n app -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
endpoints.kubernetes.io/last-change-trigger-time: "2023-08-28T05:26:53Z"
creationTimestamp: "2023-08-28T02:33:38Z"
labels:
app: nginx
name: nginx-service
namespace: app
resourceVersion: "214715"
uid: 04d89ad4-c0be-4107-8595-847b56e50072
subsets:
- addresses:
- ip: 10.244.1.20
nodeName: k8sn1.stonecoding.net
targetRef:
kind: Pod
name: nginx-deployment-9456bbbf9-b5b79
namespace: app
resourceVersion: "193892"
uid: 15681a8a-7a57-48bc-b7c3-f569a3be0ccd
- ip: 10.244.1.21
nodeName: k8sn1.stonecoding.net
targetRef:
kind: Pod
name: nginx-deployment-9456bbbf9-6hn6v
namespace: app
resourceVersion: "214711"
uid: b833c8ad-42bf-4f1e-bb77-6ba5a0c39e95
- ip: 10.244.2.19
nodeName: k8sn2.stonecoding.net
targetRef:
kind: Pod
name: nginx-deployment-9456bbbf9-f4dxq
namespace: app
resourceVersion: "193991"
uid: 384b52ec-1c49-4c68-b76a-572b0560ec29
- ip: 10.244.2.20
nodeName: k8sn2.stonecoding.net
targetRef:
kind: Pod
name: nginx-deployment-9456bbbf9-j95b6
namespace: app
resourceVersion: "193988"
uid: e9be4521-9f4a-4e09-9d63-2b5dd6857a75
ports:
- name: http
port: 80
protocol: TCP
将 Deployment 缩容为 3 副本：
[root@k8sm1 ~]# kubectl scale deployment/nginx-deployment --replicas=2 -n app
deployment.apps/nginx-deployment scaled
查看 Endpoint，就只有剩下的 Pod 了：
[root@k8sm1 ~]# kubectl get endpoints nginx-service -n app -o yaml
apiVersion: v1
kind: Endpoints
metadata:
creationTimestamp: "2023-08-28T02:33:38Z"
labels:
app: nginx
name: nginx-service
namespace: app
resourceVersion: "215281"
uid: 04d89ad4-c0be-4107-8595-847b56e50072
subsets:
- addresses:
- ip: 10.244.1.20
nodeName: k8sn1.stonecoding.net
targetRef:
kind: Pod
name: nginx-deployment-9456bbbf9-b5b79
namespace: app
resourceVersion: "193892"
uid: 15681a8a-7a57-48bc-b7c3-f569a3be0ccd
- ip: 10.244.2.19
nodeName: k8sn2.stonecoding.net
targetRef:
kind: Pod
name: nginx-deployment-9456bbbf9-f4dxq
namespace: app
resourceVersion: "193991"
uid: 384b52ec-1c49-4c68-b76a-572b0560ec29
ports:
- name: http
port: 80
protocol: TCP
服务发现
对于集群内的 Pod，Kubernetes 支持两种服务发现模式：环境变量和 DNS。
环境变量
创建 Service 之后创建的 Pod 会被注入之前 Service 对应的环境变量，格式为 {SVCNAME}_SERVICE_HOST 和 {SVCNAME}_SERVICE_PORT（将服务名称转换为大小，横线转换为下划线），这样 Pod 就可以通过其环境变量访问对应的服务了。
创建一个 Service 及其对应的一个 Nginx 镜像版本为 1.18 的 Deployment：
[root@k8sm1 ~]# vi nginx-v118.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx-service-v118
namespace: app
labels:
app: nginx-v118
spec:
type: ClusterIP
selector:
app: nginx-v118
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
sessionAffinity: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment-v118
namespace: app
labels:
app: nginx-v118
spec:
replicas: 1
selector:
matchLabels:
app: nginx-v118
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: nginx-v118
spec:
containers:
- name: nginx
image: nginx:1.18
ports:
- containerPort: 80
[root@k8sm1 ~]# kubectl apply -f nginx-v118.yaml
service/nginx-service-v118 created
deployment.apps/nginx-deployment-v118 created
[root@k8sm1 ~]# kubectl get all -n app
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-9456bbbf9-5cdgr 1/1 Running 0 8m3s
pod/nginx-deployment-9456bbbf9-87rfh 1/1 Running 0 8m
pod/nginx-deployment-9456bbbf9-jf8zw 1/1 Running 0 7m59s
pod/nginx-deployment-v118-69584f9576-d4cw6 1/1 Running 0 15s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx-service ClusterIP 10.97.154.99 <none> 80/TCP 3h52m
service/nginx-service-v118 ClusterIP 10.111.251.40 <none> 80/TCP 15s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 3/3 3 3 3d
deployment.apps/nginx-deployment-v118 1/1 1 1 15s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-6dfc66cc6c 0 0 0 10m
replicaset.apps/nginx-deployment-9456bbbf9 3 3 3 3d
replicaset.apps/nginx-deployment-v118-69584f9576 1 1 1 15s
进入 Pod nginx-deployment-v118-69584f9576-d4cw6 查看环境变量：
[root@k8sm1 ~]# kubectl exec -it nginx-deployment-v118-69584f9576-d4cw6 sh -n app
# env | grep NGINX_SERVICE_V118
NGINX_SERVICE_V118_SERVICE_HOST=10.111.251.40
NGINX_SERVICE_V118_PORT=tcp://10.111.251.40:80
NGINX_SERVICE_V118_SERVICE_PORT=80
NGINX_SERVICE_V118_PORT_80_TCP_ADDR=10.111.251.40
NGINX_SERVICE_V118_PORT_80_TCP_PORT=80
NGINX_SERVICE_V118_PORT_80_TCP_PROTO=tcp
NGINX_SERVICE_V118_PORT_80_TCP=tcp://10.111.251.40:80
NGINX_SERVICE_V118_SERVICE_PORT_HTTP=80
DNS
使用环境变量作为服务发现必须先创建服务，再创建 Pod。更常用的是使用组件 CoreDNS 提供的 DNS 服务来进行服务发现。
[root@k8sm1 ~]# kubectl get pods -n kube-system -o wide | grep dns
coredns-6d8c4cb4d-bwwxw 1/1 Running 5 (5h31m ago) 7d3h 10.244.0.13 k8sm1.stonecoding.net <none> <none>
coredns-6d8c4cb4d-zp7nr 1/1 Running 5 (5h31m ago) 7d3h 10.244.0.12 k8sm1.stonecoding.net <none> <none>
[root@k8sm1 ~]# kubectl get services -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 7d3h
[root@k8sm1 ~]# kubectl describe services kube-dns -n kube-system
Name: kube-dns
Namespace: kube-system
Labels: k8s-app=kube-dns
kubernetes.io/cluster-service=true
kubernetes.io/name=CoreDNS
Annotations: prometheus.io/port: 9153
prometheus.io/scrape: true
Selector: k8s-app=kube-dns
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.96.0.10
IPs: 10.96.0.10
Port: dns 53/UDP
TargetPort: 53/UDP
Endpoints: 10.244.0.12:53,10.244.0.13:53
Port: dns-tcp 53/TCP
TargetPort: 53/TCP
Endpoints: 10.244.0.12:53,10.244.0.13:53
Port: metrics 9153/TCP
TargetPort: 9153/TCP
Endpoints: 10.244.0.12:9153,10.244.0.13:9153
Session Affinity: None
Events: <none>
创建一个镜像为 busybox 的 Pod：
[root@k8sm1 ~]# vi busybox-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: busybox
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
[root@k8sm1 ~]# kubectl apply -f busybox-pod.yaml
pod/busybox created
[root@k8sm1 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 24s
进入容器查看服务：
[root@k8sm1 ~]# kubectl exec -it busybox -- /bin/sh
/ # nslookup 10.111.251.40
Server: 10.96.0.10
Address: 10.96.0.10:53
40.251.111.10.in-addr.arpa name = nginx-service-v118.app.svc.cluster.local
服务代理
Kubernetes 集群的每个节点都有 kube-proxy 组件，作为服务代理，为服务实现 Virtual IP（VIP），使发往 Service 的流量负载均衡到正确的后端 Pod。
kube-proxy 监听 API Server 中资源对象的变化情况，包括以下三种：
- Service
- Endpoint/EndpointSlices（如果使用了 EndpointSlice，则会监听 EndpointSlice，否则监听 Endpoint）
- Node
然后根据资源变化情况，为服务配置负载均衡。
在 Linux 上目前 Kube-proxy 可用模式有：
- iptables
- ipvs
iptables
Kube-proxy 默认使用 iptables 模式：
[root@k8sm1 ~]# curl http://localhost:10249/proxyMode
iptables
在这种模式下，Kube-proxy 监视 Kubernetes 控制平面，获知对 Service 和 EndpointSlice 的添加和删除操作。 对于每个 Service，Kube-proxy 会添加 iptables 规则，这些规则捕获流向 Service 的 clusterIP 和 port 的流量， 并将这些流量重定向到 Service 后端 Pod 中的其中之一。对于每个 Endpoint，会添加指向某个后端 Pod 的 iptables 规则。
默认情况下，iptables 模式下的 Kube-proxy 会随机选择一个后端 Pod。
使用 iptables 处理流量的系统开销较低，因为流量由 Linux netfilter 处理，无需在用户空间和内核空间之间切换，也更为可靠。
如果 Kube-proxy 以 iptables 模式运行，并且选择的第一个 Pod 没有响应，那么连接会失败。可以使用 Pod 的就绪探针来验证后端 Pod 是否健康，以避免 Kube-proxy 将流量发送到失败的 Pod 上。
当创建 Service 时，Kubernetes 控制平面会分配一个虚拟 IP 地址，例如 10.97.154.99。当节点上的 Kube-proxy 观察到新 Service 时，会添加一系列 iptables 规则，以便将访问这个地址的流量转到真实的 Pod 上。
[root@k8sm1 ~]# iptables -t nat -nL OUTPUT
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
第一条规则会匹配所有的流量，然后跳到 KUBE-SERVICES 这条链上。看一下 KUBE-SERVICES 的具体内容：
[root@k8sm1 ~]# iptables -t nat -nL KUBE-SERVICES
Chain KUBE-SERVICES (2 references)
target prot opt source destination
KUBE-SVC-EDNDUDH2C75GIR6O tcp -- 0.0.0.0/0 10.98.51.149 /* ingress-nginx/ingress-nginx-controller:https cluster IP */ tcp dpt:443
KUBE-SVC-EZYNCFY2F7N6OQA2 tcp -- 0.0.0.0/0 10.101.134.29 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
KUBE-SVC-CEZPIJSAUFW5MYPQ tcp -- 0.0.0.0/0 10.107.3.191 /* kubernetes-dashboard/kubernetes-dashboard cluster IP */ tcp dpt:443
KUBE-SVC-CB4FKN7PLFL7IROR tcp -- 0.0.0.0/0 10.97.154.99 /* app/nginx-service:http cluster IP */ tcp dpt:80
KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- 0.0.0.0/0 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-CG5I4G2RS3ZVWGLK tcp -- 0.0.0.0/0 10.98.51.149 /* ingress-nginx/ingress-nginx-controller:http cluster IP */ tcp dpt:80
KUBE-SVC-JD5MR3NA4I4DYORP tcp -- 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
KUBE-SVC-TCOU7JCQXEZGVUNU udp -- 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-SVC-Z6GDYMWE5TV2NNJN tcp -- 0.0.0.0/0 10.110.86.109 /* kubernetes-dashboard/dashboard-metrics-scraper cluster IP */ tcp dpt:8000
KUBE-SVC-BMEYHMRRLQBXSPQR tcp -- 0.0.0.0/0 10.111.251.40 /* app/nginx-service-v118:http cluster IP */ tcp dpt:80
KUBE-NODEPORTS all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
这里前面的 KUBE-SVC-* 都是根据 destination，protocol 和目的端口号来匹配的，根据 Service 地址和端口号以及协议，可以匹配到 KUBE-SVC-CB4FKN7PLFL7IROR 这条规则，然后跳到这条链上。接着看这条链的定义：
[root@k8sm1 ~]# iptables -t nat -nL KUBE-SVC-CB4FKN7PLFL7IROR
Chain KUBE-SVC-CB4FKN7PLFL7IROR (1 references)
target prot opt source destination
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.97.154.99 /* app/nginx-service:http cluster IP */ tcp dpt:80
KUBE-SEP-6IWAUY3HDSBJG6JK all -- 0.0.0.0/0 0.0.0.0/0 /* app/nginx-service:http */ statistic mode random probability 0.33333333349
KUBE-SEP-6SVYFH2G76OTHT2L all -- 0.0.0.0/0 0.0.0.0/0 /* app/nginx-service:http */ statistic mode random probability 0.50000000000
KUBE-SEP-L2XRLFTEOGVDOKM7 all -- 0.0.0.0/0 0.0.0.0/0 /* app/nginx-service:http */
最下面的 2-4 条规则就是使用 iptables 实现负载均衡的地方了。第 2 条规则有 33% 的匹配几率。如果匹配到了其中一条，就会跳到这个链上。比如：
[root@k8sm1 ~]# iptables -t nat -nL KUBE-SEP-6IWAUY3HDSBJG6JK
Chain KUBE-SEP-6IWAUY3HDSBJG6JK (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.1.25 0.0.0.0/0 /* app/nginx-service:http */
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* app/nginx-service:http */ tcp to:10.244.1.25:80
第 2 条规则将目的 IP 和 Port 改写成 10.244.1.25:80，也就是 Pod 的 IP 和 Port，这样流量就经过负载均衡指向了后端的 Pod。
IPVS
在 IPVS 模式下，Kube-proxy 监视 Kubernetes Service 和 EndpointSlice，然后调用 netlink 接口创建 IPVS 规则， 并定期与 Kubernetes Service 和 EndpointSlice 同步 IPVS 规则，以确保 IPVS 状态与期望的状态保持一致。 访问 Service 时，IPVS 会将流量导向到某个后端 Pod。
IPVS 代理模式基于 netfilter 回调函数，类似于 iptables 模式， 但使用哈希表作为底层数据结构，工作在内核空间。 这意味着 IPVS 模式下的 Kube-proxy 比 iptables 模式下的 Kube-proxy 重定向流量的延迟更低，同步代理规则时性能也更好。 当服务数量超过 10000 个时，建议使用 IPVS。
IPVS 可以使用的负载均衡算法有：
rr：轮询lc：最少连接（打开连接数最少）dh：目标地址哈希sh：源地址哈希sed：最短预期延迟nq：最少队列
注意： 要在 IPVS 模式下运行 Kube-proxy，必须在启动 Kube-proxy 之前确保节点上的 IPVS 可用。
当 Kube-proxy 以 IPVS 代理模式启动时，将验证 IPVS 内核模块是否可用。 如果未检测到 IPVS 内核模块，则 Kube-proxy 将退回到以 iptables 代理模式运行。
Ingress
可以使用 Service 的 NodePort 将服务暴露在宿主机的端口上供外部访问，但随着服务的增多，会大量占用宿主机的端口，维护大量的 NodePort 也非常麻烦。此时就需要使用 Ingress 来作为反向代理（类似于 Nginx）处理外部对集群服务的 HTTP 请求。
Ingress 是 Kubernetes 中的一个资源对象，作为集群内服务对外暴露的统一入口，将外部流量路由到 Kubernetes 集群内的服务。可以通过 Ingress 资源来配置不同的转发规则，从而达到根据不同的规则设置访问集群内不同的 Service 所对应的后端 Pod。
为了使 Ingress 正常工作，集群中必须运行 Ingress 控制器，Ingress 控制器有很多种，这里使用前面已经安装好的 Ingress-Nginx。
[root@k8sm1 ~]# kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-xltbp 0/1 Completed 0 127m
pod/ingress-nginx-admission-patch-gh9p5 0/1 Completed 0 127m
pod/ingress-nginx-controller-j49qh 1/1 Running 0 127m
pod/ingress-nginx-controller-whd5f 1/1 Running 0 127m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.99.217.210 <pending> 80:31412/TCP,443:30818/TCP 127m
service/ingress-nginx-controller-admission ClusterIP 10.109.220.22 <none> 443/TCP 127m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/ingress-nginx-controller 2 2 2 2 2 ingress=nginx,kubernetes.io/os=linux 127m
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 4s 127m
job.batch/ingress-nginx-admission-patch 1/1 5s 127m
[root@k8sm1 ~]# kubectl get ingressclass
NAME CONTROLLER PARAMETERS AGE
nginx k8s.io/ingress-nginx <none> 167m
其中 Pod 有：
ingress-nginx-controller是 Ingress-Nginx 的控制器组件，负责监视 Kubernetes API Server 上的 Ingress 对象，并根据配置动态地更新 Nginx 配置文件，实现 HTTP(S) 的负载均衡和路由。ingress-nginx-admission-create是一个 Kubernetes Admission Controller，可以拦截 Kubernetes 集群内的 Ingress 对象的创建操作，并在 Ingress 对象创建之前，对其进行一些额外的验证和处理。ingress-nginx-admission-patch也是一个 Kubernetes Admission Controller，可以在 Ingress 对象更新之前，对其进行额外的验证和处理，类似于ingress-nginx-admission-create。
Service 有：
ingress-nginx-controller负责将请求转发到ingress-nginx-controllerPods。它通常会将流量分发到ingress-nginx-controller的多个副本中，并确保副本集的负载平衡。这个 Service 可以被配置为使用NodePort、LoadBalancer或ClusterIP类型，根据需要进行暴露。ingress-nginx-controller-admission用于 Kubernetes Admission Webhooks，允许在创建、更新或删除资源时，对其进行校验或修改。它提供了一个 API Endpoint，用于与 Kubernetes API Server 进行通信，以便进行这些校验或修改。这个 Service 可以被配置为使用NodePort、LoadBalancer或ClusterIP类型，根据需要进行暴露。
创建
使用如下资源配置清单文件 web-ingress.yaml 定义 Ingress：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web
namespace: app
spec:
ingressClassName: nginx
rules:
- host: web.stonecoding.net
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
其中：
ingressClassName：指定使用的 Ingress 控制器spec.rules[*].http.paths[*].pathType：有 3 种路径类型：ImplementationSpecific：匹配方法取决于 IngressClassExact：精确匹配 URL 路径，且区分大小写Prefix：基于以/分隔的 URL 路径前缀匹配，区分大小写
匹配示例如下：
| Kind | Path(s) | Request path(s) | Matches? |
|---|---|---|---|
| Prefix | / | (all paths) | Yes |
| Exact | /foo | /foo | Yes |
| Exact | /foo | /bar | No |
| Exact | /foo | /foo/ | No |
| Exact | /foo/ | /foo | No |
| Prefix | /foo | /foo, /foo/ | Yes |
| Prefix | /foo/ | /foo, /foo/ | Yes |
| Prefix | /aaa/bb | /aaa/bbb | No |
| Prefix | /aaa/bbb | /aaa/bbb | Yes |
| Prefix | /aaa/bbb/ | /aaa/bbb | Yes, ignores trailing slash |
| Prefix | /aaa/bbb | /aaa/bbb/ | Yes, matches trailing slash |
| Prefix | /aaa/bbb | /aaa/bbb/ccc | Yes, matches subpath |
| Prefix | /aaa/bbb | /aaa/bbbxyz | No, does not match string prefix |
| Prefix | /, /aaa | /aaa/ccc | Yes, matches /aaa prefix |
| Prefix | /, /aaa, /aaa/bbb | /aaa/bbb | Yes, matches /aaa/bbb prefix |
| Prefix | /, /aaa, /aaa/bbb | /ccc | Yes, matches / prefix |
| Prefix | /aaa | /ccc | No, uses default backend |
| Mixed | /foo (Prefix), /foo (Exact) | /foo | Yes, prefers Exact |
创建 Ingress：
[root@k8sm1 ~]# kubectl apply -f web-ingress.yaml
ingress.networking.k8s.io/web created
查看 Ingress：
[root@k8sm1 ~]# kubectl get ingress -n app
NAME CLASS HOSTS ADDRESS PORTS AGE
web nginx web.stonecoding.net 80 22m
[root@k8sm1 ~]# kubectl describe ingress web -n app
Name: web
Labels: <none>
Namespace: app
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
web.stonecoding.net
/ nginx-service:80 (10.244.1.30:80,10.244.1.31:80,10.244.2.32:80)
Annotations: nginx.ingress.kubernetes.io/rewrite-target: /$1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 50m nginx-ingress-controller Scheduled for sync
Normal Sync 50m nginx-ingress-controller Scheduled for sync
出现 error: endpoints "default-http-backend" not found 报错，需要创建服务 default-http-backend，资源配置清单文件如下：
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: kube-system
spec:
selector:
app: ingress-nginx-controller
ports:
- protocol: TCP
port: 80
targetPort: 80
创建服务：
[root@k8sm1 ~]# kubectl apply -f default-http-backend-service.yaml
service/default-http-backend created
再次查看 Ingress：
[root@k8sm1 ~]# kubectl describe ingress web -n app
Name: web
Labels: <none>
Namespace: app
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
web.stonecoding.net
/ nginx-service:80 (10.244.1.30:80,10.244.1.31:80,10.244.2.32:80)
Annotations: nginx.ingress.kubernetes.io/rewrite-target: /$1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 5m44s nginx-ingress-controller Scheduled for sync
Normal Sync 5m44s nginx-ingress-controller Scheduled for sync
配置好 web.stonecoding.net 域名解析，指向地址 192.168.92.140，即可使用该域名进行访问：
Node
Node 是 Kubernetes 集群的物理节点。
查看所有节点信息：
[root@k8sm1 ~]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8sm1.stonecoding.net Ready control-plane,master 23h v1.23.17 192.168.92.140 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://20.10.23
k8sn1.stonecoding.net Ready <none> 23h v1.23.17 192.168.92.141 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://20.10.23
k8sn2.stonecoding.net Ready <none> 23h v1.23.17 192.168.92.142 <none> CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://20.10.23
查看指定节点信息：
[root@k8sm1 ~]# kubectl describe node k8sm1
Name: k8sm1.stonecoding.net
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=k8sm1.stonecoding.net
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=
node-role.kubernetes.io/master=
node.kubernetes.io/exclude-from-external-load-balancers=
Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"fa:9d:c6:a2:46:3a"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 192.168.92.140
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 21 Aug 2023 11:28:29 +0800
Taints: node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: k8sm1.stonecoding.net
AcquireTime: <unset>
RenewTime: Wed, 23 Aug 2023 14:39:40 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Wed, 23 Aug 2023 08:51:27 +0800 Wed, 23 Aug 2023 08:51:27 +0800 FlannelIsUp Flannel is running on this node
MemoryPressure False Wed, 23 Aug 2023 14:36:58 +0800 Mon, 21 Aug 2023 11:28:20 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 23 Aug 2023 14:36:58 +0800 Mon, 21 Aug 2023 11:28:20 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 23 Aug 2023 14:36:58 +0800 Mon, 21 Aug 2023 11:28:20 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 23 Aug 2023 14:36:58 +0800 Mon, 21 Aug 2023 15:27:04 +0800 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.92.140
Hostname: k8sm1.stonecoding.net
Capacity:
cpu: 2
ephemeral-storage: 17394Mi
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3861292Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 16415037823
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3758892Ki
pods: 110
System Info:
Machine ID: 25ffa45cb6834fb0a7f70f8567485af8
System UUID: 564D0B5B-3F3A-3F0C-7016-A769F6FA5BE5
Boot ID: c11c3bf2-62da-41be-a6a3-4b5c5c2b75af
Kernel Version: 3.10.0-1160.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.23
Kubelet Version: v1.23.17
Kube-Proxy Version: v1.23.17
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (8 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-flannel kube-flannel-ds-h25mg 100m (5%) 0 (0%) 50Mi (1%) 0 (0%) 2d3h
kube-system coredns-6d8c4cb4d-bwwxw 100m (5%) 0 (0%) 70Mi (1%) 170Mi (4%) 2d3h
kube-system coredns-6d8c4cb4d-zp7nr 100m (5%) 0 (0%) 70Mi (1%) 170Mi (4%) 2d3h
kube-system etcd-k8sm1.stonecoding.net 100m (5%) 0 (0%) 100Mi (2%) 0 (0%) 2d3h
kube-system kube-apiserver-k8sm1.stonecoding.net 250m (12%) 0 (0%) 0 (0%) 0 (0%) 2d3h
kube-system kube-controller-manager-k8sm1.stonecoding.net 200m (10%) 0 (0%) 0 (0%) 0 (0%) 2d3h
kube-system kube-proxy-4dl9s 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2d3h
kube-system kube-scheduler-k8sm1.stonecoding.net 100m (5%) 0 (0%) 0 (0%) 0 (0%) 2d3h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 950m (47%) 0 (0%)
memory 290Mi (7%) 340Mi (9%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events: <none>
或者：
[root@k8sm1 ~]# kubectl get nodes k8sm1.stonecoding.net -o yaml
apiVersion: v1
kind: Node
metadata:
annotations:
flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"ca:f3:8a:05:77:ab"}'
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: "true"
flannel.alpha.coreos.com/public-ip: 192.168.92.140
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: "0"
volumes.kubernetes.io/controller-managed-attach-detach: "true"
creationTimestamp: "2023-08-21T03:28:29Z"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: k8sm1.stonecoding.net
kubernetes.io/os: linux
node-role.kubernetes.io/control-plane: ""
node-role.kubernetes.io/master: ""
node.kubernetes.io/exclude-from-external-load-balancers: ""
name: k8sm1.stonecoding.net
resourceVersion: "45356"
uid: c013d0a0-08cd-4afe-abca-3640079df59f
spec:
podCIDR: 10.244.0.0/24
podCIDRs:
- 10.244.0.0/24
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
status:
addresses:
- address: 192.168.92.140
type: InternalIP
- address: k8sm1.stonecoding.net
type: Hostname
allocatable:
cpu: "2"
ephemeral-storage: "16415037823"
hugepages-1Gi: "0"
hugepages-2Mi: "0"
memory: 3758900Ki
pods: "110"
capacity:
cpu: "2"
ephemeral-storage: 17394Mi
hugepages-1Gi: "0"
hugepages-2Mi: "0"
memory: 3861300Ki
pods: "110"
conditions:
- lastHeartbeatTime: "2023-08-22T01:05:38Z"
lastTransitionTime: "2023-08-22T01:05:38Z"
message: Flannel is running on this node
reason: FlannelIsUp
status: "False"
type: NetworkUnavailable
- lastHeartbeatTime: "2023-08-22T03:44:04Z"
lastTransitionTime: "2023-08-21T03:28:20Z"
message: kubelet has sufficient memory available
reason: KubeletHasSufficientMemory
status: "False"
type: MemoryPressure
- lastHeartbeatTime: "2023-08-22T03:44:04Z"
lastTransitionTime: "2023-08-21T03:28:20Z"
message: kubelet has no disk pressure
reason: KubeletHasNoDiskPressure
status: "False"
type: DiskPressure
- lastHeartbeatTime: "2023-08-22T03:44:04Z"
lastTransitionTime: "2023-08-21T03:28:20Z"
message: kubelet has sufficient PID available
reason: KubeletHasSufficientPID
status: "False"
type: PIDPressure
- lastHeartbeatTime: "2023-08-22T03:44:04Z"
lastTransitionTime: "2023-08-21T07:27:04Z"
message: kubelet is posting ready status
reason: KubeletReady
status: "True"
type: Ready
daemonEndpoints:
kubeletEndpoint:
Port: 10250
images:
- names:
- registry.aliyuncs.com/google_containers/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c
- registry.aliyuncs.com/google_containers/etcd:3.5.6-0
sizeBytes: 299475478
- names:
- registry.aliyuncs.com/google_containers/kube-apiserver@sha256:5f6f091e7b3886b9068eb354fe29b4fbd3384e821930e405e47166065e5e859b
- registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.17
sizeBytes: 130012137
- names:
- registry.aliyuncs.com/google_containers/kube-controller-manager@sha256:5495a48e110d4800f2e0dbd27dc2acb7e77cabb1340733519c349e8e04e6566f
- registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.17
sizeBytes: 119948546
- names:
- registry.aliyuncs.com/google_containers/kube-proxy@sha256:4b3d1119f5fdd8230965c758a467bdca7fdef84c05b3f45605095f67bd64b5ff
- registry.aliyuncs.com/google_containers/kube-proxy:v1.23.17
sizeBytes: 110840983
- names:
- flannel/flannel@sha256:c7214e3ce66191e45b8c2808c703a2a5674751e90f0f65aef0b404db0a22400c
- flannel/flannel:v0.22.2
sizeBytes: 70179967
- names:
- registry.aliyuncs.com/google_containers/kube-scheduler@sha256:eb140d570d1f3d441f49c18141a8d81fee03525e7882441cfe8823d81b634659
- registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.17
sizeBytes: 51856655
- names:
- registry.aliyuncs.com/google_containers/coredns@sha256:5b6ec0d6de9baaf3e92d0f66cd96a25b9edbce8716f5f15dcd1a616b3abd590e
- registry.aliyuncs.com/google_containers/coredns:v1.8.6
sizeBytes: 46829283
- names:
- flannel/flannel-cni-plugin@sha256:ca6779c6ad63b77af8a00151cefc08578241197b9a6fe144b0e55484bc52b852
- flannel/flannel-cni-plugin:v1.2.0
sizeBytes: 8038625
- names:
- registry.aliyuncs.com/google_containers/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db
- registry.aliyuncs.com/google_containers/pause:3.6
sizeBytes: 682696
nodeInfo:
architecture: amd64
bootID: 8e821c8e-7934-4402-ad15-461d87588c60
containerRuntimeVersion: docker://20.10.23
kernelVersion: 3.10.0-1160.el7.x86_64
kubeProxyVersion: v1.23.17
kubeletVersion: v1.23.17
machineID: 25ffa45cb6834fb0a7f70f8567485af8
operatingSystem: linux
osImage: CentOS Linux 7 (Core)
systemUUID: 564D0B5B-3F3A-3F0C-7016-A769F6FA5BE5
查看节点的 Label：
[root@k8sm1 ~]# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
k8sm1.stonecoding.net Ready control-plane,master 24h v1.23.17 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8sm1.stonecoding.net,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
k8sn1.stonecoding.net Ready <none> 24h v1.23.17 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8sn1.stonecoding.net,kubernetes.io/os=linux
k8sn2.stonecoding.net Ready <none> 24h v1.23.17 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8sn2.stonecoding.net,kubernetes.io/os=linux
维护
维护节点的步骤如下：
- 使用
kubectl cordon命令将节点标记为不可调度
[root@k8sm1 ~]# kubectl cordon k8sn2.stonecoding.net
node/k8sn2.stonecoding.net cordoned
- 使用
kubectl drain安全驱逐节点上面所有 Pod 到其他节点
[root@k8sm1 ~]# kubectl drain k8sn2.stonecoding.net --force --ignore-daemonsets --delete-emptydir-data
node/k8sn2.stonecoding.net already cordoned
WARNING: ignoring DaemonSet-managed Pods: ingress-nginx/ingress-nginx-controller-m5qqj, kube-flannel/kube-flannel-ds-5tf8x, kube-system/kube-proxy-t62d9
evicting pod kubernetes-dashboard/kubernetes-dashboard-fb8648fd9-t4c7k
evicting pod app/nginx-deployment-v118-69584f9576-d4cw6
evicting pod app/nginx-deployment-9456bbbf9-5w9qt
evicting pod ingress-nginx/ingress-nginx-admission-patch-gh9p5
pod/ingress-nginx-admission-patch-gh9p5 evicted
pod/nginx-deployment-9456bbbf9-5w9qt evicted
pod/kubernetes-dashboard-fb8648fd9-t4c7k evicted
pod/nginx-deployment-v118-69584f9576-d4cw6 evicted
node/k8sn2.stonecoding.net drained
- 维护完成后，使用
kubectl uncordon恢复节点可调度
[root@k8sm1 ~]# kubectl uncordon k8sn2.stonecoding.net
node/k8sn2.stonecoding.net uncordoned
排错
查看日志：
[root@k8sm1 ~]# kubectl logs -f nginx-deployment-9456bbbf9-5w9qt -n app
