Install and Configure Kubernetes (k8s) on debian10
跳到导航
跳到搜索
目录
- 1 思路2020
- 2 info
- 3 FQ
- 4 pre
- 5 Set Hostname and update hosts file
- 6 翻墙
- 7 ins docker
- 8 docker代理设置
- 9 ins 在所有节点上
- 10 在 Master 节点上配置 kubelet 所需的 cgroup 驱动
- 11 初始化master
- 12 配置kubectl认证信息
- 13 安装pod网络on master
- 14 添加节点
- 15 master 也当作node
- 16 下面的是不是可以不要翻墙了呢
- 17 chpater4 k8s architecture
- 18 下面 关闭ss docker 代理 polipo
- 19 chapter 5 run apps
- 20 等待
- 21 进阶
- 22 What is new
- 23 trouble
- 24 see also
- 25 see also
- 26 see also
思路2020
先在围墙外的机器 pull下来 然后 push到自己的hub.docker 最后在内网的机器再pull 下来 再tag一下
初始化时 指定aliyun mirrors 本来是指定 1。17。1版本的 我改了新的 kubeadm init --apiserver-advertise-address=192.168.11.184 --image-repository registry.aliyuncs.com/google_containers --ignore-preflight-errors=all --kubernetes-version v1.17.3 --service-cidr=10.96.0.0/16 --pod-network-cidr=10.244.0.0/16
info
cat >>/etc/hosts <<EOF 192.168.11.184 k8s-master 192.168.88.31 k8s-node1 192.168.88.32 k8s-node2 EOF 每台机器最少2GB内存，2CPUs。 集群中所有机器之间网络连接正常。 打开相应的端口，详见： [ Check required ports https://kubernetes.io/docs/setup/independent/install-kubeadm/#check-required-ports] Kubernetes要求集群中所有机器具有不同的Mac地址、产品uuid、Hostname。可以使用如下命令查看： # UUID cat /sys/class/dmi/id/product_uuid # Mac地址 ip link
FQ
Ubuntu利用shadowsocks和polipo终端翻墙
pre
搞个离线的吧 可以先学习着
Set Hostname and update hosts file
sudo hostnamectl set-hostname "k8s-master" sudo hostnamectl set-hostname k8s-node1 sudo hostnamectl set-hostname k8s-node2 #Add the following lines in /etc/hosts file on all three systems,
翻墙
Ubuntu利用shadowsocks和polipo终端翻墙
cat /etc/profile #最好就是这个写成 polipo那台机器的IP就行了，其它机器就不用再搞 ss polipo了 #这 这里填写 polipo那台机器的ip export http_proxy="http://polipoIP:8123/" export https_proxy=$http_proxy #export no_proxy="localhost,127.0.0.1,192.168.88.58,10.96.0.0,10.224.0.0" export no_proxy="localhost,127.0.0.1,192.168.88.58,10.96.0.0,10.224.0.0,10.224.*"
如果不想翻墙 请参考使用Kubeadm搭建Kubernetes(1.12.2)集群
ins docker
#docker-compose 直接用官方的二进制包
debian9 or 10
.Install using the repository on debian
apt install software-properties-common
apt-get remove docker docker-engine docker.io containerd runc
sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common -y
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable"
apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
https://docs.docker.com/install/linux/docker-ce/debian/
2.install-from-a-package on debian
Go to https://download.docker.com/linux/debian/dists/, choose your Debian version, browse to pool/stable/, choose either amd64 or armhf, and download the .deb file for the Docker CE version you want to install.
I am stretch so
apt install libltdl7
http://mirrors.aliyun.com/docker-ce/linux/debian/dists/stretch/pool/stable/amd64/
docker代理设置
#不要少了开头的service 还要记得check一个代理成功不 昨天就是 另外两台机器 也写了 127.0.0.1 丢人 mkdir -p /etc/systemd/system/docker.service.d vi /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTPS_PROXY=http://127.0.0.1:8123/" "HTTP_PROXY=http://127.0.0.1:8123/" "NO_PROXY=localhost,127.0.0.1,192.168.88.67,10.96.0.0,10.224.0.0" #Environment="HTTP_PROXY=http://proxy.example.com:80/" "HTTPS_PROXY=http://proxy.example.com:80/""NO_PROXY=localhost,127.0.0.1,docker-registry.somecorporation.com" systemctl daemon-reload systemctl restart docker systemctl enable docker systemctl show --property=Environment docker other evan@k8s-master:~$ sudo systemctl enable docker Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable docker
ins 在所有节点上
swapoff -a; sudo usermod -a -G docker $USER
修改docker cgroup driver为systemd
根据文档CRI installation中的内容，对于使用systemd作为init system的Linux的发行版，使用systemd作为docker的cgroup driver可以确保服务器节点在资源紧张的情况更加稳定，因此这里修改各个节点上docker的cgroup driver为systemd。
创建或修改/etc/docker/daemon.json：
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
重启docker：
systemctl restart docker
docker info | grep Cgroup
Cgroup Driver: systemd
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
#init 之前不要启动
#systemctl start kubelet&& systemctl enable kubelet.service
启动不了
原来是kubelet 的cgroup dirver 与 docker的不一样。docker默认使用cgroupfs,keubelet 默认使用systemd。
https://kubernetes.io/docs/setup/cri/
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl restart docker
#这个有改的 18.04上成功了的
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
systemctl daemon-reload && systemctl restart kubelet && systemctl enable kubelet.service
在 Master 节点上配置 kubelet 所需的 cgroup 驱动
使用 Docker 时，kubeadm 会自动为其检测 cgroup 驱动在运行时对 /var/lib/kubelet/kubeadm-flags.env 文件进行配置。 如果您使用了不同的 CRI， 您得把 /etc/default/kubelet 文件中的 cgroup-driver 位置改为对应的值，像这样： KUBELET_EXTRA_ARGS=--cgroup-driver=<value> 这个文件将会被 kubeadm init 和 kubeadm join 用于为 kubelet 获取 额外的用户参数。 请注意，您只需要在您的 cgroup driver 不是 cgroupfs 时这么做，因为 cgroupfs 已经是 kubelet 的默认值了。 systemctl daemon-reload; systemctl restart kubelet #需要重启 kubelet： /etc/systemd/system/kubelet.service.d/10-kubeadm.conf This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) #me 2020 evan@k8s-master:~$ cat /var/lib/kubelet/kubeadm-flags.env KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf
初始化master
#可以用国内阿里节点 不用FQ了
kubeadm init --apiserver-advertise-address=192.168.11.184 --image-repository registry.aliyuncs.com/google_containers --ignore-preflight-errors=all --kubernetes-version v1.17.1 --service-cidr=10.96.0.0/16 --pod-network-cidr=10.244.0.0/16
#14:25:52--14:47:55 kubelet 其实是没启动的 在init之前
kubeadm init --apiserver-advertise-address=192.168.88.30 --pod-network-cidr=10.224.0.0/16 # --apiserver-advertise-address=masterip
kubeadm join 192.168.88.58:6443 --token fuwhe0.ro0c8u82u4xtmn8q \
--discovery-token-ca-cert-hash sha256:83bd9c19486c44fde674f4ccf0a7382848cd7bfeff8c361d54e7a2955a4dbd60
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
另外有一个小技巧，在init的过程中，另开一个终端，运行
journalctl -f -u kubelet.service
可以查看具体是什么愿意卡住了
配置kubectl认证信息
cat /etc/sudoers.d/evan echo 'evan ALL=(ALL) NOPASSWD:NOPASSWD:ALL' > /etc/sudoers.d/evan su - evan mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config echo "source <(kubectl completion bash)" >> ~/.bashrc exit # 对于root用户 这省不能少 不然 # kubectl apply -f kube-flannel.yml The connection to the server localhost:8080 was refused - did you specify the right host or port? export KUBECONFIG=/etc/kubernetes/admin.conf #也可以直接放到~/.bash_profile echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
安装pod网络on master
- 普通用户 不要翻墙
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
添加节点
不要翻墙了 新起个窗口
# on node
kubeadm join 192.168.88.58:6443 --token fuwhe0.ro0c8u82u4xtmn8q \
--discovery-token-ca-cert-hash sha256:83bd9c19486c44fde674f4ccf0a7382848cd7bfeff8c361d54e7a2955a4dbd60
evan@k8s-master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s NotReady master 5h12m v1.14.2
u16 NotReady <none> 106m v1.14.2
evan@k8s-master:~$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-fb8b8dccf-nprqq 0/1 Terminating 16 5h11m
kube-system coredns-fb8b8dccf-qn85f 0/1 Pending 0 5m4s
kube-system coredns-fb8b8dccf-sgtw4 0/1 Terminating 16 5h11m
kube-system coredns-fb8b8dccf-wsnkg 0/1 Pending 0 5m5s
kube-system etcd-k8s 1/1 Running 0 5h11m
kube-system kube-apiserver-k8s 1/1 Running 0 5h11m
kube-system kube-controller-manager-k8s 1/1 Running 0 5h11m
kube-system kube-flannel-ds-amd64-8vvn6 0/1 Init:0/1 0 107m
kube-system kube-flannel-ds-amd64-q92vz 1/1 Running 0 112m
kube-system kube-proxy-85vkt 0/1 ContainerCreating 0 107m
kube-system kube-proxy-fr7lv 1/1 Running 0 5h11m
kube-system kube-scheduler-k8s 1/1 Running 0 5h11m
evan@k8s-master:~$ kubectl describe pod kube-proxy-85vkt --namespace=kube-system
Name: kube-proxy-85vkt
Namespace: kube-system
Priority: 2000001000
PriorityClassName: system-node-critical
Node: u16/192.168.88.66
****
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 109m default-scheduler Successfully assigned kube-system/kube-proxy-85vkt to u16
Normal Pulling 108m kubelet, u16 Pulling image "k8s.gcr.io/kube-proxy:v1.14.2"
Normal Pulled 107m kubelet, u16 Successfully pulled image "k8s.gcr.io/kube-proxy:v1.14.2"
Normal Created 107m kubelet, u16 Created container kube-proxy
Normal Started 107m kubelet, u16 Started container kube-proxy
Warning FailedCreatePodSandBox 52m (x119 over 107m) kubelet, u16 Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
放了一个晚上 早上还是坏的 突然打开已是好的了
evan@ubuntu18:~$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-fb8b8dccf-2rbwc 1/1 Running 3 18h
kube-system coredns-fb8b8dccf-67zc2 1/1 Running 3 18h
kube-system etcd-ubuntu18 1/1 Running 10 18h
kube-system kube-apiserver-ubuntu18 1/1 Running 4 18h
kube-system kube-controller-manager-ubuntu18 1/1 Running 5 18h
kube-system kube-flannel-ds-amd64-b6bn8 1/1 Running 45 16h
kube-system kube-flannel-ds-amd64-v9wxm 1/1 Running 46 16h
kube-system kube-flannel-ds-amd64-zn4xd 1/1 Running 3 16h
kube-system kube-proxy-d7pmb 1/1 Running 4 18h
kube-system kube-proxy-gcddr 1/1 Running 0 16h
kube-system kube-proxy-lv8cb 1/1 Running 0 16h
kube-system kube-scheduler-ubuntu18 1/1 Running 5 18h
master 也当作node 这里的master hostname 为 ubuntu18OB
evan@ubuntu18:~$ kubectl taint node ubuntu18 node-role.kubernetes.io/master-
node/ubuntu18 untainted
#master only
kubectl taint node ubuntu18 node-role.kubernetes.io/master="":NoSchedule
master 也当作node
[root@master tomcat]# hostname master [root@master tomcat]# kubectl taint node master node-role.kubernetes.io/master- node/master untainted
下面的是不是可以不要翻墙了呢
chpater4 k8s architecture
#唯一不是容器形式运行的k8s 组件
evan@k8s-master:~$ sudo systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Mon 2019-05-27 07:26:18 UTC; 21min ago
Docs: https://kubernetes.io/docs/home/
Main PID: 817 (kubelet)
Tasks: 19 (limit: 3499)
CGroup: /system.slice/kubelet.service
└─817 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf -
在master节点上发起个创建应用请求
这里我们创建个名为httpd-app的应用，镜像为httpd，有两个副本pod
evan@k8s-master:~$ kubectl run httpd-app --image=httpd --replicas=2
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/httpd-app created
evan@k8s-master:~$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
httpd-app 0/2 2 0 103s
evan@k8s-master:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-app-6df58645c6-bvg9w 0/1 ContainerCreating 0 2m10s <none> k8s-node1 <none> <none>
httpd-app-6df58645c6-n9xdj 0/1 ContainerCreating 0 2m10s <none> k8s-node2 <none> <none>
evan@k8s-master:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-app-6df58645c6-bvg9w 0/1 ContainerCreating 0 3m58s <none> k8s-node1 <none> <none>
httpd-app-6df58645c6-n9xdj 1/1 Running 0 3m58s 10.224.1.2 k8s-node2 <none> <none>
#OK了
evan@k8s-master:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-app-6df58645c6-bvg9w 1/1 Running 0 6m8s 10.224.2.3 k8s-node1 <none> <none>
httpd-app-6df58645c6-n9xdj 1/1 Running 0 6m8s 10.224.1.2 k8s-node2 <none> <none>
下面 关闭ss docker 代理 polipo
chapter 5 run apps
evan@k8s-master:~$ kubectl run nginx-deployment --image=nginx:1.7.9 --replicas=2 kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. deployment.apps/nginx-deployment created 上面的命令将部署包含两个副本的 Deployment nginx-deployment，容器的 image 为 nginx:1.7.9。 等待一段时间 kubectl get deployment nginx-deployment NAME READY UP-TO-DATE AVAILABLE AGE nginx-deployment 2/2 2 2 36m 接下来我们用 kubectl describe deployment 了解更详细的信息
等待
sudo sslocal -c /root/shadowsocks.json -d start sslocal -c shadowsocks.json -d start sslocal -c shadowsocks.json -d start
进阶
K8S 源码探秘 之 kubeadm init 执行流程分析
What is new
在Kubernetes 1.11中，CoreDNS已经实现了基于DNS的服务发现的GA，可作为kube-dns插件的替代品。这意味着CoreDNS将作为各种安装工具未来发布版本中的一个选项来提供。 事实上，kubeadm团队选择将其作为Kubernetes 1.11的默认选项。
CoreDNS正式GA | kube-dns与CoreDNS有何差异？
trouble
Kubenetes服务不启动问题
重启系统后，发现kubelet服务没有起来，首先检查： 1.vim /etc/fstab #注释掉里面的swap一行。 2 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 文件加入KUBELET_CGROUP_ARGS和KUBELET_EXTRA_ARGS参数， 3.注意在启动参数中也要加入，如下： [Service] Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd" Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false" ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_CGROUP_ARGS $KUBELET_EXTRA_ARGS systemctl daemon-reload systemctl restart kubelet
trouble2 重启一下机器就坏
为什么重启一下机器就坏了呢
systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Fri 2019-05-24 20:27:22 CST; 1s ago
Docs: https://kubernetes.io/docs/home/
Process: 1889 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (cod
Main PID: 1889 (code=exited, status=255)
kubelet.service: Main process exited, code=exited, status=255
journalctl -xefu kubelet
原来是kubelet 的cgroup dirver 与 docker的不一样。docker默认使用cgroupfs,keubelet 默认使用systemd。
简单地说就是在kubeadm init 之前kubelet会不断重启。
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
在集群初始化遇到问题，可以使用下面的命令进行清理后重新再初始化：
kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/
K8S 初始化问题，有哪位遇到过，求解！timed out waiting for the condition
trouble3
evan@k8s-master:~$ docker pull gcr.io/kubernetes-helm/tiller:v2.14.0
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.39/images/create?fromImage=gcr.io%2Fkubernetes-helm%2Ftiller&tag=v2.14.0: dial unix /var/run/docker.sock: connect: permission denied
sudo usermod -a -G docker $USER #普通用户添加天docker 组
Docker pull Get Permission Denied
trouble 3
docker 223.6.6.6 有时有问题 建议用8.8.4.4
see also
使用Kubeadm搭建Kubernetes(1.12.2)集群
Debian 9 使用kubeadm创建 k8s 集群（上）
Debian 9 使用kubeadm创建 k8s 集群（下）
Install and Configure Kubernetes (k8s) 1.13 on Ubuntu 18.04 LTS / Ubuntu 18.10
Ubuntu 18.04 离线安装Kubernetes v1.11.1
see also
https://www.kubernetes.org.cn/course/install
Install and Configure Kubernetes (k8s) on ubuntu
docker/kubernetes国内源/镜像源解决方式 k8s常见报错解决--持续更新
