查看“︁K8s trouble shooting”︁的源代码
←
K8s trouble shooting
跳转到导航
跳转到搜索
因为以下原因,您没有权限编辑该页面:
您请求的操作仅限属于该用户组的用户执行:
用户
您可以查看和复制此页面的源代码。
==summy== <pre> 总结:症状 → 第一条命令 → 根因 症状 第一条命令 看什么 典型根因 ImagePullBackOff describe pod Events tag 错 / 无权限 / 网络 Pending describe pod FailedScheduling requests 过大 / taint / PVC CrashLoopBackOff logs --previous 应用报错 + Exit Code 配置 / 参数 / 依赖 OOMKilled describe pod Reason + Exit Code 137 limit 太小 / 内存泄漏 Running 但 0/1 describe pod Readiness probe failed 探针路径/端口/超时 Pod 正常但不通 get endpoints 是否为空 selector / targetPort 错 三个核心习惯 先 describe 看 Events,再 logs:很多问题(调度、拉镜像、OOM、探针)日志里根本没有。 改坏了先 rollout undo:生产环境先恢复服务,再慢慢查根因。 从流量路径逐层排查:Pod → Endpoints → Service → Ingress,每层单独验证,用"绕过上一层直连"的方法缩小范围。 如果想再进阶,可以继续练:ConfigMap/Secret 缺失(CreateContainerConfigError)、PVC Pending、NetworkPolicy 误拦截、节点 NotReady。需要的话我可以按同样格式再出一套 </pre> ==6场景 6:Pod 全正常,但 Service 访问不通(selector 不匹配)== <pre> 这是最容易让人困惑的一类:Pod 全是 1/1 Running,但就是访问不了。 制造故障 kubectl -n lab patch svc web -p '{"spec":{"selector":{"app":"web-v2"}}}' 现象 kubectl -n lab run curl --rm -it --restart=Never --image=curlimages/curl -- curl -sv --max-time 5 http://web # 超时 / Connection refused 排查(按流量路径逐层验证) bash # 1. Service 有没有 Endpoints? kubectl -n lab get endpoints web # NAME ENDPOINTS AGE # web <none> 10m ← 空!问题就在 Service 到 Pod 这一层 # 2. 对比 Service 的 selector 和 Pod 的 labels kubectl -n lab get svc web -o jsonpath='{.spec.selector}{"\n"}' kubectl -n lab get pods --show-labels # selector 是 app=web-v2,Pod 的 label 是 app=web → 对不上 ➜ ~ kubectl -n lab get svc web -o jsonpath='{.spec.selector}{"\n"}' {"app":"web-v2"} ➜ ~ kubectl -n lab get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS oom-94fdcb47f-sfpzw 1/1 Running 0 9h app=oom,pod-template-hash=94fdcb47f web-7bccbc594-k969p 1/1 Running 0 9h app=web,pod-template-hash=7bccbc594 web-7bccbc594-wgql9 1/1 Running 0 9h app=web,pod-template-hash=7bccbc594 # 3. 绕过 Service,直连 Pod IP 验证 Pod 本身是好的 kubectl -n lab get pods -o wide kubectl -n lab run curl --rm -it --restart=Never --image=curlimages/curl -- curl -sI http://<pod-ip> # 200 OK → 证明 Pod 没问题,故障在 Service 层 kubectl -n lab run curl --rm -it --restart=Never --image=curlimages/curl -- curl -sI http://10.42.2.4 All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt. If you don't see a command prompt, try pressing enter. warning: couldn't attach to pod/curl, falling back to streaming logs: unable to upgrade connection: container curl not found in pod curl_lab HTTP/1.1 200 OK Server: nginx/1.25.5 恢复 kubectl -n lab patch svc web -p '{"spec":{"selector":{"app":"web"}}}' kubectl -n lab get endpoints web # 两个 IP 回来了 Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice NAME ENDPOINTS AGE web 10.42.2.251:80,10.42.2.4:80 23h 最终验证 & 清理 # 全部恢复后的健康检查 kubectl -n lab get deploy,pods,svc,endpoints kubectl -n lab rollout history deploy/web # 能看到刚才折腾的所有版本 # 清理 kubectl delete ns lab </pre> == 5== <pre> 场景 5:Running 但 0/1(Readiness 探针失败) 制造故障:探针路径写错 bash kubectl -n lab patch deploy web --type=json -p='[ {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/healthz-wrong"} ]' 现象 bash kubectl -n lab get pods # STATUS 是 Running,但 READY 是 0/1,没有重启 kubectl -n lab get endpoints web # ENDPOINTS 对新 Pod 为空 💡 对比:liveness 失败会重启容器;readiness 失败只会摘除流量,不重启。所以这里 RESTARTS 是 0。 排查 bash kubectl -n lab describe pod <pod> | grep -A5 Events Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 404 手动验证探针到底返回什么: bash kubectl -n lab exec <pod> -- curl -s -o /dev/null -w "%{http_code}\n" localhost/healthz-wrong # 404 kubectl -n lab exec <pod> -- curl -s -o /dev/null -w "%{http_code}\n" localhost/ # 200 ➜ k8s kubectl -n lab exec web-6d6594866-2c7rn -- curl -s -o /dev/null -w "%{http_code}\n" localhost/healthz-wrong 404 ➜ k8s kubectl -n lab exec web-6d6594866-2c7rn -- curl -s -o /dev/null -w "%{http_code}\n" localhost/ 200 结论:应用没问题,是探针路径配错。 恢复 bash kubectl -n lab rollout undo deploy/web kubectl -n lab rollout status deploy/web </pre> ==last== <pre> kubectl -n lab describe pod web-6d78696589-z776p | grep -A8 "Last State" Last State: Terminated Reason: Error Exit Code: 1 Started: Thu, 19 Nov 2026 14:32:38 +0800 Finished: Thu, 19 Nov 2026 14:32:38 +0800 Ready: False Restart Count: 36 Limits: cpu: 100 kubectl describe pod 拉取该 Pod 的全量状态 + 事件,比 get 详细得多 -n lab 指定命名空间 lab,不加就默认去 default 找,找不到会报 NotFound web-6d78696589-z776p Pod 名。中间 6d78696589 是 ReplicaSet 的哈希,后缀 z776p 是随机串——说明这个 Pod 由 Deployment web 管理,不是裸跑的 grep -A8 "Last State" 匹配到 Last State 那行,并附带其后 8 行(-A = After context)。这就是为什么输出正好从 Last State 截到 Limits 小技巧:换成 grep -B2 -A12 "Last State" 能看到上下文更全,包括当前 State 和容器名。 Last State vs State K8s 给每个容器留两个槽位:State 是当前,Last State 是上一次终止那次的。出现 Last State 有内容,本身就说明这容器重启过。你现在看的不是它此刻的样子,是它的"遗照" 2. 如果 Pod 有多个容器,指定容器名 kubectl -n lab logs web-6d78696589-z776p -c web --previous resume kubectl -n lab rollout status deploy/web Exit Code 典型含义 1 通用失败。配置错、连不上依赖、参数错、脚本里某条命令失败——绝大数业务崩溃长这样 137 128+9,收到 SIGKILL,几乎等于 OOM 被杀 或被人为 kill -9 143 128+15,收到 SIGTERM,正常优雅终止(如滚动更新) 126 / 127 命令不可执行 / 命令找不到(镜像里没这个二进制) 0 正常退出。但主进程正常退出对长期服务是坏事 → K8s 会不断重启它 </pre> =rollout before version= <pre> k8s kubectl -n lab patch deploy web --type=json -p='[ {"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"50m"} ]' kubectl -n lab rollout status deploy/web ➜ k8s kubectl -n lab rollout status deploy/web error: deployment "web" exceeded its progress deadline ➜ k8s kubectl -n lab get deploy, rs,pods error: arguments in resource/name form must have a single resource and name ➜ k8s kubectl -n lab get deploy,rs,pods NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/web 2/2 1 2 3h31m NAME DESIRED CURRENT READY AGE replicaset.apps/web-65c6db8bd4 0 0 0 3h23m replicaset.apps/web-6d78696589 1 1 0 179m replicaset.apps/web-7bccbc594 2 2 2 3h1m replicaset.apps/web-b8d8c8889 0 0 0 3h31m replicaset.apps/web-d5477b897 0 0 0 3h10m NAME READY STATUS RESTARTS AGE pod/web-6d78696589-z776p 0/1 CrashLoopBackOff 40 (51s ago) 179m pod/web-7bccbc594-6w65z 1/1 Running 0 3h1m pod/web-7bccbc594-92rgn 1/1 Running 0 3h1m ➜ k8s kubectl -n lab rollout history deploy/web deployment.apps/web REVISION CHANGE-CAUSE 2 <none> 3 <none> 4 <none> 5 <none> 6 <none> ➜ k8s kubectl -n lab get rs -o custom-columns=NAME:.metadata.name,REV:.metadata.annotations.deployment\\.kubernetes\\.io/revision,DESIRED:.spec.replicas,READY:.status.readyReplicas NAME REV DESIRED READY web-65c6db8bd4 2 0 <none> web-6d78696589 6 1 <none> web-7bccbc594 5 2 2 web-b8d8c8889 3 0 <none> web-d5477b897 4 0 <none> ➜ k8s kubectl -n lab rollout history deploy/web --revision=5 deployment.apps/web with revision #5 Pod Template: Labels: app=web pod-template-hash=7bccbc594 Containers: nginx: Image: nginx:1.25 Port: 80/TCP Host Port: 0/TCP Limits: cpu: 100 memory: 128Mi Requests: cpu: 50m memory: 64Mi Readiness: http-get http://:80/ delay=3s timeout=1s period=5s #success=1 #failure=3 Environment: <none> Mounts: <none> Volumes: <none> Node-Selectors: <none> Tolerations: <none> ➜ k8s kubectl -n lab get pod NAME READY STATUS RESTARTS AGE web-6d78696589-z776p 0/1 CrashLoopBackOff 40 (4m34s ago) 3h3m web-7bccbc594-6w65z 1/1 Running 0 3h4m web-7bccbc594-92rgn 1/1 Running 0 3h5m ➜ k8s kubectl -n lab rollout history deploy/web --revision=5 deployment.apps/web with revision #5 Pod Template: Labels: app=web pod-template-hash=7bccbc594 Containers: nginx: Image: nginx:1.25 Port: 80/TCP Host Port: 0/TCP Limits: cpu: 100 memory: 128Mi Requests: cpu: 50m memory: 64Mi Readiness: http-get http://:80/ delay=3s timeout=1s period=5s #success=1 #failure=3 Environment: <none> Mounts: <none> Volumes: <none> Node-Selectors: <none> Tolerations: <none> ➜ k8s kubectl -n lab rollout undo deploy/web --to-revision=5 deployment.apps/web rolled back ➜ k8s kubectl -n lab get deploy,rs,pods NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/web 2/2 2 2 3h36m NAME DESIRED CURRENT READY AGE replicaset.apps/web-65c6db8bd4 0 0 0 3h28m replicaset.apps/web-6d78696589 0 0 0 3h4m replicaset.apps/web-7bccbc594 2 2 2 3h6m replicaset.apps/web-b8d8c8889 0 0 0 3h36m replicaset.apps/web-d5477b897 0 0 0 3h15m NAME READY STATUS RESTARTS AGE pod/web-7bccbc594-6w65z 1/1 Running 0 3h6m pod/web-7bccbc594-92rgn 1/1 Running 0 3h6m </pre>
返回
K8s trouble shooting
。
导航菜单
个人工具
登录
命名空间
页面
讨论
大陆简体
查看
阅读
查看源代码
查看历史
更多
搜索
导航
首页
我的导航
关于我
shell
python
ops
linuxchina.net
blog.linuxchina
最近更改
随机页面
帮助
工具
链入页面
相关更改
特殊页面
页面信息