K8s trouble shooting
跳转到导航
跳转到搜索
summy
总结:症状 → 第一条命令 → 根因 症状 第一条命令 看什么 典型根因 ImagePullBackOff describe pod Events tag 错 / 无权限 / 网络 Pending describe pod FailedScheduling requests 过大 / taint / PVC CrashLoopBackOff logs --previous 应用报错 + Exit Code 配置 / 参数 / 依赖 OOMKilled describe pod Reason + Exit Code 137 limit 太小 / 内存泄漏 Running 但 0/1 describe pod Readiness probe failed 探针路径/端口/超时 Pod 正常但不通 get endpoints 是否为空 selector / targetPort 错 三个核心习惯 先 describe 看 Events,再 logs:很多问题(调度、拉镜像、OOM、探针)日志里根本没有。 改坏了先 rollout undo:生产环境先恢复服务,再慢慢查根因。 从流量路径逐层排查:Pod → Endpoints → Service → Ingress,每层单独验证,用"绕过上一层直连"的方法缩小范围。 如果想再进阶,可以继续练:ConfigMap/Secret 缺失(CreateContainerConfigError)、PVC Pending、NetworkPolicy 误拦截、节点 NotReady。需要的话我可以按同样格式再出一套
6场景 6:Pod 全正常,但 Service 访问不通(selector 不匹配)
这是最容易让人困惑的一类:Pod 全是 1/1 Running,但就是访问不了。
制造故障
kubectl -n lab patch svc web -p '{"spec":{"selector":{"app":"web-v2"}}}'
现象
kubectl -n lab run curl --rm -it --restart=Never --image=curlimages/curl -- curl -sv --max-time 5 http://web
# 超时 / Connection refused
排查(按流量路径逐层验证)
bash
# 1. Service 有没有 Endpoints?
kubectl -n lab get endpoints web
# NAME ENDPOINTS AGE
# web <none> 10m ← 空!问题就在 Service 到 Pod 这一层
# 2. 对比 Service 的 selector 和 Pod 的 labels
kubectl -n lab get svc web -o jsonpath='{.spec.selector}{"\n"}'
kubectl -n lab get pods --show-labels
# selector 是 app=web-v2,Pod 的 label 是 app=web → 对不上
➜ ~ kubectl -n lab get svc web -o jsonpath='{.spec.selector}{"\n"}'
{"app":"web-v2"}
➜ ~ kubectl -n lab get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
oom-94fdcb47f-sfpzw 1/1 Running 0 9h app=oom,pod-template-hash=94fdcb47f
web-7bccbc594-k969p 1/1 Running 0 9h app=web,pod-template-hash=7bccbc594
web-7bccbc594-wgql9 1/1 Running 0 9h app=web,pod-template-hash=7bccbc594
# 3. 绕过 Service,直连 Pod IP 验证 Pod 本身是好的
kubectl -n lab get pods -o wide
kubectl -n lab run curl --rm -it --restart=Never --image=curlimages/curl -- curl -sI http://<pod-ip>
# 200 OK → 证明 Pod 没问题,故障在 Service 层
kubectl -n lab run curl --rm -it --restart=Never --image=curlimages/curl -- curl -sI http://10.42.2.4
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/curl, falling back to streaming logs: unable to upgrade connection: container curl not found in pod curl_lab
HTTP/1.1 200 OK
Server: nginx/1.25.5
恢复
kubectl -n lab patch svc web -p '{"spec":{"selector":{"app":"web"}}}'
kubectl -n lab get endpoints web # 两个 IP 回来了
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME ENDPOINTS AGE
web 10.42.2.251:80,10.42.2.4:80 23h
最终验证 & 清理
# 全部恢复后的健康检查
kubectl -n lab get deploy,pods,svc,endpoints
kubectl -n lab rollout history deploy/web # 能看到刚才折腾的所有版本
# 清理
kubectl delete ns lab
5
场景 5:Running 但 0/1(Readiness 探针失败)
制造故障:探针路径写错
bash
kubectl -n lab patch deploy web --type=json -p='[
{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/healthz-wrong"}
]'
现象
bash
kubectl -n lab get pods
# STATUS 是 Running,但 READY 是 0/1,没有重启
kubectl -n lab get endpoints web
# ENDPOINTS 对新 Pod 为空
💡 对比:liveness 失败会重启容器;readiness 失败只会摘除流量,不重启。所以这里 RESTARTS 是 0。
排查
bash
kubectl -n lab describe pod <pod> | grep -A5 Events
Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 404
手动验证探针到底返回什么:
bash
kubectl -n lab exec <pod> -- curl -s -o /dev/null -w "%{http_code}\n" localhost/healthz-wrong # 404
kubectl -n lab exec <pod> -- curl -s -o /dev/null -w "%{http_code}\n" localhost/ # 200
➜ k8s kubectl -n lab exec web-6d6594866-2c7rn -- curl -s -o /dev/null -w "%{http_code}\n" localhost/healthz-wrong
404
➜ k8s kubectl -n lab exec web-6d6594866-2c7rn -- curl -s -o /dev/null -w "%{http_code}\n" localhost/
200
结论:应用没问题,是探针路径配错。
恢复
bash
kubectl -n lab rollout undo deploy/web
kubectl -n lab rollout status deploy/web
last
kubectl -n lab describe pod web-6d78696589-z776p | grep -A8 "Last State"
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 19 Nov 2026 14:32:38 +0800
Finished: Thu, 19 Nov 2026 14:32:38 +0800
Ready: False
Restart Count: 36
Limits:
cpu: 100
kubectl describe pod
拉取该 Pod 的全量状态 + 事件,比 get 详细得多
-n lab
指定命名空间 lab,不加就默认去 default 找,找不到会报 NotFound
web-6d78696589-z776p
Pod 名。中间 6d78696589 是 ReplicaSet 的哈希,后缀 z776p 是随机串——说明这个 Pod 由 Deployment web 管理,不是裸跑的
grep -A8 "Last State"
匹配到 Last State 那行,并附带其后 8 行(-A = After context)。这就是为什么输出正好从 Last State 截到 Limits
小技巧:换成 grep -B2 -A12 "Last State" 能看到上下文更全,包括当前 State 和容器名。
Last State vs State
K8s 给每个容器留两个槽位:State 是当前,Last State 是上一次终止那次的。出现 Last State 有内容,本身就说明这容器重启过。你现在看的不是它此刻的样子,是它的"遗照"
2. 如果 Pod 有多个容器,指定容器名
kubectl -n lab logs web-6d78696589-z776p -c web --previous
resume
kubectl -n lab rollout status deploy/web
Exit Code 典型含义
1 通用失败。配置错、连不上依赖、参数错、脚本里某条命令失败——绝大数业务崩溃长这样
137 128+9,收到 SIGKILL,几乎等于 OOM 被杀 或被人为 kill -9
143 128+15,收到 SIGTERM,正常优雅终止(如滚动更新)
126 / 127 命令不可执行 / 命令找不到(镜像里没这个二进制)
0 正常退出。但主进程正常退出对长期服务是坏事 → K8s 会不断重启它
rollout before version
k8s kubectl -n lab patch deploy web --type=json -p='[
{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"50m"}
]'
kubectl -n lab rollout status deploy/web
➜ k8s kubectl -n lab rollout status deploy/web
error: deployment "web" exceeded its progress deadline
➜ k8s kubectl -n lab get deploy, rs,pods
error: arguments in resource/name form must have a single resource and name
➜ k8s kubectl -n lab get deploy,rs,pods
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/web 2/2 1 2 3h31m
NAME DESIRED CURRENT READY AGE
replicaset.apps/web-65c6db8bd4 0 0 0 3h23m
replicaset.apps/web-6d78696589 1 1 0 179m
replicaset.apps/web-7bccbc594 2 2 2 3h1m
replicaset.apps/web-b8d8c8889 0 0 0 3h31m
replicaset.apps/web-d5477b897 0 0 0 3h10m
NAME READY STATUS RESTARTS AGE
pod/web-6d78696589-z776p 0/1 CrashLoopBackOff 40 (51s ago) 179m
pod/web-7bccbc594-6w65z 1/1 Running 0 3h1m
pod/web-7bccbc594-92rgn 1/1 Running 0 3h1m
➜ k8s kubectl -n lab rollout history deploy/web
deployment.apps/web
REVISION CHANGE-CAUSE
2 <none>
3 <none>
4 <none>
5 <none>
6 <none>
➜ k8s kubectl -n lab get rs -o custom-columns=NAME:.metadata.name,REV:.metadata.annotations.deployment\\.kubernetes\\.io/revision,DESIRED:.spec.replicas,READY:.status.readyReplicas
NAME REV DESIRED READY
web-65c6db8bd4 2 0 <none>
web-6d78696589 6 1 <none>
web-7bccbc594 5 2 2
web-b8d8c8889 3 0 <none>
web-d5477b897 4 0 <none>
➜ k8s kubectl -n lab rollout history deploy/web --revision=5
deployment.apps/web with revision #5
Pod Template:
Labels: app=web
pod-template-hash=7bccbc594
Containers:
nginx:
Image: nginx:1.25
Port: 80/TCP
Host Port: 0/TCP
Limits:
cpu: 100
memory: 128Mi
Requests:
cpu: 50m
memory: 64Mi
Readiness: http-get http://:80/ delay=3s timeout=1s period=5s #success=1 #failure=3
Environment: <none>
Mounts: <none>
Volumes: <none>
Node-Selectors: <none>
Tolerations: <none>
➜ k8s kubectl -n lab get pod
NAME READY STATUS RESTARTS AGE
web-6d78696589-z776p 0/1 CrashLoopBackOff 40 (4m34s ago) 3h3m
web-7bccbc594-6w65z 1/1 Running 0 3h4m
web-7bccbc594-92rgn 1/1 Running 0 3h5m
➜ k8s kubectl -n lab rollout history deploy/web --revision=5
deployment.apps/web with revision #5
Pod Template:
Labels: app=web
pod-template-hash=7bccbc594
Containers:
nginx:
Image: nginx:1.25
Port: 80/TCP
Host Port: 0/TCP
Limits:
cpu: 100
memory: 128Mi
Requests:
cpu: 50m
memory: 64Mi
Readiness: http-get http://:80/ delay=3s timeout=1s period=5s #success=1 #failure=3
Environment: <none>
Mounts: <none>
Volumes: <none>
Node-Selectors: <none>
Tolerations: <none>
➜ k8s kubectl -n lab rollout undo deploy/web --to-revision=5
deployment.apps/web rolled back
➜ k8s kubectl -n lab get deploy,rs,pods
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/web 2/2 2 2 3h36m
NAME DESIRED CURRENT READY AGE
replicaset.apps/web-65c6db8bd4 0 0 0 3h28m
replicaset.apps/web-6d78696589 0 0 0 3h4m
replicaset.apps/web-7bccbc594 2 2 2 3h6m
replicaset.apps/web-b8d8c8889 0 0 0 3h36m
replicaset.apps/web-d5477b897 0 0 0 3h15m
NAME READY STATUS RESTARTS AGE
pod/web-7bccbc594-6w65z 1/1 Running 0 3h6m
pod/web-7bccbc594-92rgn 1/1 Running 0 3h6m