K8s trouble shooting

来自linuxsa wiki
跳转到导航 跳转到搜索

summy


总结:症状 → 第一条命令 → 根因
症状	第一条命令	看什么	典型根因
ImagePullBackOff	describe pod	Events	tag 错 / 无权限 / 网络
Pending	describe pod	FailedScheduling	requests 过大 / taint / PVC
CrashLoopBackOff	logs --previous	应用报错 + Exit Code	配置 / 参数 / 依赖
OOMKilled	describe pod	Reason + Exit Code 137	limit 太小 / 内存泄漏
Running 但 0/1	describe pod	Readiness probe failed	探针路径/端口/超时
Pod 正常但不通	get endpoints	是否为空	selector / targetPort 错
三个核心习惯
先 describe 看 Events,再 logs:很多问题(调度、拉镜像、OOM、探针)日志里根本没有。
改坏了先 rollout undo:生产环境先恢复服务,再慢慢查根因。
从流量路径逐层排查:Pod → Endpoints → Service → Ingress,每层单独验证,用"绕过上一层直连"的方法缩小范围。

如果想再进阶,可以继续练:ConfigMap/Secret 缺失(CreateContainerConfigError)、PVC Pending、NetworkPolicy 误拦截、节点 NotReady。需要的话我可以按同样格式再出一套

6场景 6:Pod 全正常,但 Service 访问不通(selector 不匹配)

这是最容易让人困惑的一类:Pod 全是 1/1 Running,但就是访问不了。

制造故障

kubectl -n lab patch svc web -p '{"spec":{"selector":{"app":"web-v2"}}}'

现象

kubectl -n lab run curl --rm -it --restart=Never --image=curlimages/curl -- curl -sv --max-time 5 http://web
# 超时 / Connection refused



排查(按流量路径逐层验证)
bash
# 1. Service 有没有 Endpoints?
kubectl -n lab get endpoints web
#    NAME   ENDPOINTS   AGE
#    web    <none>      10m          ← 空!问题就在 Service 到 Pod 这一层

# 2. 对比 Service 的 selector 和 Pod 的 labels
kubectl -n lab get svc web -o jsonpath='{.spec.selector}{"\n"}'
kubectl -n lab get pods --show-labels
#    selector 是 app=web-v2,Pod 的 label 是 app=web → 对不上

➜  ~ kubectl -n lab get svc web -o jsonpath='{.spec.selector}{"\n"}'
{"app":"web-v2"}
➜  ~ kubectl -n lab get pods --show-labels
NAME                  READY   STATUS    RESTARTS   AGE   LABELS
oom-94fdcb47f-sfpzw   1/1     Running   0          9h    app=oom,pod-template-hash=94fdcb47f
web-7bccbc594-k969p   1/1     Running   0          9h    app=web,pod-template-hash=7bccbc594
web-7bccbc594-wgql9   1/1     Running   0          9h    app=web,pod-template-hash=7bccbc594


# 3. 绕过 Service,直连 Pod IP 验证 Pod 本身是好的
kubectl -n lab get pods -o wide
kubectl -n lab run curl --rm -it --restart=Never --image=curlimages/curl -- curl -sI http://<pod-ip>
#    200 OK → 证明 Pod 没问题,故障在 Service 层
kubectl -n lab run curl --rm -it --restart=Never --image=curlimages/curl -- curl -sI http://10.42.2.4
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/curl, falling back to streaming logs: unable to upgrade connection: container curl not found in pod curl_lab
HTTP/1.1 200 OK
Server: nginx/1.25.5


恢复

kubectl -n lab patch svc web -p '{"spec":{"selector":{"app":"web"}}}'
kubectl -n lab get endpoints web      # 两个 IP 回来了

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME   ENDPOINTS                     AGE
web    10.42.2.251:80,10.42.2.4:80   23h



最终验证 & 清理

# 全部恢复后的健康检查
kubectl -n lab get deploy,pods,svc,endpoints
kubectl -n lab rollout history deploy/web      # 能看到刚才折腾的所有版本

# 清理
kubectl delete ns lab

5


场景 5:Running 但 0/1(Readiness 探针失败)
制造故障:探针路径写错
bash
kubectl -n lab patch deploy web --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/healthz-wrong"}
]'
现象
bash
kubectl -n lab get pods
# STATUS 是 Running,但 READY 是 0/1,没有重启
kubectl -n lab get endpoints web
# ENDPOINTS 对新 Pod 为空

💡 对比:liveness 失败会重启容器;readiness 失败只会摘除流量,不重启。所以这里 RESTARTS 是 0。

排查
bash
kubectl -n lab describe pod <pod> | grep -A5 Events
Warning  Unhealthy  Readiness probe failed: HTTP probe failed with statuscode: 404

手动验证探针到底返回什么:

bash
kubectl -n lab exec <pod> -- curl -s -o /dev/null -w "%{http_code}\n" localhost/healthz-wrong   # 404
kubectl -n lab exec <pod> -- curl -s -o /dev/null -w "%{http_code}\n" localhost/                # 200

➜  k8s kubectl -n lab exec web-6d6594866-2c7rn  -- curl -s -o /dev/null -w "%{http_code}\n" localhost/healthz-wrong 
404
➜  k8s kubectl -n lab exec web-6d6594866-2c7rn  -- curl -s -o /dev/null -w  "%{http_code}\n" localhost/             
200


结论:应用没问题,是探针路径配错。

恢复
bash
kubectl -n lab rollout undo deploy/web
kubectl -n lab rollout status deploy/web

last



kubectl -n lab describe pod web-6d78696589-z776p | grep -A8 "Last State"
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 19 Nov 2026 14:32:38 +0800
      Finished:     Thu, 19 Nov 2026 14:32:38 +0800
    Ready:          False
    Restart Count:  36
    Limits:
      cpu:     100

kubectl describe pod	
拉取该 Pod 的全量状态 + 事件,比 get 详细得多

-n lab	
指定命名空间 lab,不加就默认去 default 找,找不到会报 NotFound

web-6d78696589-z776p	
Pod 名。中间 6d78696589 是 ReplicaSet 的哈希,后缀 z776p 是随机串——说明这个 Pod 由 Deployment web 管理,不是裸跑的

grep -A8 "Last State"
	
匹配到 Last State 那行,并附带其后 8 行(-A = After context)。这就是为什么输出正好从 Last State 截到 Limits
小技巧:换成 grep -B2 -A12 "Last State" 能看到上下文更全,包括当前 State 和容器名。


 Last State vs State
K8s 给每个容器留两个槽位:State 是当前,Last State 是上一次终止那次的。出现 Last State 有内容,本身就说明这容器重启过。你现在看的不是它此刻的样子,是它的"遗照"

2. 如果 Pod 有多个容器,指定容器名
kubectl -n lab logs web-6d78696589-z776p -c web --previous

resume 
 kubectl -n lab rollout status deploy/web




Exit Code	典型含义
1​	通用失败。配置错、连不上依赖、参数错、脚本里某条命令失败——绝大数业务崩溃长这样
137	128+9,收到 SIGKILL,几乎等于 OOM 被杀​ 或被人为 kill -9
143	128+15,收到 SIGTERM,正常优雅终止(如滚动更新)
126 / 127	命令不可执行 / 命令找不到(镜像里没这个二进制)
0	正常退出。但主进程正常退出对长期服务是坏事​ → K8s 会不断重启它

rollout before version

k8s kubectl -n lab patch deploy web --type=json -p='[                       
  {"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"50m"}
]'
kubectl -n lab rollout status deploy/web
➜  k8s kubectl -n lab rollout status deploy/web
error: deployment "web" exceeded its progress deadline
➜  k8s kubectl -n  lab get deploy, rs,pods 
error: arguments in resource/name form must have a single resource and name
➜  k8s kubectl -n  lab get deploy,rs,pods  
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web   2/2     1            2           3h31m

NAME                             DESIRED   CURRENT   READY   AGE
replicaset.apps/web-65c6db8bd4   0         0         0       3h23m
replicaset.apps/web-6d78696589   1         1         0       179m
replicaset.apps/web-7bccbc594    2         2         2       3h1m
replicaset.apps/web-b8d8c8889    0         0         0       3h31m
replicaset.apps/web-d5477b897    0         0         0       3h10m

NAME                       READY   STATUS             RESTARTS       AGE
pod/web-6d78696589-z776p   0/1     CrashLoopBackOff   40 (51s ago)   179m
pod/web-7bccbc594-6w65z    1/1     Running            0              3h1m
pod/web-7bccbc594-92rgn    1/1     Running            0              3h1m
➜  k8s kubectl -n lab rollout history deploy/web 
deployment.apps/web 
REVISION  CHANGE-CAUSE
2         <none>
3         <none>
4         <none>
5         <none>
6         <none>

➜  k8s kubectl -n lab get rs -o custom-columns=NAME:.metadata.name,REV:.metadata.annotations.deployment\\.kubernetes\\.io/revision,DESIRED:.spec.replicas,READY:.status.readyReplicas
NAME             REV   DESIRED   READY
web-65c6db8bd4   2     0         <none>
web-6d78696589   6     1         <none>
web-7bccbc594    5     2         2
web-b8d8c8889    3     0         <none>
web-d5477b897    4     0         <none>
➜  k8s kubectl -n lab rollout history deploy/web --revision=5
deployment.apps/web with revision #5
Pod Template:
  Labels:	app=web
	pod-template-hash=7bccbc594
  Containers:
   nginx:
    Image:	nginx:1.25
    Port:	80/TCP
    Host Port:	0/TCP
    Limits:
      cpu:	100
      memory:	128Mi
    Requests:
      cpu:	50m
      memory:	64Mi
    Readiness:	http-get http://:80/ delay=3s timeout=1s period=5s #success=1 #failure=3
    Environment:	<none>
    Mounts:	<none>
  Volumes:	<none>
  Node-Selectors:	<none>
  Tolerations:	<none>

➜  k8s kubectl -n lab  get pod 
NAME                   READY   STATUS             RESTARTS         AGE
web-6d78696589-z776p   0/1     CrashLoopBackOff   40 (4m34s ago)   3h3m
web-7bccbc594-6w65z    1/1     Running            0                3h4m
web-7bccbc594-92rgn    1/1     Running            0                3h5m
➜  k8s kubectl -n lab rollout history deploy/web --revision=5
deployment.apps/web with revision #5
Pod Template:
  Labels:	app=web
	pod-template-hash=7bccbc594
  Containers:
   nginx:
    Image:	nginx:1.25
    Port:	80/TCP
    Host Port:	0/TCP
    Limits:
      cpu:	100
      memory:	128Mi
    Requests:
      cpu:	50m
      memory:	64Mi
    Readiness:	http-get http://:80/ delay=3s timeout=1s period=5s #success=1 #failure=3
    Environment:	<none>
    Mounts:	<none>
  Volumes:	<none>
  Node-Selectors:	<none>
  Tolerations:	<none>

➜  k8s kubectl -n lab rollout undo deploy/web --to-revision=5 
deployment.apps/web rolled back
➜  k8s kubectl -n lab get deploy,rs,pods 
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web   2/2     2            2           3h36m

NAME                             DESIRED   CURRENT   READY   AGE
replicaset.apps/web-65c6db8bd4   0         0         0       3h28m
replicaset.apps/web-6d78696589   0         0         0       3h4m
replicaset.apps/web-7bccbc594    2         2         2       3h6m
replicaset.apps/web-b8d8c8889    0         0         0       3h36m
replicaset.apps/web-d5477b897    0         0         0       3h15m

NAME                      READY   STATUS    RESTARTS   AGE
pod/web-7bccbc594-6w65z   1/1     Running   0          3h6m
pod/web-7bccbc594-92rgn   1/1     Running   0          3h6m