Kubernetes 安全折腾手记

作者: 醉月思发布日期: 2022-05-19 分类: 技术实践

Kubernetes 安全一旦进项目，好看的架构图就没那么管用了。

这次做 Kubernetes 安全加固，从 RBAC 到安全策略，。

基础安全

集群访问控制

# API Server 认证配置
apiVersion: v1
kind: Config
clusters:
- cluster:
    server: https://kubernetes-api-server
    certificate-authority-data: <ca-cert>
  name: kubernetes
users:
- name: admin
  user:
    client-certificate-data: <client-cert>
    client-key-data: <client-key>
contexts:
- context:
    cluster: kubernetes
    user: admin
  name: admin-context
current-context: admin-context

RBAC 基础

# 创建 ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: default

---
# 创建 Role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: myapp-role
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "create", "update", "delete"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "create", "update", "delete"]

---
# 创建 RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-rolebinding
  namespace: default
subjects:
- kind: ServiceAccount
  name: myapp-sa
  namespace: default
roleRef:
  kind: Role
  name: myapp-role
  apiGroup: rbac.authorization.k8s.io

Pod 安全

安全上下文

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 1000
  containers:
  - name: myapp
    image: myapp:latest
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
        - ALL
      volumeMounts:
      - name: tmp
        mountPath: /tmp
  volumes:
  - name: tmp
    emptyDir: {}

Pod 安全策略

# 创建 PodSecurityPolicy
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restricted-psp
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  volumes:
  - 'configMap'
  - 'emptyDir'
  - 'projected'
  - 'secret'
  - 'downwardAPI'
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    rule: 'MustRunAsNonRoot'
  seLinux:
    rule: 'RunAsAny'
  fsGroup:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'RunAsAny'
  readOnlyRootFilesystem: false

---
# 创建 ClusterRole for PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psp-restricted-role
rules:
- apiGroups: ['policy']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['restricted-psp']

---
# 创建 RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: psp-restricted-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: myapp-sa
  namespace: default
roleRef:
  kind: ClusterRole
  name: psp-restricted-role
  apiGroup: rbac.authorization.k8s.io

网络策略

# 拒绝所有流量
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

---
# 允许特定流量
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-specific
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: myapp
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
    ports:
    - protocol: TCP
      port: 80
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: database
    ports:
    - protocol: TCP
      port: 5432

Secret 管理

Kubernetes Secret

# 创建 Secret
apiVersion: v1
kind: Secret
metadata:
  name: myapp-secret
type: Opaque
stringData:
  database-url: "postgres://user:password@db:5432/myapp"
  api-key: "your-api-key"

---
# 使用 Secret
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: myapp
    image: myapp:latest
    env:
    - name: DATABASE_URL
      valueFrom:
        secretKeyRef:
          name: myapp-secret
          key: database-url
    - name: API_KEY
      valueFrom:
        secretKeyRef:
          name: myapp-secret
          key: api-key

External Secrets

# 使用 External Secrets Operator
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: myapp-secret
    creationPolicy: Owner
  data:
  - secretKey: database-url
    remoteRef:
      key: myapp/credentials
      property: database_url
  - secretKey: api-key
    remoteRef:
      key: myapp/credentials
      property: api_key

Vault 集成

# 使用 Vault Agent Injector
apiVersion: v1
kind: Pod
metadata:
  name: myapp
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "myapp-role"
    vault.hashicorp.com/agent-inject-secret-database-url: "secret/data/myapp#database_url"
spec:
  containers:
  - name: myapp
    image: myapp:latest
    env:
    - name: DATABASE_URL
      value: "vault:secret/data/myapp#database_url"

镜像安全

镜像扫描

# 使用 Trivy 扫描镜像
trivy image myapp:latest

# 使用 Clair 扫描镜像
clairctl scan myapp:latest

# 使用 Grype 扫描镜像
grype myapp:latest

镜像签名验证

# 使用 Notary 签名镜像
# 1. 生成密钥对
notary key generate myapp

# 2. 签名镜像
notary sign myapp:latest

# 3. 验证签名
notary verify myapp:latest

# 在 Kubernetes 中配置镜像策略
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sImageTagDig
metadata:
  name: image-tag-policy
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    namespaces: ["default"]
  parameters:
    tags:
    - digest

审计日志

审计配置

# 审计策略
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  namespaces: ["default"]
- level: RequestResponse
  resources:
  - group: ""
    resources: ["pods"]
  verbs: ["create", "update", "delete"]
- level: Request
  users: ["system:anonymous"]
  verbs: ["get", "list", "watch"]

---
# 启用审计日志
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - name: kube-apiserver
    command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    volumeMounts:
    - name: audit-config
      mountPath: /etc/kubernetes/audit-policy.yaml
    - name: audit-log
      mountPath: /var/log/kubernetes

审计日志分析

# 使用 kubectl audit
kubectl audit logs /var/log/kubernetes/audit.log

# 使用 Falco 监控安全事件
falco --config /etc/falco/falco.yaml

踩过的坑

坑一：权限过大

给 ServiceAccount 分配了过大权限。

解决：遵循最小权限原则。

# 错误：给 ServiceAccount 所有权限
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: too-permissive
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

# 正确：只给必要的权限
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: minimal-permissions
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]

坑二：镜像漏洞

使用了有漏洞的镜像。

解决：定期扫描镜像，及时更新。

# CI/CD 中集成镜像扫描
trivy image --severity HIGH,CRITICAL myapp:latest

# 或者在 Kubernetes 中使用准入控制器
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/trivy-operator/main/deploy/static/trivy-operator.yaml

坑三：网络策略配置错误

网络策略配置错误，导致服务无法通信。

解决：仔细测试网络策略，逐步放开。

# 先测试默认拒绝策略
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

# 然后逐步添加允许策略
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-traffic
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  ingress:
  - ports:
    - protocol: TCP
      port: 80

安全检查清单

集群安全

RBAC 配置正确
网络策略配置
镜像安全扫描
审计日志启用

Pod 安全

SecurityContext 配置
只读根文件系统
非 root 用户
资源限制

Secret 安全

Secret 加密存储
不使用明文 Secret
定期轮换密钥

网络安全

网络策略配置
入口/出口控制
TLS 加密

写在最后

Kubernetes 安全这东西，不只是技术，是运维文化和流程。

解决了：

集群安全
容器安全
数据安全

带来了：

复杂度增加
运维成本
性能影响

安全加固之前先评估：

安全要求
团队能力
预算
性能要求

不是所有集群都需要严格的安全配置，但基本安全不能少。

这次 Kubernetes 安全加固花了一个月，从 RBAC 到安全策略。加固完成后，安全事件减少了 80%，合规性检查通过率从 60% 提升到 95%。

版权声明: 本文首发于指尖魔法屋-Kubernetes 安全折腾手记（https://blog.thinkmoon.cn/post/92-kubernetes-security-rbac-security-policies-practice/）转载或引用必须申明原指尖魔法屋来源及源地址！

把精简换到安全时踩过的坑

CI 流水线里最拖时间的一步是拉镜像：一个基于 Ubuntu 的全量 Python 镜像 1.2GB，冷启动要八分钟。

📁 技术实践

📅 2023-10-28

容器网络技术：Docker 网络不够用了之后

从简单的 Docker 网络到复杂的 Kubernetes 集群网络，容器网络技术经历了显著的演进。

📁 技术实践

📅 2019-01-11

容器网络实践笔记

这次做 Kubernetes 迁移，从 Docker 的简单网络到 Kubernetes 的 CNI，再到 S…

📁 技术实践

📅 2019-05-06

Kubernetes 安全折腾手记

基础安全

集群访问控制

RBAC 基础

Pod 安全

安全上下文

Pod 安全策略

网络策略

Secret 管理

Kubernetes Secret

External Secrets

Vault 集成

镜像安全

镜像扫描

镜像签名验证

审计日志

审计配置

审计日志分析

踩过的坑

坑一：权限过大

坑二：镜像漏洞

坑三：网络策略配置错误

安全检查清单

集群安全

Pod 安全

Secret 安全

网络安全

写在最后

评论

相关阅读