Kubernetes provides an object called Secret that is meant for storing sensitive data like passwords, tokens or keys. Secrets are decoupled (distinct) from Pods to decrease the risk of exposing sensitive data while creating, viewing and updating Pods. Containers in a Pod can access secrets via environment variables or files mounted through volumes.
Let’s create a Secret named mypassword holding a
key/value pair password=s3cret!:
$ kubectl create secret generic mypassword --from-literal password=s3cret!NOTE: When you create secrets from command line they get persisted in
your shell history file, e.g. ~/.bash_history. To prevent
this add space in front of the command. The secret is also visible in
the processes listing, like ps aux. So it’s best not to
create production secrets from command line.
Ok, now, how secure is the secret we’ve created? It turns out that by default, not very. Let’s have a look.
Getting secrets from the API server
Anyone who has access to the Kubernetes API server can get the secret:
$ kubectl get secrets mypassword -o yaml
...
data:
password: czNjcmV0IQ==
...Oh, but we can’t read it. Is it encrypted? No, it’s just base64 decoded:
$ echo czNjcmV0IQ== | base64 -d -
s3cret!Getting secrets from etcd
Secrets, like other Kubernetes objects, are persisted in the etcd data store; by default unencrypted. So if we can access the data store, we can see the secrets. On a minikube cluster, we can do it like this:
$ minikube ssh
$ sudo -i
$ cat << "EOF" | bash
export ETCDCTL_CACERT=/var/lib/minikube/certs/etcd/ca.crt
export ETCDCTL_CERT=/var/lib/minikube/certs/etcd/peer.crt
export ETCDCTL_KEY=/var/lib/minikube/certs/etcd/peer.key
export ETCDCTL_API=3
ETCDCTL_BIN=$(find / -name etcdctl | grep bin | head -1)
$ETCDCTL_BIN get /registry/secrets/default/mypassword
EOF
...
passwords3cret!▒Opaque▒"
...Getting secrets from within a Pod
Let’s suppose that we can’t access the API server or the etcd database because the cluster operator put some authorization in place. And that’s the way to do it on a production cluster. The authorization mechanism in Kubernetes is called RBAC (Role Based Access Control). It’s composed of the following primitives
- User represents a “normal user” connecting to the cluster (there’s no API resource for User)
- ServiceAccount represents a program running in a pod and there is a
pre-created
defaultservice account for each namespace assigned to each created pod - Role defines a set of permissions on a namespace (or cluster) level
- RoleBinding maps roles to users or service accounts on a namespace (or cluster) level
Let’s create a service account that is allowed to read (list allows for implicit reading) all secrets within the default namespace:
$ kubectl create serviceaccount secrets-reader
$ kubectl create role read-secrets --resource secrets --verb list
$ kubectl create rolebinding secrets-reader --serviceaccount default:secrets-reader --role read-secretsHere’s a pod using the service account we have created (instead of the default service account):
$ cat << EOF | k apply -f -
apiVersion: v1
kind: Pod
metadata:
labels:
run: nginx
name: nginx
spec:
serviceAccount: secrets-reader
containers:
- image: nginx
name: nginx
EOFService account authenticates to the Kubernetes API via a JWT token that is mounted inside pod containers. If an attacker gains access to a container (for example by exploiting a vulnerability inside a web application or a web server) she can get all secrets in a namespace (or on the whole cluster if clusterrolebinding was used). Like this:
$ cat << 'EOF' | kubectl exec -i nginx -- bash | jq -r '.items[].data.password' | base64 -d
SAPATH=/var/run/secrets/kubernetes.io/serviceaccount
TOKEN=$(cat ${SAPATH}/token)
CACERT=${SAPATH}/ca.crt
URLPATH=https://kubernetes.default.svc/api/v1/namespaces/default/secrets
curl -s --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" $URLPATH
EOFFinding risky Roles
There’s a tool called KubiScan that can find risky roles (and other objects) for you:
$ git clone git@github.com:cyberark/KubiScan.git
$ cd KubiScan
$ ./docker_run.sh ~/.kube/config
$ kubiscan --risky-roles # -r to show also rules (permissions)
...
+------------+
|Risky Roles |
+----------+------+-------------+------------------------------------+-----------------------------------+
| Priority | Kind | Namespace | Name | Creation Time |
+----------+------+-------------+------------------------------------+-----------------------------------+
| CRITICAL | Role | default | read-secrets | Sat Jan 27 16:30:21 2024 (0 days) |
| CRITICAL | Role | kube-system | system:controller:bootstrap-signer | Sat Jan 27 16:21:17 2024 (0 days) |
| CRITICAL | Role | kube-system | system:controller:token-cleaner | Sat Jan 27 16:21:17 2024 (0 days) |
+----------+------+-------------+------------------------------------+-----------------------------------+