Debugging Kubernetes Secrets: Why `kubectl get secret` lies to you

8 min read #kubernetes #k8s #base64 #secrets #debugging

You run kubectl get secret my-app -o yaml and the response looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: my-app
data:
  DATABASE_URL: cG9zdGdyZXM6Ly91c2VyOnBhc3N3b3JkQGRiLm15LWFwcC5pbnRlcm5hbDo1NDMyL3Byb2Q=
  API_KEY: c2tfbGl2ZV81MWFiY2RlZmdoaWprbG1ub3BxcnN0dXY=
type: Opaque

It looks encrypted. It isn’t. Those are base64-encoded plaintext values, and anyone who can read the YAML can decode them in two seconds.

This is the single most important thing about Kubernetes Secrets to internalize: base64 is not encryption. It’s an encoding. It’s there because YAML can’t carry arbitrary binary safely; not because the values are protected from you.

This post is the practical playbook for inspecting, debugging, and working with Kubernetes Secrets without getting bitten.

Decoding a secret in 30 seconds

# Decode a single key
kubectl get secret my-app -o jsonpath='{.data.DATABASE_URL}' | base64 -d
# → postgres://user:password@db.my-app.internal:5432/prod

# Decode all keys at once (modern kubectl)
kubectl get secret my-app -o jsonpath='{.data}' | jq 'map_values(@base64d)'
# → {
#     "DATABASE_URL": "postgres://user:password@db.my-app.internal:5432/prod",
#     "API_KEY": "sk_live_51abcdefghijklmnopqrstuv"
#   }

Or in two steps with our Base64 decoder:

kubectl get secret my-app -o yaml
Copy the encoded value, paste into the decoder.

The @base64d filter in jq is the magic — it’s been there since jq 1.6 (released 2018) but a lot of K8s tutorials still teach the extract-and-pipe-to-base64 -d flow.

The “stringData” gotcha

When you create a Secret with the data: field, you provide base64-encoded values. When you use stringData:, you provide plain text and Kubernetes encodes it for you on apply:

# These two manifests create the same Secret
---
apiVersion: v1
kind: Secret
metadata: { name: foo }
data:
  password: cGFzc3dvcmQ=
---
apiVersion: v1
kind: Secret
metadata: { name: foo }
stringData:
  password: password

Mistakes here:

stringData overrides data for the same key. If you specify both with conflicting values, stringData wins on apply.
stringData is write-only. When you read the Secret back, you’ll always see the data form. There’s no way to “look at the stringData” of a stored Secret.
Trailing newlines. When you kubectl create secret generic foo --from-file=key=key.txt, the file’s trailing \n is included. Either strip it (tr -d '\n') or be aware that it’s there and your secret is now password\n.

Generating Secrets safely from the CLI

The single command everyone learns:

kubectl create secret generic my-app \
  --from-literal=DATABASE_URL='postgres://user:pass@host/db' \
  --from-literal=API_KEY='sk_live_abc123'

Three things to know:

The --from-literal value lives in your shell history. This is where most “I committed the password to disk by accident” incidents come from. Use --from-file and a .gitignore’d tempfile, or the next pattern.
Secrets created this way go through your shell. If set -x or shell logging is on, the value gets logged.
Special characters need quoting carefully. --from-literal=A=$pass without quotes will be empty if $pass contains a ;. Always single-quote literal strings.

The safer pattern for production-adjacent work is to read from a file:

echo -n 'sk_live_abc123' > /tmp/api-key
kubectl create secret generic my-app --from-file=API_KEY=/tmp/api-key
shred -u /tmp/api-key   # or just rm

Note echo -n to skip the trailing newline.

When `kubectl get secret` is forbidden

Sometimes RBAC blocks you from reading Secrets directly:

Error from server (Forbidden): secrets "my-app" is forbidden:
User "user@example.com" cannot get resource "secrets" in API group "" in the namespace "prod"

You can still inspect what a running pod sees:

# List all env vars in a running pod (including from Secrets)
kubectl exec -it <pod> -- env

# Look at a specific value
kubectl exec -it <pod> -- printenv DATABASE_URL

If you’re forbidden from kubectl exec too, you have no business debugging this in prod. Get the actual SRE involved.

Common debugging scenarios

”My app says the database password is wrong”

Three possible causes, ordered by likelihood:

Trailing newline in the Secret value. kubectl create secret generic --from-file includes file newlines. Decode:
```
kubectl get secret my-db -o jsonpath='{.data.PASSWORD}' | base64 -d | xxd
```
Look for 0a at the end. If there is one, recreate the Secret without the newline.
Whitespace at the start or end of the literal value. Same technique with xxd.
Secret value uses a character that the database connection string parsed differently. @ in passwords needs URL-encoding when embedded in a connection string; & ends a query parameter; # starts a fragment. Try logging the URL-decoded form.

”Helm rendered my Secret wrong”

helm template my-chart | grep -A 5 "kind: Secret"

Then base64-decode the relevant data: field and compare to what you expected. Common cause: a Helm value containing : or other YAML metacharacters that needed | toString | b64enc instead of just b64enc.

”I rolled the Secret but the pod still has the old value”

Pods don’t reload Secret env vars without a restart. Either:

kubectl rollout restart deployment my-app

Or use Secret volume mounts instead of env vars — files mounted from a Secret get updated automatically when the Secret changes (with a kubelet sync delay of up to ~60s by default).

When base64 isn’t enough

Recap: Kubernetes Secrets at rest are stored in etcd, which by default writes plain text to disk. Anyone who can read the etcd database can extract the same plaintext as kubectl get secret.

The standard fixes:

Encryption at rest — configure EncryptionConfiguration for the API server. Secrets are encrypted in etcd; kubectl still decrypts on read.
External secrets — keep secrets in AWS Secrets Manager / GCP Secret Manager / HashiCorp Vault, sync into K8s via External Secrets Operator. The K8s Secret becomes a cache, not the source of truth.
SOPS / Sealed Secrets for storing encrypted Secrets in git. The repository can be public; only the cluster’s private key can decrypt.
Don’t store certain things as Secrets at all — TLS certs go in cert-manager-managed Secrets that rotate automatically; database credentials should be issued by Vault per pod with TTLs.

The single most important point

If you remember one thing:

The “encoding” you see in kubectl get secret is not a security boundary. Anyone who has read access to your cluster’s Secrets already has read access to the plain text.

The boundary is RBAC: who can read Secrets in this namespace? Audit that, not the encoding.

Programmatic equivalents

# Python with kubernetes client
from kubernetes import client, config
import base64

config.load_kube_config()
v1 = client.CoreV1Api()
secret = v1.read_namespaced_secret("my-app", "default")
plain = {k: base64.b64decode(v).decode() for k, v in secret.data.items()}

// Go with client-go
import (
    "context"
    "encoding/base64"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
)
secret, _ := clientset.CoreV1().Secrets("default").
    Get(context.TODO(), "my-app", metav1.GetOptions{})
for k, v := range secret.Data {
    fmt.Printf("%s = %s\n", k, string(v))   // already decoded
}

Note the Go client returns already-decoded []byte because the client decodes for you. Python and the YAML output do not.

Try it

Paste any base64-encoded Secret value into our decoder — it runs entirely in your browser, so you can decode prod credentials without leaking them to a third-party site. The “URL-safe” toggle is irrelevant for K8s (which uses standard base64), but the file input is useful for decoding entire Secret YAML manifests at once.