Is the only reason left for preferring files is that integrations prefer it as they can easily implement it as init pod? or that they have less chance to appear in ENV dumps in logs?
It’s the fear of env dumps and the fact that env variables are readable by other processes (but only inside the container) and that docker records the env vars as well. One thing I am wondering is: how much does the situation improve if you unset the Env vars after reading them?
My take is that it is noticeably more secure to use the cloud provider secret manager (or hachicorp vault).
There’s not much difference between a configmap and a k8s secret. The secret doesn’t have a separate decrypt operation so you have to lock down even list access and can’t separately view it’s metadata so you don’t get much benefit from k8s anyways.
If you use a k8s secret adapter to the external secret manager tool rather than loading directly in the container out of band, the security picture gets more complicated.
My main reason for avoiding passing secrets via the environment is that updating the secret requires a process restart. Using in-process mechanisms to retrieve secrets from a vault or KMS lets long running processes benefit from frequently rotated keys.
This isn't always possible to do without some poor UX for things like multiplayer games or VoIP where the pod keeps state and latency is a noticeable factor.
> There’s not much difference between a configmap and a k8s secret. The secret doesn’t have a separate decrypt operation so you have to lock down even list access and can’t separately view it’s metadata so you don’t get much benefit from k8s anyways.
Even though Secrets and ConfigMaps are used exactly the same way, there's a big difference between them. The most obvious one being that Secrets contain values that are sensitive. Of course, by the time these sensitive values have made their way into a Kubernetes Secret stored in a Namespace, they appear in cleartext. But the point of a Kubernetes Secret is to offer cluster administrators an Object that is explicitly meant to store Secret values, which can then, _for example_, be stored in ETCD differently (encrypted at rest for example).
> My take is that it is noticeably more secure to use the cloud provider secret manager (or hachicorp vault).
Yes, you should use Vault (or whichever is your favorite secret management tool), but this has no bearing on the Kubernetes side: ultimately, your sensitive values will end up inside a k8s Secret object, that is then being used by your Workloads (i.e. Pods).
> If you use a k8s secret adapter to the external secret manager tool rather than loading directly in the container out of band, the security picture gets more complicated.
Loading sensitive values out of band should be avoided since it is non-standard and forces your users to implement a method of retrieving secrets (which will be 1- painful, 2- buggy, 3- a security incident soon).
From the point of view of the process running inside the container, there is absolutely no difference between a sensitive configuration value and a non-sensitive configuration value. When this value reaches your process it is in cleartext.
What matters is the transport and persistent storage of your value. For a non-sensitive value, the storage can be done in VCS and transport over non-secure channels. For a sensitive value, the storage must be done encrypted (in hashicorp Vault for example) and transport be done over a secure channel (with authentication/authorization).
By using an adapter ([0]External Secrets for example), you actually abstract away the transport and storage of secret values from your users; allowing them to treat all configuration values the same (by reading an ENV variable, or a file), regardless of whether they are sensitive or not, and regardless of where they are stored at rest.
I agree that K8s secrets are more convenient for applications and separate out deployment concerns. But I don't agree with any of the statements made about security properties.
It is possible to not use K8s and make things just as convenient. For example by encrypting the secrets and putting them into the docker image and having a tool run at startup that decrypts them into the same file format as K8s (or as env variables). More equivalent and with less exposure of encrypted values would probably be to run a sidecar container that pulls in the secrets from the secret manager. If you want to represent the secrets in K8s you still can as a CRD that doesn't actually expose the secrets when listed or read.
What is the issue with Kubernetes Secrets that makes you want to go through all this complexity and avoid using a facility explicitly meant to manage secrets efficiently?
Security. K8s secrets present their data in the clear to anyone who can view it or list or watch secrets. Cloud secret managers provide a separate decrypt operation, along with builtin features like auditing and versioning.
> _for example_, be stored in ETCD differently (encrypted at rest for example).
Cloud provider k8s clusters are deployed with encryption at rest for ETCD now. If secrets were doubly encrypted at rest that wouldn't fix how their secret values are exposed in what should be just a metadata listing.
> From the point of view of the process running inside the container, there is absolutely no difference between a sensitive configuration value and a non-sensitive configuration value. When this value reaches your process it is in cleartext.
This model is desirable for convenience, but not for security. In order to securley handle a secret properly, the process needs to know that it is handling a secret. This can help to ensure secrets are not accidentally logged. For higher levels of security, it also helps to periodically refresh secrets so that they can be given as expiring tokens.
Environments tend to leak in places you don't expect. Mostly because they weren't designed to hold secrets in the first place. It's also another class of vulnerabilities applicable, where environment variables tends to be used for session data and can affect logging formats and other external integration points, generally in the web and perhaps specifically in the java ecosystems.
Files on the other hand has access control as a primary feature. They carry meta data that makes it obvious which uid can access the data and which gets included with backups and storage.
Apart from that, there are also possible encoding issues of secrets in environments. Secrets must be kept verbatim for integrity reasons better more suited for binary formats.
Then there are the size restrictions. An environment of several kilo- or even megabytes seems plenty until for some ridiculous reason it's suddenly not, with not at all obvious results.
One other consideration: If you use something like the Secrets Store CSI Driver and map the resulting k8s Secret to an env variable, it won't update even if the CSI Driver updates the underlying k8s Secret. If it's mounted as a file it will update and you could potentially detect the change and reload your app.
Good to learn, Thanks. But I agree more with those who suggest to bind key/secret rotation with config versions (= you can only delete an old secret after all pods using it redeployed to next version with new config).
Another approach that sometime is supported, is to use a proxy that handles secret based connections. Like a MySQL pod proxy that you connect to which take care of reconnecting to his upstream on secret change.
Both are transparent to the app pod (and its developers)
>Avoid exposing your Kubernetes nodes directly to public networks.
What's the best course of action if you need to connect to a specific pod? (You might want to do this if you're connecting users or to the same server instance, for example). Managing reverse proxies can be pretty cumbersome if you want to autoscale. Exposing an external node IP is easiest solution but if its a security risk, what are the alternatives?
One way to do it is using Session Manager and the document AWS-StartPortForwardingSessionToRemoteHost. This way you can forward a port on one of the EKS node to the control plane and access it on localhost:yourport.
A good article about it is https://aws.amazon.com/blogs/aws/new-port-forwarding-using-a.... Since that article they introduced the capability for forwarding to the remote host by using the document written above and by adding the host key and value in the parameters.
* Accessible to someone inside the pod
* Not Accessible to other pods in the cluster without cluster admin
* Can be retrieved outside EKS from API/ETCD or AWS Secret with enough permission
* Persistent across the pod lifetime (you cannot delete a secret file after use: https://stackoverflow.com/a/54404149/1997873)
Is the only reason left for preferring files is that integrations prefer it as they can easily implement it as init pod? or that they have less chance to appear in ENV dumps in logs?