Good writup. One thing I would add for bastions if you wanted to harden them would be to disable session multiplexing if you are using MFA/2FA.
MaxSessions 1
The default is 10. The plus side of multiplexing is that subsequent connections using the same ssh connection channels are not validated against the authorization mechanisms such as login or 2FA. This reduces friction and speeds up the login process because login is not actually occurring. The trade-off of multiplexing is that all subsequent logins using that ssh connection are not logged nor are they validated with MFA. This means a person phishing your team members can easily hijack their connections without needing a password or 2FA and there are no lastlog entries. SSH Session multiplexing combined with passwordless sudo makes taking over a company trivial even if they have 2FA and strong passwords.
Another risk with a bastion model is port forwarding. As an organization you have to decide what is appropriate for that bastion. Unrestricted forwarding? Restricted? Denied?
AllowAgentForwarding no
AllowTcpForwarding yes
PermitOpen 192.168.1.2:22
If this bastion is for a PCI environment then one may want tighter restrictions. If it is for a development environment then maybe less restrictions and just better auditing on each host to enable forensic remediation.
If your bastion is also used for automation to drop files into a staging area, you can limit that automation to file transfers and even limit what it may do with files. This prevents the automation from having a shell or performing port forwarding.
The keys should be outside of the home directories to prevent malicious tools from appending additional authorized_keys into the account. Make use of automation to manage key trusts and add a comment to keys to map them to an internal tracking system like Jira. This assumes your MFA/2FA is excluding specific accounts or groups via PAM and permitting the use of ssh keys with specific groups or accounts.
AuthorizedKeysFile /etc/ssh/keys/%u
Match Group sftpusers
Banner /etc/ssh/banner_sftp.txt
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
GatewayPorts no
ChrootDirectory /data/sftphome/%u
ForceCommand internal-sftp -l DEBUG1 -f AUTHPRIV -P symlink,hardlink,fsync,rmdir,remove,rename,posix-rename
AllowTcpForwarding no
AllowAgentForwarding no
-P sets limits on what may not be done in sftp. -p does the inverse and limits what may be done. [1] -l DEBUG1 or VERBOSE will give you syslog entries of what commands were executed on the files. This is useful for audits. Some redundant settings above are also useful to set explicitly for audits.
Another thing mentioned in the article is iptables. In a PCI environment one may want to also have explicit outbound rules using the owner module to limit what users or groups are permitted to ssh out. So if your organization have a group of people allowed to use this host as a bastions, then one could write a rule like
Or specify what CIDR blocks, ports, protocols may be used. You can use REJECT rules after this rule to make it obvious a connection was not allowed so that people do not spend hours debugging. This module is also handy for limiting which daemons may speak to your infrastructure. How strict or liberal the rule is entirely at the needs of your organization.
Lastly I would add that bastions should have as minimal an OS install possible and have SELinux enforcing. Actions denied by SELinux should go to a security operations center after you spend some time tuning out the noise and false positives.
Having seen this at too many companies, we at fair.com decided to adopt stronger policies to prevent this, viz:
- all inbound API requests first go to our API proxy in the secure layer.
- the API proxy encrypts all PII using the encryption service in the secure layer
- then API proxy sends the request on to the appropriate service, having swapped all PII for tokens.
- all services in the general layer are not able to talk to the encryption service to decrypt data.
- thus all data that would normally be considered very sensitive (e.g. credit reports) can be stored or passed around the services because values like SSN and others are tokenized.
- our services with UI's like our CRM, wherein the customer service rep needs so see the data unencrypted, works perfectly bc the response from the API proxy outbound decrypts the data, but selectively so based on the person's permissions through our abstracted auth layer.
Thus in the Lyft example, the majority of employees could have access to "God view" but with all the PII encrypted (so they couldn't search their friend's account by email, for example) but they can still look at rides and transactions, while just those who need to see the decrypted PII could be given those permissions.
Of course, this assumes that the encrypted PII is sufficient for anonymization. If you can look for all rides within a block of an address in their God view, then you could quickly figure out which person was your friend by narrowing down to rides originating from his house and his work. But again that comes down to properly limiting certain search capabilities in the UI.
I don't get why more companies don't follow our approach. I've seen way too much personally sensitive data in plaintext in databases over the years.
Rookie pilot here. Large airplanes have drive by wire systems where the plane pretty much flies itself. But when certain instruments like the Pitot tube don't work then the control is handed over to pilots and they operate in alternate law, where they are responsible for the actions.
If instruments cannot measure key environmental indicators such as velocity, temperature etc - no amount of automation will save the plane.
Instrument meteorological conditions (IMC) / Instrument Flight Rating (IFR) flights are when the plane is flying through darkness, or through conditions that do not allow for a judgement of the visual elements and therefore pilots can easily make incorrect judgement calls on the position of the plane, leading to a crash.
The pitot tube is a primitive equipment to measure wind velocity and easily can be jammed by ice, insects etc. I think it was the Pitot tube malfunction in this plane that caused the incident.
Another risk with a bastion model is port forwarding. As an organization you have to decide what is appropriate for that bastion. Unrestricted forwarding? Restricted? Denied?
If this bastion is for a PCI environment then one may want tighter restrictions. If it is for a development environment then maybe less restrictions and just better auditing on each host to enable forensic remediation.If your bastion is also used for automation to drop files into a staging area, you can limit that automation to file transfers and even limit what it may do with files. This prevents the automation from having a shell or performing port forwarding.
The keys should be outside of the home directories to prevent malicious tools from appending additional authorized_keys into the account. Make use of automation to manage key trusts and add a comment to keys to map them to an internal tracking system like Jira. This assumes your MFA/2FA is excluding specific accounts or groups via PAM and permitting the use of ssh keys with specific groups or accounts.
-P sets limits on what may not be done in sftp. -p does the inverse and limits what may be done. [1] -l DEBUG1 or VERBOSE will give you syslog entries of what commands were executed on the files. This is useful for audits. Some redundant settings above are also useful to set explicitly for audits.Another thing mentioned in the article is iptables. In a PCI environment one may want to also have explicit outbound rules using the owner module to limit what users or groups are permitted to ssh out. So if your organization have a group of people allowed to use this host as a bastions, then one could write a rule like
Or specify what CIDR blocks, ports, protocols may be used. You can use REJECT rules after this rule to make it obvious a connection was not allowed so that people do not spend hours debugging. This module is also handy for limiting which daemons may speak to your infrastructure. How strict or liberal the rule is entirely at the needs of your organization.Lastly I would add that bastions should have as minimal an OS install possible and have SELinux enforcing. Actions denied by SELinux should go to a security operations center after you spend some time tuning out the noise and false positives.
[1] - https://man7.org/linux/man-pages/man8/sftp-server.8.html