AWS Security Best Practices — AWS Secrets Manager — #CloudSecurity
Where do I keep my secrets / passwords in AWS
If this is the first post you are reading in the #CloudSecurity series, Welcome! :) You will find some of the other topics covered under #CloudSecurity by me interesting too. Please feel free to follow my medium profile to see the remaining Cloud Security articles and be updated on upcoming posts on the same topic. I love connecting and talking security so do head to www.ashishrajan.com and connect with me to continue the conversation. Ok, back to the post.
If you prefer consuming educational content in video format, I have you covered. For everyone else keep reading.
From this point onwards, I assume that you have basic working knowledge of AWS to be able to create an AWS Account and understand what is an AWS IAM User, Roles and AWS Access Keys. If you don’t and would want me to point you to resources that can help, feel free to comment and I can point you to them.
In this blog post we will be talking all things AWS Secrets Manager, including What does the service do? Why would you need AWS Secrets Manager? How can you use AWS Secrets Manager? AWS Secrets Manager Best Practice — Pattern and Anti-Pattern. Let’s start…
What is AWS Secrets Manager?
AWS Secret Manager is a services from AWS to store, retrieve and manage secrets throughout its lifecycle. This services can store, retrieve, rotate, encrypt and monitor the use of secrets. The services uses AWS KMS for encryption with IAM roles to restrict access to the services and CloudTrial for recording the API calls made for the secrets.
Nomenclature:
Secrets refers to database credentials, API Keys (including aws access keys), tokens, SSH keys, SSL private key or passwords.
Secret Lifecyle refers to the generation, rotation, deletion, disabling of secrets listed above.
Why would you “want to use” AWS Secrets Manager over other secret manager?
AWS Secrets Manager, is AWS’s answer to the popular secret management products like Hashicorp Vault and Unicreds etc. The secret manager in the market with AWS integration mostly via AWS Access Keys or IAM roles to work with AWS. The AWS Secret Manager service is API driven which makes creation, rotation and storage of secrets automatic to an extent. However, do consider the following before deciding to use it.
- AWS Integrated Tool
At the time of writing this AWS Secret Manager has built-in integration for rotating database credentials of MySQL, Postgres SQL, Amazon Aurora and RDS. For services that it doesn’t have integration with, it allows the use of Lambda functions to rotate these other forms of stored secrets. Similar to other AWS services, the built-in integration will only grow to include more AWS services in the future. This means eventually you will be able to manage all secrets of your entire AWS platform from one place..well eventually.
- Ease of Secret Rotation
This services will remove the need for developers to on-goingly worry about manual periodic secret rotation in all their sensitive AWS environments. They would be able to simply point their application code to this service to use secrets in the secret manager and let AWS take care of rotation. - Auditing Secrets
Visibility of who took any actions against secrets is available with AWS CloudTrail — the auditing service of AWS. This allows the admin to identity and audit the use of secrets in their environment. - Monitoring Secrets
Monitor and raise CloudWatch alerts for attempted actions on high value secrets e.g request of production secrets, scheduling secrets for deletion or even deletion of production secrets. - Building Secret Management Capability
If you currently do not have a secrets manager for your AWS environment then this will be a great place to start building the capability in your organisation. You can use the in-built integration to update and start secret lifecycle fairly quickly and in an automated fashion. You can find a sample piece of code here to get started. - Central Secure Storage for Secrets
Your secrets will be stored on a AWS service and not on someone’s laptop. This means you can audit and restrict access to the sensitive secrets with IAM roles and CloudTrail. - Automation Enabled
AWS Secrets Manager is cli enabled and cloudformaton enabled, which means it can be easily integrated into existing infrastructure as code being used in your environment.
Why would you “not want to use” AWS Secrets Manager over other secret managers?
- AWS integration just one of the capability
You can use some of the other opensource or paid secret managers in the industry e.g Hashicorp Vault which can be used for managing secrets from Azure, GCP, on-prem services and AWS. This means no need to jump on a special secrets manager just for AWS. - Custom Lambda for non-integrated AWS Services.
If you are starting to use this service today, you would have to write Lambda functions for all the services that are not integrated today. You can copy the sample lambda functions that AWS will share,eventually you may end up in the too many lambdas to manage state. This also means when the integration does become available you would have to update your application code. - Too many Lambdas
Your secret manager will be split into the multiple Lambda code snippets that you would have to maintain for all your environments apart from the other lambda that you may already be using your organisation. You may ask if it;s adding unwanted complexity to problems solved with some tools in the market. - Local aka non cloud Storage
Your secrets will be stored only in AWS instead of your on-prem HSM. This also means that if you want to use AWS Secret Manager for all your IT assests including non-AWS hosted services, you have to write custom code with AWS SDK. If you are at this point, I would argue if you are better off moving those service onto AWS instead of writing the custom code. As most of AWS services work best for services hosted in AWS, if you can read between the lines for what I am trying to say. :)
Use case
AWS Secrets Manager is a great services for
- managing secrets on the services which are currently integrated e.g MySQL, Postgres SQL, Amazon Aurora and RDS.
- Storing, retrieving and rotating shared secret keys shared between different AWS resources in an account.
- Storing, retrieving and rotating OAuth Secrets (Client Credential flow) which are shared between microservices hosted on an AWS Account.
- Storing, retrieving and rotating secrets which would otherwise be hard-coded in application code.
- Storing, retrieving and rotating secrets at scale in an AWS environment with IAM policies to control access to secret
- Ability to split the creation and rotation permissions between different groups.
NOTE: It is worth noting that depending how much infrastructure you are already running in your AWS Accounts, you may want to consider your use case against the cost of man hour required to refresh your current infrastructure where you have been already automatically manage secrets lifecycle in your organisation using another secret manager.
Cost:
AWS Cost:
- $0.40 per secret per month (pro-rated based on the number of hours)
- $0.05 per 10,000 API calls
- 30 day trial is currently available.
- Cost of Lambda API calls (if applicable)
Other Cost: Operational Cost/Capital Cost of updating all the application code to start using this service.
Best Practice:
This services was released last year so there are not many specific patterns or anti-patterns as examples, apart from the use cases that I have personally worked on or have been shared with me. Would love to hear (in the comments), from you if you have heard of any other use cases. Let’s start:
- Unique secret per environment, per AWS region, per Account. When working with secrets at scale, store the secrets used in the account/region/environment they are required in. My recommendation will be to use a secret manager per AWS account to ensure account level segregation between secrets e.g between a prod and non-prod environment secret. Accounts are the highest level of segregation you can have in AWS where no AWS resources are shared unless explicitly given permission to do so.
- IAM Policies should have fine grained access control on which application can retrieve and which application is allowed to rotate secrets. Special attention to be paid when defining IAM policies that provide access to high value secrets like production secrets.
- Scaling Secrets: AWS Secret Manager Tags or simply Tags are the best way to group secrets for an environment or usage based Access Control.
- Secret Recovery method and secret storage is not together: Ensure the method to recover a lost key is not stored with the secret e.g in the description field. Instead you can fill description or tags with which team owns the secret etc.
- Scaling Secrets across services: Use VPC Endpoint or Private Link based on your scenario to provide EC2 instances in a private subnet or no path to AWS services access to secrets.
- Scaling but keeping an eye out for stale secrets: When a secret is no longer required schedule it for deletion after a 24–48hr period instead of deleting it immediately incase some service is still using that key.
- Perform a regular audit on stored secrets to ensure no orphan secrets are still being stored.
- The key rotation period can be anywhere from 3month to 1yr based on the potential impact the loss of those keys can have.
- Encrypt and control access to all sensitive high value secrets (e.g production secrets, jenkins keys, access keys etc) present in your environment
- DO NOT HARDCODE SECRETS IN CODE! NEVER! Sorry I will calm down now. Why not use IAM Roles instead when you have applications that require access to AWS resources.
- Secrets should be stored in a secrets manager not on an employee’s laptop. This will prevent “secret sprawling” or “potential misuse by a disgruntle employee in possesion of keys”
Irrespective of whether you use this AWS service to manage secrets or any other, it is a good practice to have a secret rotation process, even better if, it’s an “almost automated process”, especially for credentials which have admin privileges or provide access to a sensitive part of the organisation.
Thank you for your time. Now since you are armed with Good,Bad and Ugly of this AWS service go out in the world and create secure things.
#PeaceOut
