Bug-Byte #5: Unlocking AWS S3: Overcome Access Denied with Insider Tips
Troubleshooting access to S3 buckets in AWS can be frustrating at times, especially given the generic nature of AWS Access Denied errors. These errors often stem from misconfigured policies on the S3 bucket or IAM principal. However, I have encountered a few more exotic cases in the field.
On the rare occasion that the issue is not policy-related, there are three places that I like to check:
- Does the bucket have server-side encryption (SSE) configured with a customer-managed KMS Key (CMK)?
- Does the traffic to the S3 bucket pass through any VPC S3 endpoints?
- Are you trying to configure a Public S3 bucket when all public access is blocked at the account level?
Server-Side Encryption
If an S3 bucket is configured with SSE and is using a CMK, make sure that the principals who require access to the bucket have been granted the ability to use the following actions in the KMS key policy:
- kms:GenerateDataKey
- kms:Decrypt
AWS is purposely vague about encryption errors like these, so it’s important to validate all three relevant policies in the case of CMK SSE:
- Key Policy
- Bucket Policy (aka resource policy)
- IAM Policy (role, user, permission set, etc.)
When in doubt, try troubleshooting the events in CloudTrail.
BONUS: Remember there’s a difference between the control plane and the data plane. Quite often, it can be helpful to enable S3 data events in CloudTrail to troubleshoot data plane permission errors. Remember, there’s an additional cost for data plane events in CloudTrail.
VPC Endpoints
These also come with policies and can be especially tricky to diagnose. I’ll give you an example.
I was recently tasked with troubleshooting access to an S3 bucket for a client. I could access the bucket from a workstation using the same credentials as the client, but their access was denied. I validated the S3, IAM, and KMS policies, but the traffic was still being denied for the users. This stumped me for a while, but eventually, I realized that my workspace and the clients were in separate VPCs, using different paths to reach the bucket, and the client was passing through a VPC endpoint. I checked the policy on the endpoint, and of course, it denied the traffic.
These endpoints use the same policy format as S3 buckets, so if you encounter an AccessDenied error, it’s worth verifying whether the traffic is flowing through one.
Account Level Public Access Block
If you are trying to spin up a public bucket and receive AccessDenied errors in CloudFormation, the root cause could be an account-level security setting. In the S3 service, there is an option to disable public access to S3 buckets in the account by default. If that setting is enabled, CFN will fail to create a public bucket.
Good luck out there. I hope this saves you some time.