Skip to main content

Accessing a Private S3 Bucket

In this section we will configure S3 access for Anyscale clusters. At a high level, accessing an S3 bucket requires two things:

  1. The IAM Role needs the correct policies to allow S3 operations
  2. The bucket must grant access to the given IAM Role.

The first part is always required, while the second part may be optional, if the IAM Role and the S3 Bucket are in the same AWS account.

info

If you try to read objects created from a cluster running in a different AWS account you may see "Permission Denied".

In order to fix this, you can

  • Use flag --bucket-owner-full-control from the CLI for {"ACL": "bucket-owner-full-control"} to ensure the bucket owner can access the file.
  • Enforce bucket owner to assume control of all uploaded content using bucket owner enforced configuration in S3.

Required Information

  • S3 Bucket to use. This ARN will look something like arn:aws:s3:::<BUCKET_NAME>.
  • The IAM Role that the cluster is running with. This can be determined from Overview.

Ensure IAM Role Policies

  1. Search for your Role on the AWS IAM page and select it. Search For Role

  2. Look in the "Permissions policies" section of your Role. If it has AmazonS3FullAccess (or a similar custom policy that grants s3:ListBucket, s3:GetObject & s3:PutObject on any resource), you are done! If not, continue to set up the correct policies.

Correctly Configured:

Search For Role

Missing Permissions:

Search For Role

  1. Click "Attach policies".
attach_policies
  1. Search for AmazonS3FullAccess and select the policy.

S3 Full Access

  1. Click "Attach policy".
attach_policy

Configure Bucket Access

To access an S3 bucket that exists in a different account, you will need to configure that bucket to allow this IAM Role access. You can do this by adding the following policy to that S3 Bucket:

  1. Search for your S3 bucket on the S3 Page and select it.

Search For Bucket

  1. Select the "Permissions" tab.

Permissions Tab

  1. Look in the Bucket Policy section and click Edit.

Bucket Policy Edit

  1. Paste the following JSON into the Policy editor box. Make sure to replace <ROLE_ARN> and <BUCKET_NAME> blocks with your role's ARN and your bucket name.
{
"Version": "2012-10-17",
"Id": "",
"Statement": [
{
"Sid": "ReadWriteS3",
"Effect": "Allow",
"Principal": {
"AWS": "<ROLE_ARN>"
},
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::<BUCKET_NAME>",
"arn:aws:s3:::<BUCKET_NAME>/*"
]
}
]
}

Policy Editor Box

  1. Click "Save changes".
Save changes