Accessing Amazon S3 Buckets

Check your docs version

This version of the Anyscale docs is deprecated. Go to the latest version for up to date information.

In this section we will configure S3 access for Anyscale clusters. At a high level, accessing an S3 bucket requires two things:

1. The IAM Role needs the correct policies to allow S3 operations.
2. The bucket must grant access to the given IAM Role.

The first part is always required, while the second part may be optional, if the IAM Role and the S3 Bucket are in the same AWS account.

info

If you try to read objects created from a cluster running in a different AWS account you may see Permission Denied.

In order to fix this, you can

• Use flag --bucket-owner-full-control from the CLI for {"ACL": "bucket-owner-full-control"} to ensure the bucket owner can access the file.
• Enforce bucket owner to assume control of all uploaded content using bucket owner enforced configuration in S3.

Access existing S3 Buckets

Required Information

• S3 Bucket to use. This ARN will look something like arn:aws:s3:::<BUCKET_NAME>.
• The IAM Role that the cluster is running with.

note

Determine the Cluster's IAM Role

By default, Anyscale clusters run with a cloud-specific IAM role.

If you followed instructions on how to run with a custom IAM role, use that role for the rest of the instructions.

Ensure IAM Role Policies

1. Search for your Role on the AWS IAM page and select it.

2. Look in the Permissions policies section of your Role. If it has AmazonS3FullAccess (or a similar custom policy that grants s3:ListBucket, s3:GetObject & s3:PutObject on any resource), you are done. If not, continue to set up the correct policies.

Correctly Configured:

Missing Permissions:

3. Click Attach policies.

4. Search for AmazonS3FullAccess and select the policy.

5. Click Attach policy.

Configure Bucket Access

To access an S3 bucket that exists in a different account, you will need to configure that bucket to allow this IAM Role access. You can do this by adding the following policy to that S3 Bucket:

1. Search for your S3 bucket on the S3 Page and select it.

2. Select the Permissions tab.

3. Look in the Bucket Policy section and click Edit.

3. Paste the following JSON into the Policy editor box. Make sure to replace <ROLE_ARN> and <BUCKET_NAME> blocks with your role's ARN and your bucket name.

{
    "Version": "2012-10-17",
    "Id": "",
    "Statement": [
        {
            "Sid": "ReadWriteS3",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<ROLE_ARN>"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET_NAME>",
                "arn:aws:s3:::<BUCKET_NAME>/*"
            ]
        }
    ]
}

4. Click Save changes.