Skip to main content

Configure AWS resources for an Anyscale cloud

Before you run Ray workloads on Anyscale, an Anyscale organization owner must configure AWS resources for an Anyscale cloud. This integration enables Anyscale to manage resources like compute instances and storage directly in an AWS account.

Prerequisites

  1. Register a user on Anyscale at console.anyscale.com and set up the Anyscale CLI locally.
  2. Verify your ability to launch EC2 instances in the AWS region you plan to use on Anyscale. Anyscale supports all commercially available regions. Anyscale doesn't support regions outside of the aws partition, meaning China regions and GovCloud regions.
  3. Set up AWS credentials locally by running aws configure. For more details, see the AWS configuration guide.
  4. Set up AWS credentials to correspond to the AWS account that you're using for the Anyscale cloud. They should have permissions to manage all required resources. Ensure that you have minimal IAM permissions for cloud operations.
note

The following resources have low default quota:

  • Number of VPCs per region
  • Number of internet gateways per region

Anyscale requires one of these resources per cloud. If you've reached your quota, see how you can raise it.

1. Install the Anyscale CLI

  1. Run the following command to install the Anyscale CLI and Python client package:
pip install -U anyscale
  1. To authenticate your credentials, run the following command, which fetches and updates the token that confirms your identity in the ~/.anyscale/credentials.json file.
anyscale login

If necessary, log in to the Anyscale console to complete authentication.

2. Choose a resource configuration method

Configuring AWS resources for an Anyscale cloud integrates Anyscale's capabilities into your AWS account to leverage its compute, storage, and networking resources for scalable, distributed computing.

You can use one of two different configuration methods that use the Anyscale CLI. Choose a method based on your organization's requirements:

  • anyscale cloud setup - Use for rapid configuration and a straightforward, low-maintenance solution; deploy in public subnets and access over public IP addresses without setting up additional networking infrastructure.
  • anyscale cloud register - Suitable for teams with advanced cloud expertise, seeking enhanced security, custom private networking, and specific compliance needs.

3. Configure cloud resources

Based on the configuration method selected from the previous step, configure AWS resources for your Anyscale cloud with the following instructions.

For the anyscale cloud setup method, Anyscale automatically creates and configures the necessary resources within your AWS account. You deploy Ray clusters in public subnets and access them using public IP addresses without needing to set up additional networking infrastructure like VPNs.

Note: To manually customize resources, use the (Custom) cloud register method instead.

An Anyscale cloud configured using anyscale cloud setup uses direct networking with an architecture similar to the following:

Direct Networking

Configure a new cloud

Run the following command to configure AWS resources for a new cloud:

anyscale cloud setup \
--name example_cloud_name \
--provider aws \
--region ap-southeast-1 \
--enable-head-node-fault-tolerance
🏁Optional flags

--enable-head-node-fault-tolerance: Enables head node fault tolerance in Anyscale Services by configuring an additional MemoryDB instance for the Ray Global Control Store. Note that this flag extends the setup time by approximately 20 minutes.

note

By default, Anyscale doesn't set any retention policy for the S3 bucket created by managed cloud setup. If you have any preference or concern, you can set one on your own.

4. Verify cloud resources

Anyscale provides a CLI command to verify cloud resources for both configuration methods. Anyscale runs verification automatically during cloud creation and you can also run the verification on demand.

Trigger functional verification by specifying --functional-verify workspace or --functional-verify service. Anyscale launches a workspace or a service to verify that the cloud is functional. You can also trigger both verifications (--functional-verify workspace,service).

$ anyscale cloud verify --name my-cloud-deployment

Authenticating
Loaded Anyscale authentication token from ANYSCALE_CLI_TOKEN.

Output
(anyscale +0.4s) Verifying VPC ...
(anyscale +0.8s) VPC vpc-1234 verification succeeded.
(anyscale +0.8s) Verifying subnets ...
(anyscale +1.2s) Subnets ['subnet-1234', 'subnet-2345', 'subnet-3456', 'subnet-4567'] verification succeeded.
(anyscale +1.2s) Verifying IAM roles ...
(anyscale +2.8s) IAM roles ['arn:aws:iam::999999999999:role/anyscale-iam-role-1234', 'arn:aws:iam::999999999999:role/cld_1234-cluster_node_role'] verification succeeded.
(anyscale +2.8s) Verifying security groups ...
(anyscale +3.0s) Security group ['sg-1234'] verification succeeded.
(anyscale +3.0s) Verifying S3 ...
(anyscale +3.1s) S3 anyscale-production-data-cld-1234 verification succeeded.
(anyscale +3.1s) Verifying EFS ...
(anyscale +3.3s) S3 fs-1234 verification succeeded.
(anyscale +3.3s) Start functional verification...
Functional verification for WORKSPACE is about to begin.
It will spin up one m5.xlarge instance for each function and will incur a small amount of cost.
For workspace verification, it takes about 5 minutes.
The instances will be terminated after verification. Do you want to continue? [y/N]: y
╭──────────────────────────────────────────── workspace verification ────────────────────────────────────────────╮
0:00:02 Workspace created at https://console.anyscale.com/workspaces/expwrk_abc/ses_abc │
0:01:22 Workspace is active. │
0:00:00 Workspace termination initiated. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
0:01:24 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Workspace verification succeeded!

Manage cloud resources

Update an existing cloud

If you configured your cloud using anyscale cloud setup, you can update certain cloud features:

anyscale cloud update --name <cloud-name> --enable-head-node-fault-tolerance

Delete cloud resources

To delete an Anyscale cloud and optionally its associated AWS resources:

anyscale cloud delete --name <cloud-name>
caution

For clouds created with anyscale cloud setup, this command deletes all AWS resources created by Anyscale, including:

  • VPC and subnets
  • Security groups
  • IAM roles
  • S3 buckets (and all data within them)
  • EFS file systems
  • MemoryDB clusters

For clouds registered with anyscale cloud register, this command only removes the cloud from Anyscale but doesn't delete any AWS resources. You must manually delete AWS resources or use your Terraform scripts.

IAM permissions reference

Minimal IAM permissions for cloud operations

The following sections describe the minimal IAM permissions required for different Anyscale CLI commands:

Required IAM permissions for anyscale cloud setup
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CloudformationManagement",
"Effect": "Allow",
"Action": [
"cloudformation:CreateChangeSet",
"cloudformation:CreateStack",
"cloudformation:DeleteStack",
"cloudformation:DescribeStackEvents",
"cloudformation:DescribeStacks",
"cloudformation:ListStacks"
],
"Resource": [
"*"
]
},
{
"Sid": "EC2Management",
"Effect": "Allow",
"Action": [
"ec2:AssociateRouteTable",
"ec2:AttachInternetGateway",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateInternetGateway",
"ec2:CreateRoute",
"ec2:CreateRouteTable",
"ec2:CreateSecurityGroup",
"ec2:CreateSubnet",
"ec2:CreateTags",
"ec2:CreateVpc",
"ec2:CreateVpcEndpoint",
"ec2:DeleteInternetGateway",
"ec2:DeleteRoute",
"ec2:DeleteRouteTable",
"ec2:DeleteSecurityGroup",
"ec2:DeleteSubnet",
"ec2:DeleteVpc",
"ec2:DeleteVpcEndpoints",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInternetGateways",
"ec2:DescribeNetworkAcls",
"ec2:DescribeRegions",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroupRules",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcAttribute",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeVpcs",
"ec2:DetachInternetGateway",
"ec2:DisassociateRouteTable",
"ec2:ModifySubnetAttribute",
"ec2:ModifyVpcAttribute",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupIngress"
],
"Resource": [
"*"
]
},
{
"Sid": "EFSManagement",
"Effect": "Allow",
"Action": [
"elasticfilesystem:CreateFileSystem",
"elasticfilesystem:CreateMountTarget",
"elasticfilesystem:DeleteFileSystem",
"elasticfilesystem:DeleteMountTarget",
"elasticfilesystem:DescribeBackupPolicy",
"elasticfilesystem:DescribeFileSystemPolicy",
"elasticfilesystem:DescribeFileSystems",
"elasticfilesystem:DescribeLifecycleConfiguration",
"elasticfilesystem:DescribeMountTargetSecurityGroups",
"elasticfilesystem:DescribeMountTargets",
"elasticfilesystem:DescribeReplicationConfigurations",
"elasticfilesystem:PutLifecycleConfiguration",
"elasticfilesystem:TagResource"
],
"Resource": [
"*"
]
},
{
"Sid": "IAMManagement",
"Effect": "Allow",
"Action": [
"iam:AddRoleToInstanceProfile",
"iam:AttachRolePolicy",
"iam:CreateInstanceProfile",
"iam:CreateRole",
"iam:DeleteInstanceProfile",
"iam:DeleteRole",
"iam:DeleteRolePolicy",
"iam:DetachRolePolicy",
"iam:GetInstanceProfile",
"iam:GetRole",
"iam:PassRole",
"iam:PutRolePolicy",
"iam:RemoveRoleFromInstanceProfile",
"iam:TagRole"
],
"Resource": [
"*"
]
},
{
"Sid": "S3Management",
"Effect": "Allow",
"Action": [
"s3:CreateBucket",
"s3:DeleteBucketPolicy",
"s3:GetAccelerateConfiguration",
"s3:GetBucketCors",
"s3:GetBucketLogging",
"s3:GetBucketNotification",
"s3:GetBucketObjectLockConfiguration",
"s3:GetBucketOwnershipControls",
"s3:GetBucketPolicy",
"s3:GetBucketPublicAccessBlock",
"s3:GetBucketTagging",
"s3:GetBucketVersioning",
"s3:GetBucketWebsite",
"s3:PutBucketCors",
"s3:PutBucketPolicy",
"s3:PutBucketPublicAccessBlock",
"s3:PutBucketTagging"
],
"Resource": [
"*"
]
},
{
"Sid": "Miscellaneous",
"Effect": "Allow",
"Action": [
"acm:ListCertificates",
"kms:CreateGrant",
"kms:DescribeKey",
"kms:GenerateDataKeyWithoutPlaintext",
"servicequotas:GetServiceQuota"
],
"Resource": [
"*"
]
}
]
}

Required IAM permissions for anyscale cloud register
note

When running anyscale cloud register, you will need both the register and verify IAM policies.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CloudformationManagement",
"Effect": "Allow",
"Action": [
"cloudformation:DescribeStacks",
"cloudformation:ListStacks"
],
"Resource": [
"*"
]
},
{
"Sid": "EC2Management",
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcs"
],
"Resource": [
"*"
]
},
{
"Sid": "EFSManagement",
"Effect": "Allow",
"Action": [
"elasticfilesystem:DescribeBackupPolicy",
"elasticfilesystem:DescribeFileSystemPolicy",
"elasticfilesystem:DescribeFileSystems",
"elasticfilesystem:DescribeMountTargets"
],
"Resource": [
"*"
]
},
{
"Sid": "IAMManagement",
"Effect": "Allow",
"Action": [
"acm:ListCertificates",
"iam:GetPolicy",
"iam:GetRole",
"iam:GetRolePolicy",
"iam:ListAttachedRolePolicies",
"iam:ListInstanceProfilesForRole",
"iam:ListRolePolicies",
"iam:UpdateAssumeRolePolicy"
],
"Resource": [
"*"
]
},
{
"Sid": "S3Management",
"Effect": "Allow",
"Action": [
"s3:GetBucketCors",
"s3:GetBucketLocation",
"s3:GetBucketPolicy",
"s3:ListBucket",
"s3:ListAllMyBuckets"
],
"Resource": [
"*"
]
},
{
"Sid": "Miscellaneous",
"Effect": "Allow",
"Action": [
"servicequotas:GetServiceQuota"
],
"Resource": [
"*"
]
}
]
}

Required IAM permissions for anyscale cloud verify
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CloudformationManagement",
"Effect": "Allow",
"Action": [
"cloudformation:DescribeStacks"
],
"Resource": [
"*"
]
},
{
"Sid": "EC2Management",
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcs"
],
"Resource": [
"*"
]
},
{
"Sid": "EFSManagement",
"Effect": "Allow",
"Action": [
"elasticfilesystem:DescribeBackupPolicy",
"elasticfilesystem:DescribeFileSystemPolicy",
"elasticfilesystem:DescribeFileSystems",
"elasticfilesystem:DescribeMountTargets"
],
"Resource": [
"*"
]
},
{
"Sid": "IAMManagement",
"Effect": "Allow",
"Action": [
"iam:GetRole",
"iam:ListAttachedRolePolicies",
"iam:ListRolePolicies",
"iam:GetRolePolicy",
"iam:GetPolicy",
"iam:ListInstanceProfilesForRole"
],
"Resource": [
"*"
]
},
{
"Sid": "S3Management",
"Effect": "Allow",
"Action": [
"s3:GetBucketCors",
"s3:GetBucketLocation",
"s3:GetBucketPolicy",
"s3:ListBucket",
"s3:ListAllMyBuckets"
],
"Resource": [
"*"
]
},
{
"Sid": "Miscellaneous",
"Effect": "Allow",
"Action": [
"servicequotas:GetServiceQuota"
],
"Resource": [
"*"
]
}
]
}

Control plane IAM permissions

The control plane IAM role requires extensive permissions to manage cloud resources on your behalf. The full policy includes permissions for:

  • EC2 instance management (launch, terminate, modify)
  • IAM role management for cluster nodes
  • S3 access for artifacts and logs
  • EFS access for shared storage
  • Security group management
  • Optional: MemoryDB for fault tolerance
  • Optional: Services permissions for load balancers
View full control plane IAM policy
[
{
"Sid": "IAM",
"Effect": "Allow",
"Action": [
"iam:PassRole",
"iam:GetInstanceProfile"
],
"Resource": "*"
},
{
"Sid": "RetrieveGenericAWSResources",
"Effect": "Allow",
"Action": [
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInstanceTypes",
"ec2:DescribeRegions",
"ec2:DescribeAccountAttributes"
],
"Resource": "*"
},
{
"Sid": "DescribeRunningResources",
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeSubnets",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups"
],
"Resource": "*"
},
{
"Sid": "InstanceTagMangement",
"Effect": "Allow",
"Action": [
"ec2:CreateTags",
"ec2:DeleteTags"
],
"Resource": "*"
},
{
"Sid": "InstanceStart",
"Effect": "Allow",
"Action": [
"ec2:StartInstances",
"ec2:RunInstances"
],
"Resource": "*"
},
{
"Sid": "InstanceStop",
"Effect": "Allow",
"Action": [
"ec2:TerminateInstances",
"ec2:StopInstances"
],
"Resource": "*"
},
{
"Sid": "InstanceManagementSpot",
"Effect": "Allow",
"Action": [
"ec2:CancelSpotInstanceRequests",
"ec2:ModifyImageAttribute",
"ec2:ModifyInstanceAttribute",
"ec2:RequestSpotInstances"
],
"Resource": "*"
},
{
"Sid": "ResourceManagementExtended",
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:CreateVolume",
"ec2:DescribeVolumes",
"ec2:AssociateIamInstanceProfile",
"ec2:DisassociateIamInstanceProfile",
"ec2:ReplaceIamInstanceProfileAssociation",
"ec2:CreatePlacementGroup",
"ec2:AllocateAddress",
"ec2:ReleaseAddress",
"ec2:DescribeIamInstanceProfileAssociations",
"ec2:DescribeInstanceStatus",
"ec2:DescribePlacementGroups",
"ec2:DescribePrefixLists",
"ec2:DescribeReservedInstancesOfferings",
"ec2:DescribeSpotInstanceRequests",
"ec2:DescribeSpotPriceHistory"
],
"Resource": "*"
},
{
"Sid": "EFSManagement",
"Effect": "Allow",
"Action": [
"elasticfilesystem:DescribeMountTargets"
],
"Resource": "*"
},
{
"Sid": "CreateSpotServiceLinkedRole",
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole", "iam:PutRolePolicy"],
"Resource": "arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot",
"Condition": {"StringLike": {"iam:AWSServiceName": "spot.amazonaws.com"}}
}
]

For services support, also attach:

[
{
"Sid": "CFN",
"Effect": "Allow",
"Action": [
"cloudformation:TagResource",
"cloudformation:UntagResource",
"cloudformation:CreateStack",
"cloudformation:UpdateStack",
"cloudformation:DeleteStack",
"cloudformation:DescribeStackEvents",
"cloudformation:DescribeStackResources",
"cloudformation:DescribeStacks",
"cloudformation:GetTemplate"
],
"Resource": "*"
},
{
"Sid": "ELBDescribe",
"Effect": "Allow",
"Action": [
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:DescribeListenerCertificates",
"elasticloadbalancing:DescribeTags"
],
"Resource": "*"
},
{
"Sid": "EC2Describe",
"Action": [
"ec2:DescribeVpcs",
"ec2:DescribeInternetGateways"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Sid": "ELBCerts",
"Effect": "Allow",
"Action": [
"elasticloadbalancing:AddListenerCertificates",
"elasticloadbalancing:RemoveListenerCertificates"
],
"Resource": "*"
},
{
"Sid": "ACMList",
"Effect": "Allow",
"Action": [
"acm:ListCertificates"
],
"Resource": "*"
},
{
"Sid": "ACM",
"Effect": "Allow",
"Action": [
"acm:DeleteCertificate",
"acm:RenewCertificate",
"acm:RequestCertificate",
"acm:AddTagsToCertificate",
"acm:DescribeCertificate",
"acm:GetCertificate",
"acm:ListTagsForCertificate"
],
"Resource": "*"
},
{
"Sid": "ELBWrite",
"Effect": "Allow",
"Action": [
"elasticloadbalancing:AddTags",
"elasticloadbalancing:RemoveTags",
"elasticloadbalancing:CreateRule",
"elasticloadbalancing:ModifyRule",
"elasticloadbalancing:DeleteRule",
"elasticloadbalancing:SetRulePriorities",
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:CreateTargetGroup",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:SetIpAddressType",
"elasticloadbalancing:SetSecurityGroups",
"elasticloadbalancing:SetSubnets"
],
"Resource": "*"
},
{
"Sid": "LinkELBService",
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "*",
"Condition": {
"StringLike": {
"iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
}
}
},
{
"Sid": "IAMPolicies",
"Effect": "Allow",
"Action": [
"iam:AttachRolePolicy",
"iam:PutRolePolicy",
"iam:UpdateRoleDescription",
"iam:DeleteServiceLinkedRole",
"iam:GetServiceLinkedRoleDeletionStatus"
],
"Resource": "arn:aws:iam::*:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing"
}
]

See also the Anyscale Terraform modules for AWS for infrastructure-as-code examples.

Glossary of cloud resources

ResourceDescription
VPC & SubnetsA VPC is a virtual network within your AWS account that's logically isolated from other virtual networks. A subnet is a range of IP addresses in your VPC to which AWS resources (such as EC2 instances) can be attached. Anyscale deploys workloads in your account within the VPC and subnets defined as part of setup.
Security groupSecurity groups help secure the cloud environment by controlling the traffic that's allowed to reach and leave AWS-hosted resources. Anyscale creates a security group with network rules to enable access to Anyscale's suite of components and applications.
S3 bucketAmazon S3 provides object storage to store cluster logs, workspace snapshots, Ray checkpoints, and other artifacts. Anyscale-created clusters always have access to this bucket.
Cross account IAM roleAn IAM role that enables the Anyscale control plane to configure and deploy infrastructure for your Ray clusters in your AWS account.
Cluster IAM role/Instance profileThe IAM role assumed by EC2 instances in your Ray clusters, granting them access to AWS resources from your Ray applications.
EFS (Optional)Amazon Elastic File System provides scalable file storage for use with EC2 instances. When present, Anyscale uses EFS for shared storage between cluster nodes.
MemoryDB (Optional)Amazon MemoryDB for Redis provides a Redis-compatible in-memory database service used for head node fault tolerance in Anyscale services.

Next steps