Skip to main content

Alternative Instances (Developer Preview)

Selecting the right instance for your workload is important. Cloud service providers offer a wide range of instance types, and determining the best fit isn't always straightforward. In addition, the desired instance type might sometimes be unavailable. Under these circumstances, you must find an alternative instance type.

Anyscale's Alternative Instances feature simplifies this decision. The Anyscale runtime can identify instances to potentially run your workloads "faster," based on CPU integer operations per second (IOPS) benchmarks or "cheaper," using On-Demand list pricing per integer operation (IOP). When you activate Alternative Instances, Anyscale has the ability to preferentially select instance types from a generated ranked list.

You can specify a default instance type, and Anyscale uses that as a basis for identifying similar instances. For CPUs, Anyscale determines similarity based on vCPU and memory. Anyscale ranks instances in two ways:

  1. "Faster": Anyscale finds a similar instance type with potentially better performance based on the computation power of a given instance type as measured by a CPU IOPS benchmark.
  2. "Cheaper": Anyscale finds a similar instance type with a potentially cheaper price per IOP, in comparison with the cloud service provider's On-Demand list pricing.
info

Price performance focuses only on CPU IOPS, so performance may vary depending on the workload. In addition, Anyscale updates instance pricing from the cloud providers weekly. Anyscale pricing updates may be delayed by up to one week.

Usage​

To use the Alternative Instances feature, you may elect to use the Anyscale UI for ease of use or the Python SDK for a programmatic approach. In either method, there are three keys that layer to offer more granular control:

  1. as-feature-alternates-ranking - Turn on/off the feature and toggle whether to rank by "faster" or "cheaper."
  2. as-feature-num-alternates- Choose the number of alternative instances for Anyscale to auto-rank for provisioning.
  3. as-feature-customized-alternates - Set the exact instance types for Anyscale to consider for provisioning.

Anyscale UI​

Key: as-feature-alternates-ranking​

To configure Alternative Instances, copy and paste the following JSON into the "Advanced configuration" input box and specify the "Value" to be "faster" or "cheaper":

{
  "TagSpecifications": 
  [
    {
      "Tags": 
      [
        {
          "Key": "as-feature-alternates-ranking",
          "Value": "cheaper"
        }
      ],
      "ResourceType": "instance"
    }
  ]
}

Key: as-feature-num-alternates​

For further control, you can set the number of alternative instances for Anyscale to try and provision, and you can set this by adding the as-feature-num-alternates key in the "Advanced configuration" JSON:

{
   ...
         "Key": "as-feature-num-alternates",
         "Value": "1" // This can be any positive int.
       },
       {
         "Key": "as-feature-alternates-ranking",
         "Value": "faster"
       }
   ...
}

Value OptionsDescription
as-feature-num-alternates = 0Anyscale only considers the instance type that you specified under Instance type in the compute config.
as-feature-num-alternates = 1(default value) Anyscale attempts to provision from either the original instance type or one alternative instance type.
as-feature-num-alternates = N where N > 1Anyscale attempts to provision from a pool of N + 1 options.
Example configurations
Consider the following scenarios, where you specify m6a.2xlarge as the desired instance type for a group of nodes.

as-feature-num-alternates = 1 and as-feature-alternates-ranking = "Cheaper"

Anyscale creates a ranked list with the original instance type m6a.2xlarge and an alternative instance type. In this case, the m5.2xlarge ranks as the cheapest option (price per IOP).

as-feature-num-alternates = 2 and as-feature-alternates-ranking = "Cheaper"

Anyscale identifies two alternatives, m5.2xlarge and m6i.2xlarge. Anyscale then creates a ranked list for these 3 options and attempts to provision instances in rank order, which in this case begins with the m5.2xlarge (price per IOP).

as-feature-num-alternates = 2 and as-feature-alternates-ranking = "Faster"

Anyscale identifies two alternatives, m5.2xlarge and m5zn.2xlarge. Anyscale creates a ranked list for these 3 options and attempts to provision instances in rank order (CPU IOPS benchmark), which in this case begins with m5.2xlarge.

If the highest ranked instance isn't available, Anyscale moves down the list to try the next one and so on.

Key: as-feature-customized-alternates​

To specify an exact list of alternate instance types rather than using Anyscale's list-generating functionality, you can define a list with the as-feature-customized-alternates key. Anyscale considers the original instance as first priority before moving down the user-defined list in order.

Paste and modify this JSON into the "Instance config" to apply to a group of nodes which overrides the cluster-wide configuration (including any settings for as-feature-alternates-ranking and as-feature-num-alternates) in the "Advanced configuration" input box:

{
  "TagSpecifications": [
    {
      "Tags": [
        {
          "Key": "as-feature-customized-alternates",
          "Value": "m5.2xlarge,m6a.2xlarge"
        }
      ],
      "ResourceType": "instance"
    }
  ]
}
note

Anyscale verifies that a designated alternative instance exactly matches CPU, GPU, and memory with the original desired instance. If there is a mismatch, Anyscale won't proceed to attempt provisioning. To override this, add a $ to the instance name to mark instances to try regardless of the specs. For example, {type1},{$type2},{type3}.

Python SDK and YAML configuration​

To specify compute configurations in a programmatic way suitable for scalability, automation, and integration into larger workflows, define a YAML file to define the necessary parameters:

cloud: anyscale_v2_default_cloud_vpn_us_east_2 # You may specify `cloud_id` instead
allowed_azs:
  - us-east-2a
  - us-east-2b
  - us-east-2c
head_node_type:
  name: head_node_type
  instance_type: m5.2xlarge
worker_node_types:
  - name: cpu_worker
    instance_type: c5.8xlarge
    min_workers: 2
    max_workers: 10
    use_spot: true
  - name: gpu_worker
    instance_type: g4dn.8xlarge
    min_workers: 0
    max_workers: 10
aws:
  TagSpecifications:
    - ResourceType: instance
      Tags:
        - Key: as-feature-alternates-ranking
          Value: cheaper

Then, use the Anyscale Python SDK to read, interpret, and apply these configurations to your system:

import yaml

from anyscale.sdk.anyscale_client.models import CreateClusterCompute
from anyscale import AnyscaleSDK

sdk = AnyscaleSDK()

with open('compute_config.yaml') as f:
   compute_configs = yaml.safe_load(f)

# If your config file contains `cloud`, use this to get the `cloud_id`
if "cloud" in compute_configs:
   compute_configs["cloud_id"] = sdk.search_clouds(
       {"name": {"equals": compute_configs["cloud"]}}
   ).results[0].id
   del compute_configs["cloud"]

config=sdk.create_cluster_compute(CreateClusterCompute(
   name="my-cluster-compute",
   config=compute_configs
))

Monitoring​

You can trace attempts to provision Alternate Instances in the Event Log.

Sample output log
[07/18/2023 10:34:28 AM] Cluster is starting.

[07/18/2023 10:34:28 AM] Launching 1 instance(s) of type m5.8xlarge (head-node-type).

[07/18/2023 10:34:28 AM] Launching 1 instance(s) of type m5.8xlarge (worker-node-type-0).

[07/18/2023 10:34:28 AM] Launching 1 instance(s) of type c5.4xlarge (worker-node-type-1).

[07/18/2023 10:34:28 AM] [planner] Processing first request for empty cluster...

[07/18/2023 10:34:28 AM] [planner] [worker-node-type-1] Built plan with 8 entries to acquire 1 instances.

[07/18/2023 10:34:28 AM] [planner] [worker-node-type-1|c6a.4xlarge] [us-west-2a] [on-demand] Attempting to launch 1 instances.

[07/18/2023 10:34:29 AM] [planner] [worker-node-type-1|c6a.4xlarge] [us-west-2a] [on-demand] Launched 1 instances.

[07/18/2023 10:34:29 AM] [planner] First instances of type c6a.4xlarge for empty cluster launched in zone 'us-west-2a'.

[07/18/2023 10:34:29 AM] [planner] [worker-node-type-1]  Launched all 1 instances.

[07/18/2023 10:34:29 AM] [planner] [head-node-type] Built plan with 8 entries to acquire 1 instances.

[07/18/2023 10:34:29 AM] [planner] [worker-node-type-0] Built plan with 8 entries to acquire 1 instances.

[07/18/2023 10:34:30 AM] [planner] [head-node-type|m6a.8xlarge] [us-west-2a] [on-demand] Attempting to launch 1 instances.

[07/18/2023 10:34:30 AM] [planner] [worker-node-type-0|m6a.8xlarge] [us-west-2a] [on-demand] Attempting to launch 1 instances.

[07/18/2023 10:34:31 AM] [planner] [worker-node-type-0|m6a.8xlarge] [us-west-2a] [on-demand] Launched 1 instances.

[07/18/2023 10:34:31 AM] [planner] [head-node-type|m6a.8xlarge] [us-west-2a] [on-demand] Launched 1 instances.

[07/18/2023 10:34:31 AM] Head node launched; awaiting startup (private IP: 10.0.51.233, instance ID: i-05ea93370e693a0d2).

[07/18/2023 10:34:31 AM] [planner] [head-node-type]  Launched all 1 instances.

[07/18/2023 10:34:31 AM] [planner] [worker-node-type-0]  Launched all 1 instances.

[07/18/2023 10:34:47 AM] [head] Node launched; pulling images and preparing environment.

[07/18/2023 10:34:47 AM] [head] Pulling image for Ray container.

[07/18/2023 10:34:50 AM] [head] Pulled image for Ray container, took 2.210725925s (size: 1.5 GB).

[07/18/2023 10:34:50 AM] [head] Created Ray container, took 771.794367ms.

[07/18/2023 10:34:51 AM] [head] Started Ray container, took 44.573107ms.

[07/18/2023 10:34:51 AM] [head] Executed 'init scripts' in container, took 17.005717ms.

[07/18/2023 10:34:55 AM] The core Grafana dashboard for this cluster is now available.

[07/18/2023 10:34:55 AM] [head] Executed 'ray start' in container, took 4.290089041s.

[07/18/2023 10:34:59 AM] Node Status: head-node-type: 0/1 ready, worker-node-type-0: 1/1 ready, worker-node-type-1: 0/1 ready

[07/18/2023 10:35:02 AM] Node Status: head-node-type: 0/1 ready, worker-node-type-0: 1/1 ready, worker-node-type-1: 1/1 ready

[07/18/2023 10:35:03 AM] Node Status: head-node-type: 1/1 ready, worker-node-type-0: 1/1 ready, worker-node-type-1: 1/1 ready

[07/18/2023 10:35:03 AM] [activity] Cluster is idle (last activity at 2023-07-18T17:35:03Z).

[07/18/2023 10:35:03 AM] Cluster is running.

[07/18/2023 10:35:34 AM] [activity] Cluster is idle (last activity from web terminal at 2023-07-18T17:34:54Z).

Learn more​

Similar to the Multi-zone solution, Anyscale considers dynamic variables of the underlying cloud provider like region/zone, instance types, and Spot vs. On-Demand instances to generate a ranked order of Alternative Instances that would either be "Faster" or "Cheaper."

Read about it in the latest blog post: Cloud Infrastructure for LLM and Generative AI Applications.