Skip to main content

Compute configuration options for Kubernetes

note

Cloud specific configurations are only available with self-hosted Anyscale Clouds.

Kubernetes-specific options​

The compute configuration allows you to provide workload-scoped advanced configuration settings for either a specific node or the entire cluster. Anyscale applies this configuration as a strategic merge patch to the Pod specifications generated by Anyscale before sending them to the Kubernetes API.

Node-specific configurations override any cluster-wide configurations for that node type. For a full reference on configuring these properties, see the official Kubernetes documentation.

View a sample of common advanced configuration options.

{
  "metadata": {
    // Add a new label.
    "labels": {"new-label": "example-value"},
    // Add a new annotation.
    "annotations": {"new-annotation": "example-value"}
  },
  "spec": {
    // Add a node selector.
    "nodeSelector": {"disktype": "ssd"},
    "tolerations": [{
      "effect": "NoSchedule",
      "key": "dedicated",
      "value": "example-anyscale"
    }]
    "containers": [{
      "name": "ray",
      // Enable privileged container in order to use Ray image_uri runtime env to run nested worker container via podman.
      "securityContext": {
        "privileged": true
      },
      // Add a PersistentVolumeClaim to the Ray container.
      "volumeMounts": [{
        "name": "pvc-volume",
        "mountPath": "/mnt/pvc-data"
      }]
    },{
      // Add a sidecar for exporting logs/metrics.
      "name": "monitoring-sidecar",
      "image": "timberio/vector:latest",
      "ports": [{
        "containerPort": 9000
      }],
      "volumeMounts": [{
        "name": "vector-volume",
        "mountPath": "/mnt/vector-data"
      }]
    }],
    "volumes": [{
      "name": "pvc-volume",
      "persistentVolumeClaim": {
        "claimName": "my-pvc"
      }
    },{
      "name": "vector-volume",
      "emptyDir": {}
    }]
  }
}

Advanced features​

The following options are common to any cloud.

Adjustable downscaling​

The Anyscale platform automatically downscales worker nodes that have been idle for a given period. By default, the timeout period ranges from 30 seconds to 4 minutes and is dynamically adjusted for each node group based on the workload. For example, short, bursty workloads have shorter timeouts and more aggressive downscaling. Adjustable downscaling allows users to adjust this timeout value at the cluster-level based on their workload needs.

To adjust the timeout value from the Anyscale console, use the Advanced features tab under the Advanced settings for the cluster. This example sets the timeout to 60 seconds for all nodes in the cluster.

{
  "idle_termination_seconds": 60
}

Cross-zone scaling​

Cross-zone scaling is a feature that allows Anyscale to launch your Ray cluster across multiple availability zones. By default, all worker nodes are launched in the same availability zone. With cross-zone scaling enabled, Anyscale first attempts to launch worker nodes in existing zones, but if that fails, then tries the next-best zone (based on availability).

Use this feature if:

  • You want to maximize the chances of provisioning desired instance types.
  • You want to spread Serve app replicas across multiple zones for better resilience and availability.
  • Your workloads have no heavy inter-node communication or the incurred inter-availability zone cost is acceptable.

To enable or disable this feature from the Anyscale console, use the "Enable cross-zone scaling" checkbox under the Advanced settings for the cluster.

Resource limits​

Cluster-wide resource limits allow you to define minimum and maximum values for any resource across all nodes in the cluster. There are two common use cases for this feature:

  1. Specifying the maximum number of GPUs to avoid unintentionally launching a large number of expensive instances.
  2. Specifying a custom resource for specific worker nodes and using that custom resource value to limit the number of nodes of those types.

To set the maximum number of CPUs and GPUs in a cluster from the Anyscale console, use the "Maximum CPUs" and "Maximum GPUs" fields under the Advanced settings for the cluster.

To set other resource limits, use the Advanced features tab under the Advanced settings for the cluster. To add a custom resource to a node group, use the Ray config tab under the Advanced config section for that node group.

This example limits the minimum resources to 1 GPU and 1 CUSTOM_RESOURCE and limits the maximum resources to 5 CUSTOM_RESOURCE.

{
  "min_resources": {
    "GPU": 1,
    "CUSTOM_RESOURCE": 1
  },
  "max_resources": {
    "CUSTOM_RESOURCE": 5,
  }
}

Workload starting and recovering timeouts​

The workload starting timeout allows you to configure how long a workload should attempt to acquire the minimum resources when it first starts, before Anyscale terminates it.

After a workload is running, it may enter the RECOVERING state if it's attempting to recover the minimum resources, for example, due to spot preemption. The workload recovering timeout allows you to configure how long a workload may remain in the RECOVERING state, to avoid the cost of idling existing nodes.

By default, Anyscale sets both timeouts to 25 minutes.

info

These timeouts only apply to jobs and workspaces, not services.

To configure the workload starting and recovering timeouts from the Anyscale console, use the Advanced features tab under the Advanced settings for the cluster. This example increases the workload starting timeout to 1 hour and decreases the workload recovering timeout to 10 minutes.

Valid time units are: s, m, and h. For example, 1h30m.

{
  "workload_starting_timeout": "1h",
  "workload_recovering_timeout": "10m"
}

Zonal startup timeout​

Anyscale attempts to pack cluster and worker group nodes into the same zone to improve cluster communication performance and minimize cross zone data transfer.

However, machines may not always be available because of capacity constraints, IP addresses, or other reasons. If Anyscale is unable to get the requested minimum resources, Anyscale terminates any existing nodes and sequentially tries the request in a different zone within the cloud deployment.

By default, Anyscale sets the zonal startup timeout to 10 minutes.

To configure the workload zonal startup timeout from the Anyscale console, use the Advanced features tab under the Advanced settings for the cluster. This example increases the workload zonal startup timeout to 60 seconds.

Valid time units are: s, m, and h. For example, 1h30m.

{
  "default_zone_starting_timeout": "60s"
}

Worker group ranking​

info

The worker group ranking feature is in developer preview.

Some workloads may require prioritizing specific worker groups over others. This could involve utilizing on-demand capacity reservations before scaling out to use spot instances, or influencing the utilization of a particular node type before another.

Anyscale's default ranking prioritizes CPU-only worker groups above GPU worker groups, spot above on-demand, and availability of the instance type. With the instance ranking feature, you can instead specify a custom ranking order for the worker groups. In addition, with node replacement enabled, Anyscale automatically replaces lower-ranked nodes with higher-ranked nodes when they become available.

When you provide multiple rankers, Anyscale prioritizes earlier rankers over later rankers. If an ordering in a later ranker conflicts with an earlier ranker, Anyscale ignores the conflicting ranking in the later ranker.

To specify a custom worker group ranking from the Anyscale console, use the Advanced features tab under the Advanced settings for the cluster. This example specifies a ranking for 3 worker groups named spot-worker-1, spot-worker-2, and on-demand-worker, where Anyscale prioritizes both spot groups over the on-demand group, but the two spot groups have equal priority.

{
  "instance_ranking_strategy": [
    {
      "ranker_type": "custom_group_order",
      "ranker_config": {
        "group_order": [
          ["spot-worker-1", "spot-worker-2"],
          "on-demand-worker"
        ]
      }
    },
  ]
}

Node replacement​

info

The node replacement feature is in developer preview. To enable this feature, contact Anyscale support.

When providing a custom worker group ranking, if the best worker group isn't available, Anyscale may select a lower-ranked worker group instead. With Anyscale's node replacement feature, if a better worker group later becomes available, the Anyscale Autoscaler attempts to replace nodes from lower-ranked worker groups with nodes from higher-ranked worker groups. In addition, you can configure the replacement threshold, the duration a worker node must run for before Anyscale can replace it, to match your workload checkpointing.

Note that the Anyscale Autoscaler only attempts to launch a replacement node that is at least as large as the existing node, taking into account CPUs, GPUs, memory, accelerator type, and any user-defined custom resources, to ensure that any workload running on the existing node is able to run on the replacement node.

One common use case for this feature is for prioritizing multiple spot worker groups over on-demand worker groups.

For example, suppose your compute config contains 2 PREFER_SPOT node types, worker-1 and worker-2, and you have a workload that checkpoints after 30 minutes that you would prefer to run on the spot workers, only using the on-demand workers if necessary.

With the below configuration, the Anyscale Autoscaler attempts to launch the spot workers before the on-demand workers, and attempts to replace any on-demand workers with spot workers after 30 minutes. Anyscale doesn't replace the spot workers with each other, or replace the on-demand workers with each other either.

To configure node replacement from the Anyscale console, specify a custom group order with enable_replacement set to true and the replacement_threshold, using the Advanced features tab under the Advanced settings for the cluster.

Valid time units for the replacement threshold are: s, m, and h. For example, 1h30m.

{
  "replacement_threshold": "30m",
  "instance_ranking_strategy": [
    {
      "ranker_type": "custom_group_order",
      "ranker_config": {
        "enable_replacement": true,
        "group_order": [
          ["worker-1/spot", "worker-2/spot"],
          ["worker-1/on-demand", "worker-2/on-demand"]
        ]
      }
    },
  ]
}

Instance selection strategy​

By default, to maximize utilization and minimize costs, the Anyscale Autoscaler uses a "force smallest fit" instance selection strategy: it always chooses the smallest possible node for your workload, even if a larger node is available, and doesn't upscale GPU worker nodes for CPU-only workloads. However, you may prefer a "relaxed" instance selection strategy for certain workloads, for example, when using a custom worker group ranking, where the Anyscale Autoscaler always attempts to launch the highest-ranked node type.

To configure the instance ranking strategy from the Anyscale console, use the Advanced features tab under the Advanced settings for the cluster.

{
  "instance_selection_strategy": "force_smallest_fit" | "relaxed",
}

Disabling NFS mounts​

Some large-scale jobs and services workloads may need to disable NFS mounts in order to bypass cloud-provider restrictions on the number of concurrent connections. For example, Filestore on GCP allows 500 concurrent connections by default. Note that an error will be raised if this flag is set on a non job or service cluster, as NFS is required for persistence in Anyscale Workspaces.

To disable NFS mounts from the Anyscale console, use the Advanced features tab under the Advanced settings for the cluster.

{
  "disable_nfs_mount": true,
}

Best practices​

Avoid scheduling on the head node​

The Ray head node contains a number of important system processes such as the global control service (GCS), the Ray driver process, and the API server.

If scheduling is enabled on the head node when the cluster is under heavy load:

  • The actors and tasks running on the head node contend for resources and may interrupt the operation of these system components.
  • The Ray cluster may become unstable and unable to recover from failures properly.

To protect the Ray cluster from instability, head node scheduling is disabled by default for multi-node clusters when creating a new compute config. However, you can follow the instructions below to fine-tune this behavior:

You can enable head node scheduling by specifying logical resources for the head node.

Multi-node cluster