Mount an EBS volume to the head node

View Markdown

Mount an EBS volume to the head node

This page describes how to attach an existing AWS Elastic Block Store (EBS) volume to your cluster's head node and have Anyscale mount it into the Ray container at a path you choose. You create and manage the volume yourself, and Anyscale handles the attach and mount when the head node starts.

note

Mounting EBS volumes to the head node is in beta. See Feature release maturity.

When to use head node EBS volumes

Use a head node EBS volume to persist data on a volume you manage yourself so the data survives cluster restarts. The volume mounts only on the head node, so this pattern fits data that the head node alone needs to read and write.

A common example is a large build cache, such as a Bazel or Docker cache, that you want to keep warm across sessions instead of rebuilding each time the head node starts.

For datasets, cross-workload collaboration, or any data that workers need to access, use shared storage or cloud object storage instead. See Storage on Anyscale for an overview of storage options.

How head node EBS mounting works

You create the EBS volume, format it, and manage its lifecycle, including sizing, backups, and deletion. Anyscale attaches the volume to the head node and mounts it into the Ray container when the head node starts, then unmounts and detaches it when the cluster terminates.

Anyscale mounts the volume only on the head node, at the absolute path you specify. Any files written under that path persist on the volume. Worker nodes can't access the volume, because Anyscale attaches it to the head node alone.

Support depends on your cloud provider and compute stack, as described in the following table:

Cloud	Compute stack	Behavior
AWS	Virtual machines	Supported.
AWS	Kubernetes	Anyscale accepts the configuration but doesn't attach or mount the volume.
Google Cloud, Azure, generic	Any	Anyscale accepts the configuration but doesn't attach or mount the volume.

Prerequisites

Before you mount an EBS volume, make sure the following are in place:

• The EBS volume exists in the same availability zone as the head node. EBS volumes can't cross availability zones.
• The volume already has a filesystem. If the device is raw, the head node fails to start with the error no filesystem found on device ...; create a filesystem on the device before mounting. Format the volume once before first use.
• The head node's instance IAM role has the ec2:AttachVolume, ec2:DescribeVolumes, and ec2:DetachVolume permissions.
• The volume isn't attached to another instance. EBS volumes are single-attach by default.

Configure a volume mount

You configure volume mounts in the head node's advanced features using the block_storage_volumes field. Each entry attaches one volume at one mount path, as in the following example:

{
  "block_storage_volumes": [
    { "volume_id": "vol-0abcd1234ef567890", "mount_path": "/mnt/persistent" }
  ]
}

To attach more than one volume, add an entry for each, as in the following example:

{
  "block_storage_volumes": [
    { "volume_id": "vol-abc123", "mount_path": "/mnt/data" },
    { "volume_id": "vol-def456", "mount_path": "/mnt/logs" }
  ]
}

Each entry accepts the following fields:

• volume_id: The ID of an existing EBS volume that you created, such as vol-0abcd1234ef567890.
• mount_path: The absolute path where the volume appears inside the Ray container, such as /mnt/persistent.

Mount a volume on a workspace

To mount a volume on a workspace, complete the following steps:

1. Start creating a workspace, or open an existing workspace and edit its compute config.

2. In the compute config, select the head node row, then open Advanced configuration > Advanced features.

3. Add a block_storage_volumes entry with your volume ID and mount path:

   {
     "block_storage_volumes": [
       { "volume_id": "vol-0abcd1234ef567890", "mount_path": "/mnt/persistent" }
     ]
   }

4. Set the cluster's zone to the availability zone of your EBS volume, such as us-east-1a. If you don't pin the zone, the next start might select a different availability zone and the attach fails.

5. Create or restart the workspace. After the head node starts, the volume's contents are available at your mount path inside the Ray container.

When you edit an existing workspace, restart it for the new configuration to take effect.

Requirements and limitations

• Head node EBS volumes are available only on AWS clouds backed by virtual machines. On AWS Kubernetes clouds and on Google Cloud, Azure, and generic clouds, Anyscale accepts the configuration but doesn't attach or mount the volume.
• Anyscale mounts the volume only on the head node. Worker nodes can't access it.
• You manage the EBS volume's lifecycle, including creation, sizing, formatting, backups, costs, and deletion. Anyscale only attaches and mounts the volume when the head node starts and unmounts and detaches it when the cluster terminates.
• The volume and head node must be in the same availability zone. Pin the cluster's zone so that restarts don't select a different availability zone.
• EBS volumes are single-attach by default, so you can attach a volume to only one cluster at a time.