Skip to main content

Resource quotas

Overview​

Resource quotas allow you to enforce limits on resource usage within your Anyscale environment. This document outlines the key concepts and management of resource quotas.

Understanding resource quotas​

Quota scope​

You can enforce resource quotas at different levels within your Anyscale infrastructure.

When creating a resource quota:

  • You must specify an Anyscale cloud for quota enforcement.
  • You can optionally specify a project and/or a user.
  • If you don't specify a project or user, the quota should apply to all projects or users within the specified scope.

Quota limits​

You can set resource quotas for the following resource dimensions:

  • Maximum number of CPUs
  • Maximum number of instances
  • Maximum number of GPUs
  • Maximum number of specific accelerator types
info

Keep these points in mind when setting quota limits:

  • Specify at least one limit when creating a quota.
  • Use only positive integer values for CPUs, instances, and GPUs.
  • Set a value of 0 for specific accelerators to disallow their use.
  • Any omitted limit defaults to unlimited for that resource.

Quota enforcement and exceeded errors​

Understanding how quota enforcement works is crucial for effective resource management:

  1. Resource quotas affect the ability to start new workloads or scale existing ones within the specified scope (cloud/project/user).
  2. If a user attempts to launch a job, service, or workspace that exceeds the set quota limits, Anyscale triggers a "quota exceeded" error. You can find these errors in the event log, which provides information about which quota limit was exceeded. See screenshot attached below.
  3. Enabling a quota does not automatically terminate existing workloads that are already running. However, these workloads may fail to scale up if doing so would exceed the quota limits.
  4. Quotas do not affect the ability to scale down resources or terminate existing workloads.

Quota exceeded error in event logs

By monitoring these quota exceeded errors, you can gain insights into resource usage patterns and adjust your quotas or resource allocation strategies accordingly.

Managing resource quotas​

You can use CLI to manage resource quotas.

Create resource quotas​

Create resource quotas using the CLI.

Example command:

anyscale resource-quota create -n test_quota \
 --cloud experiment_cloud \
 --project p1 \
 --user-email someone@yourorg.com \
 --num-instances 100 \
 --num-gpus 10 \
 --num-accelerators A10G 5 \
 --num-accelerators A100-80G 0

Listing resource quotas​

List resource quotas using the CLI. You can filter by quota name and cloud name.

anyscale resource-quota list \
 --cloud experiment_cloud \ # Optional
 --name test_quota # Optional

Disabling a resource quota​

To temporarily lift restrictions, disable a resource quota:

anyscale resource-quota disable --id <resource-quota-id>

Enabling a resource quota​

Re-enable a previously disabled quota:

anyscale resource-quota enable --id <resource-quota-id>
warning

Enabling a quota won't shut down existing workloads that exceed the limits. It only prevents upscaling or starting new clusters.

Deleting resource quotas​

Remove a resource quota entirely:

anyscale resource-quota delete --id <resource-quota-id>
warning

We do not support resource quota updates.

For further assistance, contact Anyscale support.