Skip to main content

Add custom tags to workloads and cloud resources

View Markdown

Add custom tags to workloads and cloud resources

This page provides an overview of configuring and using custom resource tags in Anyscale.

What are custom resource tags?​

Custom resource tags are key-value pairs that you configure as metadata for Anyscale workspaces, services, jobs, or job queues.

You use tags in the Anyscale console or CLI to filter results and organize dashboards.

Anyscale passes all custom resource tags through to underlying infrastructure in your cloud provider account. For workloads running on virtual machines, these values appear on underlying AWS resources as tags or Google Cloud resources as labels. Anyscale workloads running on Kubernetes pass tags through to your Kubernetes infrastructure using labels.

important

Anyscale assigns default resource tags to underlying cloud infrastructure. These tags start with anyscale-.

You can't use these tags to filter results in the Anyscale console or CLI, but they can be useful for monitoring, observability, and cost attribution in your cloud provider account.

Use for monitoring, observability, and permissions​

You can use either system tags or custom tags for monitoring and observability for use cases such as the following:

  • Cost management and attribution.
  • Filtering and finding virutal machines.
  • Identifying cloud infrastructure associated with a specific workload.
  • Cleanup queries or cronjobs.

Tag propagation semantics​

Resources in your cloud provider initialize with tags you include in the deploy command or configuration for your workload.

When you add or remove tags using the Anyscale console or CLI, Anyscale propagates these changes to underlying cloud resources.

For tags on running workloads, Anyscale sends commands to update tags for cloud resources after you apply changes in the console or using the CLI. Resources and logs in your cloud provider reflect these tags after a short delay.

Anyscale uses ephemeral resources for launching clusters, so when you update tags on terminated workloads, these tag updates can't propagate to underlying cloud resources. If you restart a terminated workload after changing tabs, the updated tags apply to the new resources provisioned in your cloud provider.

Custom tag requirements and limitations​

Anyscale always displays custom resource tags as key-value pairs using the standard key:value format.

The following requirements and limitations apply:

  • The maximum length for a tag is 63 characters. This limit includes the length of both the key and the value.
  • Keys and values can include lowercase letters, numbers, -, and _.
  • Each workload can only have a single tag for each unique key. Setting a tag with the same key and a different value overwrites the previous tag.
  • Each workload can have a maximum of 20 tags.
important

AWS, Google Cloud, Azure, and other cloud providers enforce their own limits for resource tags and labels. If you have additional custom tagging implemented in your cloud provider account, adjust your tagging strategy to respect these limits. For Anyscale clouds using Kubernetes resources, consult documentation for your managed Kubernetes service to learn how tags pass through to your cloud provider resources. In addition to custom tags, Anyscale sets systems tags on resources deployed in your cloud provider account. The total number of tags set by Anyscale shouldn't exceed the limits enforced by your cloud provider. Contact Anyscale Support to troubleshoot errors related to tags.

Add a custom tag during workspace creation​

You can use the Anyscale console to add tags while creating a workspace.

  1. Log in to the Anyscale console.
  2. Click Workspaces.
  3. Click Create.
  4. Click Custom blank workspace.
  5. Click Tags.
  6. Click Add a tag.
  7. Use the field to add a new tag using key:value syntax, or search for an existing tag and click to select.
    • Anyscale greys out tags with keys already set for your workload.
  8. Add multiple tags by clicking Add a tag and repeating.

Add custome tags for new workloads with the Anyscale CLI​

Anyscale jobs, service, and workspaces support adding tags during workload creation.

You can pass multiple tags using the --tag parameter as key:value pairs, as in the following example:

anyscale job submit --config-file /path/to/config.yaml --tag <key1:value> --tag <key2:value>

You can also define tags as a top-level field in a YAML config for each workload, as in the following example:

tags:
  key1: value
  key2: value

Add, update, or remove a custom tag for an existing workload​

Anyscale workspace, services, jobs, and job queues using similar flows for managing tags for existing workloads in either the Anyscale console or through the CLI.

note

You can update tags for workloads regardless of their state in Anyscale, but tags can only propagate to underlying resources in your cloud provider for running workloads. See Tag propagation semantics.

Anyscale treats each job run as a separate job. While workspaces, services, and job queues apply tags when you restart them from a terminated state, you must update job tags in the config file or in your job submit command to reflect the change in subsequent runs.

Manage tags in the Anyscale console​

To manage custom tags for an existing workload in the Anyscale console, complete the following steps:

  1. Log in to the Anyscale console.
  2. Click Workspaces, Jobs, or Services.
    • For a job queue, click Jobs then Job queues.
  3. Click the name of the workload.
  4. Click the edit icon Edit icon under Tags. The Tags flyout appears.

You can now add, update, or remove tags.

important

To update the value of a key, you must first remove the tag then add a new tag.

Tags appear in the console if they exist in any workload in your cloud. You must remove a tag from all workloads in the cloud to remove it from the list.

To add a tag, do the following:

  1. Click Add more tags.
  2. Click Add a tag.
  3. Use the field to add a new tag using key:value syntax, or search for an existing tag and click to select.
    • Anyscale greys out tags with keys already set for your workload.
  4. Add multiple tags by clicking Add a tag and repeating.
  5. Click Update.

To remove a tag, click the trash can icon next to the tag name.

Manage tags in the Anyscale CLI​

To manage custom tags for an existing workload using the Anyscale CLI, use the following syntax pattern for the appropriate workload type.

Add or update tags with the Anyscale CLI​

Use the following example syntax to add or update tags for a job with the CLI:

anyscale job tags add --id <workload-id> --tag <key1:value> --tag <key2:value>

Substitute the following values:

  • workload_id: The ID for your workspace, job, or service.
  • key:value: The key-value pair to define the tag. If a tag with the key already exists, the value updates.

Remove tags with the Anyscale CLI​

Use the following example syntax to remove tags from a job with the CLI:

anyscale job tags remove --id <workload-id> --key <key1> --key <key2>

Substitute the following values:

  • workload_id: The ID for your workspace, job, or service.
  • key: The key of the tag you want to remove. You can remove multiple keys by passing the --key parameter multiple times.

List tags with the Anyscale CLI​

Use the following example syntax to list all tags attached to a job with the CLI:

anyscale job tags list --id <workload-id>

Substitute the ID for your workspace, job, or service for workload_id.

Filter results with custom tags​

You can use custom tags to filter workspaces, services, jobs, and job queues.

Tag filtering only supports exact matching semantics for keys and values.

note

When you're using the Anyscale console to search for tags, Anyscale searches for substring matches across keys and values.

Anyscale console queries that filter using many tags might experience a slight query overhead.

When using either the console or CLI to filter using tags, Anyscale applies the following logic:

  • If you specify multiple tags with the same key, Anyscale uses OR logic to match any of the values specified.
  • If you specify multiple tags with different keys, Ayscalue use AND logic to match all values specified.

In the console, you select each tag individually from the list view for the workload type using the following steps:

  1. Log in to the Anyscale console.
  2. Click Workspaces, Jobs, or Services.
    • For a job queue, click Jobs then Job queues.
  3. Click Tags.
  4. Use the field search for an existing tag and click to select.
  5. Click Tags, search, and click to add additional tags.

Best practices​

Anyscale recommends the following best practices for custom tags:

  • Standardize a taxonomy of tags that you can apply to all workloads, for example team, env, project, and cost-center.
  • Use controlled vocabularies where possible to ensure consistency.
  • Use short, meaningful keys in kebab-case or snake_case format.

Common tag examples​

The follow table shows examples for how you might implement a custom tagging strategy:

CategoryExamples
Ownershipteam:mlops, owner:jane.doe
Environmentenv:dev, env:staging, env:prod
Cost and projectcost-center:cc-123, project:alpha
Lifecyclepurpose:batch, purpose:realtime, retention:short, retention:long